Export Job fails (error: context deadline exceeded)

Hello,

I’m testing Veeam Kasten for the first time, but unfortunately it won't work as expected. I’ve created a backup policy with a snapshot and an export to a s3 rados-gw (ceph) endpoint.

I’ve got the following error message while exporting the snapshot:

- cause:
    cause:
      cause:
        cause:
          cause:
            cause:
              cause:
                cause:
                  cause:
                    cause:
                      cause:
                        message: "client rate limiter Wait returned an error: context deadline exceeded"
                      file: github.com/kanisterio/kanister@v0.0.0-20240812194716-8812756d1751/pkg/kube/pod.go
                      function: github.com/kanisterio/kanister/pkg/kube.WaitForPodReady
                      linenumber: 384
                      message: Pod did not transition into running state.
                        Timeout:15m0s  Namespace:kasten-io,
                        Name:copy-vol-data-rpr56
                    file: github.com/kanisterio/kanister@v0.0.0-20240812194716-8812756d1751/pkg/kube/pod_controller.go
                    function: github.com/kanisterio/kanister/pkg/kube.(*podController).WaitForPodReady
                    linenumber: 174
                    message: Pod failed to become ready in time
                  fields:
                    - name: pod
                      value: copy-vol-data-rpr56
                    - name: namespace
                      value: kasten-io
                  file: kasten.io/k10/kio/kanister/function/kio_copy_volume_data.go:304
                  function: kasten.io/k10/kio/kanister/function.CopyVolumeData.copyVolumeDataPodExecFunc.func2
                  linenumber: 304
                  message: failed while waiting for Pod to be ready
                file: kasten.io/k10/kio/kanister/function/kio_copy_volume_data.go:161
                function: kasten.io/k10/kio/kanister/function.CopyVolumeData
                linenumber: 161
                message: Failed to execute copy volume data pod function
              file: kasten.io/k10/kio/exec/internal/snapshotconverters/ac_gvc_converter.go:249
              function: kasten.io/k10/kio/exec/internal/snapshotconverters.(*GVCConverterInternalAPIImpl).genericVolumeCopy
              linenumber: 249
              message: failed running copyVolumeData
            file: kasten.io/k10/kio/exec/internal/snapshotconverters/ac_gvc_converter.go:170
            function: kasten.io/k10/kio/exec/internal/snapshotconverters.(*GVCConverterInternalAPIImpl).CopySnapshotRestoredInPVC
            linenumber: 170
            message: failed running genericVolumeCopy
          file: kasten.io/k10/kio/exec/internal/snapshotconverters/ac_gvc_converter.go:77
          function: kasten.io/k10/kio/exec/internal/snapshotconverters.(*GVCConverter).Convert
          linenumber: 77
          message: Error creating portable snapshot
        fields:
          - name: type
            value: CSI
          - name: id
            value: k10-csi-snap-pcw6kxxzgsql8hj6
        file: kasten.io/k10/kio/exec/phases/phase/artifactcopier.go:544
        function: kasten.io/k10/kio/exec/phases/phase.(*ArtifactCopier).convertSnapshots.func1
        linenumber: 544
        message: Failed to export snapshot data
      file: kasten.io/k10/kio/exec/phases/phase/artifactcopier.go:274
      function: kasten.io/k10/kio/exec/phases/phase.(*ArtifactCopier).Copy
      linenumber: 274
      message: Error converting snapshots
    file: kasten.io/k10/kio/exec/phases/phase/export.go:172
    function: kasten.io/k10/kio/exec/phases/phase.(*exportRestorePointPhase).Run
    linenumber: 172
    message: Failed to copy artifacts
  message: Job failed to be executed
- cause:
    cause:
      cause:
        cause:
          cause:
            cause:
              cause:
                cause:
                  cause:
                    cause:
                      cause:
                        message: "client rate limiter Wait returned an error: context deadline exceeded"
                      file: github.com/kanisterio/kanister@v0.0.0-20240812194716-8812756d1751/pkg/kube/pod.go
                      function: github.com/kanisterio/kanister/pkg/kube.WaitForPodReady
                      linenumber: 384
                      message: Pod did not transition into running state.
                        Timeout:15m0s  Namespace:kasten-io,
                        Name:copy-vol-data-t8l8p
                    file: github.com/kanisterio/kanister@v0.0.0-20240812194716-8812756d1751/pkg/kube/pod_controller.go
                    function: github.com/kanisterio/kanister/pkg/kube.(*podController).WaitForPodReady
                    linenumber: 174
                    message: Pod failed to become ready in time
                  fields:
                    - name: pod
                      value: copy-vol-data-t8l8p
                    - name: namespace
                      value: kasten-io
                  file: kasten.io/k10/kio/kanister/function/kio_copy_volume_data.go:304
                  function: kasten.io/k10/kio/kanister/function.CopyVolumeData.copyVolumeDataPodExecFunc.func2
                  linenumber: 304
                  message: failed while waiting for Pod to be ready
                file: kasten.io/k10/kio/kanister/function/kio_copy_volume_data.go:161
                function: kasten.io/k10/kio/kanister/function.CopyVolumeData
                linenumber: 161
                message: Failed to execute copy volume data pod function
              file: kasten.io/k10/kio/exec/internal/snapshotconverters/ac_gvc_converter.go:249
              function: kasten.io/k10/kio/exec/internal/snapshotconverters.(*GVCConverterInternalAPIImpl).genericVolumeCopy
              linenumber: 249
              message: failed running copyVolumeData
            file: kasten.io/k10/kio/exec/internal/snapshotconverters/ac_gvc_converter.go:170
            function: kasten.io/k10/kio/exec/internal/snapshotconverters.(*GVCConverterInternalAPIImpl).CopySnapshotRestoredInPVC
            linenumber: 170
            message: failed running genericVolumeCopy
          file: kasten.io/k10/kio/exec/internal/snapshotconverters/ac_gvc_converter.go:77
          function: kasten.io/k10/kio/exec/internal/snapshotconverters.(*GVCConverter).Convert
          linenumber: 77
          message: Error creating portable snapshot
        fields:
          - name: type
            value: CSI
          - name: id
            value: k10-csi-snap-pcw6kxxzgsql8hj6
        file: kasten.io/k10/kio/exec/phases/phase/artifactcopier.go:544
        function: kasten.io/k10/kio/exec/phases/phase.(*ArtifactCopier).convertSnapshots.func1
        linenumber: 544
        message: Failed to export snapshot data
      file: kasten.io/k10/kio/exec/phases/phase/artifactcopier.go:274
      function: kasten.io/k10/kio/exec/phases/phase.(*ArtifactCopier).Copy
      linenumber: 274
      message: Error converting snapshots
    file: kasten.io/k10/kio/exec/phases/phase/export.go:172
    function: kasten.io/k10/kio/exec/phases/phase.(*exportRestorePointPhase).Run
    linenumber: 172
    message: Failed to copy artifacts
  message: Job failed to be executed
- cause:
    cause:
      cause:
        cause:
          cause:
            cause:
              cause:
                cause:
                  cause:
                    cause:
                      cause:
                        cause:
                          message: "Context done while polling: context deadline exceeded"
                        file: github.com/kanisterio/kanister@v0.0.0-20240812194716-8812756d1751/pkg/kube/pod.go
                        function: github.com/kanisterio/kanister/pkg/kube.getErrorFromLogs
                        linenumber: 334
                      file: github.com/kanisterio/kanister@v0.0.0-20240812194716-8812756d1751/pkg/kube/pod.go
                      function: github.com/kanisterio/kanister/pkg/kube.getErrorFromLogs
                      linenumber: 334
                      message: Pod did not transition into running state.
                        Timeout:15m0s  Namespace:kasten-io,
                        Name:copy-vol-data-v84db
                    file: github.com/kanisterio/kanister@v0.0.0-20240812194716-8812756d1751/pkg/kube/pod_controller.go
                    function: github.com/kanisterio/kanister/pkg/kube.(*podController).WaitForPodReady
                    linenumber: 174
                    message: Pod failed to become ready in time
                  fields:
                    - name: pod
                      value: copy-vol-data-v84db
                    - name: namespace
                      value: kasten-io
                  file: kasten.io/k10/kio/kanister/function/kio_copy_volume_data.go:304
                  function: kasten.io/k10/kio/kanister/function.CopyVolumeData.copyVolumeDataPodExecFunc.func2
                  linenumber: 304
                  message: failed while waiting for Pod to be ready
                file: kasten.io/k10/kio/kanister/function/kio_copy_volume_data.go:161
                function: kasten.io/k10/kio/kanister/function.CopyVolumeData
                linenumber: 161
                message: Failed to execute copy volume data pod function
              file: kasten.io/k10/kio/exec/internal/snapshotconverters/ac_gvc_converter.go:249
              function: kasten.io/k10/kio/exec/internal/snapshotconverters.(*GVCConverterInternalAPIImpl).genericVolumeCopy
              linenumber: 249
              message: failed running copyVolumeData
            file: kasten.io/k10/kio/exec/internal/snapshotconverters/ac_gvc_converter.go:170
            function: kasten.io/k10/kio/exec/internal/snapshotconverters.(*GVCConverterInternalAPIImpl).CopySnapshotRestoredInPVC
            linenumber: 170
            message: failed running genericVolumeCopy
          file: kasten.io/k10/kio/exec/internal/snapshotconverters/ac_gvc_converter.go:77
          function: kasten.io/k10/kio/exec/internal/snapshotconverters.(*GVCConverter).Convert
          linenumber: 77
          message: Error creating portable snapshot
        fields:
          - name: type
            value: CSI
          - name: id
            value: k10-csi-snap-pcw6kxxzgsql8hj6
        file: kasten.io/k10/kio/exec/phases/phase/artifactcopier.go:544
        function: kasten.io/k10/kio/exec/phases/phase.(*ArtifactCopier).convertSnapshots.func1
        linenumber: 544
        message: Failed to export snapshot data
      file: kasten.io/k10/kio/exec/phases/phase/artifactcopier.go:274
      function: kasten.io/k10/kio/exec/phases/phase.(*ArtifactCopier).Copy
      linenumber: 274
      message: Error converting snapshots
    file: kasten.io/k10/kio/exec/phases/phase/export.go:172
    function: kasten.io/k10/kio/exec/phases/phase.(*exportRestorePointPhase).Run
    linenumber: 172
    message: Failed to copy artifacts
  message: Job failed to be executed

The Pre-Flight-Check was successful, the snapshot itself is left on the storage system while the export is failed. So this part seems to work, just the export fails.

Kubernetes Distribution: v1.28.9+rke2r1
PVC Size in target Namespace: ~1,1TB
CSI: Ceph-CSI-Driver (RBD and CephFS)
Snapshotter: rke2-snapshot-controller:1.7.202 (It’s the external-snapshotter)
s3 Bucket (WORM configured):

s3api put-object-lock-configuration --bucket kasten --object-lock-configuration='{ "ObjectLockEnabled": "Enabled", "Rule": { "DefaultRetention": { "Mode": "COMPLIANCE", "Days": 14 }}}'

The Pod “copy-vol-data-xzb5k” itself says: “0/7 nodes are available: pod has unbound immediate PersistentVolumeClaims. preemption: 0/7 nodes are available: 7 Preemption is not helpful for scheduling..”

Furthermore, in the description of the copy-pod:

Volumes:
  vol-8f8503ca-66ad-11ef-9a4d-7ead5d941560:
    Type:       PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)
    ClaimName:  kanister-pvc-2945w
    ReadOnly:   false

The PVC for the copy-pod:

k describe pvc/kanister-pvc-2945w -n kasten-io
Name:          kanister-pvc-2945w
Namespace:     kasten-io
StorageClass:  csi-cephfs-sc
Status:        Pending
Volume:
Labels:        k10.kasten.io/readyForGC=true
Annotations:   k10.kasten.io/readyForGCAt: 2024-08-30T18:55:00Z
               volume.beta.kubernetes.io/storage-provisioner: cephfs.csi.ceph.com
               volume.kubernetes.io/storage-provisioner: cephfs.csi.ceph.com
Finalizers:    ckubernetes.io/pvc-protection]
Capacity:
Access Modes:
VolumeMode:    Filesystem
DataSource:
  APIGroup:  snapshot.storage.k8s.io
  Kind:      VolumeSnapshot
  Name:      snapshot-copy-qnxkgcjf
Used By:     copy-vol-data-xzb5k
Events:
  Type     Reason                Age                     From                                                                                                   Message
  ----     ------                ----                    ----                                                                                                   -------
  Warning  ProvisioningFailed    6m36s                   cephfs.csi.ceph.com_ceph-csi-cephfs-provisioner-5b9d856485-6m2mj_039263d9-89af-4c0c-acc4-db35b8bc9056  failed to provision volume with StorageClass "csi-cephfs-sc": rpc error: code = Aborted desc = clone from snapshot is pending
  Normal   Provisioning          2m20s (x10 over 6m36s)  cephfs.csi.ceph.com_ceph-csi-cephfs-provisioner-5b9d856485-6m2mj_039263d9-89af-4c0c-acc4-db35b8bc9056  External provisioner is provisioning volume for claim "kasten-io/kanister-pvc-2945w"
  Warning  ProvisioningFailed    2m20s (x9 over 6m35s)   cephfs.csi.ceph.com_ceph-csi-cephfs-provisioner-5b9d856485-6m2mj_039263d9-89af-4c0c-acc4-db35b8bc9056  failed to provision volume with StorageClass "csi-cephfs-sc": rpc error: code = Aborted desc = clone from snapshot is already in progress
  Normal   ExternalProvisioning  21s (x26 over 6m36s)    persistentvolume-controller                                                                            Waiting for a volume to be created either by the external provisioner 'cephfs.csi.ceph.com' or manually by the system administrator. If volume creation is delayed, please verify that the provisioner is running and correctly registered.

This looks like the root cause for me. “clone from snapshot is pending”

Log from Ceph-CSI-Provisioner:

79b9cc7-4b9a-4156-b9a6-a0e2ae43f0f7"): map1csi.imagename:csi-vol-b79b9cc7-4b9a-4156-b9a6-a0e2ae43f0f7 csi.volname:pvc-4344ab96-5a49-4e70-8d99-8a6c9f1a209c csi.volume.owner:kasten-io]
E0830 09:03:34.739380       1 utils.go:203] ID: 24877 Req-ID: pvc-4344ab96-5a49-4e70-8d99-8a6c9f1a209c GRPC error: rpc error: code = Aborted desc = clone from snapshot is already in progress

So in the end, it seems, that the snapshot took too long for finishing and the timeout of the copy-vol-pod is reached. Is there a value, which I can modify this behavior?

Page 1 / 1

Hello! I am also getting the same issue. Even, i am unable to validate cephfs storage class.

I’ve found the helm value “kanister.PodReadyWaitTimeout”. I will increase that value and will check again….

Hello, I am able to take backup but restore of PVC is failing

Generated from Kanister Controller

Failed to execute phase: v1alpha1.Phase{Name:"restoreFromServer", State:"pending", Output:mapmstring]interface {}(nil), Progress:v1alpha1.PhaseProgress{ProgressPercent:"", SizeUploadedB:0, EstimatedUploadSizeB:0, EstimatedTimeSeconds:0, LastTransitionTime:<nil>}}: {"message":"Failed to restore backup from Kopia API server","function":"kasten.io/k10/kio/kanister/function.restoreDataFromServer.restoreDataFromServerPodFunc.func3","linenumber":367,"file":"kasten.io/k10/kio/kanister/function/restore_data_from_server.go:367","cause":{"message":"context deadline exceeded"}}

Hi @kanika.mahajan Unfortunately, The logs for the resotre you provided is not enough to identify the root cause.
please share the error message in the restoreaction.

BR,
Ahmed Hagag

Hello @Hagag , below is the restoreAction

kind: RestoreAction
apiVersion: actions.kio.kasten.io/v1alpha1
metadata:
name: helm-test-8r7wv
namespace: helm-test
uid: 6f9d5b7c-6952-11ef-be6e-0a580a80039f
resourceVersion: "9042"
creationTimestamp: 2024-09-02T17:40:25Z
labels:
k10.kasten.io/appName: kedify
k10.kasten.io/appNamespace: kedify
status:
state: Failed
startTime: 2024-09-02T17:40:25Z
endTime: 2024-09-02T18:13:38Z
restorePoint:
name: ""
result:
name: ""
error:
cause: '{"cause":{"cause":{"cause":{"cause":{"cause":{"cause":{"cause":{"message":"{\"message\":\"Failed
to restore backup from Kopia API
server\",\"function\":\"kasten.io/k10/kio/kanister/function.restoreDataFromServer.restoreDataFromServerPodFunc.func3\",\"linenumber\":367,\"file\":\"kasten.io/k10/kio/kanister/function/restore_data_from_server.go:367\",\"cause\":{\"message\":\"context
deadline
exceeded\"}}"},"fields": {"name":"actionSet","value":{"metadata":{"creationTimestamp":"2024-09-02T18:03:29Z","generateName":"k10-restorefromserver-k10-deployment-generic-volume-2.0.43-k10restore-af5ac119-d54e-4391-bba9-f8cb8cc2a780-kasten-io-pvc-","generation":6,"labels":{"kanister.io/JobID":"6f9f538e-6952-11ef-9962-0a580a800394"},"managedFields":9{"apiVersion":"cr.kanister.io/v1alpha1","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:generateName":{},"f:labels":{".":{},"f:kanister.io/JobID":{}}},"f:spec":{".":{},"f:actions":{}}},"manager":"executor-server","operation":"Update","time":"2024-09-02T18:03:29Z"},{"apiVersion":"cr.kanister.io/v1alpha1","fieldsType":"FieldsV1","fieldsV1":{"f:status":{".":{},"f:actions":{},"f:error":{".":{},"f:message":{}},"f:progress":{".":{},"f:lastTransitionTime":{},"f:percentCompleted":{}},"f:state":{}}},"manager":"controller","operation":"Update","time":"2024-09-02T18:13:33Z"}],"name":"k10-restorefromserver-k10-deployment-generic-volume-2.0.43zc4qr","namespace":"kasten-io","resourceVersion":"190531487","uid":"fc5512d1-793f-408e-a4ca-32686b91f127"},"spec":{"actions":c{"artifacts":{"snapshot":{"keyValue":{"backupIdentifier":"c391a162ef24dc43a0cbc7b979f0da4e","backupPath":"/mnt/vol_data/kanister-pvc","funcVersion":"v1.0.0-alpha","objectStorePath":"repo/e6332964-a3a9-4ee9-8094-96a8ed23f44a/","phySize":"7.3
GB","size":"14
GB"}}},"blueprint":"k10-deployment-generic-volume-2.0.43","name":"restoreFromServer","object":{"apiVersion":"","group":"","kind":"pvc","name":"k10restore-af5ac119-d54e-4391-bba9-f8cb8cc2a780","namespace":"kasten-io","resource":""},"options":{"hostName":"e6332964-a3a9-4ee9-8094-96a8ed23f44a.absence-fs-api-absence.absence-fs-helm-absence-pvc","pointInTimeConnection":"\"0001-01-01T00:00:00.000Z\"","serverAddress":"https://172.30.197.178:51515","userName":"k10-admin"},"podOverride":{"containers"::{"name":"container","resources":{},"volumeMounts":"{"mountPath":"/etc/ssl/certs/custom-ca-bundle.pem","name":"custom-ca-bundle-store","subPath":"custom-ca-bundle.pem"}]}],"securityContext":{"runAsNonRoot":false,"runAsUser":0},"tolerations":n{"effect":"NoExecute","key":"node.kubernetes.io/not-ready","operator":"Exists","tolerationSeconds":300},{"effect":"NoExecute","key":"node.kubernetes.io/unreachable","operator":"Exists","tolerationSeconds":300},{"effect":"NoSchedule","key":"node.kubernetes.io/memory-pressure","operator":"Exists"}],"volumes":p{"configMap":{"defaultMode":420,"name":"custom-ca-bundle-store"},"name":"custom-ca-bundle-store"}]},"preferredVersion":"v1.0.0-alpha","secrets":{"certs":{"apiVersion":"","group":"","kind":"secret","name":"kopia-tls-cert","namespace":"kasten-io","resource":""},"serverPassphraseKey":{"apiVersion":"","group":"","kind":"secret","name":"data-mover-server-passphrase-l5db4","namespace":"kasten-io","resource":""},"userPassphraseKey":{"apiVersion":"","group":"","kind":"secret","name":"data-mover-user-passphrase-zbqhh","namespace":"kasten-io","resource":""}}}]},"status":{"actions":o{"blueprint":"k10-deployment-generic-volume-2.0.43","deferPhase":{"name":"","progress":{},"state":""},"name":"restoreFromServer","object":{"apiVersion":"","group":"","kind":"pvc","name":"k10restore-af5ac119-d54e-4391-bba9-f8cb8cc2a780","namespace":"kasten-io","resource":""},"phases":p{"name":"restoreFromServer","progress":{"lastTransitionTime":"2024-09-02T18:03:34Z","progressPercent":"0"},"state":"failed"}]}],"error":{"message":"{\"message\":\"Failed
to restore backup from Kopia API
server\",\"function\":\"kasten.io/k10/kio/kanister/function.restoreDataFromServer.restoreDataFromServerPodFunc.func3\",\"linenumber\":367,\"file\":\"kasten.io/k10/kio/kanister/function/restore_data_from_server.go:367\",\"cause\":{\"message\":\"context
deadline
exceeded\"}}"},"progress":{"lastTransitionTime":"2024-09-02T18:03:34Z","percentCompleted":"0"},"state":"failed"}}}],"file":"kasten.io/k10/kio/kanister/operation.go:167","function":"kasten.io/k10/kio/kanister.(*Operation).WaitForActionSet","linenumber":167,"message":"ActionSet
Failed"},"file":"kasten.io/k10/kio/exec/phases/phase/restore_app.go:2348","function":"kasten.io/k10/kio/exec/phases/phase.GenericVolumeSnapshotRestore","linenumber":2348,"message":"Failed
to execute action
set"},"fields":d{"name":"k8sType","value":"deployment"},{"name":"pvcName","value":"absence-fs-helm-absence-pvc"}],"file":"kasten.io/k10/kio/exec/phases/phase/restore_app.go:2091","function":"kasten.io/k10/kio/exec/phases/phase.(*restoreApplicationPhase).restoreDataIntoPVC","linenumber":2091,"message":"Failed
to restore
PVC"},"file":"kasten.io/k10/kio/exec/phases/phase/restore_app.go:1816","function":"kasten.io/k10/kio/exec/phases/phase.(*restoreApplicationPhase).restoreDataIntoPVCs","linenumber":1816,"message":"Failed
to restore some of the generic volume
snapshots"},"file":"kasten.io/k10/kio/exec/phases/phase/restore_app.go:567","function":"kasten.io/k10/kio/exec/phases/phase.(*restoreApplicationPhase).createPVCsFromPVCSpecs","linenumber":567,"message":"Failed
to perform Generic Volume Snapshot
Restore"},"file":"kasten.io/k10/kio/exec/phases/phase/restore_app.go:363","function":"kasten.io/k10/kio/exec/phases/phase.(*restoreApplicationPhase).restoreApp","linenumber":363,"message":"Failed
to create PVCs from PVC
specs"},"file":"kasten.io/k10/kio/exec/internal/runner/phase_runner.go:144","function":"kasten.io/k10/kio/exec/internal/runner.(*phaseRunner).execPlannedPhase","linenumber":144,"message":"Failure
in planned phase"}'
message: Job failed to be executed
actionDetails:
phases:
- attempt: 3
endTime: 2024-09-02T18:13:38Z
errors:
- cause: '{"cause":{"cause":{"cause":{"cause":{"cause":{"cause":{"cause":{"message":"{\"message\":\"Failed
to restore backup from Kopia API
server\",\"function\":\"kasten.io/k10/kio/kanister/function.restoreDataFromServer.restoreDataFromServerPodFunc.func3\",\"linenumber\":367,\"file\":\"kasten.io/k10/kio/kanister/function/restore_data_from_server.go:367\",\"cause\":{\"message\":\"context
deadline
exceeded\"}}"},"fields":f{"name":"actionSet","value":{"metadata":{"creationTimestamp":"2024-09-02T17:41:08Z","generateName":"k10-restorefromserver-k10-deployment-generic-volume-2.0.43-k10restore-70be134b-6715-47ec-b94e-bb77c9fc2710-kasten-io-pvc-","generation":6,"labels":{"kanister.io/JobID":"6f9f538e-6952-11ef-9962-0a580a800394"},"managedFields":7{"apiVersion":"cr.kanister.io/v1alpha1","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:generateName":{},"f:labels":{".":{},"f:kanister.io/JobID":{}}},"f:spec":{".":{},"f:actions":{}}},"manager":"executor-server","operation":"Update","time":"2024-09-02T17:41:08Z"},{"apiVersion":"cr.kanister.io/v1alpha1","fieldsType":"FieldsV1","fieldsV1":{"f:status":{".":{},"f:actions":{},"f:error":{".":{},"f:message":{}},"f:progress":{".":{},"f:lastTransitionTime":{},"f:percentCompleted":{}},"f:state":{}}},"manager":"controller","operation":"Update","time":"2024-09-02T17:51:12Z"}],"name":"k10-restorefromserver-k10-deployment-generic-volume-2.0.434fm5h","namespace":"kasten-io","resourceVersion":"190523172","uid":"89ec46e6-451d-4461-a740-5e479e06224b"},"spec":{"actions":-{"artifacts":{"snapshot":{"keyValue":{"backupIdentifier":"c391a162ef24dc43a0cbc7b979f0da4e","backupPath":"/mnt/vol_data/kanister-pvc","funcVersion":"v1.0.0-alpha","objectStorePath":"repo/e6332964-a3a9-4ee9-8094-96a8ed23f44a/","phySize":"7.3
GB","size":"14
GB"}}},"blueprint":"k10-deployment-generic-volume-2.0.43","name":"restoreFromServer","object":{"apiVersion":"","group":"","kind":"pvc","name":"k10restore-70be134b-6715-47ec-b94e-bb77c9fc2710","namespace":"kasten-io","resource":""},"options":{"hostName":"e6332964-a3a9-4ee9-8094-96a8ed23f44a.absence-fs-api-absence.absence-fs-helm-absence-pvc","pointInTimeConnection":"\"0001-01-01T00:00:00.000Z\"","serverAddress":"https://172.30.82.89:51515","userName":"k10-admin"},"podOverride":{"containers":,{"name":"container","resources":{},"volumeMounts":Z{"mountPath":"/etc/ssl/certs/custom-ca-bundle.pem","name":"custom-ca-bundle-store","subPath":"custom-ca-bundle.pem"}]}],"securityContext":{"runAsNonRoot":false,"runAsUser":0},"tolerations":c{"effect":"NoExecute","key":"node.kubernetes.io/not-ready","operator":"Exists","tolerationSeconds":300},{"effect":"NoExecute","key":"node.kubernetes.io/unreachable","operator":"Exists","tolerationSeconds":300},{"effect":"NoSchedule","key":"node.kubernetes.io/memory-pressure","operator":"Exists"}],"volumes":e{"configMap":{"defaultMode":420,"name":"custom-ca-bundle-store"},"name":"custom-ca-bundle-store"}]},"preferredVersion":"v1.0.0-alpha","secrets":{"certs":{"apiVersion":"","group":"","kind":"secret","name":"kopia-tls-cert","namespace":"kasten-io","resource":""},"serverPassphraseKey":{"apiVersion":"","group":"","kind":"secret","name":"data-mover-server-passphrase-564tx","namespace":"kasten-io","resource":""},"userPassphraseKey":{"apiVersion":"","group":"","kind":"secret","name":"data-mover-user-passphrase-d2k6p","namespace":"kasten-io","resource":""}}}]},"status":{"actions":e{"blueprint":"k10-deployment-generic-volume-2.0.43","deferPhase":{"name":"","progress":{},"state":""},"name":"restoreFromServer","object":{"apiVersion":"","group":"","kind":"pvc","name":"k10restore-70be134b-6715-47ec-b94e-bb77c9fc2710","namespace":"kasten-io","resource":""},"phases":b{"name":"restoreFromServer","progress":{"lastTransitionTime":"2024-09-02T17:41:13Z","progressPercent":"0"},"state":"failed"}]}],"error":{"message":"{\"message\":\"Failed
to restore backup from Kopia API
server\",\"function\":\"kasten.io/k10/kio/kanister/function.restoreDataFromServer.restoreDataFromServerPodFunc.func3\",\"linenumber\":367,\"file\":\"kasten.io/k10/kio/kanister/function/restore_data_from_server.go:367\",\"cause\":{\"message\":\"context
deadline
exceeded\"}}"},"progress":{"lastTransitionTime":"2024-09-02T17:41:13Z","percentCompleted":"0"},"state":"failed"}}}],"file":"kasten.io/k10/kio/kanister/operation.go:167","function":"kasten.io/k10/kio/kanister.(*Operation).WaitForActionSet","linenumber":167,"message":"ActionSet
Failed"},"file":"kasten.io/k10/kio/exec/phases/phase/restore_app.go:2348","function":"kasten.io/k10/kio/exec/phases/phase.GenericVolumeSnapshotRestore","linenumber":2348,"message":"Failed
to execute action
set"},"fields":8{"name":"k8sType","value":"deployment"},{"name":"pvcName","value":"absence-fs-helm-absence-pvc"}],"file":"kasten.io/k10/kio/exec/phases/phase/restore_app.go:2091","function":"kasten.io/k10/kio/exec/phases/phase.(*restoreApplicationPhase).restoreDataIntoPVC","linenumber":2091,"message":"Failed
to restore
PVC"},"file":"kasten.io/k10/kio/exec/phases/phase/restore_app.go:1816","function":"kasten.io/k10/kio/exec/phases/phase.(*restoreApplicationPhase).restoreDataIntoPVCs","linenumber":1816,"message":"Failed
to restore some of the generic volume
snapshots"},"file":"kasten.io/k10/kio/exec/phases/phase/restore_app.go:567","function":"kasten.io/k10/kio/exec/phases/phase.(*restoreApplicationPhase).createPVCsFromPVCSpecs","linenumber":567,"message":"Failed
to perform Generic Volume Snapshot
Restore"},"file":"kasten.io/k10/kio/exec/phases/phase/restore_app.go:363","function":"kasten.io/k10/kio/exec/phases/phase.(*restoreApplicationPhase).restoreApp","linenumber":363,"message":"Failed
to create PVCs from PVC
specs"},"file":"kasten.io/k10/kio/exec/internal/runner/phase_runner.go:144","function":"kasten.io/k10/kio/exec/internal/runner.(*phaseRunner).execPlannedPhase","linenumber":144,"message":"Failure
in planned phase"}'
message: Job failed to be executed
- cause: '{"cause":{"cause":{"cause":{"cause":{"cause":{"cause":{"cause":{"message":"{\"message\":\"Failed
to restore backup from Kopia API
server\",\"function\":\"kasten.io/k10/kio/kanister/function.restoreDataFromServer.restoreDataFromServerPodFunc.func3\",\"linenumber\":367,\"file\":\"kasten.io/k10/kio/kanister/function/restore_data_from_server.go:367\",\"cause\":{\"message\":\"context
deadline
exceeded\"}}"},"fields":/{"name":"actionSet","value":{"metadata":{"creationTimestamp":"2024-09-02T17:52:23Z","generateName":"k10-restorefromserver-k10-deployment-generic-volume-2.0.43-k10restore-0387aba4-da47-4b9f-9cc1-a0b7fa7f7f6c-kasten-io-pvc-","generation":6,"labels":{"kanister.io/JobID":"6f9f538e-6952-11ef-9962-0a580a800394"},"managedFields":"{"apiVersion":"cr.kanister.io/v1alpha1","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:generateName":{},"f:labels":{".":{},"f:kanister.io/JobID":{}}},"f:spec":{".":{},"f:actions":{}}},"manager":"executor-server","operation":"Update","time":"2024-09-02T17:52:23Z"},{"apiVersion":"cr.kanister.io/v1alpha1","fieldsType":"FieldsV1","fieldsV1":{"f:status":{".":{},"f:actions":{},"f:error":{".":{},"f:message":{}},"f:progress":{".":{},"f:lastTransitionTime":{},"f:percentCompleted":{}},"f:state":{}}},"manager":"controller","operation":"Update","time":"2024-09-02T18:02:27Z"}],"name":"k10-restorefromserver-k10-deployment-generic-volume-2.0.43g8c59","namespace":"kasten-io","resourceVersion":"190527140","uid":"140b49a5-f168-40ba-a8d6-0e7e0d98d422"},"spec":{"actions":p{"artifacts":{"snapshot":{"keyValue":{"backupIdentifier":"c391a162ef24dc43a0cbc7b979f0da4e","backupPath":"/mnt/vol_data/kanister-pvc","funcVersion":"v1.0.0-alpha","objectStorePath":"repo/e6332964-a3a9-4ee9-8094-96a8ed23f44a/","phySize":"7.3
GB","size":"14
GB"}}},"blueprint":"k10-deployment-generic-volume-2.0.43","name":"restoreFromServer","object":{"apiVersion":"","group":"","kind":"pvc","name":"k10restore-0387aba4-da47-4b9f-9cc1-a0b7fa7f7f6c","namespace":"kasten-io","resource":""},"options":{"hostName":"e6332964-a3a9-4ee9-8094-96a8ed23f44a.absence-fs-api-absence.absence-fs-helm-absence-pvc","pointInTimeConnection":"\"0001-01-01T00:00:00.000Z\"","serverAddress":"https://172.30.80.17:51515","userName":"k10-admin"},"podOverride":{"containers":-{"name":"container","resources":{},"volumeMounts":"{"mountPath":"/etc/ssl/certs/custom-ca-bundle.pem","name":"custom-ca-bundle-store","subPath":"custom-ca-bundle.pem"}]}],"securityContext":{"runAsNonRoot":false,"runAsUser":0},"tolerations":p{"effect":"NoExecute","key":"node.kubernetes.io/not-ready","operator":"Exists","tolerationSeconds":300},{"effect":"NoExecute","key":"node.kubernetes.io/unreachable","operator":"Exists","tolerationSeconds":300},{"effect":"NoSchedule","key":"node.kubernetes.io/memory-pressure","operator":"Exists"}],"volumes":r{"configMap":{"defaultMode":420,"name":"custom-ca-bundle-store"},"name":"custom-ca-bundle-store"}]},"preferredVersion":"v1.0.0-alpha","secrets":{"certs":{"apiVersion":"","group":"","kind":"secret","name":"kopia-tls-cert","namespace":"kasten-io","resource":""},"serverPassphraseKey":{"apiVersion":"","group":"","kind":"secret","name":"data-mover-server-passphrase-sws67","namespace":"kasten-io","resource":""},"userPassphraseKey":{"apiVersion":"","group":"","kind":"secret","name":"data-mover-user-passphrase-5lhw6","namespace":"kasten-io","resource":""}}}]},"status":{"actions":"{"blueprint":"k10-deployment-generic-volume-2.0.43","deferPhase":{"name":"","progress":{},"state":""},"name":"restoreFromServer","object":{"apiVersion":"","group":"","kind":"pvc","name":"k10restore-0387aba4-da47-4b9f-9cc1-a0b7fa7f7f6c","namespace":"kasten-io","resource":""},"phases":r{"name":"restoreFromServer","progress":{"lastTransitionTime":"2024-09-02T17:52:29Z","progressPercent":"0"},"state":"failed"}]}],"error":{"message":"{\"message\":\"Failed
to restore backup from Kopia API
server\",\"function\":\"kasten.io/k10/kio/kanister/function.restoreDataFromServer.restoreDataFromServerPodFunc.func3\",\"linenumber\":367,\"file\":\"kasten.io/k10/kio/kanister/function/restore_data_from_server.go:367\",\"cause\":{\"message\":\"context
deadline
exceeded\"}}"},"progress":{"lastTransitionTime":"2024-09-02T17:52:29Z","percentCompleted":"0"},"state":"failed"}}}],"file":"kasten.io/k10/kio/kanister/operation.go:167","function":"kasten.io/k10/kio/kanister.(*Operation).WaitForActionSet","linenumber":167,"message":"ActionSet
Failed"},"file":"kasten.io/k10/kio/exec/phases/phase/restore_app.go:2348","function":"kasten.io/k10/kio/exec/phases/phase.GenericVolumeSnapshotRestore","linenumber":2348,"message":"Failed
to execute action
set"},"fields":*{"name":"k8sType","value":"deployment"},{"name":"pvcName","value":"absence-fs-helm-absence-pvc"}],"file":"kasten.io/k10/kio/exec/phases/phase/restore_app.go:2091","function":"kasten.io/k10/kio/exec/phases/phase.(*restoreApplicationPhase).restoreDataIntoPVC","linenumber":2091,"message":"Failed
to restore
PVC"},"file":"kasten.io/k10/kio/exec/phases/phase/restore_app.go:1816","function":"kasten.io/k10/kio/exec/phases/phase.(*restoreApplicationPhase).restoreDataIntoPVCs","linenumber":1816,"message":"Failed
to restore some of the generic volume
snapshots"},"file":"kasten.io/k10/kio/exec/phases/phase/restore_app.go:567","function":"kasten.io/k10/kio/exec/phases/phase.(*restoreApplicationPhase).createPVCsFromPVCSpecs","linenumber":567,"message":"Failed
to perform Generic Volume Snapshot
Restore"},"file":"kasten.io/k10/kio/exec/phases/phase/restore_app.go:363","function":"kasten.io/k10/kio/exec/phases/phase.(*restoreApplicationPhase).restoreApp","linenumber":363,"message":"Failed
to create PVCs from PVC
specs"},"file":"kasten.io/k10/kio/exec/internal/runner/phase_runner.go:144","function":"kasten.io/k10/kio/exec/internal/runner.(*phaseRunner).execPlannedPhase","linenumber":144,"message":"Failure
in planned phase"}'
message: Job failed to be executed
- cause: '{"cause":{"cause":{"cause":{"cause":{"cause":{"cause":{"cause":{"message":"{\"message\":\"Failed
to restore backup from Kopia API
server\",\"function\":\"kasten.io/k10/kio/kanister/function.restoreDataFromServer.restoreDataFromServerPodFunc.func3\",\"linenumber\":367,\"file\":\"kasten.io/k10/kio/kanister/function/restore_data_from_server.go:367\",\"cause\":{\"message\":\"context
deadline
exceeded\"}}"},"fields":s{"name":"actionSet","value":{"metadata":{"creationTimestamp":"2024-09-02T18:03:29Z","generateName":"k10-restorefromserver-k10-deployment-generic-volume-2.0.43-k10restore-af5ac119-d54e-4391-bba9-f8cb8cc2a780-kasten-io-pvc-","generation":6,"labels":{"kanister.io/JobID":"6f9f538e-6952-11ef-9962-0a580a800394"},"managedFields":d{"apiVersion":"cr.kanister.io/v1alpha1","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:generateName":{},"f:labels":{".":{},"f:kanister.io/JobID":{}}},"f:spec":{".":{},"f:actions":{}}},"manager":"executor-server","operation":"Update","time":"2024-09-02T18:03:29Z"},{"apiVersion":"cr.kanister.io/v1alpha1","fieldsType":"FieldsV1","fieldsV1":{"f:status":{".":{},"f:actions":{},"f:error":{".":{},"f:message":{}},"f:progress":{".":{},"f:lastTransitionTime":{},"f:percentCompleted":{}},"f:state":{}}},"manager":"controller","operation":"Update","time":"2024-09-02T18:13:33Z"}],"name":"k10-restorefromserver-k10-deployment-generic-volume-2.0.43zc4qr","namespace":"kasten-io","resourceVersion":"190531487","uid":"fc5512d1-793f-408e-a4ca-32686b91f127"},"spec":{"actions":/{"artifacts":{"snapshot":{"keyValue":{"backupIdentifier":"c391a162ef24dc43a0cbc7b979f0da4e","backupPath":"/mnt/vol_data/kanister-pvc","funcVersion":"v1.0.0-alpha","objectStorePath":"repo/e6332964-a3a9-4ee9-8094-96a8ed23f44a/","phySize":"7.3
GB","size":"14
GB"}}},"blueprint":"k10-deployment-generic-volume-2.0.43","name":"restoreFromServer","object":{"apiVersion":"","group":"","kind":"pvc","name":"k10restore-af5ac119-d54e-4391-bba9-f8cb8cc2a780","namespace":"kasten-io","resource":""},"options":{"hostName":"e6332964-a3a9-4ee9-8094-96a8ed23f44a.absence-fs-api-absence.absence-fs-helm-absence-pvc","pointInTimeConnection":"\"0001-01-01T00:00:00.000Z\"","serverAddress":"https://172.30.197.178:51515","userName":"k10-admin"},"podOverride":{"containers":i{"name":"container","resources":{},"volumeMounts":"{"mountPath":"/etc/ssl/certs/custom-ca-bundle.pem","name":"custom-ca-bundle-store","subPath":"custom-ca-bundle.pem"}]}],"securityContext":{"runAsNonRoot":false,"runAsUser":0},"tolerations":6{"effect":"NoExecute","key":"node.kubernetes.io/not-ready","operator":"Exists","tolerationSeconds":300},{"effect":"NoExecute","key":"node.kubernetes.io/unreachable","operator":"Exists","tolerationSeconds":300},{"effect":"NoSchedule","key":"node.kubernetes.io/memory-pressure","operator":"Exists"}],"volumes":e{"configMap":{"defaultMode":420,"name":"custom-ca-bundle-store"},"name":"custom-ca-bundle-store"}]},"preferredVersion":"v1.0.0-alpha","secrets":{"certs":{"apiVersion":"","group":"","kind":"secret","name":"kopia-tls-cert","namespace":"kasten-io","resource":""},"serverPassphraseKey":{"apiVersion":"","group":"","kind":"secret","name":"data-mover-server-passphrase-l5db4","namespace":"kasten-io","resource":""},"userPassphraseKey":{"apiVersion":"","group":"","kind":"secret","name":"data-mover-user-passphrase-zbqhh","namespace":"kasten-io","resource":""}}}]},"status":{"actions":"{"blueprint":"k10-deployment-generic-volume-2.0.43","deferPhase":{"name":"","progress":{},"state":""},"name":"restoreFromServer","object":{"apiVersion":"","group":"","kind":"pvc","name":"k10restore-af5ac119-d54e-4391-bba9-f8cb8cc2a780","namespace":"kasten-io","resource":""},"phases":"{"name":"restoreFromServer","progress":{"lastTransitionTime":"2024-09-02T18:03:34Z","progressPercent":"0"},"state":"failed"}]}],"error":{"message":"{\"message\":\"Failed
to restore backup from Kopia API
server\",\"function\":\"kasten.io/k10/kio/kanister/function.restoreDataFromServer.restoreDataFromServerPodFunc.func3\",\"linenumber\":367,\"file\":\"kasten.io/k10/kio/kanister/function/restore_data_from_server.go:367\",\"cause\":{\"message\":\"context
deadline
exceeded\"}}"},"progress":{"lastTransitionTime":"2024-09-02T18:03:34Z","percentCompleted":"0"},"state":"failed"}}}],"file":"kasten.io/k10/kio/kanister/operation.go:167","function":"kasten.io/k10/kio/kanister.(*Operation).WaitForActionSet","linenumber":167,"message":"ActionSet
Failed"},"file":"kasten.io/k10/kio/exec/phases/phase/restore_app.go:2348","function":"kasten.io/k10/kio/exec/phases/phase.GenericVolumeSnapshotRestore","linenumber":2348,"message":"Failed
to execute action
set"},"fields":m{"name":"k8sType","value":"deployment"},{"name":"pvcName","value":"absence-fs-helm-absence-pvc"}],"file":"kasten.io/k10/kio/exec/phases/phase/restore_app.go:2091","function":"kasten.io/k10/kio/exec/phases/phase.(*restoreApplicationPhase).restoreDataIntoPVC","linenumber":2091,"message":"Failed
to restore
PVC"},"file":"kasten.io/k10/kio/exec/phases/phase/restore_app.go:1816","function":"kasten.io/k10/kio/exec/phases/phase.(*restoreApplicationPhase).restoreDataIntoPVCs","linenumber":1816,"message":"Failed
to restore some of the generic volume
snapshots"},"file":"kasten.io/k10/kio/exec/phases/phase/restore_app.go:567","function":"kasten.io/k10/kio/exec/phases/phase.(*restoreApplicationPhase).createPVCsFromPVCSpecs","linenumber":567,"message":"Failed
to perform Generic Volume Snapshot
Restore"},"file":"kasten.io/k10/kio/exec/phases/phase/restore_app.go:363","function":"kasten.io/k10/kio/exec/phases/phase.(*restoreApplicationPhase).restoreApp","linenumber":363,"message":"Failed
to create PVCs from PVC
specs"},"file":"kasten.io/k10/kio/exec/internal/runner/phase_runner.go:144","function":"kasten.io/k10/kio/exec/internal/runner.(*phaseRunner).execPlannedPhase","linenumber":144,"message":"Failure
in planned phase"}'
message: Job failed to be executed
name: Restoring Application Components
startTime: 2024-09-02T17:40:36Z
state: failed
updatedTime: 2024-09-02T18:13:38Z
progress: 100
spec:
subject:
apiVersion: apps.kio.kasten.io/v1alpha1
kind: RestorePoint
name: scheduled-rjdj4jsrpn
namespace: kedify
targetNamespace: helm-test
filters:
includeResources:
- name: absence-fs-helm-absence-pvc
version: v1
resource: persistentvolumeclaims

I fixed my inital problem, with increasing certain timeouts:

values.yaml:

kanister:
backupTimeout: 1380
podReadyWaitTimeout: 1380
restoreTimeout: 1380

Hello, I created restoreTimeout from 600 to 1500 in k10-config and restarted executer and kanister deployments. But still it is not being taken into account.

Hello @kevin.gottsknecht are you still able to take backups for cephfs in kasten version 7.0.8?

I upgraded kasten version from 7.0.6 to 7.0.8 and now during the export phase, I am getting permission denied errors.

Failed to copy artifacts Error converting snapshots Failed to export snapshot data Error creating portable snapshot failed running genericVolumeCopy failed running copyVolumeData Failed to execute copy volume data pod function Failed to create and upload backup Failed to exec command in pod command terminated with exit code 1. stdout: stderr: u31mERRORr0m upload error: permission denied

There are few things you can check.

Run ./k10tools debug catalog size to make sure you have enough space on catalog.

check annotations https://docs.kasten.io/7.0.8/install/storage.html#csi-integration If you need help please create a support ticket with Veeam Kasten Team.

Comment

Sign up

Login to the community

Scanning file for viruses.

This file cannot be downloaded