Skip to main content
Solved

Export Job fails (error: context deadline exceeded)

K

Hello, 

I’m testing Veeam Kasten for the first time, but unfortunately it won't work as expected. I’ve created a backup policy with a snapshot and an export to a s3 rados-gw (ceph) endpoint.

I’ve got the following error message while exporting the snapshot:

1- cause:
2    cause:
3      cause:
4        cause:
5          cause:
6            cause:
7              cause:
8                cause:
9                  cause:
10                    cause:
11                      cause:
12                        message: "client rate limiter Wait returned an error: context deadline exceeded"
13                      file: github.com/kanisterio/kanister@v0.0.0-20240812194716-8812756d1751/pkg/kube/pod.go
14                      function: github.com/kanisterio/kanister/pkg/kube.WaitForPodReady
15                      linenumber: 384
16                      message: Pod did not transition into running state.
17                        Timeout:15m0s  Namespace:kasten-io,
18                        Name:copy-vol-data-rpr56
19                    file: github.com/kanisterio/kanister@v0.0.0-20240812194716-8812756d1751/pkg/kube/pod_controller.go
20                    function: github.com/kanisterio/kanister/pkg/kube.(*podController).WaitForPodReady
21                    linenumber: 174
22                    message: Pod failed to become ready in time
23                  fields:
24                    - name: pod
25                      value: copy-vol-data-rpr56
26                    - name: namespace
27                      value: kasten-io
28                  file: kasten.io/k10/kio/kanister/function/kio_copy_volume_data.go:304
29                  function: kasten.io/k10/kio/kanister/function.CopyVolumeData.copyVolumeDataPodExecFunc.func2
30                  linenumber: 304
31                  message: failed while waiting for Pod to be ready
32                file: kasten.io/k10/kio/kanister/function/kio_copy_volume_data.go:161
33                function: kasten.io/k10/kio/kanister/function.CopyVolumeData
34                linenumber: 161
35                message: Failed to execute copy volume data pod function
36              file: kasten.io/k10/kio/exec/internal/snapshotconverters/ac_gvc_converter.go:249
37              function: kasten.io/k10/kio/exec/internal/snapshotconverters.(*GVCConverterInternalAPIImpl).genericVolumeCopy
38              linenumber: 249
39              message: failed running copyVolumeData
40            file: kasten.io/k10/kio/exec/internal/snapshotconverters/ac_gvc_converter.go:170
41            function: kasten.io/k10/kio/exec/internal/snapshotconverters.(*GVCConverterInternalAPIImpl).CopySnapshotRestoredInPVC
42            linenumber: 170
43            message: failed running genericVolumeCopy
44          file: kasten.io/k10/kio/exec/internal/snapshotconverters/ac_gvc_converter.go:77
45          function: kasten.io/k10/kio/exec/internal/snapshotconverters.(*GVCConverter).Convert
46          linenumber: 77
47          message: Error creating portable snapshot
48        fields:
49          - name: type
50            value: CSI
51          - name: id
52            value: k10-csi-snap-pcw6kxxzgsql8hj6
53        file: kasten.io/k10/kio/exec/phases/phase/artifactcopier.go:544
54        function: kasten.io/k10/kio/exec/phases/phase.(*ArtifactCopier).convertSnapshots.func1
55        linenumber: 544
56        message: Failed to export snapshot data
57      file: kasten.io/k10/kio/exec/phases/phase/artifactcopier.go:274
58      function: kasten.io/k10/kio/exec/phases/phase.(*ArtifactCopier).Copy
59      linenumber: 274
60      message: Error converting snapshots
61    file: kasten.io/k10/kio/exec/phases/phase/export.go:172
62    function: kasten.io/k10/kio/exec/phases/phase.(*exportRestorePointPhase).Run
63    linenumber: 172
64    message: Failed to copy artifacts
65  message: Job failed to be executed
66- cause:
67    cause:
68      cause:
69        cause:
70          cause:
71            cause:
72              cause:
73                cause:
74                  cause:
75                    cause:
76                      cause:
77                        message: "client rate limiter Wait returned an error: context deadline exceeded"
78                      file: github.com/kanisterio/kanister@v0.0.0-20240812194716-8812756d1751/pkg/kube/pod.go
79                      function: github.com/kanisterio/kanister/pkg/kube.WaitForPodReady
80                      linenumber: 384
81                      message: Pod did not transition into running state.
82                        Timeout:15m0s  Namespace:kasten-io,
83                        Name:copy-vol-data-t8l8p
84                    file: github.com/kanisterio/kanister@v0.0.0-20240812194716-8812756d1751/pkg/kube/pod_controller.go
85                    function: github.com/kanisterio/kanister/pkg/kube.(*podController).WaitForPodReady
86                    linenumber: 174
87                    message: Pod failed to become ready in time
88                  fields:
89                    - name: pod
90                      value: copy-vol-data-t8l8p
91                    - name: namespace
92                      value: kasten-io
93                  file: kasten.io/k10/kio/kanister/function/kio_copy_volume_data.go:304
94                  function: kasten.io/k10/kio/kanister/function.CopyVolumeData.copyVolumeDataPodExecFunc.func2
95                  linenumber: 304
96                  message: failed while waiting for Pod to be ready
97                file: kasten.io/k10/kio/kanister/function/kio_copy_volume_data.go:161
98                function: kasten.io/k10/kio/kanister/function.CopyVolumeData
99                linenumber: 161
100                message: Failed to execute copy volume data pod function
101              file: kasten.io/k10/kio/exec/internal/snapshotconverters/ac_gvc_converter.go:249
102              function: kasten.io/k10/kio/exec/internal/snapshotconverters.(*GVCConverterInternalAPIImpl).genericVolumeCopy
103              linenumber: 249
104              message: failed running copyVolumeData
105            file: kasten.io/k10/kio/exec/internal/snapshotconverters/ac_gvc_converter.go:170
106            function: kasten.io/k10/kio/exec/internal/snapshotconverters.(*GVCConverterInternalAPIImpl).CopySnapshotRestoredInPVC
107            linenumber: 170
108            message: failed running genericVolumeCopy
109          file: kasten.io/k10/kio/exec/internal/snapshotconverters/ac_gvc_converter.go:77
110          function: kasten.io/k10/kio/exec/internal/snapshotconverters.(*GVCConverter).Convert
111          linenumber: 77
112          message: Error creating portable snapshot
113        fields:
114          - name: type
115            value: CSI
116          - name: id
117            value: k10-csi-snap-pcw6kxxzgsql8hj6
118        file: kasten.io/k10/kio/exec/phases/phase/artifactcopier.go:544
119        function: kasten.io/k10/kio/exec/phases/phase.(*ArtifactCopier).convertSnapshots.func1
120        linenumber: 544
121        message: Failed to export snapshot data
122      file: kasten.io/k10/kio/exec/phases/phase/artifactcopier.go:274
123      function: kasten.io/k10/kio/exec/phases/phase.(*ArtifactCopier).Copy
124      linenumber: 274
125      message: Error converting snapshots
126    file: kasten.io/k10/kio/exec/phases/phase/export.go:172
127    function: kasten.io/k10/kio/exec/phases/phase.(*exportRestorePointPhase).Run
128    linenumber: 172
129    message: Failed to copy artifacts
130  message: Job failed to be executed
131- cause:
132    cause:
133      cause:
134        cause:
135          cause:
136            cause:
137              cause:
138                cause:
139                  cause:
140                    cause:
141                      cause:
142                        cause:
143                          message: "Context done while polling: context deadline exceeded"
144                        file: github.com/kanisterio/kanister@v0.0.0-20240812194716-8812756d1751/pkg/kube/pod.go
145                        function: github.com/kanisterio/kanister/pkg/kube.getErrorFromLogs
146                        linenumber: 334
147                      file: github.com/kanisterio/kanister@v0.0.0-20240812194716-8812756d1751/pkg/kube/pod.go
148                      function: github.com/kanisterio/kanister/pkg/kube.getErrorFromLogs
149                      linenumber: 334
150                      message: Pod did not transition into running state.
151                        Timeout:15m0s  Namespace:kasten-io,
152                        Name:copy-vol-data-v84db
153                    file: github.com/kanisterio/kanister@v0.0.0-20240812194716-8812756d1751/pkg/kube/pod_controller.go
154                    function: github.com/kanisterio/kanister/pkg/kube.(*podController).WaitForPodReady
155                    linenumber: 174
156                    message: Pod failed to become ready in time
157                  fields:
158                    - name: pod
159                      value: copy-vol-data-v84db
160                    - name: namespace
161                      value: kasten-io
162                  file: kasten.io/k10/kio/kanister/function/kio_copy_volume_data.go:304
163                  function: kasten.io/k10/kio/kanister/function.CopyVolumeData.copyVolumeDataPodExecFunc.func2
164                  linenumber: 304
165                  message: failed while waiting for Pod to be ready
166                file: kasten.io/k10/kio/kanister/function/kio_copy_volume_data.go:161
167                function: kasten.io/k10/kio/kanister/function.CopyVolumeData
168                linenumber: 161
169                message: Failed to execute copy volume data pod function
170              file: kasten.io/k10/kio/exec/internal/snapshotconverters/ac_gvc_converter.go:249
171              function: kasten.io/k10/kio/exec/internal/snapshotconverters.(*GVCConverterInternalAPIImpl).genericVolumeCopy
172              linenumber: 249
173              message: failed running copyVolumeData
174            file: kasten.io/k10/kio/exec/internal/snapshotconverters/ac_gvc_converter.go:170
175            function: kasten.io/k10/kio/exec/internal/snapshotconverters.(*GVCConverterInternalAPIImpl).CopySnapshotRestoredInPVC
176            linenumber: 170
177            message: failed running genericVolumeCopy
178          file: kasten.io/k10/kio/exec/internal/snapshotconverters/ac_gvc_converter.go:77
179          function: kasten.io/k10/kio/exec/internal/snapshotconverters.(*GVCConverter).Convert
180          linenumber: 77
181          message: Error creating portable snapshot
182        fields:
183          - name: type
184            value: CSI
185          - name: id
186            value: k10-csi-snap-pcw6kxxzgsql8hj6
187        file: kasten.io/k10/kio/exec/phases/phase/artifactcopier.go:544
188        function: kasten.io/k10/kio/exec/phases/phase.(*ArtifactCopier).convertSnapshots.func1
189        linenumber: 544
190        message: Failed to export snapshot data
191      file: kasten.io/k10/kio/exec/phases/phase/artifactcopier.go:274
192      function: kasten.io/k10/kio/exec/phases/phase.(*ArtifactCopier).Copy
193      linenumber: 274
194      message: Error converting snapshots
195    file: kasten.io/k10/kio/exec/phases/phase/export.go:172
196    function: kasten.io/k10/kio/exec/phases/phase.(*exportRestorePointPhase).Run
197    linenumber: 172
198    message: Failed to copy artifacts
199  message: Job failed to be executed
200

The Pre-Flight-Check was successful, the snapshot itself is left on the storage system while the export is failed. So this part seems to work, just the export fails.

Kubernetes Distribution: v1.28.9+rke2r1
PVC Size in target Namespace: ~1,1TB
CSI: Ceph-CSI-Driver (RBD and CephFS)
Snapshotter: rke2-snapshot-controller:1.7.202 (It’s the external-snapshotter)
s3 Bucket (WORM configured): 

1s3api put-object-lock-configuration --bucket kasten --object-lock-configuration='{ "ObjectLockEnabled": "Enabled", "Rule": { "DefaultRetention": { "Mode": "COMPLIANCE", "Days": 14 }}}'


The Pod “copy-vol-data-xzb5k” itself says: “0/7 nodes are available: pod has unbound immediate PersistentVolumeClaims. preemption: 0/7 nodes are available: 7 Preemption is not helpful for scheduling..”

 

Furthermore, in the description of the copy-pod:
 

1Volumes:
2  vol-8f8503ca-66ad-11ef-9a4d-7ead5d941560:
3    Type:       PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)
4    ClaimName:  kanister-pvc-2945w
5    ReadOnly:   false
6

The PVC for the copy-pod:
 

1k describe pvc/kanister-pvc-2945w -n kasten-io
2Name:          kanister-pvc-2945w
3Namespace:     kasten-io
4StorageClass:  csi-cephfs-sc
5Status:        Pending
6Volume:
7Labels:        k10.kasten.io/readyForGC=true
8Annotations:   k10.kasten.io/readyForGCAt: 2024-08-30T18:55:00Z
9               volume.beta.kubernetes.io/storage-provisioner: cephfs.csi.ceph.com
10               volume.kubernetes.io/storage-provisioner: cephfs.csi.ceph.com
11Finalizers:    [kubernetes.io/pvc-protection]
12Capacity:
13Access Modes:
14VolumeMode:    Filesystem
15DataSource:
16  APIGroup:  snapshot.storage.k8s.io
17  Kind:      VolumeSnapshot
18  Name:      snapshot-copy-qnxkgcjf
19Used By:     copy-vol-data-xzb5k
20Events:
21  Type     Reason                Age                     From                                                                                                   Message
22  ----     ------                ----                    ----                                                                                                   -------
23  Warning  ProvisioningFailed    6m36s                   cephfs.csi.ceph.com_ceph-csi-cephfs-provisioner-5b9d856485-6m2mj_039263d9-89af-4c0c-acc4-db35b8bc9056  failed to provision volume with StorageClass "csi-cephfs-sc": rpc error: code = Aborted desc = clone from snapshot is pending
24  Normal   Provisioning          2m20s (x10 over 6m36s)  cephfs.csi.ceph.com_ceph-csi-cephfs-provisioner-5b9d856485-6m2mj_039263d9-89af-4c0c-acc4-db35b8bc9056  External provisioner is provisioning volume for claim "kasten-io/kanister-pvc-2945w"
25  Warning  ProvisioningFailed    2m20s (x9 over 6m35s)   cephfs.csi.ceph.com_ceph-csi-cephfs-provisioner-5b9d856485-6m2mj_039263d9-89af-4c0c-acc4-db35b8bc9056  failed to provision volume with StorageClass "csi-cephfs-sc": rpc error: code = Aborted desc = clone from snapshot is already in progress
26  Normal   ExternalProvisioning  21s (x26 over 6m36s)    persistentvolume-controller                                                                            Waiting for a volume to be created either by the external provisioner 'cephfs.csi.ceph.com' or manually by the system administrator. If volume creation is delayed, please verify that the provisioner is running and correctly registered.

This looks like the root cause for me. “clone from snapshot is pending”

Log from Ceph-CSI-Provisioner:

179b9cc7-4b9a-4156-b9a6-a0e2ae43f0f7"): map[csi.imagename:csi-vol-b79b9cc7-4b9a-4156-b9a6-a0e2ae43f0f7 csi.volname:pvc-4344ab96-5a49-4e70-8d99-8a6c9f1a209c csi.volume.owner:kasten-io]
2E0830 09:03:34.739380       1 utils.go:203] ID: 24877 Req-ID: pvc-4344ab96-5a49-4e70-8d99-8a6c9f1a209c GRPC error: rpc error: code = Aborted desc = clone from snapshot is already in progress
3


So in the end, it seems, that the snapshot took too long for finishing and the timeout of the copy-vol-pod is reached. Is there a value, which I can modify this behavior?
 

Best answer by kevin.gottsknecht

I fixed my inital problem, with increasing certain timeouts:

values.yaml:

kanister:
  backupTimeout: 1380
  podReadyWaitTimeout: 1380
  restoreTimeout: 1380
 

View original
Did this topic help you find an answer to your question?

9 comments

K
Forum|alt.badge.img

Hello! I am also getting the same issue. Even, i am unable to validate cephfs storage class.

K

I’ve found the helm value “kanister.PodReadyWaitTimeout”. I will increase that value and will check again….

K
Forum|alt.badge.img

Hello, I am able to take backup but restore of PVC is failing

 

 

 

Generated from Kanister Controller

Failed to execute phase: v1alpha1.Phase{Name:"restoreFromServer", State:"pending", Output:map[string]interface {}(nil), Progress:v1alpha1.PhaseProgress{ProgressPercent:"", SizeUploadedB:0, EstimatedUploadSizeB:0, EstimatedTimeSeconds:0, LastTransitionTime:<nil>}}: {"message":"Failed to restore backup from Kopia API server","function":"kasten.io/k10/kio/kanister/function.restoreDataFromServer.restoreDataFromServerPodFunc.func3","linenumber":367,"file":"kasten.io/k10/kio/kanister/function/restore_data_from_server.go:367","cause":{"message":"context deadline exceeded"}}

Hagag
Forum|alt.badge.img+2
  • Hagag
  • Experienced User
  • 165 comments
  • September 3, 2024

Hi @kanika.mahajan Unfortunately, The logs for the resotre you provided is not enough to identify the root cause.
please share the error message in the restoreaction.

BR,
Ahmed Hagag

K
Forum|alt.badge.img
  • kanika.mahajan
  • Comes here often
  • 30 comments
  • September 3, 2024

Hello @Hagag , below is the restoreAction

 

kind: RestoreAction
apiVersion: actions.kio.kasten.io/v1alpha1
metadata:
  name: helm-test-8r7wv
  namespace: helm-test
  uid: 6f9d5b7c-6952-11ef-be6e-0a580a80039f
  resourceVersion: "9042"
  creationTimestamp: 2024-09-02T17:40:25Z
  labels:
    k10.kasten.io/appName: kedify
    k10.kasten.io/appNamespace: kedify
status:
  state: Failed
  startTime: 2024-09-02T17:40:25Z
  endTime: 2024-09-02T18:13:38Z
  restorePoint:
    name: ""
  result:
    name: ""
  error:
    cause: '{"cause":{"cause":{"cause":{"cause":{"cause":{"cause":{"cause":{"message":"{\"message\":\"Failed
      to restore backup from Kopia API
      server\",\"function\":\"kasten.io/k10/kio/kanister/function.restoreDataFromServer.restoreDataFromServerPodFunc.func3\",\"linenumber\":367,\"file\":\"kasten.io/k10/kio/kanister/function/restore_data_from_server.go:367\",\"cause\":{\"message\":\"context
      deadline
      exceeded\"}}"},"fields":[{"name":"actionSet","value":{"metadata":{"creationTimestamp":"2024-09-02T18:03:29Z","generateName":"k10-restorefromserver-k10-deployment-generic-volume-2.0.43-k10restore-af5ac119-d54e-4391-bba9-f8cb8cc2a780-kasten-io-pvc-","generation":6,"labels":{"kanister.io/JobID":"6f9f538e-6952-11ef-9962-0a580a800394"},"managedFields":[{"apiVersion":"cr.kanister.io/v1alpha1","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:generateName":{},"f:labels":{".":{},"f:kanister.io/JobID":{}}},"f:spec":{".":{},"f:actions":{}}},"manager":"executor-server","operation":"Update","time":"2024-09-02T18:03:29Z"},{"apiVersion":"cr.kanister.io/v1alpha1","fieldsType":"FieldsV1","fieldsV1":{"f:status":{".":{},"f:actions":{},"f:error":{".":{},"f:message":{}},"f:progress":{".":{},"f:lastTransitionTime":{},"f:percentCompleted":{}},"f:state":{}}},"manager":"controller","operation":"Update","time":"2024-09-02T18:13:33Z"}],"name":"k10-restorefromserver-k10-deployment-generic-volume-2.0.43zc4qr","namespace":"kasten-io","resourceVersion":"190531487","uid":"fc5512d1-793f-408e-a4ca-32686b91f127"},"spec":{"actions":[{"artifacts":{"snapshot":{"keyValue":{"backupIdentifier":"c391a162ef24dc43a0cbc7b979f0da4e","backupPath":"/mnt/vol_data/kanister-pvc","funcVersion":"v1.0.0-alpha","objectStorePath":"repo/e6332964-a3a9-4ee9-8094-96a8ed23f44a/","phySize":"7.3
      GB","size":"14
      GB"}}},"blueprint":"k10-deployment-generic-volume-2.0.43","name":"restoreFromServer","object":{"apiVersion":"","group":"","kind":"pvc","name":"k10restore-af5ac119-d54e-4391-bba9-f8cb8cc2a780","namespace":"kasten-io","resource":""},"options":{"hostName":"e6332964-a3a9-4ee9-8094-96a8ed23f44a.absence-fs-api-absence.absence-fs-helm-absence-pvc","pointInTimeConnection":"\"0001-01-01T00:00:00.000Z\"","serverAddress":"https://172.30.197.178:51515","userName":"k10-admin"},"podOverride":{"containers":[{"name":"container","resources":{},"volumeMounts":[{"mountPath":"/etc/ssl/certs/custom-ca-bundle.pem","name":"custom-ca-bundle-store","subPath":"custom-ca-bundle.pem"}]}],"securityContext":{"runAsNonRoot":false,"runAsUser":0},"tolerations":[{"effect":"NoExecute","key":"node.kubernetes.io/not-ready","operator":"Exists","tolerationSeconds":300},{"effect":"NoExecute","key":"node.kubernetes.io/unreachable","operator":"Exists","tolerationSeconds":300},{"effect":"NoSchedule","key":"node.kubernetes.io/memory-pressure","operator":"Exists"}],"volumes":[{"configMap":{"defaultMode":420,"name":"custom-ca-bundle-store"},"name":"custom-ca-bundle-store"}]},"preferredVersion":"v1.0.0-alpha","secrets":{"certs":{"apiVersion":"","group":"","kind":"secret","name":"kopia-tls-cert","namespace":"kasten-io","resource":""},"serverPassphraseKey":{"apiVersion":"","group":"","kind":"secret","name":"data-mover-server-passphrase-l5db4","namespace":"kasten-io","resource":""},"userPassphraseKey":{"apiVersion":"","group":"","kind":"secret","name":"data-mover-user-passphrase-zbqhh","namespace":"kasten-io","resource":""}}}]},"status":{"actions":[{"blueprint":"k10-deployment-generic-volume-2.0.43","deferPhase":{"name":"","progress":{},"state":""},"name":"restoreFromServer","object":{"apiVersion":"","group":"","kind":"pvc","name":"k10restore-af5ac119-d54e-4391-bba9-f8cb8cc2a780","namespace":"kasten-io","resource":""},"phases":[{"name":"restoreFromServer","progress":{"lastTransitionTime":"2024-09-02T18:03:34Z","progressPercent":"0"},"state":"failed"}]}],"error":{"message":"{\"message\":\"Failed
      to restore backup from Kopia API
      server\",\"function\":\"kasten.io/k10/kio/kanister/function.restoreDataFromServer.restoreDataFromServerPodFunc.func3\",\"linenumber\":367,\"file\":\"kasten.io/k10/kio/kanister/function/restore_data_from_server.go:367\",\"cause\":{\"message\":\"context
      deadline
      exceeded\"}}"},"progress":{"lastTransitionTime":"2024-09-02T18:03:34Z","percentCompleted":"0"},"state":"failed"}}}],"file":"kasten.io/k10/kio/kanister/operation.go:167","function":"kasten.io/k10/kio/kanister.(*Operation).WaitForActionSet","linenumber":167,"message":"ActionSet
      Failed"},"file":"kasten.io/k10/kio/exec/phases/phase/restore_app.go:2348","function":"kasten.io/k10/kio/exec/phases/phase.GenericVolumeSnapshotRestore","linenumber":2348,"message":"Failed
      to execute action
      set"},"fields":[{"name":"k8sType","value":"deployment"},{"name":"pvcName","value":"absence-fs-helm-absence-pvc"}],"file":"kasten.io/k10/kio/exec/phases/phase/restore_app.go:2091","function":"kasten.io/k10/kio/exec/phases/phase.(*restoreApplicationPhase).restoreDataIntoPVC","linenumber":2091,"message":"Failed
      to restore
      PVC"},"file":"kasten.io/k10/kio/exec/phases/phase/restore_app.go:1816","function":"kasten.io/k10/kio/exec/phases/phase.(*restoreApplicationPhase).restoreDataIntoPVCs","linenumber":1816,"message":"Failed
      to restore some of the generic volume
      snapshots"},"file":"kasten.io/k10/kio/exec/phases/phase/restore_app.go:567","function":"kasten.io/k10/kio/exec/phases/phase.(*restoreApplicationPhase).createPVCsFromPVCSpecs","linenumber":567,"message":"Failed
      to perform Generic Volume Snapshot
      Restore"},"file":"kasten.io/k10/kio/exec/phases/phase/restore_app.go:363","function":"kasten.io/k10/kio/exec/phases/phase.(*restoreApplicationPhase).restoreApp","linenumber":363,"message":"Failed
      to create PVCs from PVC
      specs"},"file":"kasten.io/k10/kio/exec/internal/runner/phase_runner.go:144","function":"kasten.io/k10/kio/exec/internal/runner.(*phaseRunner).execPlannedPhase","linenumber":144,"message":"Failure
      in planned phase"}'
    message: Job failed to be executed
  actionDetails:
    phases:
      - attempt: 3
        endTime: 2024-09-02T18:13:38Z
        errors:
          - cause: '{"cause":{"cause":{"cause":{"cause":{"cause":{"cause":{"cause":{"message":"{\"message\":\"Failed
              to restore backup from Kopia API
              server\",\"function\":\"kasten.io/k10/kio/kanister/function.restoreDataFromServer.restoreDataFromServerPodFunc.func3\",\"linenumber\":367,\"file\":\"kasten.io/k10/kio/kanister/function/restore_data_from_server.go:367\",\"cause\":{\"message\":\"context
              deadline
              exceeded\"}}"},"fields":[{"name":"actionSet","value":{"metadata":{"creationTimestamp":"2024-09-02T17:41:08Z","generateName":"k10-restorefromserver-k10-deployment-generic-volume-2.0.43-k10restore-70be134b-6715-47ec-b94e-bb77c9fc2710-kasten-io-pvc-","generation":6,"labels":{"kanister.io/JobID":"6f9f538e-6952-11ef-9962-0a580a800394"},"managedFields":[{"apiVersion":"cr.kanister.io/v1alpha1","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:generateName":{},"f:labels":{".":{},"f:kanister.io/JobID":{}}},"f:spec":{".":{},"f:actions":{}}},"manager":"executor-server","operation":"Update","time":"2024-09-02T17:41:08Z"},{"apiVersion":"cr.kanister.io/v1alpha1","fieldsType":"FieldsV1","fieldsV1":{"f:status":{".":{},"f:actions":{},"f:error":{".":{},"f:message":{}},"f:progress":{".":{},"f:lastTransitionTime":{},"f:percentCompleted":{}},"f:state":{}}},"manager":"controller","operation":"Update","time":"2024-09-02T17:51:12Z"}],"name":"k10-restorefromserver-k10-deployment-generic-volume-2.0.434fm5h","namespace":"kasten-io","resourceVersion":"190523172","uid":"89ec46e6-451d-4461-a740-5e479e06224b"},"spec":{"actions":[{"artifacts":{"snapshot":{"keyValue":{"backupIdentifier":"c391a162ef24dc43a0cbc7b979f0da4e","backupPath":"/mnt/vol_data/kanister-pvc","funcVersion":"v1.0.0-alpha","objectStorePath":"repo/e6332964-a3a9-4ee9-8094-96a8ed23f44a/","phySize":"7.3
              GB","size":"14
              GB"}}},"blueprint":"k10-deployment-generic-volume-2.0.43","name":"restoreFromServer","object":{"apiVersion":"","group":"","kind":"pvc","name":"k10restore-70be134b-6715-47ec-b94e-bb77c9fc2710","namespace":"kasten-io","resource":""},"options":{"hostName":"e6332964-a3a9-4ee9-8094-96a8ed23f44a.absence-fs-api-absence.absence-fs-helm-absence-pvc","pointInTimeConnection":"\"0001-01-01T00:00:00.000Z\"","serverAddress":"https://172.30.82.89:51515","userName":"k10-admin"},"podOverride":{"containers":[{"name":"container","resources":{},"volumeMounts":[{"mountPath":"/etc/ssl/certs/custom-ca-bundle.pem","name":"custom-ca-bundle-store","subPath":"custom-ca-bundle.pem"}]}],"securityContext":{"runAsNonRoot":false,"runAsUser":0},"tolerations":[{"effect":"NoExecute","key":"node.kubernetes.io/not-ready","operator":"Exists","tolerationSeconds":300},{"effect":"NoExecute","key":"node.kubernetes.io/unreachable","operator":"Exists","tolerationSeconds":300},{"effect":"NoSchedule","key":"node.kubernetes.io/memory-pressure","operator":"Exists"}],"volumes":[{"configMap":{"defaultMode":420,"name":"custom-ca-bundle-store"},"name":"custom-ca-bundle-store"}]},"preferredVersion":"v1.0.0-alpha","secrets":{"certs":{"apiVersion":"","group":"","kind":"secret","name":"kopia-tls-cert","namespace":"kasten-io","resource":""},"serverPassphraseKey":{"apiVersion":"","group":"","kind":"secret","name":"data-mover-server-passphrase-564tx","namespace":"kasten-io","resource":""},"userPassphraseKey":{"apiVersion":"","group":"","kind":"secret","name":"data-mover-user-passphrase-d2k6p","namespace":"kasten-io","resource":""}}}]},"status":{"actions":[{"blueprint":"k10-deployment-generic-volume-2.0.43","deferPhase":{"name":"","progress":{},"state":""},"name":"restoreFromServer","object":{"apiVersion":"","group":"","kind":"pvc","name":"k10restore-70be134b-6715-47ec-b94e-bb77c9fc2710","namespace":"kasten-io","resource":""},"phases":[{"name":"restoreFromServer","progress":{"lastTransitionTime":"2024-09-02T17:41:13Z","progressPercent":"0"},"state":"failed"}]}],"error":{"message":"{\"message\":\"Failed
              to restore backup from Kopia API
              server\",\"function\":\"kasten.io/k10/kio/kanister/function.restoreDataFromServer.restoreDataFromServerPodFunc.func3\",\"linenumber\":367,\"file\":\"kasten.io/k10/kio/kanister/function/restore_data_from_server.go:367\",\"cause\":{\"message\":\"context
              deadline
              exceeded\"}}"},"progress":{"lastTransitionTime":"2024-09-02T17:41:13Z","percentCompleted":"0"},"state":"failed"}}}],"file":"kasten.io/k10/kio/kanister/operation.go:167","function":"kasten.io/k10/kio/kanister.(*Operation).WaitForActionSet","linenumber":167,"message":"ActionSet
              Failed"},"file":"kasten.io/k10/kio/exec/phases/phase/restore_app.go:2348","function":"kasten.io/k10/kio/exec/phases/phase.GenericVolumeSnapshotRestore","linenumber":2348,"message":"Failed
              to execute action
              set"},"fields":[{"name":"k8sType","value":"deployment"},{"name":"pvcName","value":"absence-fs-helm-absence-pvc"}],"file":"kasten.io/k10/kio/exec/phases/phase/restore_app.go:2091","function":"kasten.io/k10/kio/exec/phases/phase.(*restoreApplicationPhase).restoreDataIntoPVC","linenumber":2091,"message":"Failed
              to restore
              PVC"},"file":"kasten.io/k10/kio/exec/phases/phase/restore_app.go:1816","function":"kasten.io/k10/kio/exec/phases/phase.(*restoreApplicationPhase).restoreDataIntoPVCs","linenumber":1816,"message":"Failed
              to restore some of the generic volume
              snapshots"},"file":"kasten.io/k10/kio/exec/phases/phase/restore_app.go:567","function":"kasten.io/k10/kio/exec/phases/phase.(*restoreApplicationPhase).createPVCsFromPVCSpecs","linenumber":567,"message":"Failed
              to perform Generic Volume Snapshot
              Restore"},"file":"kasten.io/k10/kio/exec/phases/phase/restore_app.go:363","function":"kasten.io/k10/kio/exec/phases/phase.(*restoreApplicationPhase).restoreApp","linenumber":363,"message":"Failed
              to create PVCs from PVC
              specs"},"file":"kasten.io/k10/kio/exec/internal/runner/phase_runner.go:144","function":"kasten.io/k10/kio/exec/internal/runner.(*phaseRunner).execPlannedPhase","linenumber":144,"message":"Failure
              in planned phase"}'
            message: Job failed to be executed
          - cause: '{"cause":{"cause":{"cause":{"cause":{"cause":{"cause":{"cause":{"message":"{\"message\":\"Failed
              to restore backup from Kopia API
              server\",\"function\":\"kasten.io/k10/kio/kanister/function.restoreDataFromServer.restoreDataFromServerPodFunc.func3\",\"linenumber\":367,\"file\":\"kasten.io/k10/kio/kanister/function/restore_data_from_server.go:367\",\"cause\":{\"message\":\"context
              deadline
              exceeded\"}}"},"fields":[{"name":"actionSet","value":{"metadata":{"creationTimestamp":"2024-09-02T17:52:23Z","generateName":"k10-restorefromserver-k10-deployment-generic-volume-2.0.43-k10restore-0387aba4-da47-4b9f-9cc1-a0b7fa7f7f6c-kasten-io-pvc-","generation":6,"labels":{"kanister.io/JobID":"6f9f538e-6952-11ef-9962-0a580a800394"},"managedFields":[{"apiVersion":"cr.kanister.io/v1alpha1","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:generateName":{},"f:labels":{".":{},"f:kanister.io/JobID":{}}},"f:spec":{".":{},"f:actions":{}}},"manager":"executor-server","operation":"Update","time":"2024-09-02T17:52:23Z"},{"apiVersion":"cr.kanister.io/v1alpha1","fieldsType":"FieldsV1","fieldsV1":{"f:status":{".":{},"f:actions":{},"f:error":{".":{},"f:message":{}},"f:progress":{".":{},"f:lastTransitionTime":{},"f:percentCompleted":{}},"f:state":{}}},"manager":"controller","operation":"Update","time":"2024-09-02T18:02:27Z"}],"name":"k10-restorefromserver-k10-deployment-generic-volume-2.0.43g8c59","namespace":"kasten-io","resourceVersion":"190527140","uid":"140b49a5-f168-40ba-a8d6-0e7e0d98d422"},"spec":{"actions":[{"artifacts":{"snapshot":{"keyValue":{"backupIdentifier":"c391a162ef24dc43a0cbc7b979f0da4e","backupPath":"/mnt/vol_data/kanister-pvc","funcVersion":"v1.0.0-alpha","objectStorePath":"repo/e6332964-a3a9-4ee9-8094-96a8ed23f44a/","phySize":"7.3
              GB","size":"14
              GB"}}},"blueprint":"k10-deployment-generic-volume-2.0.43","name":"restoreFromServer","object":{"apiVersion":"","group":"","kind":"pvc","name":"k10restore-0387aba4-da47-4b9f-9cc1-a0b7fa7f7f6c","namespace":"kasten-io","resource":""},"options":{"hostName":"e6332964-a3a9-4ee9-8094-96a8ed23f44a.absence-fs-api-absence.absence-fs-helm-absence-pvc","pointInTimeConnection":"\"0001-01-01T00:00:00.000Z\"","serverAddress":"https://172.30.80.17:51515","userName":"k10-admin"},"podOverride":{"containers":[{"name":"container","resources":{},"volumeMounts":[{"mountPath":"/etc/ssl/certs/custom-ca-bundle.pem","name":"custom-ca-bundle-store","subPath":"custom-ca-bundle.pem"}]}],"securityContext":{"runAsNonRoot":false,"runAsUser":0},"tolerations":[{"effect":"NoExecute","key":"node.kubernetes.io/not-ready","operator":"Exists","tolerationSeconds":300},{"effect":"NoExecute","key":"node.kubernetes.io/unreachable","operator":"Exists","tolerationSeconds":300},{"effect":"NoSchedule","key":"node.kubernetes.io/memory-pressure","operator":"Exists"}],"volumes":[{"configMap":{"defaultMode":420,"name":"custom-ca-bundle-store"},"name":"custom-ca-bundle-store"}]},"preferredVersion":"v1.0.0-alpha","secrets":{"certs":{"apiVersion":"","group":"","kind":"secret","name":"kopia-tls-cert","namespace":"kasten-io","resource":""},"serverPassphraseKey":{"apiVersion":"","group":"","kind":"secret","name":"data-mover-server-passphrase-sws67","namespace":"kasten-io","resource":""},"userPassphraseKey":{"apiVersion":"","group":"","kind":"secret","name":"data-mover-user-passphrase-5lhw6","namespace":"kasten-io","resource":""}}}]},"status":{"actions":[{"blueprint":"k10-deployment-generic-volume-2.0.43","deferPhase":{"name":"","progress":{},"state":""},"name":"restoreFromServer","object":{"apiVersion":"","group":"","kind":"pvc","name":"k10restore-0387aba4-da47-4b9f-9cc1-a0b7fa7f7f6c","namespace":"kasten-io","resource":""},"phases":[{"name":"restoreFromServer","progress":{"lastTransitionTime":"2024-09-02T17:52:29Z","progressPercent":"0"},"state":"failed"}]}],"error":{"message":"{\"message\":\"Failed
              to restore backup from Kopia API
              server\",\"function\":\"kasten.io/k10/kio/kanister/function.restoreDataFromServer.restoreDataFromServerPodFunc.func3\",\"linenumber\":367,\"file\":\"kasten.io/k10/kio/kanister/function/restore_data_from_server.go:367\",\"cause\":{\"message\":\"context
              deadline
              exceeded\"}}"},"progress":{"lastTransitionTime":"2024-09-02T17:52:29Z","percentCompleted":"0"},"state":"failed"}}}],"file":"kasten.io/k10/kio/kanister/operation.go:167","function":"kasten.io/k10/kio/kanister.(*Operation).WaitForActionSet","linenumber":167,"message":"ActionSet
              Failed"},"file":"kasten.io/k10/kio/exec/phases/phase/restore_app.go:2348","function":"kasten.io/k10/kio/exec/phases/phase.GenericVolumeSnapshotRestore","linenumber":2348,"message":"Failed
              to execute action
              set"},"fields":[{"name":"k8sType","value":"deployment"},{"name":"pvcName","value":"absence-fs-helm-absence-pvc"}],"file":"kasten.io/k10/kio/exec/phases/phase/restore_app.go:2091","function":"kasten.io/k10/kio/exec/phases/phase.(*restoreApplicationPhase).restoreDataIntoPVC","linenumber":2091,"message":"Failed
              to restore
              PVC"},"file":"kasten.io/k10/kio/exec/phases/phase/restore_app.go:1816","function":"kasten.io/k10/kio/exec/phases/phase.(*restoreApplicationPhase).restoreDataIntoPVCs","linenumber":1816,"message":"Failed
              to restore some of the generic volume
              snapshots"},"file":"kasten.io/k10/kio/exec/phases/phase/restore_app.go:567","function":"kasten.io/k10/kio/exec/phases/phase.(*restoreApplicationPhase).createPVCsFromPVCSpecs","linenumber":567,"message":"Failed
              to perform Generic Volume Snapshot
              Restore"},"file":"kasten.io/k10/kio/exec/phases/phase/restore_app.go:363","function":"kasten.io/k10/kio/exec/phases/phase.(*restoreApplicationPhase).restoreApp","linenumber":363,"message":"Failed
              to create PVCs from PVC
              specs"},"file":"kasten.io/k10/kio/exec/internal/runner/phase_runner.go:144","function":"kasten.io/k10/kio/exec/internal/runner.(*phaseRunner).execPlannedPhase","linenumber":144,"message":"Failure
              in planned phase"}'
            message: Job failed to be executed
          - cause: '{"cause":{"cause":{"cause":{"cause":{"cause":{"cause":{"cause":{"message":"{\"message\":\"Failed
              to restore backup from Kopia API
              server\",\"function\":\"kasten.io/k10/kio/kanister/function.restoreDataFromServer.restoreDataFromServerPodFunc.func3\",\"linenumber\":367,\"file\":\"kasten.io/k10/kio/kanister/function/restore_data_from_server.go:367\",\"cause\":{\"message\":\"context
              deadline
              exceeded\"}}"},"fields":[{"name":"actionSet","value":{"metadata":{"creationTimestamp":"2024-09-02T18:03:29Z","generateName":"k10-restorefromserver-k10-deployment-generic-volume-2.0.43-k10restore-af5ac119-d54e-4391-bba9-f8cb8cc2a780-kasten-io-pvc-","generation":6,"labels":{"kanister.io/JobID":"6f9f538e-6952-11ef-9962-0a580a800394"},"managedFields":[{"apiVersion":"cr.kanister.io/v1alpha1","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:generateName":{},"f:labels":{".":{},"f:kanister.io/JobID":{}}},"f:spec":{".":{},"f:actions":{}}},"manager":"executor-server","operation":"Update","time":"2024-09-02T18:03:29Z"},{"apiVersion":"cr.kanister.io/v1alpha1","fieldsType":"FieldsV1","fieldsV1":{"f:status":{".":{},"f:actions":{},"f:error":{".":{},"f:message":{}},"f:progress":{".":{},"f:lastTransitionTime":{},"f:percentCompleted":{}},"f:state":{}}},"manager":"controller","operation":"Update","time":"2024-09-02T18:13:33Z"}],"name":"k10-restorefromserver-k10-deployment-generic-volume-2.0.43zc4qr","namespace":"kasten-io","resourceVersion":"190531487","uid":"fc5512d1-793f-408e-a4ca-32686b91f127"},"spec":{"actions":[{"artifacts":{"snapshot":{"keyValue":{"backupIdentifier":"c391a162ef24dc43a0cbc7b979f0da4e","backupPath":"/mnt/vol_data/kanister-pvc","funcVersion":"v1.0.0-alpha","objectStorePath":"repo/e6332964-a3a9-4ee9-8094-96a8ed23f44a/","phySize":"7.3
              GB","size":"14
              GB"}}},"blueprint":"k10-deployment-generic-volume-2.0.43","name":"restoreFromServer","object":{"apiVersion":"","group":"","kind":"pvc","name":"k10restore-af5ac119-d54e-4391-bba9-f8cb8cc2a780","namespace":"kasten-io","resource":""},"options":{"hostName":"e6332964-a3a9-4ee9-8094-96a8ed23f44a.absence-fs-api-absence.absence-fs-helm-absence-pvc","pointInTimeConnection":"\"0001-01-01T00:00:00.000Z\"","serverAddress":"https://172.30.197.178:51515","userName":"k10-admin"},"podOverride":{"containers":[{"name":"container","resources":{},"volumeMounts":[{"mountPath":"/etc/ssl/certs/custom-ca-bundle.pem","name":"custom-ca-bundle-store","subPath":"custom-ca-bundle.pem"}]}],"securityContext":{"runAsNonRoot":false,"runAsUser":0},"tolerations":[{"effect":"NoExecute","key":"node.kubernetes.io/not-ready","operator":"Exists","tolerationSeconds":300},{"effect":"NoExecute","key":"node.kubernetes.io/unreachable","operator":"Exists","tolerationSeconds":300},{"effect":"NoSchedule","key":"node.kubernetes.io/memory-pressure","operator":"Exists"}],"volumes":[{"configMap":{"defaultMode":420,"name":"custom-ca-bundle-store"},"name":"custom-ca-bundle-store"}]},"preferredVersion":"v1.0.0-alpha","secrets":{"certs":{"apiVersion":"","group":"","kind":"secret","name":"kopia-tls-cert","namespace":"kasten-io","resource":""},"serverPassphraseKey":{"apiVersion":"","group":"","kind":"secret","name":"data-mover-server-passphrase-l5db4","namespace":"kasten-io","resource":""},"userPassphraseKey":{"apiVersion":"","group":"","kind":"secret","name":"data-mover-user-passphrase-zbqhh","namespace":"kasten-io","resource":""}}}]},"status":{"actions":[{"blueprint":"k10-deployment-generic-volume-2.0.43","deferPhase":{"name":"","progress":{},"state":""},"name":"restoreFromServer","object":{"apiVersion":"","group":"","kind":"pvc","name":"k10restore-af5ac119-d54e-4391-bba9-f8cb8cc2a780","namespace":"kasten-io","resource":""},"phases":[{"name":"restoreFromServer","progress":{"lastTransitionTime":"2024-09-02T18:03:34Z","progressPercent":"0"},"state":"failed"}]}],"error":{"message":"{\"message\":\"Failed
              to restore backup from Kopia API
              server\",\"function\":\"kasten.io/k10/kio/kanister/function.restoreDataFromServer.restoreDataFromServerPodFunc.func3\",\"linenumber\":367,\"file\":\"kasten.io/k10/kio/kanister/function/restore_data_from_server.go:367\",\"cause\":{\"message\":\"context
              deadline
              exceeded\"}}"},"progress":{"lastTransitionTime":"2024-09-02T18:03:34Z","percentCompleted":"0"},"state":"failed"}}}],"file":"kasten.io/k10/kio/kanister/operation.go:167","function":"kasten.io/k10/kio/kanister.(*Operation).WaitForActionSet","linenumber":167,"message":"ActionSet
              Failed"},"file":"kasten.io/k10/kio/exec/phases/phase/restore_app.go:2348","function":"kasten.io/k10/kio/exec/phases/phase.GenericVolumeSnapshotRestore","linenumber":2348,"message":"Failed
              to execute action
              set"},"fields":[{"name":"k8sType","value":"deployment"},{"name":"pvcName","value":"absence-fs-helm-absence-pvc"}],"file":"kasten.io/k10/kio/exec/phases/phase/restore_app.go:2091","function":"kasten.io/k10/kio/exec/phases/phase.(*restoreApplicationPhase).restoreDataIntoPVC","linenumber":2091,"message":"Failed
              to restore
              PVC"},"file":"kasten.io/k10/kio/exec/phases/phase/restore_app.go:1816","function":"kasten.io/k10/kio/exec/phases/phase.(*restoreApplicationPhase).restoreDataIntoPVCs","linenumber":1816,"message":"Failed
              to restore some of the generic volume
              snapshots"},"file":"kasten.io/k10/kio/exec/phases/phase/restore_app.go:567","function":"kasten.io/k10/kio/exec/phases/phase.(*restoreApplicationPhase).createPVCsFromPVCSpecs","linenumber":567,"message":"Failed
              to perform Generic Volume Snapshot
              Restore"},"file":"kasten.io/k10/kio/exec/phases/phase/restore_app.go:363","function":"kasten.io/k10/kio/exec/phases/phase.(*restoreApplicationPhase).restoreApp","linenumber":363,"message":"Failed
              to create PVCs from PVC
              specs"},"file":"kasten.io/k10/kio/exec/internal/runner/phase_runner.go:144","function":"kasten.io/k10/kio/exec/internal/runner.(*phaseRunner).execPlannedPhase","linenumber":144,"message":"Failure
              in planned phase"}'
            message: Job failed to be executed
        name: Restoring Application Components
        startTime: 2024-09-02T17:40:36Z
        state: failed
        updatedTime: 2024-09-02T18:13:38Z
  progress: 100
spec:
  subject:
    apiVersion: apps.kio.kasten.io/v1alpha1
    kind: RestorePoint
    name: scheduled-rjdj4jsrpn
    namespace: kedify
  targetNamespace: helm-test
  filters:
    includeResources:
      - name: absence-fs-helm-absence-pvc
        version: v1
        resource: persistentvolumeclaims
 

K
  • kevin.gottsknecht
  • Author
  • Not a newbie anymore
  • 2 comments
  • Answer
  • September 4, 2024

I fixed my inital problem, with increasing certain timeouts:

values.yaml:

kanister:
  backupTimeout: 1380
  podReadyWaitTimeout: 1380
  restoreTimeout: 1380
 

Hagag
N
2 people like this
  • Hagag
    Hagag
  • N
    NPatel
K
Forum|alt.badge.img
  • kanika.mahajan
  • Comes here often
  • 30 comments
  • September 4, 2024

Hello, I created restoreTimeout from 600  to 1500 in k10-config and restarted executer and kanister deployments. But still it is not being taken into account.

K
Forum|alt.badge.img
  • kanika.mahajan
  • Comes here often
  • 30 comments
  • September 9, 2024

Hello @kevin.gottsknecht are you still able to take backups for cephfs in kasten version 7.0.8? 

I upgraded kasten version from 7.0.6 to 7.0.8 and now during the export phase, I am getting permission denied errors.

 

Failed to copy artifacts Error converting snapshots Failed to export snapshot data Error creating portable snapshot failed running genericVolumeCopy failed running copyVolumeData Failed to execute copy volume data pod function Failed to create and upload backup Failed to exec command in pod command terminated with exit code 1. stdout: stderr: [31mERROR[0m upload error: permission denied

M
Forum|alt.badge.img
  • michaelxue
  • Comes here often
  • 19 comments
  • November 18, 2024

There are few things you can check. 

  1. Run ./k10tools debug catalog size to make sure you have enough space on catalog.
  2. check annotations https://docs.kasten.io/7.0.8/install/storage.html#csi-integration  If you need help please create a support ticket with Veeam Kasten Team. 

Comment


Terms & ConditionsAccessibility statement