unable to restore cephfs pvc | Veeam Community Resource Hub

Hello! I am also facing a problem during restore of a Cephfs PVC. When the restore operation is launched, it creates restore data and data mover pod, and I verified some data existence in the volume attached with restore data pod, but after 10-12 minutes, the restore data pod is getting deleted, and kasten retries and recreates the restore data pod three times and fails. In the kasten dashboard, I am seeing the error.

kind: RestoreAction

apiVersion: actions.kio.kasten.io/v1alpha1

metadata:

namespace: kedify-restore-fs

uid: 4161dbb1-66f2-11ef-96ef-0a580a80039f

resourceVersion: "8104"

creationTimestamp: 2024-08-30T17:06:53Z

labels:

k10.kasten.io/appName: kedify

k10.kasten.io/appNamespace: kedify

status:

state: Failed

startTime: 2024-08-30T17:06:53Z

endTime: 2024-08-30T17:33:23Z

restorePoint:

result:

error:

cause: '{"cause":{"cause":{"cause":{"cause":{"cause":{"cause":{"cause":{"message":"{\"message\":\"Failed

to restore backup from Kopia API

server\",\"function\":\"kasten.io/k10/kio/kanister/function.restoreDataFromServer.restoreDataFromServerPodFunc.func3\",\"linenumber\":367,\"file\":\"kasten.io/k10/kio/kanister/function/restore_data_from_server.go:367\",\"cause\":{\"message\":\"context

deadline

exceeded\"}}"},"fields":>{"name":"actionSet","value":{"metadata":{"creationTimestamp":"2024-08-30T17:23:07Z","generateName":"k10-restorefromserver-k10-deployment-generic-volume-2.0.43-k10restore-e2001d4e-4a45-44db-b242-ef37e167135c-kasten-io-pvc-","generation":6,"labels":{"kanister.io/JobID":"4165c935-66f2-11ef-9e00-0a580a800394"},"managedFields":-{"apiVersion":"cr.kanister.io/v1alpha1","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:generateName":{},"f:labels":{".":{},"f:kanister.io/JobID":{}}},"f:spec":{".":{},"f:actions":{}}},"manager":"executor-server","operation":"Update","time":"2024-08-30T17:23:07Z"},{"apiVersion":"cr.kanister.io/v1alpha1","fieldsType":"FieldsV1","fieldsV1":{"f:status":{".":{},"f:actions":{},"f:error":{".":{},"f:message":{}},"f:progress":{".":{},"f:lastTransitionTime":{},"f:percentCompleted":{}},"f:state":{}}},"manager":"controller","operation":"Update","time":"2024-08-30T17:33:12Z"}],"name":"k10-restorefromserver-k10-deployment-generic-volume-2.0.43f8k89","namespace":"kasten-io","resourceVersion":"189097467","uid":"d1b20006-e369-49c3-b004-1e66cc9d30e8"},"spec":{"actions":0{"artifacts":{"snapshot":{"keyValue":{"backupIdentifier":"db391b2e3b607b96328a99936fa43027","backupPath":"/mnt/vol_data/kanister-pvc","funcVersion":"v1.0.0-alpha","objectStorePath":"repo/e6332964-a3a9-4ee9-8094-96a8ed23f44a/","phySize":"7.3

GB","size":"15

GB"}}},"blueprint":"k10-deployment-generic-volume-2.0.43","name":"restoreFromServer","object":{"apiVersion":"","group":"","kind":"pvc","name":"k10restore-e2001d4e-4a45-44db-b242-ef37e167135c","namespace":"kasten-io","resource":""},"options":{"hostName":"e6332964-a3a9-4ee9-8094-96a8ed23f44a.absence-fs-api-absence.absence-fs-helm-absence-pvc","pointInTimeConnection":"\"0001-01-01T00:00:00.000Z\"","serverAddress":"https://172.30.88.67:51515","userName":"k10-admin"},"podOverride":{"containers":"{"name":"container","resources":{},"volumeMounts"::{"mountPath":"/etc/ssl/certs/custom-ca-bundle.pem","name":"custom-ca-bundle-store","subPath":"custom-ca-bundle.pem"}]}],"securityContext":{"runAsNonRoot":false,"runAsUser":0},"tolerations":o{"effect":"NoExecute","key":"node.kubernetes.io/not-ready","operator":"Exists","tolerationSeconds":300},{"effect":"NoExecute","key":"node.kubernetes.io/unreachable","operator":"Exists","tolerationSeconds":300},{"effect":"NoSchedule","key":"node.kubernetes.io/memory-pressure","operator":"Exists"}],"volumes":-{"configMap":{"defaultMode":420,"name":"custom-ca-bundle-store"},"name":"custom-ca-bundle-store"}]},"preferredVersion":"v1.0.0-alpha","secrets":{"certs":{"apiVersion":"","group":"","kind":"secret","name":"kopia-tls-cert","namespace":"kasten-io","resource":""},"serverPassphraseKey":{"apiVersion":"","group":"","kind":"secret","name":"data-mover-server-passphrase-qvghr","namespace":"kasten-io","resource":""},"userPassphraseKey":{"apiVersion":"","group":"","kind":"secret","name":"data-mover-user-passphrase-dvdmd","namespace":"kasten-io","resource":""}}}]},"status":{"actions":i{"blueprint":"k10-deployment-generic-volume-2.0.43","deferPhase":{"name":"","progress":{},"state":""},"name":"restoreFromServer","object":{"apiVersion":"","group":"","kind":"pvc","name":"k10restore-e2001d4e-4a45-44db-b242-ef37e167135c","namespace":"kasten-io","resource":""},"phases":s{"name":"restoreFromServer","progress":{"lastTransitionTime":"2024-08-30T17:23:12Z","progressPercent":"0"},"state":"failed"}]}],"error":{"message":"{\"message\":\"Failed

to restore backup from Kopia API

server\",\"function\":\"kasten.io/k10/kio/kanister/function.restoreDataFromServer.restoreDataFromServerPodFunc.func3\",\"linenumber\":367,\"file\":\"kasten.io/k10/kio/kanister/function/restore_data_from_server.go:367\",\"cause\":{\"message\":\"context

deadline

exceeded\"}}"},"progress":{"lastTransitionTime":"2024-08-30T17:23:12Z","percentCompleted":"0"},"state":"failed"}}}],"file":"kasten.io/k10/kio/kanister/operation.go:167","function":"kasten.io/k10/kio/kanister.(*Operation).WaitForActionSet","linenumber":167,"message":"ActionSet

Failed"},"file":"kasten.io/k10/kio/exec/phases/phase/restore_app.go:2348","function":"kasten.io/k10/kio/exec/phases/phase.GenericVolumeSnapshotRestore","linenumber":2348,"message":"Failed

to execute action

set"},"fields":<{"name":"k8sType","value":"deployment"},{"name":"pvcName","value":"absence-fs-helm-absence-pvc"}],"file":"kasten.io/k10/kio/exec/phases/phase/restore_app.go:2091","function":"kasten.io/k10/kio/exec/phases/phase.(*restoreApplicationPhase).restoreDataIntoPVC","linenumber":2091,"message":"Failed

to restore

PVC"},"file":"kasten.io/k10/kio/exec/phases/phase/restore_app.go:1816","function":"kasten.io/k10/kio/exec/phases/phase.(*restoreApplicationPhase).restoreDataIntoPVCs","linenumber":1816,"message":"Failed

to restore some of the generic volume

snapshots"},"file":"kasten.io/k10/kio/exec/phases/phase/restore_app.go:567","function":"kasten.io/k10/kio/exec/phases/phase.(*restoreApplicationPhase).createPVCsFromPVCSpecs","linenumber":567,"message":"Failed

to perform Generic Volume Snapshot

Restore"},"file":"kasten.io/k10/kio/exec/phases/phase/restore_app.go:363","function":"kasten.io/k10/kio/exec/phases/phase.(*restoreApplicationPhase).restoreApp","linenumber":363,"message":"Failed

to create PVCs from PVC

specs"},"file":"kasten.io/k10/kio/exec/internal/runner/phase_runner.go:144","function":"kasten.io/k10/kio/exec/internal/runner.(*phaseRunner).execPlannedPhase","linenumber":144,"message":"Failure

in planned phase"}'

message: Job failed to be executed

actionDetails:

phases:

- attempt: 3

endTime: 2024-08-30T17:33:23Z

errors:

- cause: '{"cause":{"cause":{"cause":{"cause":{"cause":{"cause":{"cause":{"message":"{\"message\":\"Failed

to restore backup from Kopia API

server\",\"function\":\"kasten.io/k10/kio/kanister/function.restoreDataFromServer.restoreDataFromServerPodFunc.func3\",\"linenumber\":367,\"file\":\"kasten.io/k10/kio/kanister/function/restore_data_from_server.go:367\",\"cause\":{\"message\":\"context

deadline

exceeded\"}}"},"fields":r{"name":"actionSet","value":{"metadata":{"creationTimestamp":"2024-08-30T17:07:28Z","generateName":"k10-restorefromserver-k10-deployment-generic-volume-2.0.43-k10restore-18cc9fd5-21f2-40a6-b976-3815ee99092e-kasten-io-pvc-","generation":6,"labels":{"kanister.io/JobID":"4165c935-66f2-11ef-9e00-0a580a800394"},"managedFields":4{"apiVersion":"cr.kanister.io/v1alpha1","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:generateName":{},"f:labels":{".":{},"f:kanister.io/JobID":{}}},"f:spec":{".":{},"f:actions":{}}},"manager":"executor-server","operation":"Update","time":"2024-08-30T17:07:28Z"},{"apiVersion":"cr.kanister.io/v1alpha1","fieldsType":"FieldsV1","fieldsV1":{"f:status":{".":{},"f:actions":{},"f:error":{".":{},"f:message":{}},"f:progress":{".":{},"f:lastTransitionTime":{},"f:percentCompleted":{}},"f:state":{}}},"manager":"controller","operation":"Update","time":"2024-08-30T17:17:32Z"}],"name":"k10-restorefromserver-k10-deployment-generic-volume-2.0.43kkczb","namespace":"kasten-io","resourceVersion":"189092050","uid":"c4d93d66-8170-44dc-a3de-335a17687bb1"},"spec":{"actions":c{"artifacts":{"snapshot":{"keyValue":{"backupIdentifier":"db391b2e3b607b96328a99936fa43027","backupPath":"/mnt/vol_data/kanister-pvc","funcVersion":"v1.0.0-alpha","objectStorePath":"repo/e6332964-a3a9-4ee9-8094-96a8ed23f44a/","phySize":"7.3

GB","size":"15

GB"}}},"blueprint":"k10-deployment-generic-volume-2.0.43","name":"restoreFromServer","object":{"apiVersion":"","group":"","kind":"pvc","name":"k10restore-18cc9fd5-21f2-40a6-b976-3815ee99092e","namespace":"kasten-io","resource":""},"options":{"hostName":"e6332964-a3a9-4ee9-8094-96a8ed23f44a.absence-fs-api-absence.absence-fs-helm-absence-pvc","pointInTimeConnection":"\"0001-01-01T00:00:00.000Z\"","serverAddress":"https://172.30.192.33:51515","userName":"k10-admin"},"podOverride":{"containers":,{"name":"container","resources":{},"volumeMounts":Z{"mountPath":"/etc/ssl/certs/custom-ca-bundle.pem","name":"custom-ca-bundle-store","subPath":"custom-ca-bundle.pem"}]}],"securityContext":{"runAsNonRoot":false,"runAsUser":0},"tolerations":-{"effect":"NoExecute","key":"node.kubernetes.io/not-ready","operator":"Exists","tolerationSeconds":300},{"effect":"NoExecute","key":"node.kubernetes.io/unreachable","operator":"Exists","tolerationSeconds":300},{"effect":"NoSchedule","key":"node.kubernetes.io/memory-pressure","operator":"Exists"}],"volumes":r{"configMap":{"defaultMode":420,"name":"custom-ca-bundle-store"},"name":"custom-ca-bundle-store"}]},"preferredVersion":"v1.0.0-alpha","secrets":{"certs":{"apiVersion":"","group":"","kind":"secret","name":"kopia-tls-cert","namespace":"kasten-io","resource":""},"serverPassphraseKey":{"apiVersion":"","group":"","kind":"secret","name":"data-mover-server-passphrase-jwqhq","namespace":"kasten-io","resource":""},"userPassphraseKey":{"apiVersion":"","group":"","kind":"secret","name":"data-mover-user-passphrase-4q4k2","namespace":"kasten-io","resource":""}}}]},"status":{"actions":K{"blueprint":"k10-deployment-generic-volume-2.0.43","deferPhase":{"name":"","progress":{},"state":""},"name":"restoreFromServer","object":{"apiVersion":"","group":"","kind":"pvc","name":"k10restore-18cc9fd5-21f2-40a6-b976-3815ee99092e","namespace":"kasten-io","resource":""},"phases":o{"name":"restoreFromServer","progress":{"lastTransitionTime":"2024-08-30T17:07:33Z","progressPercent":"0"},"state":"failed"}]}],"error":{"message":"{\"message\":\"Failed

to restore backup from Kopia API

server\",\"function\":\"kasten.io/k10/kio/kanister/function.restoreDataFromServer.restoreDataFromServerPodFunc.func3\",\"linenumber\":367,\"file\":\"kasten.io/k10/kio/kanister/function/restore_data_from_server.go:367\",\"cause\":{\"message\":\"context

deadline

exceeded\"}}"},"progress":{"lastTransitionTime":"2024-08-30T17:07:33Z","percentCompleted":"0"},"state":"failed"}}}],"file":"kasten.io/k10/kio/kanister/operation.go:167","function":"kasten.io/k10/kio/kanister.(*Operation).WaitForActionSet","linenumber":167,"message":"ActionSet

Failed"},"file":"kasten.io/k10/kio/exec/phases/phase/restore_app.go:2348","function":"kasten.io/k10/kio/exec/phases/phase.GenericVolumeSnapshotRestore","linenumber":2348,"message":"Failed

to execute action

set"},"fields":"{"name":"k8sType","value":"deployment"},{"name":"pvcName","value":"absence-fs-helm-absence-pvc"}],"file":"kasten.io/k10/kio/exec/phases/phase/restore_app.go:2091","function":"kasten.io/k10/kio/exec/phases/phase.(*restoreApplicationPhase).restoreDataIntoPVC","linenumber":2091,"message":"Failed

to restore

PVC"},"file":"kasten.io/k10/kio/exec/phases/phase/restore_app.go:1816","function":"kasten.io/k10/kio/exec/phases/phase.(*restoreApplicationPhase).restoreDataIntoPVCs","linenumber":1816,"message":"Failed

to restore some of the generic volume

snapshots"},"file":"kasten.io/k10/kio/exec/phases/phase/restore_app.go:567","function":"kasten.io/k10/kio/exec/phases/phase.(*restoreApplicationPhase).createPVCsFromPVCSpecs","linenumber":567,"message":"Failed

to perform Generic Volume Snapshot

Restore"},"file":"kasten.io/k10/kio/exec/phases/phase/restore_app.go:363","function":"kasten.io/k10/kio/exec/phases/phase.(*restoreApplicationPhase).restoreApp","linenumber":363,"message":"Failed

to create PVCs from PVC

specs"},"file":"kasten.io/k10/kio/exec/internal/runner/phase_runner.go:144","function":"kasten.io/k10/kio/exec/internal/runner.(*phaseRunner).execPlannedPhase","linenumber":144,"message":"Failure

in planned phase"}'

message: Job failed to be executed

- cause: '{"cause":{"cause":{"cause":{"cause":{"cause":{"cause":{"cause":{"message":"{\"message\":\"Failed

to restore backup from Kopia API

server\",\"function\":\"kasten.io/k10/kio/kanister/function.restoreDataFromServer.restoreDataFromServerPodFunc.func3\",\"linenumber\":367,\"file\":\"kasten.io/k10/kio/kanister/function/restore_data_from_server.go:367\",\"cause\":{\"message\":\"Failed

to exec command in pod: command terminated with exit code

1.\\nstdout: \\nstderr: \\u001bD31mERROR\\u001bd0m error

restoring: restore error: copy file: error creating file: cannot

write data to file %q

/mnt/vol_data/kanister-pvc/archive_13_2021-01-08.tar.gz:

unexpected content error: GetContent: 500 Internal Server Error:

internal server error: error getting cached content: failed to get

blob with ID

p3f243aefb34b1f3d4ce594fd104d7675-scbdd835f3e79a76712c: unable to

complete

GetBlobFromPath:/mnt/data/outcastv2-test/repo/e6332964-a3a9-4ee9-8094-96a8ed23f44a/p/3f2/43aefb34b1f3d4ce594fd104d7675-scbdd835f3e79a76712c.f

despite 10 retries: open

/mnt/data/outcastv2-test/repo/e6332964-a3a9-4ee9-8094-96a8ed23f44a/p/3f2/43aefb34b1f3d4ce594fd104d7675-scbdd835f3e79a76712c.f:

permission

denied\"}}"},"fields":t{"name":"actionSet","value":{"metadata":{"creationTimestamp":"2024-08-30T17:18:37Z","generateName":"k10-restorefromserver-k10-deployment-generic-volume-2.0.43-k10restore-2e3d5b67-24f3-439a-8869-d241b0519522-kasten-io-pvc-","generation":6,"labels":{"kanister.io/JobID":"4165c935-66f2-11ef-9e00-0a580a800394"},"managedFields":�{"apiVersion":"cr.kanister.io/v1alpha1","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:generateName":{},"f:labels":{".":{},"f:kanister.io/JobID":{}}},"f:spec":{".":{},"f:actions":{}}},"manager":"executor-server","operation":"Update","time":"2024-08-30T17:18:37Z"},{"apiVersion":"cr.kanister.io/v1alpha1","fieldsType":"FieldsV1","fieldsV1":{"f:status":{".":{},"f:actions":{},"f:error":{".":{},"f:message":{}},"f:progress":{".":{},"f:lastTransitionTime":{},"f:percentCompleted":{}},"f:state":{}}},"manager":"controller","operation":"Update","time":"2024-08-30T17:21:50Z"}],"name":"k10-restorefromserver-k10-deployment-generic-volume-2.0.43j9cqr","namespace":"kasten-io","resourceVersion":"189093596","uid":"0613b011-8c69-4bf2-ac0d-bd6ffbdf039d"},"spec":{"actions":o{"artifacts":{"snapshot":{"keyValue":{"backupIdentifier":"db391b2e3b607b96328a99936fa43027","backupPath":"/mnt/vol_data/kanister-pvc","funcVersion":"v1.0.0-alpha","objectStorePath":"repo/e6332964-a3a9-4ee9-8094-96a8ed23f44a/","phySize":"7.3

GB","size":"15

GB"}}},"blueprint":"k10-deployment-generic-volume-2.0.43","name":"restoreFromServer","object":{"apiVersion":"","group":"","kind":"pvc","name":"k10restore-2e3d5b67-24f3-439a-8869-d241b0519522","namespace":"kasten-io","resource":""},"options":{"hostName":"e6332964-a3a9-4ee9-8094-96a8ed23f44a.absence-fs-api-absence.absence-fs-helm-absence-pvc","pointInTimeConnection":"\"0001-01-01T00:00:00.000Z\"","serverAddress":"https://172.30.205.147:51515","userName":"k10-admin"},"podOverride":{"containers":b{"name":"container","resources":{},"volumeMounts":m{"mountPath":"/etc/ssl/certs/custom-ca-bundle.pem","name":"custom-ca-bundle-store","subPath":"custom-ca-bundle.pem"}]}],"securityContext":{"runAsNonRoot":false,"runAsUser":0},"tolerations":m{"effect":"NoExecute","key":"node.kubernetes.io/not-ready","operator":"Exists","tolerationSeconds":300},{"effect":"NoExecute","key":"node.kubernetes.io/unreachable","operator":"Exists","tolerationSeconds":300},{"effect":"NoSchedule","key":"node.kubernetes.io/memory-pressure","operator":"Exists"}],"volumes":a{"configMap":{"defaultMode":420,"name":"custom-ca-bundle-store"},"name":"custom-ca-bundle-store"}]},"preferredVersion":"v1.0.0-alpha","secrets":{"certs":{"apiVersion":"","group":"","kind":"secret","name":"kopia-tls-cert","namespace":"kasten-io","resource":""},"serverPassphraseKey":{"apiVersion":"","group":"","kind":"secret","name":"data-mover-server-passphrase-5b2pb","namespace":"kasten-io","resource":""},"userPassphraseKey":{"apiVersion":"","group":"","kind":"secret","name":"data-mover-user-passphrase-sqjsx","namespace":"kasten-io","resource":""}}}]},"status":{"actions":"{"blueprint":"k10-deployment-generic-volume-2.0.43","deferPhase":{"name":"","progress":{},"state":""},"name":"restoreFromServer","object":{"apiVersion":"","group":"","kind":"pvc","name":"k10restore-2e3d5b67-24f3-439a-8869-d241b0519522","namespace":"kasten-io","resource":""},"phases"::{"name":"restoreFromServer","progress":{"lastTransitionTime":"2024-08-30T17:18:43Z","progressPercent":"0"},"state":"failed"}]}],"error":{"message":"{\"message\":\"Failed

to restore backup from Kopia API

server\",\"function\":\"kasten.io/k10/kio/kanister/function.restoreDataFromServer.restoreDataFromServerPodFunc.func3\",\"linenumber\":367,\"file\":\"kasten.io/k10/kio/kanister/function/restore_data_from_server.go:367\",\"cause\":{\"message\":\"Failed

to exec command in pod: command terminated with exit code

1.\\nstdout: \\nstderr: \\u001bp31mERROR\\u001b<0m error

restoring: restore error: copy file: error creating file: cannot

write data to file %q

/mnt/vol_data/kanister-pvc/archive_13_2021-01-08.tar.gz:

unexpected content error: GetContent: 500 Internal Server Error:

internal server error: error getting cached content: failed to get

blob with ID

p3f243aefb34b1f3d4ce594fd104d7675-scbdd835f3e79a76712c: unable to

complete

GetBlobFromPath:/mnt/data/outcastv2-test/repo/e6332964-a3a9-4ee9-8094-96a8ed23f44a/p/3f2/43aefb34b1f3d4ce594fd104d7675-scbdd835f3e79a76712c.f

despite 10 retries: open

/mnt/data/outcastv2-test/repo/e6332964-a3a9-4ee9-8094-96a8ed23f44a/p/3f2/43aefb34b1f3d4ce594fd104d7675-scbdd835f3e79a76712c.f:

permission

denied\"}}"},"progress":{"lastTransitionTime":"2024-08-30T17:18:43Z","percentCompleted":"0"},"state":"failed"}}}],"file":"kasten.io/k10/kio/kanister/operation.go:167","function":"kasten.io/k10/kio/kanister.(*Operation).WaitForActionSet","linenumber":167,"message":"ActionSet

Failed"},"file":"kasten.io/k10/kio/exec/phases/phase/restore_app.go:2348","function":"kasten.io/k10/kio/exec/phases/phase.GenericVolumeSnapshotRestore","linenumber":2348,"message":"Failed

Page 1 / 1

Hi @kanika.mahajan

what is your location profile type? i think it is a permission issue
from the logs you can see that “GetContent” throws permission denied issue.

unexpected content error: GetContent: 500 Internal Server Error:

internal server error: error getting cached content: failed to getblob with ID p3f243aefb34b1f3d4ce594fd104d7675-scbdd835f3e79a76712c: unable to complete GetBlobFromPath:/mnt/data/outcastv2-test/repo/e6332964-a3a9-4ee9-8094-96a8ed23f44a/p/3f2/43aefb34b1f3d4ce594fd104d7675-scbdd835f3e79a76712c.f

despite 10 retries: open

/mnt/data/outcastv2-test/repo/e6332964-a3a9-4ee9-8094-96a8ed23f44a/p/3f2/43aefb34b1f3d4ce594fd104d7675-scbdd835f3e79a76712c.f:permission denied

BR,
Ahmed Hagag

Hello, On my test cluster, I am using a PVC RBD (RWO) for the location profile. And the error I am getting is

error:
cause: '{"cause":{"cause":{"cause":{"cause":{"cause":{"cause":{"cause":{"message":"{\"message\":\"Failed
to restore backup from Kopia API
server\",\"function\":\"kasten.io/k10/kio/kanister/function.restoreDataFromServer.restoreDataFromServerPodFunc.func3\",\"linenumber\":367,\"file\":\"kasten.io/k10/kio/kanister/function/restore_data_from_server.go:367\",\"cause\":{\"message\":\"context
deadline
exceeded\"}}"},"fields":\{"name":"actionSet","value":{"metadata":{"creationTimestamp":"2024-09-02T18:03:29Z","generateName":"k10-restorefromserver-k10-deployment-generic-volume-2.0.43-k10restore-af5ac119-d54e-4391-bba9-f8cb8cc2a780-kasten-io-pvc-","generation":6,"labels":{"kanister.io/JobID":"6f9f538e-6952-11ef-9962-0a580a800394"},"managedFields":"{"apiVersion":"cr.kanister.io/v1alpha1","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:generateName":{},"f:labels":{".":{},"f:kanister.io/JobID":{}}},"f:spec":{".":{},"f:actions":{}}},"manager":"executor-server","operation":"Update","time":"2024-09-02T18:03:29Z"},{"apiVersion":"cr.kanister.io/v1alpha1","fieldsType":"FieldsV1","fieldsV1":{"f:status":{".":{},"f:actions":{},"f:error":{".":{},"f:message":{}},"f:progress":{".":{},"f:lastTransitionTime":{},"f:percentCompleted":{}},"f:state":{}}},"manager":"controller","operation":"Update","time":"2024-09-02T18:13:33Z"}],"name":"k10-restorefromserver-k10-deployment-generic-volume-2.0.43zc4qr","namespace":"kasten-io","resourceVersion":"190531487","uid":"fc5512d1-793f-408e-a4ca-32686b91f127"},"spec":{"actions":p{"artifacts":{"snapshot":{"keyValue":{"backupIdentifier":"c391a162ef24dc43a0cbc7b979f0da4e","backupPath":"/mnt/vol_data/kanister-pvc","funcVersion":"v1.0.0-alpha","objectStorePath":"repo/e6332964-a3a9-4ee9-8094-96a8ed23f44a/","phySize":"7.3
GB","size":"14
GB"}}},"blueprint":"k10-deployment-generic-volume-2.0.43","name":"restoreFromServer","object":{"apiVersion":"","group":"","kind":"pvc","name":"k10restore-af5ac119-d54e-4391-bba9-f8cb8cc2a780","namespace":"kasten-io","resource":""},"options":{"hostName":"e6332964-a3a9-4ee9-8094-96a8ed23f44a.absence-fs-api-absence.absence-fs-helm-absence-pvc","pointInTimeConnection":"\"0001-01-01T00:00:00.000Z\"","serverAddress":"https://172.30.197.178:51515","userName":"k10-admin"},"podOverride":{"containers":e{"name":"container","resources":{},"volumeMounts":e{"mountPath":"/etc/ssl/certs/custom-ca-bundle.pem","name":"custom-ca-bundle-store","subPath":"custom-ca-bundle.pem"}]}],"securityContext":{"runAsNonRoot":false,"runAsUser":0},"tolerations":s{"effect":"NoExecute","key":"node.kubernetes.io/not-ready","operator":"Exists","tolerationSeconds":300},{"effect":"NoExecute","key":"node.kubernetes.io/unreachable","operator":"Exists","tolerationSeconds":300},{"effect":"NoSchedule","key":"node.kubernetes.io/memory-pressure","operator":"Exists"}],"volumes"::{"configMap":{"defaultMode":420,"name":"custom-ca-bundle-store"},"name":"custom-ca-bundle-store"}]},"preferredVersion":"v1.0.0-alpha","secrets":{"certs":{"apiVersion":"","group":"","kind":"secret","name":"kopia-tls-cert","namespace":"kasten-io","resource":""},"serverPassphraseKey":{"apiVersion":"","group":"","kind":"secret","name":"data-mover-server-passphrase-l5db4","namespace":"kasten-io","resource":""},"userPassphraseKey":{"apiVersion":"","group":"","kind":"secret","name":"data-mover-user-passphrase-zbqhh","namespace":"kasten-io","resource":""}}}]},"status":{"actions":}{"blueprint":"k10-deployment-generic-volume-2.0.43","deferPhase":{"name":"","progress":{},"state":""},"name":"restoreFromServer","object":{"apiVersion":"","group":"","kind":"pvc","name":"k10restore-af5ac119-d54e-4391-bba9-f8cb8cc2a780","namespace":"kasten-io","resource":""},"phases":e{"name":"restoreFromServer","progress":{"lastTransitionTime":"2024-09-02T18:03:34Z","progressPercent":"0"},"state":"failed"}]}],"error":{"message":"{\"message\":\"Failed
to restore backup from Kopia API
server\",\"function\":\"kasten.io/k10/kio/kanister/function.restoreDataFromServer.restoreDataFromServerPodFunc.func3\",\"linenumber\":367,\"file\":\"kasten.io/k10/kio/kanister/function/restore_data_from_server.go:367\",\"cause\":{\"message\":\"context
deadline
exceeded\"}}"},"progress":{"lastTransitionTime":"2024-09-02T18:03:34Z","percentCompleted":"0"},"state":"failed"}}}],"file":"kasten.io/k10/kio/kanister/operation.go:167","function":"kasten.io/k10/kio/kanister.(*Operation).WaitForActionSet","linenumber":167,"message":"ActionSet
Failed"},"file":"kasten.io/k10/kio/exec/phases/phase/restore_app.go:2348","function":"kasten.io/k10/kio/exec/phases/phase.GenericVolumeSnapshotRestore","linenumber":2348,"message":"Failed
to execute action
set"},"fields":x{"name":"k8sType","value":"deployment"},{"name":"pvcName","value":"absence-fs-helm-absence-pvc"}],"file":"kasten.io/k10/kio/exec/phases/phase/restore_app.go:2091","function":"kasten.io/k10/kio/exec/phases/phase.(*restoreApplicationPhase).restoreDataIntoPVC","linenumber":2091,"message":"Failed
to restore
PVC"},"file":"kasten.io/k10/kio/exec/phases/phase/restore_app.go:1816","function":"kasten.io/k10/kio/exec/phases/phase.(*restoreApplicationPhase).restoreDataIntoPVCs","linenumber":1816,"message":"Failed
to restore some of the generic volume
snapshots"},"file":"kasten.io/k10/kio/exec/phases/phase/restore_app.go:567","function":"kasten.io/k10/kio/exec/phases/phase.(*restoreApplicationPhase).createPVCsFromPVCSpecs","linenumber":567,"message":"Failed
to perform Generic Volume Snapshot
Restore"},"file":"kasten.io/k10/kio/exec/phases/phase/restore_app.go:363","function":"kasten.io/k10/kio/exec/phases/phase.(*restoreApplicationPhase).restoreApp","linenumber":363,"message":"Failed
to create PVCs from PVC
specs"},"file":"kasten.io/k10/kio/exec/internal/runner/phase_runner.go:144","function":"kasten.io/k10/kio/exec/internal/runner.(*phaseRunner).execPlannedPhase","linenumber":144,"message":"Failure
in planned phase"}'
message: Job failed to be executed

Hello @Hagag , on my test cluster, I am using PVC RBD (RWO) for the location profile. And I am getting below error:-

error:
cause: '{"cause":{"cause":{"cause":{"cause":{"cause":{"cause":{"cause":{"message":"{\"message\":\"Failed
to restore backup from Kopia API
server\",\"function\":\"kasten.io/k10/kio/kanister/function.restoreDataFromServer.restoreDataFromServerPodFunc.func3\",\"linenumber\":367,\"file\":\"kasten.io/k10/kio/kanister/function/restore_data_from_server.go:367\",\"cause\":{\"message\":\"context
deadline
exceeded\"}}"},"fields":\{"name":"actionSet","value":{"metadata":{"creationTimestamp":"2024-09-02T18:03:29Z","generateName":"k10-restorefromserver-k10-deployment-generic-volume-2.0.43-k10restore-af5ac119-d54e-4391-bba9-f8cb8cc2a780-kasten-io-pvc-","generation":6,"labels":{"kanister.io/JobID":"6f9f538e-6952-11ef-9962-0a580a800394"},"managedFields":"{"apiVersion":"cr.kanister.io/v1alpha1","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:generateName":{},"f:labels":{".":{},"f:kanister.io/JobID":{}}},"f:spec":{".":{},"f:actions":{}}},"manager":"executor-server","operation":"Update","time":"2024-09-02T18:03:29Z"},{"apiVersion":"cr.kanister.io/v1alpha1","fieldsType":"FieldsV1","fieldsV1":{"f:status":{".":{},"f:actions":{},"f:error":{".":{},"f:message":{}},"f:progress":{".":{},"f:lastTransitionTime":{},"f:percentCompleted":{}},"f:state":{}}},"manager":"controller","operation":"Update","time":"2024-09-02T18:13:33Z"}],"name":"k10-restorefromserver-k10-deployment-generic-volume-2.0.43zc4qr","namespace":"kasten-io","resourceVersion":"190531487","uid":"fc5512d1-793f-408e-a4ca-32686b91f127"},"spec":{"actions":p{"artifacts":{"snapshot":{"keyValue":{"backupIdentifier":"c391a162ef24dc43a0cbc7b979f0da4e","backupPath":"/mnt/vol_data/kanister-pvc","funcVersion":"v1.0.0-alpha","objectStorePath":"repo/e6332964-a3a9-4ee9-8094-96a8ed23f44a/","phySize":"7.3
GB","size":"14
GB"}}},"blueprint":"k10-deployment-generic-volume-2.0.43","name":"restoreFromServer","object":{"apiVersion":"","group":"","kind":"pvc","name":"k10restore-af5ac119-d54e-4391-bba9-f8cb8cc2a780","namespace":"kasten-io","resource":""},"options":{"hostName":"e6332964-a3a9-4ee9-8094-96a8ed23f44a.absence-fs-api-absence.absence-fs-helm-absence-pvc","pointInTimeConnection":"\"0001-01-01T00:00:00.000Z\"","serverAddress":"https://172.30.197.178:51515","userName":"k10-admin"},"podOverride":{"containers":e{"name":"container","resources":{},"volumeMounts":e{"mountPath":"/etc/ssl/certs/custom-ca-bundle.pem","name":"custom-ca-bundle-store","subPath":"custom-ca-bundle.pem"}]}],"securityContext":{"runAsNonRoot":false,"runAsUser":0},"tolerations":s{"effect":"NoExecute","key":"node.kubernetes.io/not-ready","operator":"Exists","tolerationSeconds":300},{"effect":"NoExecute","key":"node.kubernetes.io/unreachable","operator":"Exists","tolerationSeconds":300},{"effect":"NoSchedule","key":"node.kubernetes.io/memory-pressure","operator":"Exists"}],"volumes"::{"configMap":{"defaultMode":420,"name":"custom-ca-bundle-store"},"name":"custom-ca-bundle-store"}]},"preferredVersion":"v1.0.0-alpha","secrets":{"certs":{"apiVersion":"","group":"","kind":"secret","name":"kopia-tls-cert","namespace":"kasten-io","resource":""},"serverPassphraseKey":{"apiVersion":"","group":"","kind":"secret","name":"data-mover-server-passphrase-l5db4","namespace":"kasten-io","resource":""},"userPassphraseKey":{"apiVersion":"","group":"","kind":"secret","name":"data-mover-user-passphrase-zbqhh","namespace":"kasten-io","resource":""}}}]},"status":{"actions":}{"blueprint":"k10-deployment-generic-volume-2.0.43","deferPhase":{"name":"","progress":{},"state":""},"name":"restoreFromServer","object":{"apiVersion":"","group":"","kind":"pvc","name":"k10restore-af5ac119-d54e-4391-bba9-f8cb8cc2a780","namespace":"kasten-io","resource":""},"phases":e{"name":"restoreFromServer","progress":{"lastTransitionTime":"2024-09-02T18:03:34Z","progressPercent":"0"},"state":"failed"}]}],"error":{"message":"{\"message\":\"Failed
to restore backup from Kopia API
server\",\"function\":\"kasten.io/k10/kio/kanister/function.restoreDataFromServer.restoreDataFromServerPodFunc.func3\",\"linenumber\":367,\"file\":\"kasten.io/k10/kio/kanister/function/restore_data_from_server.go:367\",\"cause\":{\"message\":\"context
deadline
exceeded\"}}"},"progress":{"lastTransitionTime":"2024-09-02T18:03:34Z","percentCompleted":"0"},"state":"failed"}}}],"file":"kasten.io/k10/kio/kanister/operation.go:167","function":"kasten.io/k10/kio/kanister.(*Operation).WaitForActionSet","linenumber":167,"message":"ActionSet
Failed"},"file":"kasten.io/k10/kio/exec/phases/phase/restore_app.go:2348","function":"kasten.io/k10/kio/exec/phases/phase.GenericVolumeSnapshotRestore","linenumber":2348,"message":"Failed
to execute action
set"},"fields":x{"name":"k8sType","value":"deployment"},{"name":"pvcName","value":"absence-fs-helm-absence-pvc"}],"file":"kasten.io/k10/kio/exec/phases/phase/restore_app.go:2091","function":"kasten.io/k10/kio/exec/phases/phase.(*restoreApplicationPhase).restoreDataIntoPVC","linenumber":2091,"message":"Failed
to restore
PVC"},"file":"kasten.io/k10/kio/exec/phases/phase/restore_app.go:1816","function":"kasten.io/k10/kio/exec/phases/phase.(*restoreApplicationPhase).restoreDataIntoPVCs","linenumber":1816,"message":"Failed
to restore some of the generic volume
snapshots"},"file":"kasten.io/k10/kio/exec/phases/phase/restore_app.go:567","function":"kasten.io/k10/kio/exec/phases/phase.(*restoreApplicationPhase).createPVCsFromPVCSpecs","linenumber":567,"message":"Failed
to perform Generic Volume Snapshot
Restore"},"file":"kasten.io/k10/kio/exec/phases/phase/restore_app.go:363","function":"kasten.io/k10/kio/exec/phases/phase.(*restoreApplicationPhase).restoreApp","linenumber":363,"message":"Failed
to create PVCs from PVC
specs"},"file":"kasten.io/k10/kio/exec/internal/runner/phase_runner.go:144","function":"kasten.io/k10/kio/exec/internal/runner.(*phaseRunner).execPlannedPhase","linenumber":144,"message":"Failure
in planned phase"}'
message: Job failed to be executed

Comment

Sign up

Login to the community

Scanning file for viruses.

This file cannot be downloaded