K10 7.5.4
kubernetes version v1.29.8-eks-47985ba
I’m interested in moving my velero backups over to Kasten, but so far its been quite unreliable and very inconsistent experience for me, debugging has been a huge hussle due to the fact that the logs almost never points at an issue properly, but i’m fully open to the possibility that this is a skill issue on my side… so please enlighten me.
Additionally the kubernetes is using a private cloud environment that is similar to AWS itself.
I have a local pypi server, with 25gb of data in total, a local object storage (aws s3 like) that seem to be consistent in being able to back up so far, compressed to 14gb after upload all good… even restore worked once.
as part of my effort to migrate into kasten i tried to backup to s3 too in addition to my local object storage.
i made two attempts in total, both resulted in the following errors :
kubectl get --raw /apis/actions.kio.kasten.io/v1alpha1/namespaces/tlv-infra/exportactions/scheduled-sdzjhs8vxf/details
{"kind":"ExportAction","apiVersion":"actions.kio.kasten.io/v1alpha1","metadata":{"name":"scheduled-sdzjhs8vxf","namespace":"tlv-infra","uid":"67f25862-fa8b-11ef-a7ae-5e0c7035baf5","resourceVersion":"43554","creationTimestamp":"2025-03-06T13:03:32Z","labels":{"k10.kasten.io/appName":"tlv-infra","k10.kasten.io/appNamespace":"tlv-infra","k10.kasten.io/policyName":"devpi-hourly-s3","k10.kasten.io/policyNamespace":"kasten-io","k10.kasten.io/runActionName":"run-m5dhhdvtvq","k10.kasten.io/runActionNamespace":"kasten-io"}},"status":{"state":"Failed","startTime":"2025-03-06T13:03:32Z","endTime":"2025-03-06T13:30:57Z","restorePoint":{"name":""},"result":{"name":""},"error":{"cause":"{\"cause\":{\"cause\":{\"cause\":{\"cause\":{\"cause\":{\"cause\":{\"cause\":{\"cause\":{\"file\":\"github.com/kanisterio/kanister@v0.0.0-20250127193535-3fb7f42ff723/pkg/kube/snapshot/snapshot_stable.go\",\"function\":\"github.com/kanisterio/kanister/pkg/kube/snapshot.isReadyToUse\",\"linenumber\":356,\"message\":\"Failed to check and update snapshot content: failed to list snapshot for content snapshot-copy-k6qpcds6-content-2cb91212-53e0-4167-b7e6-a612cf48105c: \\\"can not find snapshot for snapshotID snap-7f992c136188412689b159d7bbb5fb8c\\\"\"},\"file\":\"kasten.io/k10/kio/exec/internal/phaseutils/phaseutils.go:398\",\"function\":\"kasten.io/k10/kio/exec/internal/phaseutils.CreatePVCandSnapshotFromSource\",\"linenumber\":398,\"message\":\"Failed to create snapshot from source\"},\"file\":\"kasten.io/k10/kio/exec/internal/backupobjects/snapshot_based_pvc_creator.go:230\",\"function\":\"kasten.io/k10/kio/exec/internal/backupobjects.(*SnapshotBasedPVCCreatorImpl).createPVCFromCSISnapshot\",\"linenumber\":230,\"message\":\"Failed to clone volume and snapshot from source snapshot\"},\"fields\":[{\"name\":\"snapshotType\",\"value\":\"CSI\"}],\"file\":\"kasten.io/k10/kio/exec/internal/backupobjects/snapshot_based_pvc_creator.go:131\",\"function\":\"kasten.io/k10/kio/exec/internal/backupobjects.(*SnapshotBasedPVCCreatorImpl).CreateVolumeForSnapshot\",\"linenumber\":131,\"message\":\"failed creating snapshot\"},\"file\":\"kasten.io/k10/kio/exec/internal/backupobjects/export_alg.go:299\",\"function\":\"kasten.io/k10/kio/exec/internal/backupobjects.(*ExportAlg).exportFileModeObject\",\"linenumber\":299,\"message\":\"failed to create volume for snapshot\"},\"fields\":[{\"name\":\"type\",\"value\":\"CSI\"},{\"name\":\"id\",\"value\":\"k10-csi-snap-njz4trcm9cxxzm2j\"}],\"file\":\"kasten.io/k10/kio/exec/internal/backupobjects/export_alg.go:197\",\"function\":\"kasten.io/k10/kio/exec/internal/backupobjects.(*ExportAlg).Process.func3\",\"linenumber\":197,\"message\":\"Failed to export snapshot data\"},\"file\":\"kasten.io/k10/kio/exec/phases/phase/artifactcopier.go:439\",\"function\":\"kasten.io/k10/kio/exec/phases/phase.(*ArtifactCopier).convertSnapshots.func2\",\"linenumber\":439,\"message\":\"algorithm failed\"},\"file\":\"kasten.io/k10/kio/exec/phases/phase/artifactcopier.go:232\",\"function\":\"kasten.io/k10/kio/exec/phases/phase.(*ArtifactCopier).Copy\",\"linenumber\":232,\"message\":\"Error converting snapshots\"},\"file\":\"kasten.io/k10/kio/exec/phases/phase/export.go:187\",\"function\":\"kasten.io/k10/kio/exec/phases/phase.(*exportRestorePointPhase).Run\",\"linenumber\":187,\"message\":\"Failed to copy artifacts\"}","message":"Job failed to be executed"},"actionDetails":{"phases":[{"attempt":3,"endTime":"2025-03-06T13:30:56Z","errors":[{"cause":"{\"cause\":{\"cause\":{\"cause\":{\"cause\":{\"cause\":{\"cause\":{\"cause\":{\"cause\":{\"file\":\"kasten.io/k10/kio/kanister/function/kio_copy_volume_data.go:402\",\"function\":\"kasten.io/k10/kio/kanister/function.CopyVolumeData.copyVolumeDataPodExecFunc.func2\",\"linenumber\":402,\"message\":\"Failed to get snapshot ID from create snapshot output\"},\"file\":\"kasten.io/k10/kio/kanister/function/kio_copy_volume_data.go:170\",\"function\":\"kasten.io/k10/kio/kanister/function.CopyVolumeData\",\"linenumber\":170,\"message\":\"Failed to execute copy volume data pod function\"},\"file\":\"kasten.io/k10/kio/exec/internal/backupobjects/file_mode_pvc_exporter.go:198\",\"function\":\"kasten.io/k10/kio/exec/internal/backupobjects.(*FileModePVCExporterImpl).doGenericVolumeCopy\",\"linenumber\":198,\"message\":\"failed running copyVolumeData\"},\"file\":\"kasten.io/k10/kio/exec/internal/backupobjects/file_mode_pvc_exporter.go:124\",\"function\":\"kasten.io/k10/kio/exec/internal/backupobjects.(*FileModePVCExporterImpl).Export\",\"linenumber\":124,\"message\":\"Generic Volume Copy failed\"},\"file\":\"kasten.io/k10/kio/exec/internal/backupobjects/export_alg.go:318\",\"function\":\"kasten.io/k10/kio/exec/internal/backupobjects.(*ExportAlg).exportFileModeObject\",\"linenumber\":318,\"message\":\"Error creating portable snapshot\"},\"fields\":[{\"name\":\"type\",\"value\":\"CSI\"},{\"name\":\"id\",\"value\":\"k10-csi-snap-njz4trcm9cxxzm2j\"}],\"file\":\"kasten.io/k10/kio/exec/internal/backupobjects/export_alg.go:197\",\"function\":\"kasten.io/k10/kio/exec/internal/backupobjects.(*ExportAlg).Process.func3\",\"linenumber\":197,\"message\":\"Failed to export snapshot data\"},\"file\":\"kasten.io/k10/kio/exec/phases/phase/artifactcopier.go:439\",\"function\":\"kasten.io/k10/kio/exec/phases/phase.(*ArtifactCopier).convertSnapshots.func2\",\"linenumber\":439,\"message\":\"algorithm failed\"},\"file\":\"kasten.io/k10/kio/exec/phases/phase/artifactcopier.go:232\",\"function\":\"kasten.io/k10/kio/exec/phases/phase.(*ArtifactCopier).Copy\",\"linenumber\":232,\"message\":\"Error converting snapshots\"},\"file\":\"kasten.io/k10/kio/exec/phases/phase/export.go:187\",\"function\":\"kasten.io/k10/kio/exec/phases/phase.(*exportRestorePointPhase).Run\",\"linenumber\":187,\"message\":\"Failed to copy artifacts\"}","message":"Job failed to be executed"},{"cause":"{\"cause\":{\"cause\":{\"cause\":{\"cause\":{\"cause\":{\"cause\":{\"cause\":{\"cause\":{\"file\":\"github.com/kanisterio/kanister@v0.0.0-20250127193535-3fb7f42ff723/pkg/kube/snapshot/snapshot_stable.go\",\"function\":\"github.com/kanisterio/kanister/pkg/kube/snapshot.isReadyToUse\",\"linenumber\":356,\"message\":\"Failed to check and update snapshot content: failed to list snapshot for content snapshot-copy-q2grjw2j-content-ea561758-f452-4779-8965-2836bf61d411: \\\"can not find snapshot for snapshotID snap-7f992c136188412689b159d7bbb5fb8c\\\"\"},\"file\":\"kasten.io/k10/kio/exec/internal/phaseutils/phaseutils.go:398\",\"function\":\"kasten.io/k10/kio/exec/internal/phaseutils.CreatePVCandSnapshotFromSource\",\"linenumber\":398,\"message\":\"Failed to create snapshot from source\"},\"file\":\"kasten.io/k10/kio/exec/internal/backupobjects/snapshot_based_pvc_creator.go:230\",\"function\":\"kasten.io/k10/kio/exec/internal/backupobjects.(*SnapshotBasedPVCCreatorImpl).createPVCFromCSISnapshot\",\"linenumber\":230,\"message\":\"Failed to clone volume and snapshot from source snapshot\"},\"fields\":[{\"name\":\"snapshotType\",\"value\":\"CSI\"}],\"file\":\"kasten.io/k10/kio/exec/internal/backupobjects/snapshot_based_pvc_creator.go:131\",\"function\":\"kasten.io/k10/kio/exec/internal/backupobjects.(*SnapshotBasedPVCCreatorImpl).CreateVolumeForSnapshot\",\"linenumber\":131,\"message\":\"failed creating snapshot\"},\"file\":\"kasten.io/k10/kio/exec/internal/backupobjects/export_alg.go:299\",\"function\":\"kasten.io/k10/kio/exec/internal/backupobjects.(*ExportAlg).exportFileModeObject\",\"linenumber\":299,\"message\":\"failed to create volume for snapshot\"},\"fields\":[{\"name\":\"type\",\"value\":\"CSI\"},{\"name\":\"id\",\"value\":\"k10-csi-snap-njz4trcm9cxxzm2j\"}],\"file\":\"kasten.io/k10/kio/exec/internal/backupobjects/export_alg.go:197\",\"function\":\"kasten.io/k10/kio/exec/internal/backupobjects.(*ExportAlg).Process.func3\",\"linenumber\":197,\"message\":\"Failed to export snapshot data\"},\"file\":\"kasten.io/k10/kio/exec/phases/phase/artifactcopier.go:439\",\"function\":\"kasten.io/k10/kio/exec/phases/phase.(*ArtifactCopier).convertSnapshots.func2\",\"linenumber\":439,\"message\":\"algorithm failed\"},\"file\":\"kasten.io/k10/kio/exec/phases/phase/artifactcopier.go:232\",\"function\":\"kasten.io/k10/kio/exec/phases/phase.(*ArtifactCopier).Copy\",\"linenumber\":232,\"message\":\"Error converting snapshots\"},\"file\":\"kasten.io/k10/kio/exec/phases/phase/export.go:187\",\"function\":\"kasten.io/k10/kio/exec/phases/phase.(*exportRestorePointPhase).Run\",\"linenumber\":187,\"message\":\"Failed to copy artifacts\"}","message":"Job failed to be executed"},{"cause":"{\"cause\":{\"cause\":{\"cause\":{\"cause\":{\"cause\":{\"cause\":{\"cause\":{\"cause\":{\"file\":\"github.com/kanisterio/kanister@v0.0.0-20250127193535-3fb7f42ff723/pkg/kube/snapshot/snapshot_stable.go\",\"function\":\"github.com/kanisterio/kanister/pkg/kube/snapshot.isReadyToUse\",\"linenumber\":356,\"message\":\"Failed to check and update snapshot content: failed to list snapshot for content snapshot-copy-k6qpcds6-content-2cb91212-53e0-4167-b7e6-a612cf48105c: \\\"can not find snapshot for snapshotID snap-7f992c136188412689b159d7bbb5fb8c\\\"\"},\"file\":\"kasten.io/k10/kio/exec/internal/phaseutils/phaseutils.go:398\",\"function\":\"kasten.io/k10/kio/exec/internal/phaseutils.CreatePVCandSnapshotFromSource\",\"linenumber\":398,\"message\":\"Failed to create snapshot from source\"},\"file\":\"kasten.io/k10/kio/exec/internal/backupobjects/snapshot_based_pvc_creator.go:230\",\"function\":\"kasten.io/k10/kio/exec/internal/backupobjects.(*SnapshotBasedPVCCreatorImpl).createPVCFromCSISnapshot\",\"linenumber\":230,\"message\":\"Failed to clone volume and snapshot from source snapshot\"},\"fields\":[{\"name\":\"snapshotType\",\"value\":\"CSI\"}],\"file\":\"kasten.io/k10/kio/exec/internal/backupobjects/snapshot_based_pvc_creator.go:131\",\"function\":\"kasten.io/k10/kio/exec/internal/backupobjects.(*SnapshotBasedPVCCreatorImpl).CreateVolumeForSnapshot\",\"linenumber\":131,\"message\":\"failed creating snapshot\"},\"file\":\"kasten.io/k10/kio/exec/internal/backupobjects/export_alg.go:299\",\"function\":\"kasten.io/k10/kio/exec/internal/backupobjects.(*ExportAlg).exportFileModeObject\",\"linenumber\":299,\"message\":\"failed to create volume for snapshot\"},\"fields\":[{\"name\":\"type\",\"value\":\"CSI\"},{\"name\":\"id\",\"value\":\"k10-csi-snap-njz4trcm9cxxzm2j\"}],\"file\":\"kasten.io/k10/kio/exec/internal/backupobjects/export_alg.go:197\",\"function\":\"kasten.io/k10/kio/exec/internal/backupobjects.(*ExportAlg).Process.func3\",\"linenumber\":197,\"message\":\"Failed to export snapshot data\"},\"file\":\"kasten.io/k10/kio/exec/phases/phase/artifactcopier.go:439\",\"function\":\"kasten.io/k10/kio/exec/phases/phase.(*ArtifactCopier).convertSnapshots.func2\",\"linenumber\":439,\"message\":\"algorithm failed\"},\"file\":\"kasten.io/k10/kio/exec/phases/phase/artifactcopier.go:232\",\"function\":\"kasten.io/k10/kio/exec/phases/phase.(*ArtifactCopier).Copy\",\"linenumber\":232,\"message\":\"Error converting snapshots\"},\"file\":\"kasten.io/k10/kio/exec/phases/phase/export.go:187\",\"function\":\"kasten.io/k10/kio/exec/phases/phase.(*exportRestorePointPhase).Run\",\"linenumber\":187,\"message\":\"Failed to copy artifacts\"}","message":"Job failed to be executed"}],"name":"Exporting RestorePoint","startTime":"2025-03-06T13:03:33Z","state":"failed","updatedTime":"2025-03-06T13:30:56Z","volumeOperations":[{"namespace":"tlv-infra","pvcName":"devpi-pvc","operation":"Upload","dataFormat":"GenericVolumeCopy","exportDirective":"FileSystemMode","storageClass":"ebs-sc","storageType":"CSI","snapshotId":"k10-csi-snap-njz4trcm9cxxzm2j","volumeSnapshotClass":"ebs-vsc"}]}]},"progress":100,"progressDetails":{"processedBytes":0,"readBytes":0,"totalBytes":107374182400,"transferredBytes":0,"processingRate":0,"totalVolumes":1,"completedVolumes":0,"updatedTime":"2025-03-06T13:30:56Z"}},"spec":{"subject":{"name":"tlv-infra","namespace":"tlv-infra"},"scheduledTime":"2025-03-06T13:00:00Z","receiveString":"bIzAPpoanmEF4TF7iwlvncghfEJLHlOcyDiy9W8ia8EfMNBYllv+XM+6a9Lbmd85PQfgzKuNGU+u7zELhTr9ghZ3lAFLP9qde5Op/fmU4rWg6qE5v31VJg6mxNfSUaZgZZ4Nsx14ir1TnbOQ7NzF6kihKc/L2Uwc0lyfDEiW3/OwxKcMAhmzlFOSsliRD0NE9qUkT6agu2pM00o3MTiL9929XXSzHSjrpihH94zbaJdioYyeZS3AdosMu6SFyo31PM+SJrmJctQ/QXX7oFkmsSfK9/6SbfqB1gWRFWJ5BmdyVXvSq3ZwuDa0stXH9t1e0EEyLe5RkXNIzIgZyXzSWvBP4rwDMV8Iu8dripH6uKHy2E+L243R5DUvw1aunCdrqMntOJgFm62vRD9+y9z5d0fglK4SBe3YqRqrz5xQOWZbnPddLcjBZnQ1WBDfnP/zEh2D7rR0Od3gFkhB/hJkUnvnz4Fy47FwcDrnO35L4YIBtQYSgxTJRlhSYSxDFpjo267EXRTBiCJon8ZDkAr+g2EPrOspRPaBcWQhOpaojCR4pN5fKN/3ATwZdU66Ni2LTEJRYWPRskdIqLeT4YPqPIk/hw","profile":{"name":"tlv-infra","namespace":"kasten-io"},"migrationToken":{"name":"devpi-hourly-s3-migration-token","namespace":"kasten-io"},"exportData":{"enabled":true,"exporterStorageClassName":"ebs-sc-nondefault"}}}
➜ ~
its also important to mention that apps with minimal data or no persistent storage seems to be backing up quite consistently to both local object storage and S3.
My goal is to backup a yum repository that i have which contains approximately 400gb (which isn’t really alot but fails to on similar terms) so for now i’m trying to atleast get the smaller app data (25gb app ) to backup consistently
but this is the yum repository backup log (backup to local object storage) :
kubectl get --raw /apis/actions.kio.kasten.io/v1alpha1/namespaces/tlv-infra/exportactions/scheduled-2hkc6svznr/details
{"kind":"ExportAction","apiVersion":"actions.kio.kasten.io/v1alpha1","metadata":{"name":"scheduled-2hkc6svznr","namespace":"tlv-infra","uid":"214815c3-f93c-11ef-a7ae-5e0c7035baf5","resourceVersion":"26016","creationTimestamp":"2025-03-04T21:03:32Z","labels":{"k10.kasten.io/appName":"tlv-infra","k10.kasten.io/appNamespace":"tlv-infra","k10.kasten.io/isRunNow":"true","k10.kasten.io/policyName":"yum-repo-weekly-ngos","k10.kasten.io/policyNamespace":"kasten-io","k10.kasten.io/runActionName":"policy-run-zx2h5","k10.kasten.io/runActionNamespace":"kasten-io"}},"status":{"state":"Failed","startTime":"2025-03-04T21:03:32Z","endTime":"2025-03-04T23:32:31Z","restorePoint":{"name":""},"result":{"name":""},"error":{"cause":"{\"cause\":{\"cause\":{\"cause\":{\"cause\":{\"cause\":{\"cause\":{\"cause\":{\"cause\":{\"file\":\"github.com/kanisterio/kanister@v0.0.0-20250127193535-3fb7f42ff723/pkg/kube/snapshot/snapshot_stable.go\",\"function\":\"github.com/kanisterio/kanister/pkg/kube/snapshot.isReadyToUse\",\"linenumber\":356,\"message\":\"Failed to check and update snapshot content: failed to list snapshot for content snapshot-copy-2jbs252c-content-404dd5c6-d382-4fe8-8056-e32bdeb028ec: \\\"rpc error: code = Internal desc = Could not get snapshot ID \\\\\\\"snap-9bc2c55ab859410692a33a07a8cfa64e\\\\\\\": operation error EC2: DescribeSnapshots, https response error StatusCode: 401, RequestID: , api error UnknownError: UnknownError\\\"\"},\"file\":\"kasten.io/k10/kio/exec/internal/phaseutils/phaseutils.go:398\",\"function\":\"kasten.io/k10/kio/exec/internal/phaseutils.CreatePVCandSnapshotFromSource\",\"linenumber\":398,\"message\":\"Failed to create snapshot from source\"},\"file\":\"kasten.io/k10/kio/exec/internal/backupobjects/snapshot_based_pvc_creator.go:230\",\"function\":\"kasten.io/k10/kio/exec/internal/backupobjects.(*SnapshotBasedPVCCreatorImpl).createPVCFromCSISnapshot\",\"linenumber\":230,\"message\":\"Failed to clone volume and snapshot from source snapshot\"},\"fields\":[{\"name\":\"snapshotType\",\"value\":\"CSI\"}],\"file\":\"kasten.io/k10/kio/exec/internal/backupobjects/snapshot_based_pvc_creator.go:131\",\"function\":\"kasten.io/k10/kio/exec/internal/backupobjects.(*SnapshotBasedPVCCreatorImpl).CreateVolumeForSnapshot\",\"linenumber\":131,\"message\":\"failed creating snapshot\"},\"file\":\"kasten.io/k10/kio/exec/internal/backupobjects/export_alg.go:299\",\"function\":\"kasten.io/k10/kio/exec/internal/backupobjects.(*ExportAlg).exportFileModeObject\",\"linenumber\":299,\"message\":\"failed to create volume for snapshot\"},\"fields\":[{\"name\":\"type\",\"value\":\"CSI\"},{\"name\":\"id\",\"value\":\"k10-csi-snap-r8s52s2w4r545p29\"}],\"file\":\"kasten.io/k10/kio/exec/internal/backupobjects/export_alg.go:197\",\"function\":\"kasten.io/k10/kio/exec/internal/backupobjects.(*ExportAlg).Process.func3\",\"linenumber\":197,\"message\":\"Failed to export snapshot data\"},\"file\":\"kasten.io/k10/kio/exec/phases/phase/artifactcopier.go:439\",\"function\":\"kasten.io/k10/kio/exec/phases/phase.(*ArtifactCopier).convertSnapshots.func2\",\"linenumber\":439,\"message\":\"algorithm failed\"},\"file\":\"kasten.io/k10/kio/exec/phases/phase/artifactcopier.go:232\",\"function\":\"kasten.io/k10/kio/exec/phases/phase.(*ArtifactCopier).Copy\",\"linenumber\":232,\"message\":\"Error converting snapshots\"},\"file\":\"kasten.io/k10/kio/exec/phases/phase/export.go:187\",\"function\":\"kasten.io/k10/kio/exec/phases/phase.(*exportRestorePointPhase).Run\",\"linenumber\":187,\"message\":\"Failed to copy artifacts\"}","message":"Job failed to be executed"},"actionDetails":{"phases":[{"attempt":3,"endTime":"2025-03-04T23:32:31Z","errors":[{"cause":"{\"cause\":{\"cause\":{\"cause\":{\"cause\":{\"cause\":{\"cause\":{\"cause\":{\"cause\":{\"file\":\"kasten.io/k10/kio/kanister/function/kio_copy_volume_data.go:402\",\"function\":\"kasten.io/k10/kio/kanister/function.CopyVolumeData.copyVolumeDataPodExecFunc.func2\",\"linenumber\":402,\"message\":\"Failed to get snapshot ID from create snapshot output\"},\"file\":\"kasten.io/k10/kio/kanister/function/kio_copy_volume_data.go:170\",\"function\":\"kasten.io/k10/kio/kanister/function.CopyVolumeData\",\"linenumber\":170,\"message\":\"Failed to execute copy volume data pod function\"},\"file\":\"kasten.io/k10/kio/exec/internal/backupobjects/file_mode_pvc_exporter.go:198\",\"function\":\"kasten.io/k10/kio/exec/internal/backupobjects.(*FileModePVCExporterImpl).doGenericVolumeCopy\",\"linenumber\":198,\"message\":\"failed running copyVolumeData\"},\"file\":\"kasten.io/k10/kio/exec/internal/backupobjects/file_mode_pvc_exporter.go:124\",\"function\":\"kasten.io/k10/kio/exec/internal/backupobjects.(*FileModePVCExporterImpl).Export\",\"linenumber\":124,\"message\":\"Generic Volume Copy failed\"},\"file\":\"kasten.io/k10/kio/exec/internal/backupobjects/export_alg.go:318\",\"function\":\"kasten.io/k10/kio/exec/internal/backupobjects.(*ExportAlg).exportFileModeObject\",\"linenumber\":318,\"message\":\"Error creating portable snapshot\"},\"fields\":[{\"name\":\"type\",\"value\":\"CSI\"},{\"name\":\"id\",\"value\":\"k10-csi-snap-r8s52s2w4r545p29\"}],\"file\":\"kasten.io/k10/kio/exec/internal/backupobjects/export_alg.go:197\",\"function\":\"kasten.io/k10/kio/exec/internal/backupobjects.(*ExportAlg).Process.func3\",\"linenumber\":197,\"message\":\"Failed to export snapshot data\"},\"file\":\"kasten.io/k10/kio/exec/phases/phase/artifactcopier.go:439\",\"function\":\"kasten.io/k10/kio/exec/phases/phase.(*ArtifactCopier).convertSnapshots.func2\",\"linenumber\":439,\"message\":\"algorithm failed\"},\"file\":\"kasten.io/k10/kio/exec/phases/phase/artifactcopier.go:232\",\"function\":\"kasten.io/k10/kio/exec/phases/phase.(*ArtifactCopier).Copy\",\"linenumber\":232,\"message\":\"Error converting snapshots\"},\"file\":\"kasten.io/k10/kio/exec/phases/phase/export.go:187\",\"function\":\"kasten.io/k10/kio/exec/phases/phase.(*exportRestorePointPhase).Run\",\"linenumber\":187,\"message\":\"Failed to copy artifacts\"}","message":"Job failed to be executed"},{"cause":"{\"cause\":{\"cause\":{\"cause\":{\"cause\":{\"cause\":{\"cause\":{\"cause\":{\"cause\":{\"file\":\"kasten.io/k10/kio/kanister/function/kio_copy_volume_data.go:402\",\"function\":\"kasten.io/k10/kio/kanister/function.CopyVolumeData.copyVolumeDataPodExecFunc.func2\",\"linenumber\":402,\"message\":\"Failed to get snapshot ID from create snapshot output\"},\"file\":\"kasten.io/k10/kio/kanister/function/kio_copy_volume_data.go:170\",\"function\":\"kasten.io/k10/kio/kanister/function.CopyVolumeData\",\"linenumber\":170,\"message\":\"Failed to execute copy volume data pod function\"},\"file\":\"kasten.io/k10/kio/exec/internal/backupobjects/file_mode_pvc_exporter.go:198\",\"function\":\"kasten.io/k10/kio/exec/internal/backupobjects.(*FileModePVCExporterImpl).doGenericVolumeCopy\",\"linenumber\":198,\"message\":\"failed running copyVolumeData\"},\"file\":\"kasten.io/k10/kio/exec/internal/backupobjects/file_mode_pvc_exporter.go:124\",\"function\":\"kasten.io/k10/kio/exec/internal/backupobjects.(*FileModePVCExporterImpl).Export\",\"linenumber\":124,\"message\":\"Generic Volume Copy failed\"},\"file\":\"kasten.io/k10/kio/exec/internal/backupobjects/export_alg.go:318\",\"function\":\"kasten.io/k10/kio/exec/internal/backupobjects.(*ExportAlg).exportFileModeObject\",\"linenumber\":318,\"message\":\"Error creating portable snapshot\"},\"fields\":[{\"name\":\"type\",\"value\":\"CSI\"},{\"name\":\"id\",\"value\":\"k10-csi-snap-r8s52s2w4r545p29\"}],\"file\":\"kasten.io/k10/kio/exec/internal/backupobjects/export_alg.go:197\",\"function\":\"kasten.io/k10/kio/exec/internal/backupobjects.(*ExportAlg).Process.func3\",\"linenumber\":197,\"message\":\"Failed to export snapshot data\"},\"file\":\"kasten.io/k10/kio/exec/phases/phase/artifactcopier.go:439\",\"function\":\"kasten.io/k10/kio/exec/phases/phase.(*ArtifactCopier).convertSnapshots.func2\",\"linenumber\":439,\"message\":\"algorithm failed\"},\"file\":\"kasten.io/k10/kio/exec/phases/phase/artifactcopier.go:232\",\"function\":\"kasten.io/k10/kio/exec/phases/phase.(*ArtifactCopier).Copy\",\"linenumber\":232,\"message\":\"Error converting snapshots\"},\"file\":\"kasten.io/k10/kio/exec/phases/phase/export.go:187\",\"function\":\"kasten.io/k10/kio/exec/phases/phase.(*exportRestorePointPhase).Run\",\"linenumber\":187,\"message\":\"Failed to copy artifacts\"}","message":"Job failed to be executed"},{"cause":"{\"cause\":{\"cause\":{\"cause\":{\"cause\":{\"cause\":{\"cause\":{\"cause\":{\"cause\":{\"file\":\"github.com/kanisterio/kanister@v0.0.0-20250127193535-3fb7f42ff723/pkg/kube/snapshot/snapshot_stable.go\",\"function\":\"github.com/kanisterio/kanister/pkg/kube/snapshot.isReadyToUse\",\"linenumber\":356,\"message\":\"Failed to check and update snapshot content: failed to list snapshot for content snapshot-copy-2jbs252c-content-404dd5c6-d382-4fe8-8056-e32bdeb028ec: \\\"rpc error: code = Internal desc = Could not get snapshot ID \\\\\\\"snap-9bc2c55ab859410692a33a07a8cfa64e\\\\\\\": operation error EC2: DescribeSnapshots, https response error StatusCode: 401, RequestID: , api error UnknownError: UnknownError\\\"\"},\"file\":\"kasten.io/k10/kio/exec/internal/phaseutils/phaseutils.go:398\",\"function\":\"kasten.io/k10/kio/exec/internal/phaseutils.CreatePVCandSnapshotFromSource\",\"linenumber\":398,\"message\":\"Failed to create snapshot from source\"},\"file\":\"kasten.io/k10/kio/exec/internal/backupobjects/snapshot_based_pvc_creator.go:230\",\"function\":\"kasten.io/k10/kio/exec/internal/backupobjects.(*SnapshotBasedPVCCreatorImpl).createPVCFromCSISnapshot\",\"linenumber\":230,\"message\":\"Failed to clone volume and snapshot from source snapshot\"},\"fields\":[{\"name\":\"snapshotType\",\"value\":\"CSI\"}],\"file\":\"kasten.io/k10/kio/exec/internal/backupobjects/snapshot_based_pvc_creator.go:131\",\"function\":\"kasten.io/k10/kio/exec/internal/backupobjects.(*SnapshotBasedPVCCreatorImpl).CreateVolumeForSnapshot\",\"linenumber\":131,\"message\":\"failed creating snapshot\"},\"file\":\"kasten.io/k10/kio/exec/internal/backupobjects/export_alg.go:299\",\"function\":\"kasten.io/k10/kio/exec/internal/backupobjects.(*ExportAlg).exportFileModeObject\",\"linenumber\":299,\"message\":\"failed to create volume for snapshot\"},\"fields\":[{\"name\":\"type\",\"value\":\"CSI\"},{\"name\":\"id\",\"value\":\"k10-csi-snap-r8s52s2w4r545p29\"}],\"file\":\"kasten.io/k10/kio/exec/internal/backupobjects/export_alg.go:197\",\"function\":\"kasten.io/k10/kio/exec/internal/backupobjects.(*ExportAlg).Process.func3\",\"linenumber\":197,\"message\":\"Failed to export snapshot data\"},\"file\":\"kasten.io/k10/kio/exec/phases/phase/artifactcopier.go:439\",\"function\":\"kasten.io/k10/kio/exec/phases/phase.(*ArtifactCopier).convertSnapshots.func2\",\"linenumber\":439,\"message\":\"algorithm failed\"},\"file\":\"kasten.io/k10/kio/exec/phases/phase/artifactcopier.go:232\",\"function\":\"kasten.io/k10/kio/exec/phases/phase.(*ArtifactCopier).Copy\",\"linenumber\":232,\"message\":\"Error converting snapshots\"},\"file\":\"kasten.io/k10/kio/exec/phases/phase/export.go:187\",\"function\":\"kasten.io/k10/kio/exec/phases/phase.(*exportRestorePointPhase).Run\",\"linenumber\":187,\"message\":\"Failed to copy artifacts\"}","message":"Job failed to be executed"}],"name":"Exporting RestorePoint","startTime":"2025-03-04T21:03:32Z","state":"failed","updatedTime":"2025-03-04T23:32:31Z","volumeOperations":[{"namespace":"tlv-infra","pvcName":"yum-repos-d","operation":"Upload","dataFormat":"GenericVolumeCopy","exportDirective":"FileSystemMode","storageClass":"ebs-sc","storageType":"CSI","snapshotId":"k10-csi-snap-trcsqxc7mtx9mrgx","volumeSnapshotClass":"ebs-vsc"},{"namespace":"tlv-infra","pvcName":"yum-repo-packages","operation":"Upload","dataFormat":"GenericVolumeCopy","exportDirective":"FileSystemMode","storageClass":"ebs-sc","storageType":"CSI","snapshotId":"k10-csi-snap-r8s52s2w4r545p29","volumeSnapshotClass":"ebs-vsc"}]}]},"progress":100,"progressDetails":{"processedBytes":0,"readBytes":0,"totalBytes":483183820800,"transferredBytes":0,"processingRate":0,"totalVolumes":1,"completedVolumes":0,"updatedTime":"2025-03-04T23:32:30Z"}},"spec":{"subject":{"name":"tlv-infra","namespace":"tlv-infra"},"scheduledTime":"2025-03-04T21:01:48Z","receiveString":"bIzAPpoanmEJ5Axflw8xhdxYWU8wcmOh6kOl5nwTb64nNet5sTrtepWHev/a5/AWBCXWzKuNGU+u7zELhTr92DQBtQg8Hse8b4mOzLjQ9J7Srv15zGERBU+N4Jb7MrVfTaIy5WFB48timbOQ7NzJ73WFNcmVwVhl91HkYHir/Yin17Q9BnaLkWizlTmCKRl554glMYmPgkh600o3MTiL9929XXSzHSjrpihH94zbaJdioYyeZS3AdosHtK2Qh965Mo7HaaGIcsFhEjSypEtq73KIsafBOL2YkAqHDmx2WWg+WnHF9jVjum66tN7Q9NNky11uK9VFwzpImMsnn3zEWutP5aBEZVAB65QvnYO6/Ou1jhzMjNvDgWx00gbJyG4878fiCcoRgby9ECh1j9frMxu53L4RQuLRogz+h9gVNn0ex+EacpzVJyIjWVDOy72oJFGQ6PB/K7bkDUpQ6wl/RDfklIZ6qap6bnfwbn5bp40d9hwHjQvGW1hfLHpfO5H9nLbcMDLCnyR/uN1em03w3D9Ivec0Wb3UPTNoJd/rnD85+ZlBcJTsFhoZJQfxFiHDExBYmxlUoYc/5Sw8q7pA8MwkVYWNk9wp6DCNM8S4JOA2rv+Xmj4KOdB1LoALWYhV95gAt3Zj013E21e2EA","profile":{"name":"ngos-infra","namespace":"kasten-io"},"migrationToken":{"name":"yum-repo-weekly-ngos-migration-token","namespace":"kasten-io"},"exportData":{"enabled":true}}}
