Backup policy getting failed during export phase

@Madi.Cristil @safiya - this might be better in the Kasten section for help.

@jaiganeshjk 

 Hello @kanika.mahajan,

Please have you follow all steps from our documentation (https://docs.kasten.io/latest/install/storage.html#snapshots-as-shallow-read-only-volumes-cephfs-only).

Starting from Veeam Kasten 7.0.8 there is no need to add annotations to persist the SeLinuxOptions on OCP clusters, it is now setup by default, in this case since you are already on 7.0.8 it is not needed.

Please let me know if you have done the setup in previous version and upgraded to 7.0.8 or not.

Regards,
Rubens

Hello @FRubens ! I tried to remove below annotation and executed a backup policy, But still I am getting permission denied errors

annotations:

k10.kasten.io/sc-preserve-selinux-options: 'true'

- cause:
    cause:
      cause:
        cause:
          cause:
            cause:
              cause:
                cause:
                  cause:
                    cause:
                      message: "command terminated with exit code 1.

                        stdout:\ 

                        stderr: \e[31mERROR\e[0m upload error: permission
                        denied"
                    file: github.com/kanisterio/kanister@v0.0.0-20240920021913-d207c416a800/pkg/kube/exec.go
                    function: github.com/kanisterio/kanister/pkg/kube.ExecWithOptions
                    linenumber: 156
                    message: Failed to exec command in pod
                  file: kasten.io/k10/kio/kanister/function/kio_copy_volume_data.go:384
                  function: kasten.io/k10/kio/kanister/function.CopyVolumeData.copyVolumeDataPodExecFunc.func2
                  linenumber: 384
                  message: Failed to create and upload backup
                file: kasten.io/k10/kio/kanister/function/kio_copy_volume_data.go:166
                function: kasten.io/k10/kio/kanister/function.CopyVolumeData
                linenumber: 166
                message: Failed to execute copy volume data pod function
              file: kasten.io/k10/kio/exec/internal/snapshotconverters/ac_gvc_converter.go:249
              function: kasten.io/k10/kio/exec/internal/snapshotconverters.(*GVCConverterInternalAPIImpl).genericVolumeCopy
              linenumber: 249
              message: failed running copyVolumeData
            file: kasten.io/k10/kio/exec/internal/snapshotconverters/ac_gvc_converter.go:170
            function: kasten.io/k10/kio/exec/internal/snapshotconverters.(*GVCConverterInternalAPIImpl).CopySnapshotRestoredInPVC
            linenumber: 170
            message: failed running genericVolumeCopy
          file: kasten.io/k10/kio/exec/internal/snapshotconverters/ac_gvc_converter.go:77
          function: kasten.io/k10/kio/exec/internal/snapshotconverters.(*GVCConverter).Convert
          linenumber: 77
          message: Error creating portable snapshot
        fields:
          - name: type
            value: CSI
          - name: id
            value: k10-csi-snap-n5ptp2zcwq58t7sr
        file: kasten.io/k10/kio/exec/phases/phase/artifactcopier.go:544
        function: kasten.io/k10/kio/exec/phases/phase.(*ArtifactCopier).convertSnapshots.func1
        linenumber: 544
        message: Failed to export snapshot data
      file: kasten.io/k10/kio/exec/phases/phase/artifactcopier.go:274
      function: kasten.io/k10/kio/exec/phases/phase.(*ArtifactCopier).Copy
      linenumber: 274
      message: Error converting snapshots
    file: kasten.io/k10/kio/exec/phases/phase/export.go:172
    function: kasten.io/k10/kio/exec/phases/phase.(*exportRestorePointPhase).Run
    linenumber: 172
    message: Failed to copy artifacts
  message: Job failed to be executed
- cause:
    cause:
      cause:
        cause:
          cause:
            cause:
              cause:
                cause:
                  cause:
                    cause:
                      message: "command terminated with exit code 1.

                        stdout:\ 

                        stderr: \e[31mERROR\e[0m upload error: permission
                        denied"
                    file: github.com/kanisterio/kanister@v0.0.0-20240920021913-d207c416a800/pkg/kube/exec.go
                    function: github.com/kanisterio/kanister/pkg/kube.ExecWithOptions
                    linenumber: 156
                    message: Failed to exec command in pod
                  file: kasten.io/k10/kio/kanister/function/kio_copy_volume_data.go:384
                  function: kasten.io/k10/kio/kanister/function.CopyVolumeData.copyVolumeDataPodExecFunc.func2
                  linenumber: 384
                  message: Failed to create and upload backup
                file: kasten.io/k10/kio/kanister/function/kio_copy_volume_data.go:166
                function: kasten.io/k10/kio/kanister/function.CopyVolumeData
                linenumber: 166
                message: Failed to execute copy volume data pod function
              file: kasten.io/k10/kio/exec/internal/snapshotconverters/ac_gvc_converter.go:249
              function: kasten.io/k10/kio/exec/internal/snapshotconverters.(*GVCConverterInternalAPIImpl).genericVolumeCopy
              linenumber: 249
              message: failed running copyVolumeData
            file: kasten.io/k10/kio/exec/internal/snapshotconverters/ac_gvc_converter.go:170
            function: kasten.io/k10/kio/exec/internal/snapshotconverters.(*GVCConverterInternalAPIImpl).CopySnapshotRestoredInPVC
            linenumber: 170
            message: failed running genericVolumeCopy
          file: kasten.io/k10/kio/exec/internal/snapshotconverters/ac_gvc_converter.go:77
          function: kasten.io/k10/kio/exec/internal/snapshotconverters.(*GVCConverter).Convert
          linenumber: 77
          message: Error creating portable snapshot
        fields:
          - name: type
            value: CSI
          - name: id
            value: k10-csi-snap-n5ptp2zcwq58t7sr
        file: kasten.io/k10/kio/exec/phases/phase/artifactcopier.go:544
        function: kasten.io/k10/kio/exec/phases/phase.(*ArtifactCopier).convertSnapshots.func1
        linenumber: 544
        message: Failed to export snapshot data
      file: kasten.io/k10/kio/exec/phases/phase/artifactcopier.go:274
      function: kasten.io/k10/kio/exec/phases/phase.(*ArtifactCopier).Copy
      linenumber: 274
      message: Error converting snapshots
    file: kasten.io/k10/kio/exec/phases/phase/export.go:172
    function: kasten.io/k10/kio/exec/phases/phase.(*exportRestorePointPhase).Run
    linenumber: 172
    message: Failed to copy artifacts
  message: Job failed to be executed
- cause:
    cause:
      cause:
        cause:
          cause:
            cause:
              cause:
                cause:
                  cause:
                    cause:
                      message: "command terminated with exit code 1.

                        stdout:\ 

                        stderr: \e[31mERROR\e[0m upload error: permission
                        denied"
                    file: github.com/kanisterio/kanister@v0.0.0-20240920021913-d207c416a800/pkg/kube/exec.go
                    function: github.com/kanisterio/kanister/pkg/kube.ExecWithOptions
                    linenumber: 156
                    message: Failed to exec command in pod
                  file: kasten.io/k10/kio/kanister/function/kio_copy_volume_data.go:384
                  function: kasten.io/k10/kio/kanister/function.CopyVolumeData.copyVolumeDataPodExecFunc.func2
                  linenumber: 384
                  message: Failed to create and upload backup
                file: kasten.io/k10/kio/kanister/function/kio_copy_volume_data.go:166
                function: kasten.io/k10/kio/kanister/function.CopyVolumeData
                linenumber: 166
                message: Failed to execute copy volume data pod function
              file: kasten.io/k10/kio/exec/internal/snapshotconverters/ac_gvc_converter.go:249
              function: kasten.io/k10/kio/exec/internal/snapshotconverters.(*GVCConverterInternalAPIImpl).genericVolumeCopy
              linenumber: 249
              message: failed running copyVolumeData
            file: kasten.io/k10/kio/exec/internal/snapshotconverters/ac_gvc_converter.go:170
            function: kasten.io/k10/kio/exec/internal/snapshotconverters.(*GVCConverterInternalAPIImpl).CopySnapshotRestoredInPVC
            linenumber: 170
            message: failed running genericVolumeCopy
          file: kasten.io/k10/kio/exec/internal/snapshotconverters/ac_gvc_converter.go:77
          function: kasten.io/k10/kio/exec/internal/snapshotconverters.(*GVCConverter).Convert
          linenumber: 77
          message: Error creating portable snapshot
        fields:
          - name: type
            value: CSI
          - name: id
            value: k10-csi-snap-n5ptp2zcwq58t7sr
        file: kasten.io/k10/kio/exec/phases/phase/artifactcopier.go:544
        function: kasten.io/k10/kio/exec/phases/phase.(*ArtifactCopier).convertSnapshots.func1
        linenumber: 544
        message: Failed to export snapshot data
      file: kasten.io/k10/kio/exec/phases/phase/artifactcopier.go:274
      function: kasten.io/k10/kio/exec/phases/phase.(*ArtifactCopier).Copy
      linenumber: 274
      message: Error converting snapshots
    file: kasten.io/k10/kio/exec/phases/phase/export.go:172
    function: kasten.io/k10/kio/exec/phases/phase.(*exportRestorePointPhase).Run
    linenumber: 172
    message: Failed to copy artifacts
  message: Job failed to be executed
 

Hello @FRubens ! Yes, I did the shallow read-only setup in version 6.5.11 and it was working before upgrade

Hello @kanika.mahajan,

Thank you for the information.

Please have you already raised a support case for this issue ?

If not please I would like to ask you to raise a case attaching the debug logs, this way we can investigate further to understand the issue.

One more question, Do you setup custom SeLinuxOptions in pods/namespaces ?

Regards

Rubens

Yes @FRubens I already raised the support case and had troubleshooting session with k10 team.

No, I am on OKD(OPENSHIFT) which uses default values from security context constraints.

Thank you @kanika.mahajan .

Will proceed with the investigation in the support case and provide updates there as soon as possible.

Regards

Rubens