Backup policy getting failed during export phase

@Madi.Cristil @safiya - this might be better in the Kasten section for help.

@jaiganeshjk 

 Hello @kanika.mahajan,

Please have you follow all steps from our documentation (https://docs.kasten.io/latest/install/storage.html#snapshots-as-shallow-read-only-volumes-cephfs-only).

Starting from Veeam Kasten 7.0.8 there is no need to add annotations to persist the SeLinuxOptions on OCP clusters, it is now setup by default, in this case since you are already on 7.0.8 it is not needed.

Please let me know if you have done the setup in previous version and upgraded to 7.0.8 or not.

Regards,
Rubens

Hello @FRubens ! I tried to remove below annotation and executed a backup policy, But still I am getting permission denied errors

annotations:

k10.kasten.io/sc-preserve-selinux-options: 'true'

- cause:
    cause:
      cause:
        cause:
          cause:
            cause:
              cause:
                cause:
                  cause:
                    cause:
                      message: "command terminated with exit code 1.

                        stdout:\ 

                        stderr: \e[31mERROR\e[0m upload error: permission
                        denied"
                    file: github.com/kanisterio/kanister@v0.0.0-20240920021913-d207c416a800/pkg/kube/exec.go
                    function: github.com/kanisterio/kanister/pkg/kube.ExecWithOptions
                    linenumber: 156
                    message: Failed to exec command in pod
                  file: kasten.io/k10/kio/kanister/function/kio_copy_volume_data.go:384
                  function: kasten.io/k10/kio/kanister/function.CopyVolumeData.copyVolumeDataPodExecFunc.func2
                  linenumber: 384
                  message: Failed to create and upload backup
                file: kasten.io/k10/kio/kanister/function/kio_copy_volume_data.go:166
                function: kasten.io/k10/kio/kanister/function.CopyVolumeData
                linenumber: 166
                message: Failed to execute copy volume data pod function
              file: kasten.io/k10/kio/exec/internal/snapshotconverters/ac_gvc_converter.go:249
              function: kasten.io/k10/kio/exec/internal/snapshotconverters.(*GVCConverterInternalAPIImpl).genericVolumeCopy
              linenumber: 249
              message: failed running copyVolumeData
            file: kasten.io/k10/kio/exec/internal/snapshotconverters/ac_gvc_converter.go:170
            function: kasten.io/k10/kio/exec/internal/snapshotconverters.(*GVCConverterInternalAPIImpl).CopySnapshotRestoredInPVC
            linenumber: 170
            message: failed running genericVolumeCopy
          file: kasten.io/k10/kio/exec/internal/snapshotconverters/ac_gvc_converter.go:77
          function: kasten.io/k10/kio/exec/internal/snapshotconverters.(*GVCConverter).Convert
          linenumber: 77
          message: Error creating portable snapshot
        fields:
          - name: type
            value: CSI
          - name: id
            value: k10-csi-snap-n5ptp2zcwq58t7sr
        file: kasten.io/k10/kio/exec/phases/phase/artifactcopier.go:544
        function: kasten.io/k10/kio/exec/phases/phase.(*ArtifactCopier).convertSnapshots.func1
        linenumber: 544
        message: Failed to export snapshot data
      file: kasten.io/k10/kio/exec/phases/phase/artifactcopier.go:274
      function: kasten.io/k10/kio/exec/phases/phase.(*ArtifactCopier).Copy
      linenumber: 274
      message: Error converting snapshots
    file: kasten.io/k10/kio/exec/phases/phase/export.go:172
    function: kasten.io/k10/kio/exec/phases/phase.(*exportRestorePointPhase).Run
    linenumber: 172
    message: Failed to copy artifacts
  message: Job failed to be executed
- cause:
    cause:
      cause:
        cause:
          cause:
            cause:
              cause:
                cause:
                  cause:
                    cause:
                      message: "command terminated with exit code 1.

                        stdout:\ 

                        stderr: \e[31mERROR\e[0m upload error: permission
                        denied"
                    file: github.com/kanisterio/kanister@v0.0.0-20240920021913-d207c416a800/pkg/kube/exec.go
                    function: github.com/kanisterio/kanister/pkg/kube.ExecWithOptions
                    linenumber: 156
                    message: Failed to exec command in pod
                  file: kasten.io/k10/kio/kanister/function/kio_copy_volume_data.go:384
                  function: kasten.io/k10/kio/kanister/function.CopyVolumeData.copyVolumeDataPodExecFunc.func2
                  linenumber: 384
                  message: Failed to create and upload backup
                file: kasten.io/k10/kio/kanister/function/kio_copy_volume_data.go:166
                function: kasten.io/k10/kio/kanister/function.CopyVolumeData
                linenumber: 166
                message: Failed to execute copy volume data pod function
              file: kasten.io/k10/kio/exec/internal/snapshotconverters/ac_gvc_converter.go:249
              function: kasten.io/k10/kio/exec/internal/snapshotconverters.(*GVCConverterInternalAPIImpl).genericVolumeCopy
              linenumber: 249
              message: failed running copyVolumeData
            file: kasten.io/k10/kio/exec/internal/snapshotconverters/ac_gvc_converter.go:170
            function: kasten.io/k10/kio/exec/internal/snapshotconverters.(*GVCConverterInternalAPIImpl).CopySnapshotRestoredInPVC
            linenumber: 170
            message: failed running genericVolumeCopy
          file: kasten.io/k10/kio/exec/internal/snapshotconverters/ac_gvc_converter.go:77
          function: kasten.io/k10/kio/exec/internal/snapshotconverters.(*GVCConverter).Convert
          linenumber: 77
          message: Error creating portable snapshot
        fields:
          - name: type
            value: CSI
          - name: id
            value: k10-csi-snap-n5ptp2zcwq58t7sr
        file: kasten.io/k10/kio/exec/phases/phase/artifactcopier.go:544
        function: kasten.io/k10/kio/exec/phases/phase.(*ArtifactCopier).convertSnapshots.func1
        linenumber: 544
        message: Failed to export snapshot data
      file: kasten.io/k10/kio/exec/phases/phase/artifactcopier.go:274
      function: kasten.io/k10/kio/exec/phases/phase.(*ArtifactCopier).Copy
      linenumber: 274
      message: Error converting snapshots
    file: kasten.io/k10/kio/exec/phases/phase/export.go:172
    function: kasten.io/k10/kio/exec/phases/phase.(*exportRestorePointPhase).Run
    linenumber: 172
    message: Failed to copy artifacts
  message: Job failed to be executed
- cause:
    cause:
      cause:
        cause:
          cause:
            cause:
              cause:
                cause:
                  cause:
                    cause:
                      message: "command terminated with exit code 1.

                        stdout:\ 

                        stderr: \e[31mERROR\e[0m upload error: permission
                        denied"
                    file: github.com/kanisterio/kanister@v0.0.0-20240920021913-d207c416a800/pkg/kube/exec.go
                    function: github.com/kanisterio/kanister/pkg/kube.ExecWithOptions
                    linenumber: 156
                    message: Failed to exec command in pod
                  file: kasten.io/k10/kio/kanister/function/kio_copy_volume_data.go:384
                  function: kasten.io/k10/kio/kanister/function.CopyVolumeData.copyVolumeDataPodExecFunc.func2
                  linenumber: 384
                  message: Failed to create and upload backup
                file: kasten.io/k10/kio/kanister/function/kio_copy_volume_data.go:166
                function: kasten.io/k10/kio/kanister/function.CopyVolumeData
                linenumber: 166
                message: Failed to execute copy volume data pod function
              file: kasten.io/k10/kio/exec/internal/snapshotconverters/ac_gvc_converter.go:249
              function: kasten.io/k10/kio/exec/internal/snapshotconverters.(*GVCConverterInternalAPIImpl).genericVolumeCopy
              linenumber: 249
              message: failed running copyVolumeData
            file: kasten.io/k10/kio/exec/internal/snapshotconverters/ac_gvc_converter.go:170
            function: kasten.io/k10/kio/exec/internal/snapshotconverters.(*GVCConverterInternalAPIImpl).CopySnapshotRestoredInPVC
            linenumber: 170
            message: failed running genericVolumeCopy
          file: kasten.io/k10/kio/exec/internal/snapshotconverters/ac_gvc_converter.go:77
          function: kasten.io/k10/kio/exec/internal/snapshotconverters.(*GVCConverter).Convert
          linenumber: 77
          message: Error creating portable snapshot
        fields:
          - name: type
            value: CSI
          - name: id
            value: k10-csi-snap-n5ptp2zcwq58t7sr
        file: kasten.io/k10/kio/exec/phases/phase/artifactcopier.go:544
        function: kasten.io/k10/kio/exec/phases/phase.(*ArtifactCopier).convertSnapshots.func1
        linenumber: 544
        message: Failed to export snapshot data
      file: kasten.io/k10/kio/exec/phases/phase/artifactcopier.go:274
      function: kasten.io/k10/kio/exec/phases/phase.(*ArtifactCopier).Copy
      linenumber: 274
      message: Error converting snapshots
    file: kasten.io/k10/kio/exec/phases/phase/export.go:172
    function: kasten.io/k10/kio/exec/phases/phase.(*exportRestorePointPhase).Run
    linenumber: 172
    message: Failed to copy artifacts
  message: Job failed to be executed
 

Hello @FRubens ! Yes, I did the shallow read-only setup in version 6.5.11 and it was working before upgrade

Hello @kanika.mahajan,

Thank you for the information.

Please have you already raised a support case for this issue ?

If not please I would like to ask you to raise a case attaching the debug logs, this way we can investigate further to understand the issue.

One more question, Do you setup custom SeLinuxOptions in pods/namespaces ?

Regards

Rubens

Yes @FRubens I already raised the support case and had troubleshooting session with k10 team.

No, I am on OKD(OPENSHIFT) which uses default values from security context constraints.

Thank you @kanika.mahajan .

Will proceed with the investigation in the support case and provide updates there as soon as possible.

Regards

Rubens

Hello,
I am experiencing exactly the same problem and, despite having opened a support ticket almost two weeks ago, I am still at the same point.

If you have found a solution, would you be able to share it, please? 🙏🙏🙏

Here is my working environment:
  - Kasten: v8.0.11
  - OpenShift: v4.16.30

Regards,
Samuel

Well... since Kasten support didn't help me, and having bet on an OpenShift version upgrade that led nowhere, I ended up finding the root cause of the problem by myself, and I'm sharing it with you because I think it will definitely help some people:
 

Root cause:
The failure occurs when the protected application's pods run under the privileged SCC (or any SCC that removes MCS SELinux confinement). In that case, the data written to the CephFS PVC carries no SELinux MCS category, while the Kasten data-mover pod (copy-vol-data-*) runs confined in container_t with the MCS level of the source namespace (SELinuxLevel preservation, always enabled since K10 7.0.8). A confined process cannot read files labeled without its own MCS categories → the export fails with ERROR upload error: permission denied, even though the data-mover runs as root (POSIX permissions are irrelevant here).
This is typically the case with Harbor (my use case): its Helm chart runs containers as uid/gid 10000, and on OpenShift the usual way to make it deployable is to bind the harbor ServiceAccount to the privileged SCC. That RoleBinding is what breaks shallow read-only exports.

Clean fix (for my use case: harbor deployment):

Keep the application confined instead of privileged, so its files carry the namespace MCS level — which the data-mover then inherits:

1. Remove the privileged SCC RoleBinding for the application ServiceAccount.
2. Explicitly set the namespace SCC annotations so uid 10000 is allowed while staying confined under restricted-v2:
   ```
   apiVersion: v1
   kind: Namespace
   metadata:
     name: harbor
     annotations:
       openshift.io/sa.scc.uid-range: 10000/10000
       openshift.io/sa.scc.supplemental-groups: 10000/10000
       openshift.io/sa.scc.mcs: s0:c900,c901   # reuse the existing namespace MCS
   ```
3. Restart the workloads; verify with cat /proc/1/attr/current in the application pod (should show container_t:s0:cXX,cYY) and that the app can read/write its data.
4. Re-run the policy: the export phase now succeeds.

Notes

• When creating a new namespace with custom SCC annotations, all three must be set explicitly, otherwise pod admission fails with unable to find annotation openshift.io/sa.scc.mcs.
• !!! WARNING !!! Existing data written while the app was unconfined may be unreadable to the newly confined app; a temporary setsebool container_use_cephfs on on the nodes can serve as a transition window (or as a global workaround, at the cost of relaxing SELinux category enforcement on CephFS mounts).

⚠️ Warning: don't use namespace-level annotations (openshift.io/sa.scc.uid-range / sa.scc.supplemental-groups) as a workaround to allow a specific UID — it's not the right fix for mixed namespaces where multiple UIDs are in use (e.g. Harbor: uid 10000 for chart components, 999 for redis, plus Crunchy/Dex pods relying on allocated UIDs). It would either force the wrong UID on pods without an explicit runAsUser (breaking PVC ownership) or reject pods whose UID falls outside the custom range. Use per-UID custom SCCs bound to dedicated ServiceAccounts instead.