Question

Kasten PostgreSQL Backups Fail with permission denied, why?

7 months ago
August 6, 2024
3 comments
96 views

aburt
Not a newbie anymore
7 comments

I am having Issue with my Cluster PostgreSQL backups failing and I don’t know the cause of the failure. I keep getting a Failure to exec command into pod with “permission denied”

apiVersion: cr.kanister.io/v1alpha1
kind: Blueprint
metadata:
  name: atlassian-postgres-backups
  namespace: kasten-io
backups
actions:
  backup:
    kind: StatefulSet
    outputArtifacts:
      pgBackup:
        kopiaSnapshot: '{{ .Phases.pgDump.Output.kopiaOutput }}'
    phases:
      - args:
          command:
            - bash
            - '-o'
            - errexit
            - '-o'
            - pipefail
            - '-c'
            - >
              export PGHOST='{{ index .Object.metadata.labels
              "app.kubernetes.io/instance" }}-postgresql.{{
              .StatefulSet.Namespace }}.svc.cluster.local'

              export PGUSER='postgres'

              export PGPASSWORD='{{ index .Phases.pgDump.Secrets.pgSecret.Data
              "postgresql-postgres-password" | toString }}'

              backup_file_path="backup.sql"

              pg_dumpall --clean -U $PGUSER | kando location push --profile '{{
              toJson .Profile }}' --path "${backup_file_path}" --output-name
              "kopiaOutput" -

              echo "stopping envoy proxy"

              curl -sf -XPOST http://127.0.0.1:15020/quitquitquit
          image: ghcr.io/kanisterio/postgres-kanister-tools:0.109.0
          namespace: '{{ .StatefulSet.Namespace }}'
        func: KubeTask
        name: pgDump
        objects:
          pgSecret:
            kind: Secret
            name: atlassian-postgresdb
            namespace: '{{ .StatefulSet.Namespace }}'
  delete:
    inputArtifactNames:
      - pgBackup
    phases:
      - args:
          command:
            - bash
            - '-o'
            - errexit
            - '-o'
            - pipefail
            - '-c'
            - >
              backup_file_path="backup.sql"

              kopia_snap='{{ .ArtifactsIn.pgBackup.KopiaSnapshot }}'

              kando location delete --profile '{{ toJson .Profile }}' --path
              "${backup_file_path}" --kopia-snapshot "${kopia_snap}"

              echo "stopping envoy proxy"

              curl -sf -XPOST http://127.0.0.1:15020/quitquitquit
          image: ghcr.io/kanisterio/postgres-kanister-tools:0.109.0
          namespace: '{{ .Namespace.Name }}'
        func: KubeTask
        name: deleteDump
  restore:
    inputArtifactNames:
      - pgBackup
    kind: StatefulSet
    phases:
      - args:
          command:
            - bash
            - '-o'
            - errexit
            - '-o'
            - pipefail
            - '-c'
            - >
              export PGHOST='{{ index .Object.metadata.labels
              "app.kubernetes.io/instance" }}-postgresql.{{
              .StatefulSet.Namespace }}.svc.cluster.local'

              export PGUSER='postgres'

              export PGPASSWORD='{{ index
              .Phases.pgRestore.Secrets.pgSecret.Data
              "postgresql-postgres-password" | toString }}'

              backup_file_path="backup.sql"

              kopia_snap='{{ .ArtifactsIn.pgBackup.KopiaSnapshot }}'

              kando location pull --profile '{{ toJson .Profile }}' --path
              "${backup_file_path}" --kopia-snapshot "${kopia_snap}" - | psql -q
              -U "${PGUSER}"

              echo "stopping envoy proxy"

              curl -sf -XPOST http://127.0.0.1:15020/quitquitquit
          image: ghcr.io/kanisterio/postgres-kanister-tools:0.109.0
          namespace: '{{ .StatefulSet.Namespace }}'
        func: KubeTask
        name: pgRestore
        objects:
          pgSecret:
            kind: Secret
            name: atlassian-postgresdb
            namespace: '{{ .StatefulSet.Namespace }}'

{"kind":"ExportAction","apiVersion":"actions.kio.kasten.io/v1alpha1","metadata":{"name":"scheduled-dh6t777j7f","namespace":"jira","uid":"c32b6da2-53fc-11ef-8ae5-be28dba17685","resourceVersion":"5632","creationTimestamp":"2024-08-06T14:04:14Z","labels":{"k10.kasten.io/appName":"jira","k10.kasten.io/appNamespace":"jira","k10.kasten.io/isMetadataExport":"true","k10.kasten.io/isRunNow":"true","k10.kasten.io/policyName":"jira","k10.kasten.io/policyNamespace":"kasten-io","k10.kasten.io/runActionName":"policy-run-hj9f2","k10.kasten.io/runActionNamespace":"kasten-io"}},"status":{"state":"Failed","startTime":"2024-08-06T14:04:14Z","endTime":"2024-08-06T14:13:06Z","restorePoint":{"name":""},"result":{"name":""},"error":{"cause":"{\"cause\":{\"cause\":{\"cause\":{\"cause\":{\"cause\":{\"cause\":{\"cause\":{\"cause\":{\"message\":\"Failed to exec command in pod: command terminated with exit code 1.\\nstdout: \\nstderr: \\u001b[31mERROR\\u001b[0m upload error: permission denied\"},\"file\":\"kasten.io/k10/kio/kanister/function/kio_copy_volume_data.go:366\",\"function\":\"kasten.io/k10/kio/kanister/function.CopyVolumeData.copyVolumeDataPodExecFunc.func2\",\"linenumber\":366,\"message\":\"Failed to create and upload backup\"},\"file\":\"kasten.io/k10/kio/kanister/function/kio_copy_volume_data.go:150\",\"function\":\"kasten.io/k10/kio/kanister/function.CopyVolumeData\",\"linenumber\":150,\"message\":\"Failed to execute copy volume data pod function\"},\"file\":\"kasten.io/k10/kio/exec/internal/snapshotconverters/ac_gvc_converter.go:247\",\"function\":\"kasten.io/k10/kio/exec/internal/snapshotconverters.(*GVCConverterInternalAPIImpl).genericVolumeCopy\",\"linenumber\":247,\"message\":\"failed running copyVolumeData\"},\"file\":\"kasten.io/k10/kio/exec/internal/snapshotconverters/ac_gvc_converter.go:170\",\"function\":\"kasten.io/k10/kio/exec/internal/snapshotconverters.(*GVCConverterInternalAPIImpl).CopySnapshotRestoredInPVC\",\"linenumber\":170,\"message\":\"failed running genericVolumeCopy\"},\"file\":\"kasten.io/k10/kio/exec/internal/snapshotconverters/ac_gvc_converter.go:77\",\"function\":\"kasten.io/k10/kio/exec/internal/snapshotconverters.(*GVCConverter).Convert\",\"linenumber\":77,\"message\":\"Error creating portable snapshot\"},\"fields\":[{\"name\":\"type\",\"value\":\"CSI\"},{\"name\":\"id\",\"value\":\"k10-csi-snap-47wlkf7stshckfqz\"}],\"file\":\"kasten.io/k10/kio/exec/phases/phase/artifactcopier.go:543\",\"function\":\"kasten.io/k10/kio/exec/phases/phase.(*ArtifactCopier).convertSnapshots.func1\",\"linenumber\":543,\"message\":\"Failed to export snapshot data\"},\"file\":\"kasten.io/k10/kio/exec/phases/phase/artifactcopier.go:273\",\"function\":\"kasten.io/k10/kio/exec/phases/phase.(*ArtifactCopier).Copy\",\"linenumber\":273,\"message\":\"Error converting snapshots\"},\"file\":\"kasten.io/k10/kio/exec/phases/phase/export.go:172\",\"function\":\"kasten.io/k10/kio/exec/phases/phase.(*exportRestorePointPhase).Run\",\"linenumber\":172,\"message\":\"Failed to copy artifacts\"}","message":"Job failed to be executed"},"actionDetails":{"phases":[{"attempt":3,"endTime":"2024-08-06T14:13:06Z","errors":[{"cause":"{\"cause\":{\"cause\":{\"cause\":{\"cause\":{\"cause\":{\"cause\":{\"cause\":{\"cause\":{\"message\":\"Failed to exec command in pod: command terminated with exit code 1.\\nstdout: \\nstderr: \\u001b[31mERROR\\u001b[0m upload error: permission denied\"},\"file\":\"kasten.io/k10/kio/kanister/function/kio_copy_volume_data.go:366\",\"function\":\"kasten.io/k10/kio/kanister/function.CopyVolumeData.copyVolumeDataPodExecFunc.func2\",\"linenumber\":366,\"message\":\"Failed to create and upload backup\"},\"file\":\"kasten.io/k10/kio/kanister/function/kio_copy_volume_data.go:150\",\"function\":\"kasten.io/k10/kio/kanister/function.CopyVolumeData\",\"linenumber\":150,\"message\":\"Failed to execute copy volume data pod function\"},\"file\":\"kasten.io/k10/kio/exec/internal/snapshotconverters/ac_gvc_converter.go:247\",\"function\":\"kasten.io/k10/kio/exec/internal/snapshotconverters.(*GVCConverterInternalAPIImpl).genericVolumeCopy\",\"linenumber\":247,\"message\":\"failed running copyVolumeData\"},\"file\":\"kasten.io/k10/kio/exec/internal/snapshotconverters/ac_gvc_converter.go:170\",\"function\":\"kasten.io/k10/kio/exec/internal/snapshotconverters.(*GVCConverterInternalAPIImpl).CopySnapshotRestoredInPVC\",\"linenumber\":170,\"message\":\"failed running genericVolumeCopy\"},\"file\":\"kasten.io/k10/kio/exec/internal/snapshotconverters/ac_gvc_converter.go:77\",\"function\":\"kasten.io/k10/kio/exec/internal/snapshotconverters.(*GVCConverter).Convert\",\"linenumber\":77,\"message\":\"Error creating portable snapshot\"},\"fields\":[{\"name\":\"type\",\"value\":\"CSI\"},{\"name\":\"id\",\"value\":\"k10-csi-snap-47wlkf7stshckfqz\"}],\"file\":\"kasten.io/k10/kio/exec/phases/phase/artifactcopier.go:543\",\"function\":\"kasten.io/k10/kio/exec/phases/phase.(*ArtifactCopier).convertSnapshots.func1\",\"linenumber\":543,\"message\":\"Failed to export snapshot data\"},\"file\":\"kasten.io/k10/kio/exec/phases/phase/artifactcopier.go:273\",\"function\":\"kasten.io/k10/kio/exec/phases/phase.(*ArtifactCopier).Copy\",\"linenumber\":273,\"message\":\"Error converting snapshots\"},\"file\":\"kasten.io/k10/kio/exec/phases/phase/export.go:172\",\"function\":\"kasten.io/k10/kio/exec/phases/phase.(*exportRestorePointPhase).Run\",\"linenumber\":172,\"message\":\"Failed to copy artifacts\"}","message":"Job failed to be executed"},{"cause":"{\"cause\":{\"cause\":{\"cause\":{\"cause\":{\"cause\":{\"cause\":{\"cause\":{\"cause\":{\"message\":\"Failed to exec command in pod: command terminated with exit code 1.\\nstdout: \\nstderr: \\u001b[31mERROR\\u001b[0m upload error: permission denied\"},\"file\":\"kasten.io/k10/kio/kanister/function/kio_copy_volume_data.go:366\",\"function\":\"kasten.io/k10/kio/kanister/function.CopyVolumeData.copyVolumeDataPodExecFunc.func2\",\"linenumber\":366,\"message\":\"Failed to create and upload backup\"},\"file\":\"kasten.io/k10/kio/kanister/function/kio_copy_volume_data.go:150\",\"function\":\"kasten.io/k10/kio/kanister/function.CopyVolumeData\",\"linenumber\":150,\"message\":\"Failed to execute copy volume data pod function\"},\"file\":\"kasten.io/k10/kio/exec/internal/snapshotconverters/ac_gvc_converter.go:247\",\"function\":\"kasten.io/k10/kio/exec/internal/snapshotconverters.(*GVCConverterInternalAPIImpl).genericVolumeCopy\",\"linenumber\":247,\"message\":\"failed running copyVolumeData\"},\"file\":\"kasten.io/k10/kio/exec/internal/snapshotconverters/ac_gvc_converter.go:170\",\"function\":\"kasten.io/k10/kio/exec/internal/snapshotconverters.(*GVCConverterInternalAPIImpl).CopySnapshotRestoredInPVC\",\"linenumber\":170,\"message\":\"failed running genericVolumeCopy\"},\"file\":\"kasten.io/k10/kio/exec/internal/snapshotconverters/ac_gvc_converter.go:77\",\"function\":\"kasten.io/k10/kio/exec/internal/snapshotconverters.(*GVCConverter).Convert\",\"linenumber\":77,\"message\":\"Error creating portable snapshot\"},\"fields\":[{\"name\":\"type\",\"value\":\"CSI\"},{\"name\":\"id\",\"value\":\"k10-csi-snap-47wlkf7stshckfqz\"}],\"file\":\"kasten.io/k10/kio/exec/phases/phase/artifactcopier.go:543\",\"function\":\"kasten.io/k10/kio/exec/phases/phase.(*ArtifactCopier).convertSnapshots.func1\",\"linenumber\":543,\"message\":\"Failed to export snapshot data\"},\"file\":\"kasten.io/k10/kio/exec/phases/phase/artifactcopier.go:273\",\"function\":\"kasten.io/k10/kio/exec/phases/phase.(*ArtifactCopier).Copy\",\"linenumber\":273,\"message\":\"Error converting snapshots\"},\"file\":\"kasten.io/k10/kio/exec/phases/phase/export.go:172\",\"function\":\"kasten.io/k10/kio/exec/phases/phase.(*exportRestorePointPhase).Run\",\"linenumber\":172,\"message\":\"Failed to copy artifacts\"}","message":"Job failed to be executed"},{"cause":"{\"cause\":{\"cause\":{\"cause\":{\"cause\":{\"cause\":{\"cause\":{\"cause\":{\"cause\":{\"message\":\"Failed to exec command in pod: command terminated with exit code 1.\\nstdout: \\nstderr: \\u001b[31mERROR\\u001b[0m upload error: permission denied\"},\"file\":\"kasten.io/k10/kio/kanister/function/kio_copy_volume_data.go:366\",\"function\":\"kasten.io/k10/kio/kanister/function.CopyVolumeData.copyVolumeDataPodExecFunc.func2\",\"linenumber\":366,\"message\":\"Failed to create and upload backup\"},\"file\":\"kasten.io/k10/kio/kanister/function/kio_copy_volume_data.go:150\",\"function\":\"kasten.io/k10/kio/kanister/function.CopyVolumeData\",\"linenumber\":150,\"message\":\"Failed to execute copy volume data pod function\"},\"file\":\"kasten.io/k10/kio/exec/internal/snapshotconverters/ac_gvc_converter.go:247\",\"function\":\"kasten.io/k10/kio/exec/internal/snapshotconverters.(*GVCConverterInternalAPIImpl).genericVolumeCopy\",\"linenumber\":247,\"message\":\"failed running copyVolumeData\"},\"file\":\"kasten.io/k10/kio/exec/internal/snapshotconverters/ac_gvc_converter.go:170\",\"function\":\"kasten.io/k10/kio/exec/internal/snapshotconverters.(*GVCConverterInternalAPIImpl).CopySnapshotRestoredInPVC\",\"linenumber\":170,\"message\":\"failed running genericVolumeCopy\"},\"file\":\"kasten.io/k10/kio/exec/internal/snapshotconverters/ac_gvc_converter.go:77\",\"function\":\"kasten.io/k10/kio/exec/internal/snapshotconverters.(*GVCConverter).Convert\",\"linenumber\":77,\"message\":\"Error creating portable snapshot\"},\"fields\":[{\"name\":\"type\",\"value\":\"CSI\"},{\"name\":\"id\",\"value\":\"k10-csi-snap-47wlkf7stshckfqz\"}],\"file\":\"kasten.io/k10/kio/exec/phases/phase/artifactcopier.go:543\",\"function\":\"kasten.io/k10/kio/exec/phases/phase.(*ArtifactCopier).convertSnapshots.func1\",\"linenumber\":543,\"message\":\"Failed to export snapshot data\"},\"file\":\"kasten.io/k10/kio/exec/phases/phase/artifactcopier.go:273\",\"function\":\"kasten.io/k10/kio/exec/phases/phase.(*ArtifactCopier).Copy\",\"linenumber\":273,\"message\":\"Error converting snapshots\"},\"file\":\"kasten.io/k10/kio/exec/phases/phase/export.go:172\",\"function\":\"kasten.io/k10/kio/exec/phases/phase.(*exportRestorePointPhase).Run\",\"linenumber\":172,\"message\":\"Failed to copy artifacts\"}","message":"Job failed to be executed"}],"name":"Exporting RestorePoint","startTime":"2024-08-06T14:04:14Z","state":"failed","updatedTime":"2024-08-06T14:13:06Z","volumeOperations":[{"namespace":"jira","pvcName":"jira-shared-home","operation":"Upload","dataFormat":"Filesystem","exportDirective":"FileSystemMode","driver":"GenericVolumeCopy","storageClass":"ceph-filesystem","storageType":"CSI","snapshotId":"k10-csi-snap-47wlkf7stshckfqz","volumeSnapshotClass":"csi-cephfsplugin-snapclass"},{"namespace":"jira","pvcName":"local-home-jira-0","operation":"Upload","dataFormat":"Filesystem","exportDirective":"FileSystemMode","driver":"GenericVolumeCopy","storageClass":"ceph-block","storageType":"CSI","snapshotId":"k10-csi-snap-ktnkdg6wdh9vghzx","volumeSnapshotClass":"csi-cephrdbplugin-snapclass"}]}]},"progress":100,"progressDetails":{"processedBytes":0,"readBytes":0,"totalBytes":107374182400,"transferredBytes":0,"processingRate":0,"totalVolumes":1,"completedVolumes":0,"updatedTime":"2024-08-06T14:13:06Z"}},"spec":{"subject":{"name":"jira","namespace":"jira"},"scheduledTime":"2024-08-06T14:01:35Z","frequency":"@onDemand","receiveString":"bIzAPpoanmFN4RFPrRhEjf1TbmQkE27FxxDV2XodTqs5F+wFrXjsXoOHRMmo5aw4BlXKzKuNGU+u7zELhTr9sDo4njgtcMDIR5Ofp9WU7M6F9a5C+yJzAAz1wa/JNJlebItekWx1hckA6LOQ7NyN6miVD97gyXluwHrwAXXP0NvX6LIzJ3OVs2/PiXuDDQ952b5XM9WhgDhm00o3MTiL9929XXSzHSjrpihH94zbaJdioYyeZS3AdosHtK2Qh965Mo7HaaGIcsFhEjSyoV4msXGV4/aSY/zckwOBAnR9BWt9GHOVvXY3unq0osvK8tt4yQAjJ9cKzTQEl8cv2GTcGvVN/bFVMl9PqJgmgpaqqpTjjgzDvcHHtzohgjCukzJ7vZy0MtsJj/X8QXMpndrxb0+5kvFLSbyC6gH6m98NMXxVkq9LdYHdJHMgRQDKn+i0Kw2U+fc6cvbnS0dL+Bw0B3u7jId/0KFvezrnO19XvoQHrx0Sm1rIVQpdInpSDJzklvq2dwTCgiU+9pAe01nsln1A6v15G72ceiFyZ9So1Wlhu5ZZPcXsTRdQJUmqKSaLTFFjqgdesJZO7G9suP1Pr4c9XtXEyJ9lqWXTc4O4P+U1pcqGgSINANslZ9gZsfDkeIW2qXiyZlwHGz9dLWcQEJQZ","profile":{"name":"sf-s3","namespace":"kasten-io"},"migrationToken":{"name":"jira-migration-token","namespace":"kasten-io"},"exportData":{"enabled":true,"overrides":[{"storageClassName":"ceph-filesystem","enabled":true,"exporterStorageClassName":"ceph-shallow-filesystem"}]},"expiresAt":"2024-08-30T13:30:25Z"}}

aburt
Author
Not a newbie anymore
7 comments
7 months ago
August 8, 2024

Additional details:

using rook ceph storage.

I have VolumeSnapshotClasses created for RBD and Filesystem rook-ceph classes, and validated via the GUI.

I also have a shallow filesystem copy of ceph-filesystem with “backingSnapshot” set to true.

My Policy is configured to with storageClass Override

                exportData:
                  enabled: true
                  overrides:
                    - storageClassName: ceph-filesystem
                      enabled: true
                      exporterStorageClassName: ceph-shallow-filesystem

The policy succeeds the Backup Action, However fails on the Export Action.

I have even tried disabling the exporterStorageClass Overrides and still have a failures due to “permission denied”

with no indication on what is being denied.

Is the Blueprint using the wrong password for the database dump?
Is the Blueprint using the wrong password when uploading the dump to S3 storage?
Is the VolumeSnapshotClass not using the right secrets?
-- they are setup with snapshotter-secret-name and namespace parameteres
Is kasten failing to authenticate to S3 Storage?

There is no indication in the logs behind what is failing/denying the permission to do something

aburt
Author
Not a newbie anymore
7 comments
7 months ago
August 8, 2024

Addtionally the Failure only Occurs on applications that use a persistent storage class for a PVC.
specifically ceph-filesystem.

But that is why I have th eceph-shallow-filesystem class and storageClass override for

kanika.mahajan
Comes here often
23 comments
6 months ago
September 9, 2024

Hello, I am also getting “permission denied” error during the export phase in the version 7.0.8

Comment

Related topics

Export Job fails (error: context deadline exceeded)icon

unable to restore cephfs pvcicon

Podman backups?

k10 7.0.13 exports are failingicon

Backup policy getting failed during export phaseicon

Sign up

Login to the community

Scanning file for viruses.

This file cannot be downloaded