Skip to main content
Solved

Longhorn Generic Backup Snapshot "failed to connect to backup repository"

S

Hello everyone!

I’m currently evaluating k10 and having trouble with the generic backup on Longhorn volumes.

Pre-flight check returns no errors and I have the same setup in a test cluster which works fine, I’m not sure what the difference is.

The Deployment has the the kanister sidecar Pod and the necessary labels and the annotation “

k10.kasten.io/forcegenericbackuptrue:true” ist set.

Error Message:

'{"message":"Failed to backup data","function":"kasten.io/k10/kio/kanister/function.(*backupDataFunc).Exec","linenumber":108,"file":"kasten.io/k10/kio/kanister/function/backup_data.go:108","cause":{"message":"[\"{\\\"message\\\":\\\"Failed to create the backup repository\\\",\\\"function\\\":\\\"kasten.io/k10/kio/kopia.CreateKopiaRepository\\\",\\\"linenumber\\\":470,\\\"file\\\":\\\"kasten.io/k10/kio/kopia/repository.go:470\\\",\\\"cause\\\":{\\\"message\\\":\\\"Failed to exec command in pod: command terminated with exit code 1\\\"}}\",\"{\\\"message\\\":\\\"Failed to connect to the backup repository\\\",\\\"function\\\":\\\"kasten.io/k10/kio/kopia.ConnectToKopiaRepository\\\",\\\"linenumber\\\":558,\\\"file\\\":\\\"kasten.io/k10/kio/kopia/repository.go:558\\\",\\\"cause\\\":{\\\"message\\\":\\\"repository not found\\\",\\\"cause\\\":{\\\"message\\\":\\\"Failed to exec command in pod: command terminated with exit code 1\\\"}}}\"]"}}'

The location profile returns a valid status and the user has read/write permissions on the bucket. The k10 DR Policy backups up to the same location and does not run into any errors, so I’m unsure why this backup policy wouldn’t be able to connect.

The location profile in Advanced Settings is set.

I backup to the same location from my test cluster and haven’t run into any issues. I also tried backup up to a different location (Wasabi cloud) which results in the same error.

Interestingly before that I got a different error, complaing about a missing Kanister Profile:

- cause:
    cause:
      cause:
        cause:
          cause:
            cause:
              cause:
                message: profiles.config.kio.kasten.io "kanister-profile" not found
              fields:
                - name: profileName
                  value: kanister-profile
              file: kasten.io/k10/kio/api/profiles/clients/client.go:242
              function: kasten.io/k10/kio/api/profiles/clients.Get
              linenumber: 242
              message: Failed to get profile
            fields:
              - name: profile
                value: kanister-profile
            file: kasten.io/k10/kio/api/profiles/clients/client.go:130
            function: kasten.io/k10/kio/api/profiles/clients.fetchDefaultK10KanisterProfile
            linenumber: 130
            message: Could not get default kanister profile
          file: kasten.io/k10/kio/api/profiles/clients/client.go:100
          function: kasten.io/k10/kio/api/profiles/clients.FetchK10KanisterProfileAndKanCR
          linenumber: 100
          message: No Kanister profile found
        file: kasten.io/k10/kio/kanister/kanister_profile.go:203
        function: kasten.io/k10/kio/kanister.GetK10CRAndKanisterProfile
        linenumber: 203
        message: Could not fetch a location profile. Location profile must be specified
          in action or policy parameters
      file: kasten.io/k10/kio/exec/phases/phase/snapshot.go:96
      function: kasten.io/k10/kio/exec/phases/phase.(*SnapshotSession).PrimeSnapshotSession
      linenumber: 96
      message: Failed to prepare kopia backup session.
    file: kasten.io/k10/kio/exec/phases/backup/snapshot_data_phase.go:129
    function: kasten.io/k10/kio/exec/phases/backup.(*SnapshotDataPhase).Run
    linenumber: 129
    message: Failed to prime the snapshot session
  message: Job failed to be executed
- cause:
    cause:
      cause:
        cause:
          cause:
            cause:
              cause:
                message: profiles.config.kio.kasten.io "kanister-profile" not found
              fields:
                - name: profileName
                  value: kanister-profile
              file: kasten.io/k10/kio/api/profiles/clients/client.go:242
              function: kasten.io/k10/kio/api/profiles/clients.Get
              linenumber: 242
              message: Failed to get profile
            fields:
              - name: profile
                value: kanister-profile
            file: kasten.io/k10/kio/api/profiles/clients/client.go:130
            function: kasten.io/k10/kio/api/profiles/clients.fetchDefaultK10KanisterProfile
            linenumber: 130
            message: Could not get default kanister profile
          file: kasten.io/k10/kio/api/profiles/clients/client.go:100
          function: kasten.io/k10/kio/api/profiles/clients.FetchK10KanisterProfileAndKanCR
          linenumber: 100
          message: No Kanister profile found
        file: kasten.io/k10/kio/kanister/kanister_profile.go:203
        function: kasten.io/k10/kio/kanister.GetK10CRAndKanisterProfile
        linenumber: 203
        message: Could not fetch a location profile. Location profile must be specified
          in action or policy parameters
      file: kasten.io/k10/kio/exec/phases/phase/snapshot.go:96
      function: kasten.io/k10/kio/exec/phases/phase.(*SnapshotSession).PrimeSnapshotSession
      linenumber: 96
      message: Failed to prepare kopia backup session.
    file: kasten.io/k10/kio/exec/phases/backup/snapshot_data_phase.go:129
    function: kasten.io/k10/kio/exec/phases/backup.(*SnapshotDataPhase).Run
    linenumber: 129
    message: Failed to prime the snapshot session
  message: Job failed to be executed
- cause:
    cause:
      cause:
        cause:
          cause:
            cause:
              cause:
                message: profiles.config.kio.kasten.io "kanister-profile" not found
              fields:
                - name: profileName
                  value: kanister-profile
              file: kasten.io/k10/kio/api/profiles/clients/client.go:242
              function: kasten.io/k10/kio/api/profiles/clients.Get
              linenumber: 242
              message: Failed to get profile
            fields:
              - name: profile
                value: kanister-profile
            file: kasten.io/k10/kio/api/profiles/clients/client.go:130
            function: kasten.io/k10/kio/api/profiles/clients.fetchDefaultK10KanisterProfile
            linenumber: 130
            message: Could not get default kanister profile
          file: kasten.io/k10/kio/api/profiles/clients/client.go:100
          function: kasten.io/k10/kio/api/profiles/clients.FetchK10KanisterProfileAndKanCR
          linenumber: 100
          message: No Kanister profile found
        file: kasten.io/k10/kio/kanister/kanister_profile.go:203
        function: kasten.io/k10/kio/kanister.GetK10CRAndKanisterProfile
        linenumber: 203
        message: Could not fetch a location profile. Location profile must be specified
          in action or policy parameters
      file: kasten.io/k10/kio/exec/phases/phase/snapshot.go:96
      function: kasten.io/k10/kio/exec/phases/phase.(*SnapshotSession).PrimeSnapshotSession
      linenumber: 96
      message: Failed to prepare kopia backup session.
    file: kasten.io/k10/kio/exec/phases/backup/snapshot_data_phase.go:129
    function: kasten.io/k10/kio/exec/phases/backup.(*SnapshotDataPhase).Run
    linenumber: 129
    message: Failed to prime the snapshot session
  message: Job failed to be executed

Altough I fixed that by manually setting backupParamters and a profile in the Policy yaml, which was missing despite having set it in the UI. The generic-volume Blueprint also exists.

 

This is a fresh k10 install. It has never been installed in the cluster before.

Kubernetes 1.22.10 running on RKE 1.3.11

Installed with Helm 3.9.

Does anybody have any ideas what I’m doing wrong?

Best answer by jaiganeshjk

Glad that you were able to find the issue. We are trying to enhance this and are tracking this internally.

However, for now, kanister-sidecar uses the same security-context as the application container. You might have to manually remove the read-only root filesystem SecurityContext from the kanister-sidecar or patch the kanister-sidecar to add an emptyDir volume for the `/tmp` path. This way, K10 can write to the `/tmp` directory.

View original
    Did this topic help you find an answer to your question?

    4 comments

    jaiganeshjk
    Forum|alt.badge.img+2
    • jaiganeshjk
    • Experienced User
    • 275 comments
    • August 5, 2022

    Hi @schnapsidee , Thank you for posting this question.

    I see the error message Failed to exec command in pod: command terminated with exit code 1   in the first failure details.

    You would be able to find more details on why this exec failed in the kanister-svc pod logs.

    Would you be able to spot the error message in the kanister logs or post the logs so that I can have a look into it ?

    Second error message usually comes when you don’t have a location profile specified under Location Profile for Kanister Action. I couldn’t find any reason why setting this in the U, will still miss it in the YAML under backupParameters. I will try to see if I can reproduce the same in my test env.

     

    Jaiganesh
    Chris.Childerhose
    1 person likes this
    • Chris.Childerhose
      Chris.Childerhose
    S
    • schnapsidee
    • Author
    • Not a newbie anymore
    • 3 comments
    • August 5, 2022

    Hi, @jaiganeshjk , thanks for your help.

    The kanister-svc logs did indeed help. It turns out that the primary container had read-only filesystem set in it’s SecurityContext and as the kanister sidecar uses that SecurityContext when deploying via injector, it also had a read-only filesystem and could therefore not copy the data.

    Is there a way to set a seperate sidecar container SecurityContext with annotations and/or labels with the injector or do I have to edit deployments with read-only filesystems manually to deploy a sidecar with a different SecurityContext? I wasn’t able to find anything in the docs about this.

     

    As for the second error, I wasn’t able to reproduce it myself, so it might have just been a mistake on my part. Maybe I missclicked in the UI 😬

    Chris.Childerhose
    1 person likes this
    • Chris.Childerhose
      Chris.Childerhose
    jaiganeshjk
    Forum|alt.badge.img+2
    • jaiganeshjk
    • Experienced User
    • 275 comments
    • Answer
    • August 5, 2022

    Glad that you were able to find the issue. We are trying to enhance this and are tracking this internally.

    However, for now, kanister-sidecar uses the same security-context as the application container. You might have to manually remove the read-only root filesystem SecurityContext from the kanister-sidecar or patch the kanister-sidecar to add an emptyDir volume for the `/tmp` path. This way, K10 can write to the `/tmp` directory.

    Jaiganesh
    Geoff Burke
    1 person likes this
    • Geoff Burke
      Geoff Burke
    S
    • schnapsidee
    • Author
    • Not a newbie anymore
    • 3 comments
    • August 5, 2022

    Alright, thank you. It would be nice to be able to set this with the injector since a lot of my deployments are Helm Charts that I didn’t create myself and they often don’t allow you to just deploy a second container with a different securitycontext.

     

    Thanks four help, though!

    Comment


    Terms & Conditions