Skip to main content
Solved

Longhorn Generic Backup Snapshot "failed to connect to backup repository"

S

Hello everyone!

I’m currently evaluating k10 and having trouble with the generic backup on Longhorn volumes.

Pre-flight check returns no errors and I have the same setup in a test cluster which works fine, I’m not sure what the difference is.

The Deployment has the the kanister sidecar Pod and the necessary labels and the annotation “

k10.kasten.io/forcegenericbackuptrue:true” ist set.

Error Message:

1'{"message":"Failed to backup data","function":"kasten.io/k10/kio/kanister/function.(*backupDataFunc).Exec","linenumber":108,"file":"kasten.io/k10/kio/kanister/function/backup_data.go:108","cause":{"message":"[\"{\\\"message\\\":\\\"Failed to create the backup repository\\\",\\\"function\\\":\\\"kasten.io/k10/kio/kopia.CreateKopiaRepository\\\",\\\"linenumber\\\":470,\\\"file\\\":\\\"kasten.io/k10/kio/kopia/repository.go:470\\\",\\\"cause\\\":{\\\"message\\\":\\\"Failed to exec command in pod: command terminated with exit code 1\\\"}}\",\"{\\\"message\\\":\\\"Failed to connect to the backup repository\\\",\\\"function\\\":\\\"kasten.io/k10/kio/kopia.ConnectToKopiaRepository\\\",\\\"linenumber\\\":558,\\\"file\\\":\\\"kasten.io/k10/kio/kopia/repository.go:558\\\",\\\"cause\\\":{\\\"message\\\":\\\"repository not found\\\",\\\"cause\\\":{\\\"message\\\":\\\"Failed to exec command in pod: command terminated with exit code 1\\\"}}}\"]"}}'

The location profile returns a valid status and the user has read/write permissions on the bucket. The k10 DR Policy backups up to the same location and does not run into any errors, so I’m unsure why this backup policy wouldn’t be able to connect.

The location profile in Advanced Settings is set.

I backup to the same location from my test cluster and haven’t run into any issues. I also tried backup up to a different location (Wasabi cloud) which results in the same error.

Interestingly before that I got a different error, complaing about a missing Kanister Profile:

1- cause:
2    cause:
3      cause:
4        cause:
5          cause:
6            cause:
7              cause:
8                message: profiles.config.kio.kasten.io "kanister-profile" not found
9              fields:
10                - name: profileName
11                  value: kanister-profile
12              file: kasten.io/k10/kio/api/profiles/clients/client.go:242
13              function: kasten.io/k10/kio/api/profiles/clients.Get
14              linenumber: 242
15              message: Failed to get profile
16            fields:
17              - name: profile
18                value: kanister-profile
19            file: kasten.io/k10/kio/api/profiles/clients/client.go:130
20            function: kasten.io/k10/kio/api/profiles/clients.fetchDefaultK10KanisterProfile
21            linenumber: 130
22            message: Could not get default kanister profile
23          file: kasten.io/k10/kio/api/profiles/clients/client.go:100
24          function: kasten.io/k10/kio/api/profiles/clients.FetchK10KanisterProfileAndKanCR
25          linenumber: 100
26          message: No Kanister profile found
27        file: kasten.io/k10/kio/kanister/kanister_profile.go:203
28        function: kasten.io/k10/kio/kanister.GetK10CRAndKanisterProfile
29        linenumber: 203
30        message: Could not fetch a location profile. Location profile must be specified
31          in action or policy parameters
32      file: kasten.io/k10/kio/exec/phases/phase/snapshot.go:96
33      function: kasten.io/k10/kio/exec/phases/phase.(*SnapshotSession).PrimeSnapshotSession
34      linenumber: 96
35      message: Failed to prepare kopia backup session.
36    file: kasten.io/k10/kio/exec/phases/backup/snapshot_data_phase.go:129
37    function: kasten.io/k10/kio/exec/phases/backup.(*SnapshotDataPhase).Run
38    linenumber: 129
39    message: Failed to prime the snapshot session
40  message: Job failed to be executed
41- cause:
42    cause:
43      cause:
44        cause:
45          cause:
46            cause:
47              cause:
48                message: profiles.config.kio.kasten.io "kanister-profile" not found
49              fields:
50                - name: profileName
51                  value: kanister-profile
52              file: kasten.io/k10/kio/api/profiles/clients/client.go:242
53              function: kasten.io/k10/kio/api/profiles/clients.Get
54              linenumber: 242
55              message: Failed to get profile
56            fields:
57              - name: profile
58                value: kanister-profile
59            file: kasten.io/k10/kio/api/profiles/clients/client.go:130
60            function: kasten.io/k10/kio/api/profiles/clients.fetchDefaultK10KanisterProfile
61            linenumber: 130
62            message: Could not get default kanister profile
63          file: kasten.io/k10/kio/api/profiles/clients/client.go:100
64          function: kasten.io/k10/kio/api/profiles/clients.FetchK10KanisterProfileAndKanCR
65          linenumber: 100
66          message: No Kanister profile found
67        file: kasten.io/k10/kio/kanister/kanister_profile.go:203
68        function: kasten.io/k10/kio/kanister.GetK10CRAndKanisterProfile
69        linenumber: 203
70        message: Could not fetch a location profile. Location profile must be specified
71          in action or policy parameters
72      file: kasten.io/k10/kio/exec/phases/phase/snapshot.go:96
73      function: kasten.io/k10/kio/exec/phases/phase.(*SnapshotSession).PrimeSnapshotSession
74      linenumber: 96
75      message: Failed to prepare kopia backup session.
76    file: kasten.io/k10/kio/exec/phases/backup/snapshot_data_phase.go:129
77    function: kasten.io/k10/kio/exec/phases/backup.(*SnapshotDataPhase).Run
78    linenumber: 129
79    message: Failed to prime the snapshot session
80  message: Job failed to be executed
81- cause:
82    cause:
83      cause:
84        cause:
85          cause:
86            cause:
87              cause:
88                message: profiles.config.kio.kasten.io "kanister-profile" not found
89              fields:
90                - name: profileName
91                  value: kanister-profile
92              file: kasten.io/k10/kio/api/profiles/clients/client.go:242
93              function: kasten.io/k10/kio/api/profiles/clients.Get
94              linenumber: 242
95              message: Failed to get profile
96            fields:
97              - name: profile
98                value: kanister-profile
99            file: kasten.io/k10/kio/api/profiles/clients/client.go:130
100            function: kasten.io/k10/kio/api/profiles/clients.fetchDefaultK10KanisterProfile
101            linenumber: 130
102            message: Could not get default kanister profile
103          file: kasten.io/k10/kio/api/profiles/clients/client.go:100
104          function: kasten.io/k10/kio/api/profiles/clients.FetchK10KanisterProfileAndKanCR
105          linenumber: 100
106          message: No Kanister profile found
107        file: kasten.io/k10/kio/kanister/kanister_profile.go:203
108        function: kasten.io/k10/kio/kanister.GetK10CRAndKanisterProfile
109        linenumber: 203
110        message: Could not fetch a location profile. Location profile must be specified
111          in action or policy parameters
112      file: kasten.io/k10/kio/exec/phases/phase/snapshot.go:96
113      function: kasten.io/k10/kio/exec/phases/phase.(*SnapshotSession).PrimeSnapshotSession
114      linenumber: 96
115      message: Failed to prepare kopia backup session.
116    file: kasten.io/k10/kio/exec/phases/backup/snapshot_data_phase.go:129
117    function: kasten.io/k10/kio/exec/phases/backup.(*SnapshotDataPhase).Run
118    linenumber: 129
119    message: Failed to prime the snapshot session
120  message: Job failed to be executed
121

Altough I fixed that by manually setting backupParamters and a profile in the Policy yaml, which was missing despite having set it in the UI. The generic-volume Blueprint also exists.

 

This is a fresh k10 install. It has never been installed in the cluster before.

Kubernetes 1.22.10 running on RKE 1.3.11

Installed with Helm 3.9.

Does anybody have any ideas what I’m doing wrong?

Best answer by jaiganeshjk

Glad that you were able to find the issue. We are trying to enhance this and are tracking this internally.

However, for now, kanister-sidecar uses the same security-context as the application container. You might have to manually remove the read-only root filesystem SecurityContext from the kanister-sidecar or patch the kanister-sidecar to add an emptyDir volume for the `/tmp` path. This way, K10 can write to the `/tmp` directory.

View original
    Chris.Childerhose
    1 person likes this
    • Chris.Childerhose
      Chris.Childerhose
    Did this topic help you find an answer to your question?

    4 comments

    jaiganeshjk
    Forum|alt.badge.img+2
    • jaiganeshjk
    • Experienced User
    • 284 comments
    • August 5, 2022

    Hi @schnapsidee , Thank you for posting this question.

    I see the error message Failed to exec command in pod: command terminated with exit code 1   in the first failure details.

    You would be able to find more details on why this exec failed in the kanister-svc pod logs.

    Would you be able to spot the error message in the kanister logs or post the logs so that I can have a look into it ?

    Second error message usually comes when you don’t have a location profile specified under Location Profile for Kanister Action. I couldn’t find any reason why setting this in the U, will still miss it in the YAML under backupParameters. I will try to see if I can reproduce the same in my test env.

     

    Jaiganesh
    Chris.Childerhose
    1 person likes this
    • Chris.Childerhose
      Chris.Childerhose
    S
    • schnapsidee
    • Author
    • Not a newbie anymore
    • 3 comments
    • August 5, 2022

    Hi, @jaiganeshjk , thanks for your help.

    The kanister-svc logs did indeed help. It turns out that the primary container had read-only filesystem set in it’s SecurityContext and as the kanister sidecar uses that SecurityContext when deploying via injector, it also had a read-only filesystem and could therefore not copy the data.

    Is there a way to set a seperate sidecar container SecurityContext with annotations and/or labels with the injector or do I have to edit deployments with read-only filesystems manually to deploy a sidecar with a different SecurityContext? I wasn’t able to find anything in the docs about this.

     

    As for the second error, I wasn’t able to reproduce it myself, so it might have just been a mistake on my part. Maybe I missclicked in the UI 😬

    Chris.Childerhose
    1 person likes this
    • Chris.Childerhose
      Chris.Childerhose
    jaiganeshjk
    Forum|alt.badge.img+2
    • jaiganeshjk
    • Experienced User
    • 284 comments
    • Answer
    • August 5, 2022

    Glad that you were able to find the issue. We are trying to enhance this and are tracking this internally.

    However, for now, kanister-sidecar uses the same security-context as the application container. You might have to manually remove the read-only root filesystem SecurityContext from the kanister-sidecar or patch the kanister-sidecar to add an emptyDir volume for the `/tmp` path. This way, K10 can write to the `/tmp` directory.

    Jaiganesh
    Geoff Burke
    1 person likes this
    • Geoff Burke
      Geoff Burke
    S
    • schnapsidee
    • Author
    • Not a newbie anymore
    • 3 comments
    • August 5, 2022

    Alright, thank you. It would be nice to be able to set this with the injector since a lot of my deployments are Helm Charts that I didn’t create myself and they often don’t allow you to just deploy a second container with a different securitycontext.

     

    Thanks four help, though!

    Terms & ConditionsAccessibility statement