Solved

Longhorn Generic Backup Snapshot "failed to connect to backup repository"


Userlevel 1

Hello everyone!

I’m currently evaluating k10 and having trouble with the generic backup on Longhorn volumes.

Pre-flight check returns no errors and I have the same setup in a test cluster which works fine, I’m not sure what the difference is.

The Deployment has the the kanister sidecar Pod and the necessary labels and the annotation “

k10.kasten.io/forcegenericbackuptrue:true” ist set.

Error Message:

'{"message":"Failed to backup data","function":"kasten.io/k10/kio/kanister/function.(*backupDataFunc).Exec","linenumber":108,"file":"kasten.io/k10/kio/kanister/function/backup_data.go:108","cause":{"message":"[\"{\\\"message\\\":\\\"Failed to create the backup repository\\\",\\\"function\\\":\\\"kasten.io/k10/kio/kopia.CreateKopiaRepository\\\",\\\"linenumber\\\":470,\\\"file\\\":\\\"kasten.io/k10/kio/kopia/repository.go:470\\\",\\\"cause\\\":{\\\"message\\\":\\\"Failed to exec command in pod: command terminated with exit code 1\\\"}}\",\"{\\\"message\\\":\\\"Failed to connect to the backup repository\\\",\\\"function\\\":\\\"kasten.io/k10/kio/kopia.ConnectToKopiaRepository\\\",\\\"linenumber\\\":558,\\\"file\\\":\\\"kasten.io/k10/kio/kopia/repository.go:558\\\",\\\"cause\\\":{\\\"message\\\":\\\"repository not found\\\",\\\"cause\\\":{\\\"message\\\":\\\"Failed to exec command in pod: command terminated with exit code 1\\\"}}}\"]"}}'

The location profile returns a valid status and the user has read/write permissions on the bucket. The k10 DR Policy backups up to the same location and does not run into any errors, so I’m unsure why this backup policy wouldn’t be able to connect.

The location profile in Advanced Settings is set.

I backup to the same location from my test cluster and haven’t run into any issues. I also tried backup up to a different location (Wasabi cloud) which results in the same error.

Interestingly before that I got a different error, complaing about a missing Kanister Profile:

- cause:
cause:
cause:
cause:
cause:
cause:
cause:
message: profiles.config.kio.kasten.io "kanister-profile" not found
fields:
- name: profileName
value: kanister-profile
file: kasten.io/k10/kio/api/profiles/clients/client.go:242
function: kasten.io/k10/kio/api/profiles/clients.Get
linenumber: 242
message: Failed to get profile
fields:
- name: profile
value: kanister-profile
file: kasten.io/k10/kio/api/profiles/clients/client.go:130
function: kasten.io/k10/kio/api/profiles/clients.fetchDefaultK10KanisterProfile
linenumber: 130
message: Could not get default kanister profile
file: kasten.io/k10/kio/api/profiles/clients/client.go:100
function: kasten.io/k10/kio/api/profiles/clients.FetchK10KanisterProfileAndKanCR
linenumber: 100
message: No Kanister profile found
file: kasten.io/k10/kio/kanister/kanister_profile.go:203
function: kasten.io/k10/kio/kanister.GetK10CRAndKanisterProfile
linenumber: 203
message: Could not fetch a location profile. Location profile must be specified
in action or policy parameters
file: kasten.io/k10/kio/exec/phases/phase/snapshot.go:96
function: kasten.io/k10/kio/exec/phases/phase.(*SnapshotSession).PrimeSnapshotSession
linenumber: 96
message: Failed to prepare kopia backup session.
file: kasten.io/k10/kio/exec/phases/backup/snapshot_data_phase.go:129
function: kasten.io/k10/kio/exec/phases/backup.(*SnapshotDataPhase).Run
linenumber: 129
message: Failed to prime the snapshot session
message: Job failed to be executed
- cause:
cause:
cause:
cause:
cause:
cause:
cause:
message: profiles.config.kio.kasten.io "kanister-profile" not found
fields:
- name: profileName
value: kanister-profile
file: kasten.io/k10/kio/api/profiles/clients/client.go:242
function: kasten.io/k10/kio/api/profiles/clients.Get
linenumber: 242
message: Failed to get profile
fields:
- name: profile
value: kanister-profile
file: kasten.io/k10/kio/api/profiles/clients/client.go:130
function: kasten.io/k10/kio/api/profiles/clients.fetchDefaultK10KanisterProfile
linenumber: 130
message: Could not get default kanister profile
file: kasten.io/k10/kio/api/profiles/clients/client.go:100
function: kasten.io/k10/kio/api/profiles/clients.FetchK10KanisterProfileAndKanCR
linenumber: 100
message: No Kanister profile found
file: kasten.io/k10/kio/kanister/kanister_profile.go:203
function: kasten.io/k10/kio/kanister.GetK10CRAndKanisterProfile
linenumber: 203
message: Could not fetch a location profile. Location profile must be specified
in action or policy parameters
file: kasten.io/k10/kio/exec/phases/phase/snapshot.go:96
function: kasten.io/k10/kio/exec/phases/phase.(*SnapshotSession).PrimeSnapshotSession
linenumber: 96
message: Failed to prepare kopia backup session.
file: kasten.io/k10/kio/exec/phases/backup/snapshot_data_phase.go:129
function: kasten.io/k10/kio/exec/phases/backup.(*SnapshotDataPhase).Run
linenumber: 129
message: Failed to prime the snapshot session
message: Job failed to be executed
- cause:
cause:
cause:
cause:
cause:
cause:
cause:
message: profiles.config.kio.kasten.io "kanister-profile" not found
fields:
- name: profileName
value: kanister-profile
file: kasten.io/k10/kio/api/profiles/clients/client.go:242
function: kasten.io/k10/kio/api/profiles/clients.Get
linenumber: 242
message: Failed to get profile
fields:
- name: profile
value: kanister-profile
file: kasten.io/k10/kio/api/profiles/clients/client.go:130
function: kasten.io/k10/kio/api/profiles/clients.fetchDefaultK10KanisterProfile
linenumber: 130
message: Could not get default kanister profile
file: kasten.io/k10/kio/api/profiles/clients/client.go:100
function: kasten.io/k10/kio/api/profiles/clients.FetchK10KanisterProfileAndKanCR
linenumber: 100
message: No Kanister profile found
file: kasten.io/k10/kio/kanister/kanister_profile.go:203
function: kasten.io/k10/kio/kanister.GetK10CRAndKanisterProfile
linenumber: 203
message: Could not fetch a location profile. Location profile must be specified
in action or policy parameters
file: kasten.io/k10/kio/exec/phases/phase/snapshot.go:96
function: kasten.io/k10/kio/exec/phases/phase.(*SnapshotSession).PrimeSnapshotSession
linenumber: 96
message: Failed to prepare kopia backup session.
file: kasten.io/k10/kio/exec/phases/backup/snapshot_data_phase.go:129
function: kasten.io/k10/kio/exec/phases/backup.(*SnapshotDataPhase).Run
linenumber: 129
message: Failed to prime the snapshot session
message: Job failed to be executed

Altough I fixed that by manually setting backupParamters and a profile in the Policy yaml, which was missing despite having set it in the UI. The generic-volume Blueprint also exists.

 

This is a fresh k10 install. It has never been installed in the cluster before.

Kubernetes 1.22.10 running on RKE 1.3.11

Installed with Helm 3.9.

Does anybody have any ideas what I’m doing wrong?

icon

Best answer by jaiganeshjk 5 August 2022, 15:49

View original

4 comments

Userlevel 6
Badge +2

Hi @schnapsidee , Thank you for posting this question.

I see the error message Failed to exec command in pod: command terminated with exit code 1   in the first failure details.

You would be able to find more details on why this exec failed in the kanister-svc pod logs.

Would you be able to spot the error message in the kanister logs or post the logs so that I can have a look into it ?

Second error message usually comes when you don’t have a location profile specified under Location Profile for Kanister Action. I couldn’t find any reason why setting this in the U, will still miss it in the YAML under backupParameters. I will try to see if I can reproduce the same in my test env.

 

Userlevel 1

Hi, @jaiganeshjk , thanks for your help.

The kanister-svc logs did indeed help. It turns out that the primary container had read-only filesystem set in it’s SecurityContext and as the kanister sidecar uses that SecurityContext when deploying via injector, it also had a read-only filesystem and could therefore not copy the data.

Is there a way to set a seperate sidecar container SecurityContext with annotations and/or labels with the injector or do I have to edit deployments with read-only filesystems manually to deploy a sidecar with a different SecurityContext? I wasn’t able to find anything in the docs about this.

 

As for the second error, I wasn’t able to reproduce it myself, so it might have just been a mistake on my part. Maybe I missclicked in the UI 😬

Userlevel 6
Badge +2

Glad that you were able to find the issue. We are trying to enhance this and are tracking this internally.

However, for now, kanister-sidecar uses the same security-context as the application container. You might have to manually remove the read-only root filesystem SecurityContext from the kanister-sidecar or patch the kanister-sidecar to add an emptyDir volume for the `/tmp` path. This way, K10 can write to the `/tmp` directory.

Userlevel 1

Alright, thank you. It would be nice to be able to set this with the injector since a lot of my deployments are Helm Charts that I didn’t create myself and they often don’t allow you to just deploy a second container with a different securitycontext.

 

Thanks four help, though!

Comment