Kasten K10 backup kanister sidecar failed only for jenkins

HI,

on the logs of  kanister-svc container I have those errors :
{"Container":"kanister-sidecar","File":"pkg/format/format.go","Function":"github.com/kanisterio/kanister/pkg/format.infoWithCtx","Line":97,"LogKind":"datapath","Out":"\u001b[31mERROR\u001b[0m error connecting to API server: unable to write config file: unable to create config directory: mkdir /tmp/kopia-repository: read-only file system","Pod":"jenkins-001-0","cluster_name":"4ad9cbee-e199-48e4-b192-f937d1719039","hostname":"kanister-svc-59f76bccb4-jxk49","level":"info","msg":"Pod Update","time":"2023-08-23T09:36:16.624140221Z","version":"6.0.5"}
{"Container":"kanister-sidecar","File":"pkg/format/format.go","Function":"github.com/kanisterio/kanister/pkg/format.infoWithCtx","Line":97,"LogKind":"datapath","Out":"2023-08-23 09:36:16.619874172 +0000 UTC write error: unable to open log file: open /tmp/kopia-log/75695cc1-e6f8-46e9-a4fd-dea25afa1a6f-jenkins-001-jenkins-001/cli-logs/kopia-20230823-093616-95-repository-connect-server.1.log: no such file or directory","Pod":"jenkins-001-0","cluster_name":"4ad9cbee-e199-48e4-b192-f937d1719039","hostname":"kanister-svc-59f76bccb4-jxk49","level":"info","msg":"Pod Update","time":"2023-08-23T09:36:16.624165361Z","version":"6.0.5"}
{"ActionSet":"k10-backuptoserver-k10-statefulset-generic-volume-2.0.33-jq8p2d","File":"pkg/controller/controller.go","Function":"github.com/kanisterio/kanister/pkg/controller.(*Controller).logAndErrorEvent","Line":670,"Phase":"backupToServer","cluster_name":"4ad9cbee-e199-48e4-b192-f937d1719039","error":"{\"message\":\"Failed to backup data to Kopia API server\",\"function\":\"kasten.io/k10/kio/kanister/function.(*backupDataToServerFunc).Exec\",\"linenumber\":145,\"file\":\"kasten.io/k10/kio/kanister/function/backup_data_to_server.go:145\",\"cause\":{\"message\":\"Failed to connect to Kopia API server\",\"function\":\"kasten.io/k10/kio/kanister/function.backupDataToServer\",\"linenumber\":215,\"file\":\"kasten.io/k10/kio/kanister/function/backup_data_to_server.go:215\",\"cause\":{\"message\":\"Failed to exec command in pod: command terminated with exit code 1\"}}}","hostname":"kanister-svc-59f76bccb4-jxk49","kanister.io/JobID":"6b5fb580-4198-11ee-aacc-1abca1e12201","level":"info","msg":"Failed to execute phase: v1alpha1.Phase{Name:\"backupToServer\", State:\"pending\", Output:map[string]interface {}(nil)}:","time":"2023-08-23T09:36:16.63716963Z","version":"6.0.5"}
 

it seems that kanister can't create the folders /tmp/kopia-repository and /tmp/kopia-log/ on the container config-reload of the pod jenkins-001-0 because this container is a sidecar and have the /tmp/ readonly.

Best regards.

Hello,
Did you check the status of all pods for jenkins namespace?

HI Ahmed,

I have disable the sidecar reload-config but the k10 backup still failed with the same error:

{"Container":"kanister-sidecar","File":"pkg/format/format.go","Function":"github.com/kanisterio/kanister/pkg/format.infoWithCtx","Line":97,"LogKind":"datapath","Out":"Unable to create logs directory: mkdir /tmp/kopia-log: read-only file system","Pod":"jenkins-001-0","cluster_name":"4ad9cbee-e199-48e4-b192-f937d1719039","hostname":"kanister-svc-59f76bccb4-jxk49","level":"info","msg":"Pod Update","time":"2023-08-24T08:00:05.665260911Z","version":"6.0.5"}
{"Container":"kanister-sidecar","File":"pkg/format/format.go","Function":"github.com/kanisterio/kanister/pkg/format.infoWithCtx","Line":97,"LogKind":"datapath","Out":"2023-08-24 08:00:05.656483564 +0000 UTC write error: unable to open log file: open /tmp/kopia-log/75695cc1-e6f8-46e9-a4fd-dea25afa1a6f-jenkins-001-jenkins-001/cli-logs/kopia-20230824-080005-69-repository-connect-server.0.log: no such file or directory","Pod":"jenkins-001-0","cluster_name":"4ad9cbee-e199-48e4-b192-f937d1719039","hostname":"kanister-svc-59f76bccb4-jxk49","level":"info","msg":"Pod Update","time":"2023-08-24T08:00:05.665286151Z","version":"6.0.5"}

Can you help me please ?

Best regards.

Yes K10 needs access to /tmp filesystem to write the temporary config files. If readOnlyRootFilesystem is set in the pod/container’s securityContext, then K10 would fail to write those config files as `/tmp` is still considered root filesystem and it is set to readonly.

As you mentioned, you could either remove the securityContext readOnlyRootFilesystem or you could add an emptyDir volume to the kanister-sidecar container’s securityContext pointing to the `/tmp/` mountpoint of the container.

This way `/tmp` becomes writable and it should succeed.