Hello
Kasten has been successfully creating backups for volumes and configs but recently export jobs failing for all policies with error message “client rate limiter Wait returned an error: context deadline exceeded":
cause:
message: "client rate limiter Wait returned an error: rate: Wait(n=1) would
exceed context deadline"
file: github.com/kanisterio/kanister@v0.0.0-20240828182737-b6d930f12c93/pkg/kube/pod.go
function: github.com/kanisterio/kanister/pkg/kube.WaitForPodReady
linenumber: 412
message: Pod did not transition into running state.
Timeout:15m0s Namespace:kasten-io,
Name:copy-vol-data-7d5hj
file: github.com/kanisterio/kanister@v0.0.0-20240828182737-b6d930f12c93/pkg/kube/pod_controller.go
function: github.com/kanisterio/kanister/pkg/kube.(*podController).WaitForPodReady
linenumber: 174
message: Pod failed to become ready in time
fields:
- name: pod
value: copy-vol-data-7d5hj
- name: namespace
value: kasten-io
file: kasten.io/k10/kio/kanister/function/kio_copy_volume_data.go:304
function: kasten.io/k10/kio/kanister/function.CopyVolumeData.copyVolumeDataPodExecFunc.func2
linenumber: 304
message: failed while waiting for Pod to be ready
file: kasten.io/k10/kio/kanister/function/kio_copy_volume_data.go:161
function: kasten.io/k10/kio/kanister/function.CopyVolumeData
linenumber: 161
message: Failed to execute copy volume data pod functionTried running the job again but same results.
Also I can see lots snapshots data like 1.2 TB. It should clear itself but looks like it not deleting snapshots. Not sure export jobs are failing because of this or could be something else
I can see some snapshots are being deleted as seen in graph but keep growing exponentially
