Environment：
kasten 6.5.8
k8s 1.20
The problem：Kanister Sidecar backup 50G pvc error
Note added: ERROR MESSAGES
Job failed to be executed
-->Failed snapshots for workload
-->Failed to snapshot volumes
-->Error snapshotting volume
-->Error waiting for ActionSet
-->Timeout while polling
-->Failed to get actionset
->lient rate limiter Wait returned an
-->error: context deadline exceeded
message: "client rate limiter Wait returned an error: context deadline exceeded"
fields:
- name: actionSet
value: k10-backuptoserver-k10-deployment-generic-volume-2.0.42-nexk59f
file: kasten.io/k10/kio/kanister/operation.go:376
function: kasten.io/k10/kio/kanister.(*Operation).waitForActionSetCompletion.func1
linenumber: 376
message: Failed to get actionset
fields:
- name: duration
value: 29m59.986081382s
file: kasten.io/k10/kio/poll/poll.go:86
function: kasten.io/k10/kio/poll.waitWithBackoffWithRetries
linenumber: 86
message: Timeout while polling
fields:
- name: actionSet
value: k10-backuptoserver-k10-deployment-generic-volume-2.0.42-nexk59f
file: kasten.io/k10/kio/kanister/operation.go:396
Let's check the log
cat kopia-20250428-062734-117-repository-connect-server.0.log
2025-04-28T06:27:34.879302Z INFO kopia/cli Connecting to server 'https://10.43.164.253:51515' as 'k10-admin@3b609356-f49f-4081-94f0-2194cfcd1e25.xxxxxxxx'...
2025-04-28T06:27:34.879532Z DEBUG kopia/repo Creating cache directory '/tmp/kopia-cache' with max size 5242880000
2025-04-28T06:27:40.984678Z DEBUG cache finished initial cache scan {"cache":"cache-storage","duration":"404.200178ms","totalRetainedSize":98680193,"tooRecentBytes":35106337,"tooRecentCount":725,"maxSizeBytes":5242880000,"limitBytes":0,"inUsePercent":1}
2025-04-28T06:27:41.575988Z DEBUG client GET https://10.43.164.253:51515/api/v1/repo/parameters took 591.114317ms and returned 200 OK
2025-04-28T06:27:41.576148Z DEBUG passwordpersist Saving password to file /tmp/kopia-repository/3b609356-f49f-4081-94f0-2194cfcd1e25-xxxxxx3.config.kopia-password.
2025-04-28T06:27:41.576417Z INFO kopia/cli Connected to repository API Server.
cat latest.log
2025-04-28T06:22:56.287487Z DEBUG client PUT https://10.43.238.105:51515/api/v1/contents/3d827a4201a48a8a13c670558a3064a5?compression=1200 took 531.735µs and returned 200 OK
2025-04-28T06:22:56.288313Z DEBUG client PUT https://10.43.238.105:51515/api/v1/contents/61bd505272d3ad0f58e4591645283f35?compression=1200 took 688.839µs and returned 200 OK
2025-04-28T06:22:56.289882Z DEBUG client PUT https://10.43.238.105:51515/api/v1/contents/613cddb22e9d7c77cc7ac84c1da8f7bc?compression=1200 took 534.897µs and returned 200 OK
2025-04-28T06:22:56.290381Z DEBUG client PUT https://10.43.238.105:51515/api/v1/contents/ba5f3d7e740e8cda1ecaae76caf062c9?compression=1200 took 471.889µs and returned 200 OK
2025-04-28T06:22:56.292142Z DEBUG client PUT https://10.43.238.105:51515/api/v1/contents/290ef01a92b2bc44e00df9acd6b9a7b1?compression=1200 took 447.395µs and returned 200 OK
2025-04-28T06:22:56.293729Z DEBUG client PUT https://10.43.238.105:51515/api/v1/contents/682115559cc62a429d3216862afd35c3?compression=1200 took 467.159µs and returned 200 OK
2025-04-28T06:22:56.294904Z DEBUG client GET https://10.43.238.105:51515/api/v1/contents/e29297cd50328932eec62aaa27b7cd9d?info=1 took 392.83µs and returned 200 OK
2025-04-28T06:22:56.295787Z DEBUG client PUT https://10.43.238.105:51515/api/v1/contents/20c642f1ab49b0afc179bebd8314d78f?compression=1200 took 375.817µs and returned 200 OK
2025-04-28T06:22:56.297911Z DEBUG client PUT https://10.43.238.105:51515/api/v1/contents/358ca3906ea86d53e12114a371456afd?compression=1200 took 471.081µs and returned 200 OK
2025-04-28T06:22:56.298027Z DEBUG client PUT https://10.43.238.105:51515/api/v1/contents/00019e841031fa6fe4a49f8d6ed90490?compression=1200 took 580.496µs and returned 200 OK
2025-04-28T06:22:56.300068Z DEBUG client PUT https://10.43.238.105:51515/api/v1/contents/9b20f25840619547023c0d9ab0b52c32?compression=1200 took 590.303µs and returned 200 OK
2025-04-28T06:22:56.376693Z DEBUG client GET https://10.43.238.105:51515/api/v1/contents/d5e7200a829799bb238ee3537314a424?info=1 took 747.416µs and returned 200 OK
2025-04-28T06:22:56.376961Z DEBUG client GET https://10.43.238.105:51515/api/v1/contents/40f34496af0d8aec03588a03ffa48bc4?info=1 took 740.702µs and returned 404 Not Found
2025-04-28T06:22:56.378452Z DEBUG client PUT https://10.43.238.105:51515/api/v1/contents/c5af68bc5edc04184802d3d57a6605d2?compression=1200 took 712.216µs and returned 200 OK
2025-04-28T06:22:56.379057Z DEBUG client PUT https://10.43.238.105:51515/api/v1/contents/5ac4fad42d44efa4485c01c86d0ef257?compression=1200 took 501.76µs and returned 200 OK
2025-04-28T06:22:56.379231Z DEBUG client PUT https://10.43.238.105:51515/api/v1/contents/40f34496af0d8aec03588a03ffa48bc4?compression=1200 took 2.130583ms and returned 200 OK
cat kopia-20250428-055636-39-snapshot-create.0.log
98b78386015d942?compression=1200": round-trip error: dial tcp 10.43.238.105:51515: connect: connection refused: error writing content 5123036460fcb727c98b78386015d942: error running http request: Put "https://10.43.238.105:51515/api/v1/contents/5123036460fcb727c98b78386015d942?compression=1200": round-trip error: dial tcp 10.43.238.105:51515: connect: connection refused
2025-04-28T06:27:17.589396Z DEBUG client PUT https://10.43.238.105:51515/api/v1/contents/c9a42d17ed1363f3893728fc8cf79403?compression=1200 took 79.464771ms and failed with dial tcp 10.43.238.105:51515: connect: connection refused
2025-04-28T06:27:17.589461Z DEBUG uploader snapshotted file {"path":"blobs/default/content/vol-04/chap-46/9f266f6a-7308-44bb-add8-bcfd3e1a0aa1.bytes","error":"unable to get result: unable to write content chunk 0 of FILE:9f266f6a-7308-44bb-add8-bcfd3e1a0aa1.bytes: error writing content 5123036460fcb727c98b78386015d942: error running http request: Put \"https://10.43.238.105:51515/api/v1/contents/5123036460fcb727c98b78386015d942?compression=1200\": round-trip error: dial tcp 10.43.238.105:51515: connect: connection refused: error writing content 5123036460fcb727c98b78386015d942: error running http request: Put \"https://10.43.238.105:51515/api/v1/contents/5123036460fcb727c98b78386015d942?compression=1200\": round-trip error: dial tcp 10.43.238.105:51515: connect: connection refused"}
2025-04-28T06:27:17.589492Z ERROR object async write error: unable to write content chunk 0 of FILE:57bb5d53-332b-48cd-bed0-f840ad2479b1.properties: error writing content c9a42d17ed1363f3893728fc8cf79403: error running http request: Put "https://10.43.238.105:51515/api/v1/contents/c9a42d17ed1363f3893728fc8cf79403?compression=1200": round-trip error: dial tcp 10.43.238.105:51515: connect: connection refused: error writing content c9a42d17ed1363f3893728fc8cf79403: error running http request: Put "https://10.43.238.105:51515/api/v1/contents/c9a42d17ed1363f3893728fc8cf79403?compression=1200": round-trip error: dial tcp 10.43.238.105:51515: connect: connection refused
2025-04-28T06:27:17.589671Z DEBUG uploader snapshotted file {"path":"blobs/default/content/vol-04/chap-46/57bb5d53-332b-48cd-bed0-f840ad2479b1.properties","error":"unable to get result: unable to write content chunk 0 of FILE:57bb5d53-332b-48cd-bed0-f840ad2479b1.properties: error writing content c9a42d17ed1363f3893728fc8cf79403: error running http request: Put \"https://10.43.238.105:51515/api/v1/contents/c9a42d17ed1363f3893728fc8cf79403?compression=1200\": round-trip error: dial tcp 10.43.238.105:51515: connect: connection refused: error writing content c9a42d17ed1363f3893728fc8cf79403: error running http request: Put \"https://10.43.238.105:51515/api/v1/contents/c9a42d17ed1363f3893728fc8cf79403?compression=1200\": round-trip error: dial tcp 10.43.238.105:51515: connect: connection refused"}
2025-04-28T06:27:17.658193Z DEBUG client GET https://10.43.238.105:51515/api/v1/contents/f96995aace786dee6233c1f0f517f813?info=1 took 76.712363ms and failed with dial tcp 10.43.238.105:51515: connect: connection refused
2025-04-28T06:27:17.665761Z DEBUG client PUT https://10.43.238.105:51515/api/v1/contents/0be6351b5dade9e8ee4ef82d8d6ccf66?compression=1200 took 71.196644ms and failed with dial tcp 10.43.238.105:51515: connect: connection refused
2025-04-28T06:27:17.665836Z ERROR object async write error: unable to write content chunk 0 of FILE:4f30546d-69db-44a1-ae03-4eb6743d9a28.bytes: error writing content 0be6351b5dade9e8ee4ef82d8d6ccf66: error running http request: Put "https://10.43.238.105:51515/api/v1/contents/0be6351b5dade9e8ee4ef82d8d6ccf66?compression=1200": round-trip error: dial tcp 10.43.238.105:51515: connect: connection refused: error writing content 0be6351b5dade9e8ee4ef82d8d6ccf66: error running http request: Put "https://10.43.238.105:51515/api/v1/contents/0be6351b5dade9e8ee4ef82d8d6ccf66?compression=1200": round-trip error: dial tcp 10.43.238.105:51515: connect: connection refused
2025-04-28T06:27:17.665929Z DEBUG uploader snapshotted file {"path":"blobs/default/content/vol-04/chap-46/4f30546d-69db-44a1-ae03-4eb6743d9a28.bytes","error":"unable to get result: unable to write content chunk 0 of FILE:4f30546d-69db-44a1-ae03-4eb6743d9a28.bytes: error writing content 0be6351b5dade9e8ee4ef82d8d6ccf66: error running http request: Put \"https://10.43.238.105:51515/api/v1/contents/0be6351b5dade9e8ee4ef82d8d6ccf66?compression=1200\": round-trip error: dial tcp 10.43.238.105:51515: connect: connection refused: error writing content 0be6351b5dade9e8ee4ef82d8d6ccf66: error running http request: Put \"https://10.43.238.105:51515/api/v1/contents/0be6351b5dade9e8ee4ef82d8d6ccf66?compression=1200\": round-trip error: dial tcp 10.43.238.105:51515: connect: connection refused"}
2025-04-28T06:27:17.670107Z DEBUG client PUT https://10.43.238.105:51515/api/v1/contents/d3986ce4438c5683ea48f2323be249bc?compression=1200 took 71.425159ms and failed with dial tcp 10.43.238.105:51515: connect: connection refused
2025-04-28T06:27:17.670174Z ERROR object async write error: unable to write content chunk 0 of FILE:feabadb0-da1e-4105-a0aa-f9e162c3d983.properties: error writing content d3986ce4438c5683ea48f2323be249bc: error running http request: Put "https://10.43.238.105:51515/api/v1/contents/d3986ce4438c5683ea48f2323be249bc?compression=1200": round-trip error: dial tcp 10.43.238.105:51515: connect: connection refused: error writing content d3986ce4438c5683ea48f2323be249bc: error running http request: Put "https://10.43.238.105:51515/api/v1/contents/d3986ce4438c5683ea48f2323be249bc?compression=1200": round-trip error: dial tcp 10.43.238.105:51515: connect: connection refused
2025-04-28T06:27:17.670249Z DEBUG uploader snapshotted file {"path":"blobs/default/content/vol-04/chap-46/feabadb0-da1e-4105-a0aa-f9e162c3d983.properties","error":"unable to get result: unable to write content chunk 0 of FILE:feabadb0-da1e-4105-a0aa-f9e162c3d983.properties: error writing content d3986ce4438c5683ea48f2323be249bc: error running http request: Put \"https://10.43.238.105:51515/api/v1/contents/d3986ce4438c5683ea48f2323be249bc?compression=1200\": round-trip error: dial tcp 10.43.238.105:51515: connect: connection refused: error writing content d3986ce4438c5683ea48f2323be249bc: error running http request: Put \"https://10.43.238.105:51515/api/v1/contents/d3986ce4438c5683ea48f2323be249bc?compression=1200\": round-trip error: dial tcp 10.43.238.105:51515: connect: connection refused"}
2025-04-28T06:27:17.670315Z DEBUG client PUT https://10.43.238.105:51515/api/v1/contents/e513613df0889b124587e45baf5edc92?compression=1200 took 70.571289ms and failed with dial tcp 10.43.238.105:51515: connect: connection refused
2025-04-28T06:27:17.670360Z ERROR object async write error: unable to write content chunk 0 of FILE:a5ce01ad-c68d-495c-8940-c9bae6f3746e.properties: error writing content e513613df0889b124587e45baf5edc92: error running http request: Put "https://10.43.238.105:51515/api/v1/contents/e513613df0889b124587e45baf5edc92?compression=1200": round-trip error: dial tcp 10.43.238.105:51515: connect: connection refused: error writing content e513613df0889b124587e45baf5edc92: error running http request: Put "https://10.43.238.105:51515/api/v1/contents/e513613df0889b124587e45baf5edc92?compression=1200": round-trip error: dial tcp 10.43.238.105:51515: connect: connection refused
2025-04-28T06:27:17.670432Z DEBUG uploader snapshotted file {"path":"blobs/default/content/vol-04/chap-46/a5ce01ad-c68d-495c-8940-c9bae6f3746e.properties","error":"unable to get result: unable to write content chunk 0 of FILE:a5ce01ad-c68d-495c-8940-c9bae6f3746e.properties: error writing content e513613df0889b124587e45baf5edc92: error running http request: Put \"https://10.43.238.105:51515/api/v1/contents/e513613df0889b124587e45baf5edc92?compression=1200\": round-trip error: dial tcp 10.43.238.105:51515: connect: connection refused: error writing content e513613df0889b124587e45baf5edc92: error running http request: Put \"https://10.43.238.105:51515/api/v1/contents/e513613df0889b124587e45baf5edc92?compression=1200\": round-trip error: dial tcp 10.43.238.105:51515: connect: connection refused"}
2025-04-28T06:27:17.747962Z DEBUG client PUT https://10.43.238.105:51515/api/v1/contents/f96995aace786dee6233c1f0f517f813?compression=1200 took 71.215362ms and failed with dial tcp 10.43.238.105:51515: connect: connection refused
2025-04-28T06:27:17.748052Z ERROR object async write error: unable to write content chunk 10 of FILE:component-2024-08-03-16-15-00-3.35.0-02.bak: error writing content f96995aace786dee6233c1f0f517f813: error running http request: Put "https://10.43.238.105:51515/api/v1/contents/f96995aace786dee6233c1f0f517f813?compression=1200": round-trip error: dial tcp 10.43.238.105:51515: connect: connection refused: error writing content f96995aace786dee6233c1f0f517f813: error running http request: Put "https://10.43.238.105:51515/api/v1/contents/f96995aace786dee6233c1f0f517f813?compression=1200": round-trip error: dial tcp 10.43.238.105:51515: connect: connection refused
2025-04-28T06:27:17.750955Z DEBUG client PUT https://10.43.238.105:51515/api/v1/contents/8ac71f78ac23a614ae324d2b3d4953a4?compression=1200 took 72.444606ms and failed with dial tcp 10.43.238.105:51515: connect: connection refused
2025-04-28T06:27:17.751029Z ERROR object async write error: unable to write content chunk 0 of FILE:42493afc-4baa-43dc-99cd-b55bb64fdab4.properties: error writing content 8ac71f78ac23a614ae324d2b3d4953a4: error running http request: Put "https://10.43.238.105:51515/api/v1/contents/8ac71f78ac23a614ae324d2b3d4953a4?compression=1200": round-trip error: dial tcp 10.43.238.105:51515: connect: connection refused: error writing content 8ac71f78ac23a614ae324d2b3d4953a4: error running http request: Put "https://10.43.238.105:51515/api/v1/contents/8ac71f78ac23a614ae324d2b3d4953a4?compression=1200": round-trip error: dial tcp 10.43.238.105:51515: connect: connection refused
2025-04-28T06:27:17.751142Z DEBUG uploader snapshotted file {"path":"blobs/default/content/vol-04/chap-46/42493afc-4baa-43dc-99cd-b55bb64fdab4.properties","error":"unable to get result: unable to write content chunk 0 of FILE:42493afc-4baa-43dc-99cd-b55bb64fdab4.properties: error writing content 8ac71f78ac23a614ae324d2b3d4953a4: error running http request: Put \"https://10.43.238.105:51515/api/v1/contents/8ac71f78ac23a614ae324d2b3d4953a4?compression=1200\": round-trip error: dial tcp 10.43.238.105:51515: connect: connection refused: error writing content 8ac71f78ac23a614ae324d2b3d4953a4: error running http request: Put \"https://10.43.238.105:51515/api/v1/contents/8ac71f78ac23a614ae324d2b3d4953a4?compression=1200\": round-trip error: dial tcp 10.43.238.105:51515: connect: connection refused"}
2025-04-28T06:27:17.751494Z DEBUG client PUT https://10.43.238.105:51515/api/v1/contents/76d3ce8eaeac33dfbf6eaaf897b37d30?compression=1200 took 70.57073ms and failed with dial tcp 10.43.238.105:51515: connect: connection refused
2025-04-28T06:27:17.751536Z ERROR object async write error: unable to write content chunk 0 of FILE:9f266f6a-7308-44bb-add8-bcfd3e1a0aa1.properties: error writing content 76d3ce8eaeac33dfbf6eaaf897b37d30: error running http request: Put "https://10.43.238.105:51515/api/v1/contents/76d3ce8eaeac33dfbf6eaaf897b37d30?compression=1200": round-trip error: dial tcp 10.43.238.105:51515: connect: connection refused: error writing content 76d3ce8eaeac33dfbf6eaaf897b37d30: error running http request: Put "https://10.43.238.105:51515/api/v1/contents/76d3ce8eaeac33dfbf6eaaf897b37d30?compression=1200": round-trip error: dial tcp 10.43.238.105:51515: connect: connection refused
2025-04-28T06:27:17.751605Z DEBUG uploader snapshotted file {"path":"blobs/default/content/vol-04/chap-46/9f266f6a-7308-44bb-add8-bcfd3e1a0aa1.properties","error":"unable to get result: unable to write content chunk 0 of FILE:9f266f6a-7308-44bb-add8-bcfd3e1a0aa1.properties: error writing content 76d3ce8eaeac33dfbf6eaaf897b37d30: error running http request: Put \"https://10.43.238.105:51515/api/v1/contents/76d3ce8eaeac33dfbf6eaaf897b37d30?compression=1200\": round-trip error: dial tcp 10.43.238.105:51515: connect: connection refused: error writing content 76d3ce8eaeac33dfbf6eaaf897b37d30: error running http request: Put \"https://10.43.238.105:51515/api/v1/contents/76d3ce8eaeac33dfbf6eaaf897b37d30?compression=1200\": round-trip error: dial tcp 10.43.238.105:51515: connect: connection refused"}
2025-04-28T06:27:17.790315Z DEBUG client GET https://10.43.238.105:51515/api/v1/contents/e07f25bda225022f71e0973fa1c5228a?info=1 took 71.110443ms and failed with dial tcp 10.43.238.105:51515: connect: connection refused
2025-04-28T06:27:17.824209Z DEBUG client GET https://10.43.238.105:51515/api/v1/contents/50cdab90b8fb2adccfb213f6139228c9?info=1 took 70.746809ms and failed with dial tcp 10.43.238.105:51515: connect: connection refused
2025-04-28T06:27:17.874953Z DEBUG client PUT https://10.43.238.105:51515/api/v1/contents/e07f25bda225022f71e0973fa1c5228a?compression=1200 took 84.430302ms and failed with dial tcp 10.43.238.105:51515: connect: connection refused
2025-04-28T06:27:17.875033Z ERROR object async write error: unable to write content chunk 0 of FILE:ce2ee7e7-ec3c-40f0-bbf1-df11dfbb208c.bytes: error writing content e07f25bda225022f71e0973fa1c5228a: error running http request: Put "https://10.43.238.105:51515/api/v1/contents/e07f25bda225022f71e0973fa1c5228a?compression=1200": round-trip error: dial tcp 10.43.238.105:51515: connect: connection refused: error writing content e07f25bda225022f71e0973fa1c5228a: error running http request: Put "https://10.43.238.105:51515/api/v1/contents/e07f25bda225022f71e0973fa1c5228a?compression=1200": round-trip error: dial tcp 10.43.238.105:51515: connect: connection refused
2025-04-28T06:27:17.875126Z DEBUG uploader snapshotted file {"path":"blobs/default/content/vol-04/chap-46/ce2ee7e7-ec3c-40f0-bbf1-df11dfbb208c.bytes","error":"unable to get result: unable to write content chunk 0 of FILE:ce2ee7e7-ec3c-40f0-bbf1-df11dfbb208c.bytes: error writing content e07f25bda225022f71e0973fa1c5228a: error running http request: Put \"https://10.43.238.105:51515/api/v1/contents/e07f25bda225022f71e0973fa1c5228a?compression=1200\": round-trip error: dial tcp 10.43.238.105:51515: connect: connection refused: error writing content e07f25bda225022f71e0973fa1c5228a: error running http request: Put \"https://10.43.238.105:51515/api/v1/contents/e07f25bda225022f71e0973fa1c5228a?compression=1200\": round-trip error: dial tcp 10.43.238.105:51515: connect: connection refused"}
2025-04-28T06:27:17.899528Z DEBUG client PUT https://10.43.238.105:51515/api/v1/contents/50cdab90b8fb2adccfb213f6139228c9?compression=1200 took 74.098677ms and failed with dial tcp 10.43.238.105:51515: connect: connection refused