Hello @Florian Lacrampe 
When a pod shows the "command terminated with exit code 137" error, it indicates that the pod has exhausted its resources. To resolve this issue, you may consider raising its resource limits and trying the command again.

The following Helm parameters can be used to accomplish this:

“The following numbers provided are solely for illustrative purposes. Please adjust them according to the available resources on your cluster.”

--set genericVolumeSnapshot.resources.requests.cpu= 100m
--set genericVolumeSnapshot.resources.requests.memory= 800Mi
--set genericVolumeSnapshot.resources.limits.cpu= 1200m
--set genericVolumeSnapshot.resources.limits.memory= 4000Mi

Hello @Hagag 

Unfortunately, we have same behavior with these settings :

genericVolumeSnapshot:

  resources:

    requests:

      memory: "4G"

      cpu: "4000m"

    limits:

      memory: "4G"

      cpu: "4000m"

Here last logs from “/tmp/vmware-root/vixDiskLib-42.log” from block-mode-upload pod :

2023-05-11T06:55:57.243Z In(05) host-45 Opening file 2vsanDatastore] 84047962-d2a8-e4d6-f5c8-5cba2c166f30/_00fa/fc40568c02fd4d9d9d

faa0be8bad30db-000011.vmdk (SKIPZ-ha-nfc://3vsanDatastore] 84047962-d2a8-e4d6-f5c8-5cba2c166f30/_00fa/fc40568c02fd4d9d9dfaa0be8bad

30db-000011.vmdk@esx-xx.vmware.xx.xx:902!52797fe6-6df5-f216-652f-9e4f6a3bfb94)

******

2023-05-11T07:19:22.871Z In(05) host-68 DISKLIB-LIB : numIOs = 50000 numMergedIOs = 0 numSplitIOs = 0

command terminated with exit code 137

When i check RAM usage on node, i don’t any over-consumption nor OOMKilled.

BTW, what do you think about block-mode-upload ? I’ll try to upload without block mode, maybe that can fix this issue.

Thank you for your help

@Florian Lacrampe A block mode export accesses the content of the disk snapshot at the block level using infrastructure-specific APIs.
Snapshot content is accessed at the block level directly through the network using the VMware VDDK API.

If changed block tracking is enabled in the infrastructure for the disk volume then K10 will export just the incremental changes to the export location if possible, since you have (+ 5T) of data it will reduce the amount of data transferred during subsequent exports.

Could you attempt to run the Block mode again and verify if there are any I/O issues? It's important to note that an exit code of 137 indicates that there may be issues with resource utilization beyond just memory.

in the logs, it mentioned that the specified host (host-68) has performed a total of 50,000 input/output operations.

I’ll continue to troubleshoot to understand why block-mode-upload receive a SIGKILL signal.

I’ve planned to :

• Empty my Minio (S3) bucket
• Remove all backups
• Recreate all k10 pods

If you have any idea about log files or any stdout/stderr i can check, tell me ;)

I already check I/O perfs but i don’t see any problem.

Thank you so much for your help !

Have a nice day

Well… SIGKILL was sended due to KanisterTimeout which was not set, my bad.

For now, i continue to test large backups and also test vSAN I/O during k10kopia transfer.

Thank you for your help !

@Florian Lacrampe 

Sharing how you figured out that the Kanister timeout was missed would be helpful. Typically, this information can be found in the debug logs.