Solved

Veeam fail to backup Linux VMs on Hyper-V


Userlevel 4

Hi,

 

I am trying to back up some linux VMs, the VMs are running on Hyper-V 2016.

 

The backup job starts fine, but it freezes at 63% and fails with the error message below.

 

 

 

17/05/2023 10:00:39 :: Processing SRV-SFTP-02 Error: Shared memory connection has been forcibly closed by peer.
Failed to upload disk. Skipped arguments: [shadowSpec>];
Agent failed to process method {DataTransfer.SyncDisk}.

icon

Best answer by MMASLOUH 10 June 2023, 19:00

View original

17 comments

Userlevel 5
Badge +2

Have you checked system requirements (especially RAM) for each component like backup server, proxy and repository? We’ve seen these errors when there is insufficient RAM (see this forums thread).

 

 

Userlevel 7
Badge +16

Hi @MMASLOUH ...we need wayyyy more info on your issue here. What is your Veeam server setup? Is it an ‘all-in-one’ setup, or do you have external components (Proxy, Repo, etc)? How much resources did you give your VBR server and components? Did you meet minimum system requirements as noted in the Guide? I’ve seen some posts in the forums where folks were backing up to a local disk or some Repo which had bad blocks, so checking your storage is advisable as well. Other potential resolutions include increasing memory on some components, increasing network streams, etc. But, before suggesting any of those solutions, again… we need more info on how your environment is setup. Thanks.

Userlevel 7
Badge +20

You can also check the logs too with the above suggestions which are located here to help narrow down the component maybe causing the issue - C:\ProgramData\Veeam\Backup

Userlevel 4

Hi @coolsport00,

1 - What is your Veeam server setup?

All in one

2 - How much resources did you give your VBR server and components?

4 CPU, 12Gb Ram, 60Gb HDD “C”, 60Gb HDD “D”

Did you meet minimum system requirements as noted in the Guide?

I think it meet the minimim.

 

I can backup the Windows VMs hosted on the same hypervisor without problem, The problem is only with Linux VMs.

Userlevel 4

Here is part from the backup log.

 

[17.05.2023 10:01:22] <17> Info         Meta file saved to disk, Elapsed Time: 00:00:00.2413671, in seconds: 0,2413671
[17.05.2023 10:01:22] <17> Info         [Mutex] Releasing mutex [Global\BackupMeta[c4734c1a-e03b-4c43-8dba-a870691fa11a]]
[17.05.2023 10:01:22] <17> Info         [BackupMetaUpdater] Backup meta generated, Elapsed Time: 00:00:01.4752865, in seconds 1,4752865
[17.05.2023 10:04:30] <20> Info         [HvWmiProxy:srv-xxxxx-xxxx.zzzz.local] Closing connection
[17.05.2023 10:04:30] <20> Info         [CProxyRpcInvoker] RpcInvoker [57616766] was disposed
[17.05.2023 10:23:41] <28> Error        Processing SRV-SFTP-02
[17.05.2023 10:23:41] <28> Error        Shared memory connection has been forcibly closed by peer. (Veeam.Backup.Common.CCppComponentException)
[17.05.2023 10:23:41] <28> Error           in c++: Failed to write data into shared memory IO device.
[17.05.2023 10:23:41] <28> Error           in c++: Failed to serialize data area. ID: [3806]. Offset: [27845984256].
[17.05.2023 10:23:41] <28> Error           in c++: Failed to send next file block. Block identity: [Data block. Start offset: [27845984256], Length: [4194304], Area ID: [3806].].
[17.05.2023 10:23:41] <28> Error           in c++: Unable to asynchronously write data block. Block identity: [Data block. Start offset: [27845984256], Length: [4194304], Area ID: [3806].].
[17.05.2023 10:23:41] <28> Error           in c++: Processing of asynchronous write requests has failed. Output file: [File blocks transmission channel (sender).].
[17.05.2023 10:23:41] <28> Error           in c++: Failed to process conveyored task.
[17.05.2023 10:23:41] <28> Error        Failed to upload disk. Skipped arguments: [shadowSpec>]; (Veeam.Backup.Common.CCppComponentException)
[17.05.2023 10:23:41] <28> Error           in c++: Disk upload failed.
[17.05.2023 10:23:41] <28> Error        Agent failed to process method {DataTransfer.SyncDisk}. (Veeam.Backup.Common.CCppComponentException)
[17.05.2023 10:23:41] <28> Error           in c++ event: 7

Userlevel 7
Badge +16

Hi @MMASLOUH 12GB of RAM is not enough memory for each component you’re using on the Veeam Server. Please review the sys req’s again and add the memory Veeam recommends for each component → Veeam Server, Proxy, Repository, Mount Server, as well as Backup Console. At a minimum, you should probably have around 16GB of RAM, but that doesn’t take into effect any concurrent jobs you may have (if any). Nor does that account for the memory required for just the operating system itself, either.

Userlevel 4

Hi @coolsport00,

 

I will increase the memory to 32 GB and do a test.

Userlevel 7
Badge +16

Great. Let us know how it goes. We can then go from there if just the LNX VMs still have issues.

Userlevel 7
Badge +20

@MMASLOUH -- check out this BP site from Veeam and the Architects as it explains the VBR requirements for single server and multi-server deployments - Backup server - Veeam Backup & Replication Best Practice Guide

Userlevel 4

Hi,

 

Even after increasing memory and CPU, the problem still persists.

 

I suspect there is something wrong with these Linux VMs, as I created a new Windows Server 2016 VM on the same hypervisor and the backup went fine.

Userlevel 7
Badge +6

Hello @MMASLOUH 

 

Try to changed the storage optimization in the Backup job advanced settings from 1MB to 256KB or 512KB.

 

https://helpcenter.veeam.com/docs/backup/hyperv/backup_job_advanced_storage_hv.html?ver=120

Userlevel 7
Badge +7

Are you using Application Aware Processing on the VM? Try taking a backup with Application Aware processing enabled and see if that makes a difference. Seems like there is a time out when the backup is being processed followed by the connection being terminated. 

Userlevel 7
Badge +16

Hi @MMASLOUH ...another option you can try is create a brand new “bare” linux VM and test it in a simple job to see if it backs up ok. If so, chances are the problem is indeed with those current linux VMs.. And, at that point, would recommend getting ahold of Veeam support to take a deeper look at why you can’t back them up.

Userlevel 4

Update,

I contacted Veeam support, they asked me for log files of this job and after analyzing the logs, they found that VeeamBR server cannot communicate with hyper-v server when backup job reaches 65 %.

 

So I did a test using another hyper-v server in the same segment of VeeamBR and everything works fine.

 

I went back to our Forcepoint firewall and found that the IPS detected the NFS communications between hyper-V and our Dell data domain as a Log4j threat.

 

A rule to allow this communication solves this problem.

 

Thanks, you are the best.

Userlevel 7
Badge +22

Not the first time I have seen IPS’s making mistakes but also leading people down the wrong troubleshooting path :) 

Userlevel 7
Badge +20

Glad you were able to resolve this.  IPS can be troublesome at times for sure.

Userlevel 7
Badge +7

thanks for sharing the support answer!
for me this is what it is about, sharing experiences and knowledge!

cheers.

Comment