Skip to main content
Solved

Veeam fail to backup Linux VMs on Hyper-V


Hi,

 

I am trying to back up some linux VMs, the VMs are running on Hyper-V 2016.

 

The backup job starts fine, but it freezes at 63% and fails with the error message below.

 

 

 

17/05/2023 10:00:39 :: Processing SRV-SFTP-02 Error: Shared memory connection has been forcibly closed by peer.
Failed to upload disk. Skipped arguments: [shadowSpec>];
Agent failed to process method {DataTransfer.SyncDisk}.

Best answer by MMASLOUH

Update,

I contacted Veeam support, they asked me for log files of this job and after analyzing the logs, they found that VeeamBR server cannot communicate with hyper-v server when backup job reaches 65 %.

 

So I did a test using another hyper-v server in the same segment of VeeamBR and everything works fine.

 

I went back to our Forcepoint firewall and found that the IPS detected the NFS communications between hyper-V and our Dell data domain as a Log4j threat.

 

A rule to allow this communication solves this problem.

 

Thanks, you are the best.

View original
Did this topic help you find an answer to your question?

18 comments

MattM
Forum|alt.badge.img+2
  • Experienced User
  • 42 comments
  • May 17, 2023

Have you checked system requirements (especially RAM) for each component like backup server, proxy and repository? We’ve seen these errors when there is insufficient RAM (see this forums thread).

 

 


coolsport00
Forum|alt.badge.img+20
  • Veeam Legend
  • 4109 comments
  • May 17, 2023

Hi @MMASLOUH ...we need wayyyy more info on your issue here. What is your Veeam server setup? Is it an ‘all-in-one’ setup, or do you have external components (Proxy, Repo, etc)? How much resources did you give your VBR server and components? Did you meet minimum system requirements as noted in the Guide? I’ve seen some posts in the forums where folks were backing up to a local disk or some Repo which had bad blocks, so checking your storage is advisable as well. Other potential resolutions include increasing memory on some components, increasing network streams, etc. But, before suggesting any of those solutions, again… we need more info on how your environment is setup. Thanks.


Chris.Childerhose
Forum|alt.badge.img+21

You can also check the logs too with the above suggestions which are located here to help narrow down the component maybe causing the issue - C:\ProgramData\Veeam\Backup


  • Author
  • Comes here often
  • 11 comments
  • May 17, 2023

Hi @coolsport00,

1 - What is your Veeam server setup?

All in one

2 - How much resources did you give your VBR server and components?

4 CPU, 12Gb Ram, 60Gb HDD “C”, 60Gb HDD “D”

Did you meet minimum system requirements as noted in the Guide?

I think it meet the minimim.

 

I can backup the Windows VMs hosted on the same hypervisor without problem, The problem is only with Linux VMs.


  • Author
  • Comes here often
  • 11 comments
  • May 17, 2023

Here is part from the backup log.

 

[17.05.2023 10:01:22] <17> Info         Meta file saved to disk, Elapsed Time: 00:00:00.2413671, in seconds: 0,2413671
[17.05.2023 10:01:22] <17> Info         [Mutex] Releasing mutex [Global\BackupMeta[c4734c1a-e03b-4c43-8dba-a870691fa11a]]
[17.05.2023 10:01:22] <17> Info         [BackupMetaUpdater] Backup meta generated, Elapsed Time: 00:00:01.4752865, in seconds 1,4752865
[17.05.2023 10:04:30] <20> Info         [HvWmiProxy:srv-xxxxx-xxxx.zzzz.local] Closing connection
[17.05.2023 10:04:30] <20> Info         [CProxyRpcInvoker] RpcInvoker [57616766] was disposed
[17.05.2023 10:23:41] <28> Error        Processing SRV-SFTP-02
[17.05.2023 10:23:41] <28> Error        Shared memory connection has been forcibly closed by peer. (Veeam.Backup.Common.CCppComponentException)
[17.05.2023 10:23:41] <28> Error           in c++: Failed to write data into shared memory IO device.
[17.05.2023 10:23:41] <28> Error           in c++: Failed to serialize data area. ID: [3806]. Offset: [27845984256].
[17.05.2023 10:23:41] <28> Error           in c++: Failed to send next file block. Block identity: [Data block. Start offset: [27845984256], Length: [4194304], Area ID: [3806].].
[17.05.2023 10:23:41] <28> Error           in c++: Unable to asynchronously write data block. Block identity: [Data block. Start offset: [27845984256], Length: [4194304], Area ID: [3806].].
[17.05.2023 10:23:41] <28> Error           in c++: Processing of asynchronous write requests has failed. Output file: [File blocks transmission channel (sender).].
[17.05.2023 10:23:41] <28> Error           in c++: Failed to process conveyored task.
[17.05.2023 10:23:41] <28> Error        Failed to upload disk. Skipped arguments: [shadowSpec>]; (Veeam.Backup.Common.CCppComponentException)
[17.05.2023 10:23:41] <28> Error           in c++: Disk upload failed.
[17.05.2023 10:23:41] <28> Error        Agent failed to process method {DataTransfer.SyncDisk}. (Veeam.Backup.Common.CCppComponentException)
[17.05.2023 10:23:41] <28> Error           in c++ event: 7


coolsport00
Forum|alt.badge.img+20
  • Veeam Legend
  • 4109 comments
  • May 17, 2023

Hi @MMASLOUH 12GB of RAM is not enough memory for each component you’re using on the Veeam Server. Please review the sys req’s again and add the memory Veeam recommends for each component → Veeam Server, Proxy, Repository, Mount Server, as well as Backup Console. At a minimum, you should probably have around 16GB of RAM, but that doesn’t take into effect any concurrent jobs you may have (if any). Nor does that account for the memory required for just the operating system itself, either.


  • Author
  • Comes here often
  • 11 comments
  • May 17, 2023

Hi @coolsport00,

 

I will increase the memory to 32 GB and do a test.


coolsport00
Forum|alt.badge.img+20
  • Veeam Legend
  • 4109 comments
  • May 17, 2023

Great. Let us know how it goes. We can then go from there if just the LNX VMs still have issues.


Chris.Childerhose
Forum|alt.badge.img+21

@MMASLOUH -- check out this BP site from Veeam and the Architects as it explains the VBR requirements for single server and multi-server deployments - Backup server - Veeam Backup & Replication Best Practice Guide


  • Author
  • Comes here often
  • 11 comments
  • May 25, 2023

Hi,

 

Even after increasing memory and CPU, the problem still persists.

 

I suspect there is something wrong with these Linux VMs, as I created a new Windows Server 2016 VM on the same hypervisor and the backup went fine.


Moustafa_Hindawi
Forum|alt.badge.img+6

Hello @MMASLOUH 

 

Try to changed the storage optimization in the Backup job advanced settings from 1MB to 256KB or 512KB.

 

https://helpcenter.veeam.com/docs/backup/hyperv/backup_job_advanced_storage_hv.html?ver=120


dips
Forum|alt.badge.img+7
  • Veeam Legend
  • 808 comments
  • May 26, 2023

Are you using Application Aware Processing on the VM? Try taking a backup with Application Aware processing enabled and see if that makes a difference. Seems like there is a time out when the backup is being processed followed by the connection being terminated. 


coolsport00
Forum|alt.badge.img+20
  • Veeam Legend
  • 4109 comments
  • May 30, 2023

Hi @MMASLOUH ...another option you can try is create a brand new “bare” linux VM and test it in a simple job to see if it backs up ok. If so, chances are the problem is indeed with those current linux VMs.. And, at that point, would recommend getting ahold of Veeam support to take a deeper look at why you can’t back them up.


  • Author
  • Comes here often
  • 11 comments
  • Answer
  • June 10, 2023

Update,

I contacted Veeam support, they asked me for log files of this job and after analyzing the logs, they found that VeeamBR server cannot communicate with hyper-v server when backup job reaches 65 %.

 

So I did a test using another hyper-v server in the same segment of VeeamBR and everything works fine.

 

I went back to our Forcepoint firewall and found that the IPS detected the NFS communications between hyper-V and our Dell data domain as a Log4j threat.

 

A rule to allow this communication solves this problem.

 

Thanks, you are the best.


Geoff Burke
Forum|alt.badge.img+22
  • Veeam Legend, Veeam Vanguard
  • 1312 comments
  • June 10, 2023

Not the first time I have seen IPS’s making mistakes but also leading people down the wrong troubleshooting path :) 


Chris.Childerhose
Forum|alt.badge.img+21

Glad you were able to resolve this.  IPS can be troublesome at times for sure.


HunterLAFR
Forum|alt.badge.img+8
  • Veeam Legend
  • 421 comments
  • June 12, 2023

thanks for sharing the support answer!
for me this is what it is about, sharing experiences and knowledge!

cheers.


  • New Here
  • 2 comments
  • September 10, 2024

hi, i have the same error, but the problem is not IPS, is the demon of linux hyper-v.

i have resolv my probleme to save my debian with this link and is good now :

https://www.rkidder.com/site/index.php/2014-12-04-16-22-29/kb-linux-debian/how-to-install-hyper-v-linux-integration-services-on-debian

if help over people .. ;)