• EN
Solved

V12 Backup stuck in 90 percent

M
Userlevel 4
Badge

For the second time in two days backup is stuck at 90 or 91 percent. Software is running, but the backup job is stuck in concrete. I restarted veeam, I restarted veeam services and backup server, nothing helped, its stuck. I have two servers to backup. yesterday backup was stuck in win2012r2 exchange server backup; today its the win2016 fileserver. today veeam backuped the data disk successfully, but the system disk not. the exchange server is on the system disk. maybe that info helps. it seems to be a problem with backuping windows (and not data).

Case-Nr.05908107

icon

Best answer by Chris.Childerhose 2 March 2023, 18:08

View original

17 comments

Moustafa_Hindawi
Userlevel 7
Badge +6

Hello @Michail 

Would you specify backup type (active full, synthetic, inc.) , also agent, or agentless backup?

Could please share a screenshot for each machine backup progress to know which stage does it stuck?

Moustafa Hindawi | Blogger | Nutanix Technology Champion | NCSE | NCM-MCI | VMCE | 2xVCAP-DTM & DCV | 2xVCP-NV & CMA | LinkedIn: https://www.linkedin.com/in/moustafaadelmoustafa/ | My Blog: https://cloudwhales.co.uk/author/mhindawi/
M
Userlevel 4
Badge
its syntetic, inc.

 

Moustafa_Hindawi
Userlevel 7
Badge +6

Great, thank you.

Could you please share the utilization of the Veeam proxy which process this task from the task manager? And the proxy assigned resources? 

Moustafa Hindawi | Blogger | Nutanix Technology Champion | NCSE | NCM-MCI | VMCE | 2xVCAP-DTM & DCV | 2xVCP-NV & CMA | LinkedIn: https://www.linkedin.com/in/moustafaadelmoustafa/ | My Blog: https://cloudwhales.co.uk/author/mhindawi/
Chris.Childerhose
Userlevel 7
Badge +20

Are these virtual machines?  Which mode do you have selected for your proxy servers for backups - hot-add, NBD, etc.?  Is there any other tasks that run during your backup window in your environment?

Chris Childerhose - VMCA2022 | VMCE2023 | Veeam Vanguard 7* | Veeam Legend 3* | vExpert 5* | VCAP-DCV/VCP-DCV | Object First Ace | Cisco Champion | Twitter: @cchilderhose | Blog Site – https://just-virtualization.tech
M
Userlevel 4
Badge

Unfortunatly I didnt watch the task manager during backup job, but i will have a look at it today. Veeam ist the only software installed and running on the backup server (virtual win 2022 server). I belive that the server was almost idle while stuck in the job. I dont use any proxies. The backup job is from IP to IP in the same LAN. And btw. this backup job ran for a week like charming… I didnt change anything. Today morning I recognized that Veeam aborted the stuck job after 4:41 hours and retry1 was successfull after 18 minutes.

 

Moustafa_Hindawi
Userlevel 7
Badge +6

Hello @Michail 

As I see from your screenshot, you have bottle neck performance at Veeam Proxy, if you are not added a Veeam Proxy, then you are using the default proxy which is Veeam Backup server. Then watch the utilization of your VBR, and I believe it’s a physical server because your backup mode is NBD , you are backup your VM through Network, please prepare Windows server VM and add it as a proxy to use Hot-Add mode. Please let us know the result.

Moustafa Hindawi | Blogger | Nutanix Technology Champion | NCSE | NCM-MCI | VMCE | 2xVCAP-DTM & DCV | 2xVCP-NV & CMA | LinkedIn: https://www.linkedin.com/in/moustafaadelmoustafa/ | My Blog: https://cloudwhales.co.uk/author/mhindawi/
Moustafa_Hindawi
Userlevel 7
Badge +6

Also, “transmission pipeline hanged” error means proxy lost connection with the source VM, and you are using Network mode, then there are some assumptions, proxy is full utilized, network not stable, proxy lost connection with vcenter, there is security solution interrupt the connection. 

Moustafa Hindawi | Blogger | Nutanix Technology Champion | NCSE | NCM-MCI | VMCE | 2xVCAP-DTM & DCV | 2xVCP-NV & CMA | LinkedIn: https://www.linkedin.com/in/moustafaadelmoustafa/ | My Blog: https://cloudwhales.co.uk/author/mhindawi/
M
Userlevel 4
Badge

I am not that into all details of Veeam so I am not sure whether I understand every advice here. Just for additional Information: Our LAN is a small office network. The backup server is physically positioned right besides the storage NAS and physically connected via the same switch. There is no vCenter. Our backup server an our Veeam are absolute stand alone. The proxy: I dont understand, why i would need any proxy in our configuration (even a standard proxy)? I left the proxy option blank, so I guessed there is nothing in between the backup server and the storage NAS.

Moustafa_Hindawi
Userlevel 7
Badge +6

do a backup job, and when it stuck take a screenshot of your backup server utilization and share it please, what is your backup server resources, and network speed?

Moustafa Hindawi | Blogger | Nutanix Technology Champion | NCSE | NCM-MCI | VMCE | 2xVCAP-DTM & DCV | 2xVCP-NV & CMA | LinkedIn: https://www.linkedin.com/in/moustafaadelmoustafa/ | My Blog: https://cloudwhales.co.uk/author/mhindawi/
JMeixner
Userlevel 7
Badge +17

OK, Proxy is an integral component of Veeam. You need one for nearly every job, especial when you backup a VM. Veeam installs a default proxy on your VBR server and it seems that this is used in your case.

Does your VM run directly on the NAS or do you have a single ESXi Host? I am not really sure about your environment at the moment...

Jochen (Joe) Meixner | Veeam Vanguard 2024 | Veeam Legend 2021 - 2023 | Veeam Certified Architect (VMCA) 2022 | Twitter: @JoMeix | BlueSky: @jmeixner.bsky.social
M
Userlevel 4
Badge

Our environment is pretty easily described: Gigabit switch for all physical machines. Backup server is an ESXi host connected with 2 gigabit network cards and a virtual Windows 2022 + Veeam installed. The server I backup is an ESXi host connected with 2 gigabit cards and 2 virtual Windows servers installed (2016 file server and 2012R2 exchange server). My storage is a Synology NAS connected with 2 gigabit cards and DSM operating system (no VM on it).

M
Userlevel 4
Badge

Ok, I went through some configuration of Veeam. I have two proxies added. One VM and one Agent. Both seem to work properly. What I couldnt find are the described modes for the proxies (like Hot-Add, NBD, etc.). Where do i configure those modes?

 

Moustafa_Hindawi
Userlevel 7
Badge +6

Hello @Michail 

https://helpcenter.veeam.com/docs/backup/vsphere/vmware_proxy_server.html?ver=120

Moustafa Hindawi | Blogger | Nutanix Technology Champion | NCSE | NCM-MCI | VMCE | 2xVCAP-DTM & DCV | 2xVCP-NV & CMA | LinkedIn: https://www.linkedin.com/in/moustafaadelmoustafa/ | My Blog: https://cloudwhales.co.uk/author/mhindawi/
Chris.Childerhose
Userlevel 7
Badge +20

Ok, I went through some configuration of Veeam. I have two proxies added. One VM and one Agent. Both seem to work properly. What I couldnt find are the described modes for the proxies (like Hot-Add, NBD, etc.). Where do i configure those modes?

 

When you edit the backup server in the first screenshot for VMware you should see something during the wizard to change the mode but being the Backup server as well it is not best practice.  Are you able to deploy another VM, then add that as a VMware backup Proxy to Veeam?  You can then disable the server shown above which is the Veeam server which is also best practice so it can be the controller not the proxy too.  If you need some help at all you can check out my books on the Packt website as I wrote one for v10 and second for v11 with a third coming for v12.  Check here as they should help as I discuss the Proxy server topic -

Veeam v10 - https://subscription.packtpub.com/book/cloud-and-networking/9781838980443

Veeam v11 - https://subscription.packtpub.com/book/security/9781803236810

Chris Childerhose - VMCA2022 | VMCE2023 | Veeam Vanguard 7* | Veeam Legend 3* | vExpert 5* | VCAP-DCV/VCP-DCV | Object First Ace | Cisco Champion | Twitter: @cchilderhose | Blog Site – https://just-virtualization.tech
M
Userlevel 4
Badge

I did another test backup today without changing the configuration yet. Everything ran smooth and no backup got stuck. What I reconized was the low processing rate and a cpu load of sometimes 100%. Veeam services did cause some load, but each service not more than 5 to 15%. The highest load was caused by MS antimaleware service… I will monitor the next backups and report. When a backup job gets stuck again, I will reconfigure the proxies as adviced.

One question about the proxies: Did I get that right, I have to change the transport mode from automatic selection to “virtual appliance”? And Chris, you ment that I should setup a new VM machine to be the proxy besides the backup server or did you mean i should setup a new VM proxy within Veeam? Thankes a lot so far and in advance :)

 

Chris.Childerhose
Userlevel 7
Badge +20

I did another test backup today without changing the configuration yet. Everything ran smooth and no backup got stuck. What I reconized was the low processing rate and a cpu load of sometimes 100%. Veeam services did cause some load, but each service not more than 5 to 15%. The highest load was caused by MS antimaleware service… I will monitor the next backups and report. When a backup job gets stuck again, I will reconfigure the proxies as adviced.

One question about the proxies: Did I get that right, I have to change the transport mode from automatic selection to “virtual appliance”? And Chris, you ment that I should setup a new VM machine to be the proxy besides the backup server or did you mean i should setup a new VM proxy within Veeam? Thankes a lot so far and in advance :)

 

I meant you need a new VM to be the Proxy and configure it with 8 vCPU and 32GB of RAM - that is Best Practice configuration.  Then in the console you add this new server to be a VMware Proxy and then if needed change the mode for the Proxy - it will set to Automatic and try Hot-Add/Virtual Appliance mode first then failover to NBD if needed.   You also then disable the Veeam server in the Proxy section from being a VMware Proxy that way it can manage things as it needs to be.

Also, I would put exclusions in the MS AV as per here - KB1999: Antivirus Exclusions for Veeam Backup & Replication

Chris Childerhose - VMCA2022 | VMCE2023 | Veeam Vanguard 7* | Veeam Legend 3* | vExpert 5* | VCAP-DCV/VCP-DCV | Object First Ace | Cisco Champion | Twitter: @cchilderhose | Blog Site – https://just-virtualization.tech
Moustafa_Hindawi
Userlevel 7
Badge +6

Hello @Michail 

Leave it automatically and follow Chris's instructions, plus find below av exclusion list for backup server :

https://www.veeam.com/kb1999?ad=in-text-link

Moustafa Hindawi | Blogger | Nutanix Technology Champion | NCSE | NCM-MCI | VMCE | 2xVCAP-DTM & DCV | 2xVCP-NV & CMA | LinkedIn: https://www.linkedin.com/in/moustafaadelmoustafa/ | My Blog: https://cloudwhales.co.uk/author/mhindawi/

Comment


Terms & Conditions