• EN
Solved

Hyper-V Backup Job Issue

J
Userlevel 5
Badge

I have a Hyper-V VM in TN. Our VBR is in TX. I set the backup job for said Hyper-V VM to use on-host proxy and store data to a repo local to the Hyper-V VM in TN.  Started the job ~1940 CST and let it run, it was stuck on 50% completion, and stuck on hard disk 1 processing (like below) for almost two hours. I ended up stopping the job, not gracefully, but immediately.

 

7:41:16 PM    Hard disk 1 (931 GB) 1024 KB read at 1 MB/s    01:49:24

 

I’m using an NFR to test this because we have a Hyper-V VM in TN that we want to bring to TX but to VMware using Instant Recovery (Instant Recovery to VMware vSphere - User Guide for VMware vSphere (veeam.com))

 

I was directed here for possible help. I have logs here https://we.tl/t-CHOg5OnEjz  (make sure to click Agree if prompted, I tested it out using WeTransfer).

 

If anyone could point me to the direction “why” the job froze and didn’t process the disk, I’d be grateful.

 

EDIT: P.S. - The second hard disk processed fine..

 

7:41:16 PM    Hard disk 2 (931 GB) 5.9 GB read at 143 MB/s    00:59

icon

Best answer by jaceg23 19 February 2024, 21:52

View original

29 comments

Chris.Childerhose
Userlevel 7
Badge +20

Very interesting solution and glad you solved it.

Chris Childerhose - VMCA2022 | VMCE2023 | Veeam Vanguard 7* | Veeam Legend 3* | vExpert 5* | VCAP-DCV/VCP-DCV | Object First Ace | Cisco Champion | Twitter: @cchilderhose | Blog Site – https://just-virtualization.tech
coolsport00
Userlevel 7
Badge +17

Great idea @jaceg23 . Glad you got it sorted.

 

Shane Williford - Veeam VMCA/VMCE | Veeam Legend | VUG Leader | VCP-DCV | Twitter: @coolsport00
J
Userlevel 5
Badge

We ended up not doing this, I had this idea instead, we brought up a small VM, migrated the SQL express DBs over to it, now it’s on VMware, and not on Hyper-V. Case closed.

coolsport00
Userlevel 7
Badge +17

"The Hyper-V host was the proxy (on-host) which the VM is on. Had I used the VBR as the proxy then it would have been “off-host” and traffic would more than likely traverse the WAN. The repo is local to the Hyper-V host. After I got my initial backup, I was going to do backup copy to TX and then do instant recovery." <- I understand that, thus why (probably) your Agent job was traversing the WAN. My point was using a VM-based job with your on-host setup wouldn't 😊 My guess is you should then be good to go. Keep us posted. 

Shane Williford - Veeam VMCA/VMCE | Veeam Legend | VUG Leader | VCP-DCV | Twitter: @coolsport00
J
Userlevel 5
Badge

Ok, so in that case, they Hyper-V are different, the TN server is 2012R2 and the TX server is 2019. I’m still curious about the network layer.

wesmrt
Userlevel 7
Badge +4

Yeah.. I recommend doing just that. Agents don't use Proxies so hard telling why you were having problems. But, VM image backups do. I would make sure to explicitly assign your local Proxy to the job so data doesn't traverse the WAN. 

The Hyper-V host was the proxy (on-host) which the VM is on. Had I used the VBR as the proxy then it would have been “off-host” and traffic would more than likely traverse the WAN. The repo is local to the Hyper-V host. After I got my initial backup, I was going to do backup copy to TX and then do instant recovery.

You can use VBR as a off-host proxy, but you must enable Hyper-V feature on this server and it must be the same version of the Hyper-V where the VM being backup. Also it must be a physical server.

Wesley Martins @ Specialist Enginner at Veeam Technical Support | My blog: https://itproland.com.br (PT-BR)
wesmrt
Userlevel 7
Badge +4

I had a look at the logs and it looks like a network problem to read the data because is traffic betweens different networks. Any network problem that happens between TN and TX would affect the backup process.

I see many errors like this one:
 

[13.02.2024 22:30:08.099] <  3716>          | ERR |Asynchronous data reader has failed.
[13.02.2024 22:30:08.099] <  3716>          | ERR |Shared memory connection was closed.
[13.02.2024 22:30:08.099] <  3716>          | >>  |--tr:Failed to read data from shared memory IO device.
 

This is always a network problem.


If you have a firewall between the localities make sure the traffic between VBR and the Hyper-V is not being analyzed by firewall filters.

Any network identifiers in said logs you looked at? What were the logs you looked at and I’ll check too. 

The logs will not show any identifier, but you can check this info on Hyper-V server, then job name, Agent logs with “target”: Agent.PIC-DC.PIC-DC.Target.log

Wesley Martins @ Specialist Enginner at Veeam Technical Support | My blog: https://itproland.com.br (PT-BR)
Chris.Childerhose
Userlevel 7
Badge +20

Yeah.. I recommend doing just that. Agents don't use Proxies so hard telling why you were having problems. But, VM image backups do. I would make sure to explicitly assign your local Proxy to the job so data doesn't traverse the WAN. 

The Hyper-V host was the proxy (on-host) which the VM is on. Had I used the VBR as the proxy then it would have been “off-host” and traffic would more than likely traverse the WAN. The repo is local to the Hyper-V host. After I got my initial backup, I was going to do backup copy to TX and then do instant recovery.

Well try without the Agent installed and see how it goes.  Let us know.

Chris Childerhose - VMCA2022 | VMCE2023 | Veeam Vanguard 7* | Veeam Legend 3* | vExpert 5* | VCAP-DCV/VCP-DCV | Object First Ace | Cisco Champion | Twitter: @cchilderhose | Blog Site – https://just-virtualization.tech
J
Userlevel 5
Badge

Yeah.. I recommend doing just that. Agents don't use Proxies so hard telling why you were having problems. But, VM image backups do. I would make sure to explicitly assign your local Proxy to the job so data doesn't traverse the WAN. 

The Hyper-V host was the proxy (on-host) which the VM is on. Had I used the VBR as the proxy then it would have been “off-host” and traffic would more than likely traverse the WAN. The repo is local to the Hyper-V host. After I got my initial backup, I was going to do backup copy to TX and then do instant recovery.

coolsport00
Userlevel 7
Badge +17

Yeah.. I recommend doing just that. Agents don't use Proxies so hard telling why you were having problems. But, VM image backups do. I would make sure to explicitly assign your local Proxy to the job so data doesn't traverse the WAN. 

Shane Williford - Veeam VMCA/VMCE | Veeam Legend | VUG Leader | VCP-DCV | Twitter: @coolsport00
J
Userlevel 5
Badge

I had a look at the logs and it looks like a network problem to read the data because is traffic betweens different networks. Any network problem that happens between TN and TX would affect the backup process.

I see many errors like this one:
 

[13.02.2024 22:30:08.099] <  3716>          | ERR |Asynchronous data reader has failed.
[13.02.2024 22:30:08.099] <  3716>          | ERR |Shared memory connection was closed.
[13.02.2024 22:30:08.099] <  3716>          | >>  |--tr:Failed to read data from shared memory IO device.
 

This is always a network problem.


If you have a firewall between the localities make sure the traffic between VBR and the Hyper-V is not being analyzed by firewall filters.

Any network identifiers in said logs you looked at? What were the logs you looked at and I’ll check too. 

J
Userlevel 5
Badge

I guess I still don’t understand why the need to use Agent vs VM image. With VM image, you have less complexity, and better central backup/restore mgmt via VBR.

That being said, Chris’s link may not be the best to use here since you’re using Agent. What you moreso need is the VAW Mgmt Guide.

https://helpcenter.veeam.com/docs/backup/agents/introduction.html?ver=120

I can try to remove the agent for this test but we need to use VBR in order to get the instant recovery to go from Hyper-V to VMware.  I’m also thinking of possibly using VMware Converter to bring this VM over to VMware-world. Only a test and time will tell.

So then if you are using VBR to control the Agent on the VM to back it up then it will come across your WAN in some fashion.  If you have the Agent installed on the VM and send to a repository where the VBR server is, you can still do the instant recovery on that side like you want.  I think this is being made too complex and you need to go simple.

Agent is not controlled by VBR, it’s stand alone, something that was already in place. I’ll try and remove the agent and try the job again perhaps.

wesmrt
Userlevel 7
Badge +4

I had a look at the logs and it looks like a network problem to read the data because is traffic betweens different networks. Any network problem that happens between TN and TX would affect the backup process.

I see many errors like this one:
 

[13.02.2024 22:30:08.099] <  3716>          | ERR |Asynchronous data reader has failed.
[13.02.2024 22:30:08.099] <  3716>          | ERR |Shared memory connection was closed.
[13.02.2024 22:30:08.099] <  3716>          | >>  |--tr:Failed to read data from shared memory IO device.
 

This is always a network problem.


If you have a firewall between the localities make sure the traffic between VBR and the Hyper-V is not being analyzed by firewall filters.

Wesley Martins @ Specialist Enginner at Veeam Technical Support | My blog: https://itproland.com.br (PT-BR)
Chris.Childerhose
Userlevel 7
Badge +20

I guess I still don’t understand why the need to use Agent vs VM image. With VM image, you have less complexity, and better central backup/restore mgmt via VBR.

That being said, Chris’s link may not be the best to use here since you’re using Agent. What you moreso need is the VAW Mgmt Guide.

https://helpcenter.veeam.com/docs/backup/agents/introduction.html?ver=120

I can try to remove the agent for this test but we need to use VBR in order to get the instant recovery to go from Hyper-V to VMware.  I’m also thinking of possibly using VMware Converter to bring this VM over to VMware-world. Only a test and time will tell.

So then if you are using VBR to control the Agent on the VM to back it up then it will come across your WAN in some fashion.  If you have the Agent installed on the VM and send to a repository where the VBR server is, you can still do the instant recovery on that side like you want.  I think this is being made too complex and you need to go simple.

Chris Childerhose - VMCA2022 | VMCE2023 | Veeam Vanguard 7* | Veeam Legend 3* | vExpert 5* | VCAP-DCV/VCP-DCV | Object First Ace | Cisco Champion | Twitter: @cchilderhose | Blog Site – https://just-virtualization.tech
coolsport00
Userlevel 7
Badge +17

Ah yes..that’s right. But, you don’t need to use the Agent to do that. It’s just adding more complexity than you need. Just do a backup of the VM. Simple job. Then, use this link from a Veeam SE on how to get that hyper-v backed up VM to VMware using IR

https://rhyshammond.com/converting-hyper-v-vms-to-vmware-using-veeam/

Hope that helps @jaceg23 

Shane Williford - Veeam VMCA/VMCE | Veeam Legend | VUG Leader | VCP-DCV | Twitter: @coolsport00
J
Userlevel 5
Badge

I guess I still don’t understand why the need to use Agent vs VM image. With VM image, you have less complexity, and better central backup/restore mgmt via VBR.

That being said, Chris’s link may not be the best to use here since you’re using Agent. What you moreso need is the VAW Mgmt Guide.

https://helpcenter.veeam.com/docs/backup/agents/introduction.html?ver=120

I can try to remove the agent for this test but we need to use VBR in order to get the instant recovery to go from Hyper-V to VMware.  I’m also thinking of possibly using VMware Converter to bring this VM over to VMware-world. Only a test and time will tell.

coolsport00
Userlevel 7
Badge +17

I guess I still don’t understand why the need to use Agent vs VM image. With VM image, you have less complexity, and better central backup/restore mgmt via VBR.

That being said, Chris’s link may not be the best to use here since you’re using Agent. What you moreso need is the VAW Mgmt Guide.

https://helpcenter.veeam.com/docs/backup/agents/introduction.html?ver=120

Shane Williford - Veeam VMCA/VMCE | Veeam Legend | VUG Leader | VCP-DCV | Twitter: @coolsport00
Chris.Childerhose
Userlevel 7
Badge +20

If you are using an NFR is there any reason you cannot put the VBR server in TN not TX?  The issue based on my guess is traversing the WAN for the different components.

Also the logs if you are looking for Technical Support it is better to open a case and then post on the forums - https://forums.veeam.com

You can still create a case based on NFR licensing.

We just don’t have it setup there. The Hyper-V VM is being backed up using the Agent for Windows Free.

So, this is coming across the WAN or Internet from TN to TX.  It will take some time depending on the size of the disk.  Seems like it is probably the OS disk so that could play a part as well.  Secondary disks tend to run much faster with the Agent.

This would be an additional backup on top of the agent running on the VM itself. Could the agent components be interfering with this? If the job is configured to use on-host as proxy and the repo is local to the VM, why would it traverse the WAN? At that point, the VBR server would just be the orchestrator?

That is true and it should not.  VBR is the orchestrator but if you have VBR backing up the VM and Agent they could be in conflict.  This information is ideal to put in the initial posting when asking questions here as we are not technical support and try our best to help.  😁

Chris Childerhose - VMCA2022 | VMCE2023 | Veeam Vanguard 7* | Veeam Legend 3* | vExpert 5* | VCAP-DCV/VCP-DCV | Object First Ace | Cisco Champion | Twitter: @cchilderhose | Blog Site – https://just-virtualization.tech
J
Userlevel 5
Badge

If you are using an NFR is there any reason you cannot put the VBR server in TN not TX?  The issue based on my guess is traversing the WAN for the different components.

Also the logs if you are looking for Technical Support it is better to open a case and then post on the forums - https://forums.veeam.com

You can still create a case based on NFR licensing.

We just don’t have it setup there. The Hyper-V VM is being backed up using the Agent for Windows Free.

So, this is coming across the WAN or Internet from TN to TX.  It will take some time depending on the size of the disk.  Seems like it is probably the OS disk so that could play a part as well.  Secondary disks tend to run much faster with the Agent.

This would be an additional backup on top of the agent running on the VM itself. Could the agent components be interfering with this? If the job is configured to use on-host as proxy and the repo is local to the VM, why would it traverse the WAN? At that point, the VBR server would just be the orchestrator?

Chris.Childerhose
Userlevel 7
Badge +20

If you are using an NFR is there any reason you cannot put the VBR server in TN not TX?  The issue based on my guess is traversing the WAN for the different components.

Also the logs if you are looking for Technical Support it is better to open a case and then post on the forums - https://forums.veeam.com

You can still create a case based on NFR licensing.

We just don’t have it setup there. The Hyper-V VM is being backed up using the Agent for Windows Free.

So, this is coming across the WAN or Internet from TN to TX.  It will take some time depending on the size of the disk.  Seems like it is probably the OS disk so that could play a part as well.  Secondary disks tend to run much faster with the Agent.

Chris Childerhose - VMCA2022 | VMCE2023 | Veeam Vanguard 7* | Veeam Legend 3* | vExpert 5* | VCAP-DCV/VCP-DCV | Object First Ace | Cisco Champion | Twitter: @cchilderhose | Blog Site – https://just-virtualization.tech
J
Userlevel 5
Badge

Is there a reason for doing Agent vs VM image-based?

To have a backup of the VM basically

Sorry, that didn’t fully answer your question. At the time this was setup, we didn’t think of adding it to our VBR infrastructure. This is a system I inherited, and I’m trying to rebuild it, but I need to get this VM off Hyper-V and onto VMware like everything else.

J
Userlevel 5
Badge

Is there a reason for doing Agent vs VM image-based?

To have a backup of the VM basically

J
Userlevel 5
Badge

Can’t see the logs @jaceg23 . There are quite a few things which could cause holding the VM up. Can you copy/paste your backup job log (from the VBR server: C:\ProgramData\Veeam\Backup\<backup-job-name-folder>\Job.<job-name>.Backup.log). Just make sure to blur out any relevant info. Also, just be aware..we aren’t Veeam Support. We’re users just like you, but of course we’ll try and steer you in the right direction if we can.

There’s a 30K character limit apparently so I cannot past entire contents here. Link should have worked, I tested it in another browser and with another user. Link is https://we.tl/t-CHOg5OnEjz 

coolsport00
Userlevel 7
Badge +17

Is there a reason for doing Agent vs VM image-based?

Shane Williford - Veeam VMCA/VMCE | Veeam Legend | VUG Leader | VCP-DCV | Twitter: @coolsport00
J
Userlevel 5
Badge

If you are using an NFR is there any reason you cannot put the VBR server in TN not TX?  The issue based on my guess is traversing the WAN for the different components.

Also the logs if you are looking for Technical Support it is better to open a case and then post on the forums - https://forums.veeam.com

You can still create a case based on NFR licensing.

We just don’t have it setup there. The Hyper-V VM is being backed up using the Agent for Windows Free.

Comment


Terms & Conditions