Skip to main content

I have a Hyper-V VM in TN. Our VBR is in TX. I set the backup job for said Hyper-V VM to use on-host proxy and store data to a repo local to the Hyper-V VM in TN.  Started the job ~1940 CST and let it run, it was stuck on 50% completion, and stuck on hard disk 1 processing (like below) for almost two hours. I ended up stopping the job, not gracefully, but immediately.

 

7:41:16 PM    Hard disk 1 (931 GB) 1024 KB read at 1 MB/s    01:49:24

 

I’m using an NFR to test this because we have a Hyper-V VM in TN that we want to bring to TX but to VMware using Instant Recovery (Instant Recovery to VMware vSphere - User Guide for VMware vSphere (veeam.com))

 

I was directed here for possible help. I have logs here https://we.tl/t-CHOg5OnEjz  (make sure to click Agree if prompted, I tested it out using WeTransfer).

 

If anyone could point me to the direction “why” the job froze and didn’t process the disk, I’d be grateful.

 

EDIT: P.S. - The second hard disk processed fine..

 

7:41:16 PM    Hard disk 2 (931 GB) 5.9 GB read at 143 MB/s    00:59

Can’t see the logs @jaceg23 . There are quite a few things which could cause holding the VM up. Can you copy/paste your backup job log (from the VBR server: C:\ProgramData\Veeam\Backup\<backup-job-name-folder>\Job.<job-name>.Backup.log). Just make sure to blur out any relevant info. Also, just be aware..we aren’t Veeam Support. We’re users just like you, but of course we’ll try and steer you in the right direction if we can.


If you are using an NFR is there any reason you cannot put the VBR server in TN not TX?  The issue based on my guess is traversing the WAN for the different components.

Also the logs if you are looking for Technical Support it is better to open a case and then post on the forums - https://forums.veeam.com

You can still create a case based on NFR licensing.


Also check here for the components and other criteria for Hyper-V backups - Backup Infrastructure for Backup - User Guide for Microsoft Hyper-V (veeam.com)


Also check here for the components and other criteria for Hyper-V backups - Backup Infrastructure for Backup - User Guide for Microsoft Hyper-V (veeam.com)

I will go through this but Hard Disk 1 was the only disk which wasn’t processed, the 2nd drive processed fine.

 

7:41:16 PM    Hard disk 2 (931 GB) 5.9 GB read at 143 MB/s    00:59


If you are using an NFR is there any reason you cannot put the VBR server in TN not TX?  The issue based on my guess is traversing the WAN for the different components.

Also the logs if you are looking for Technical Support it is better to open a case and then post on the forums - https://forums.veeam.com

You can still create a case based on NFR licensing.

We just don’t have it setup there. The Hyper-V VM is being backed up using the Agent for Windows Free.


Is there a reason for doing Agent vs VM image-based?


Can’t see the logs @jaceg23 . There are quite a few things which could cause holding the VM up. Can you copy/paste your backup job log (from the VBR server: C:\ProgramData\Veeam\Backup\<backup-job-name-folder>\Job.<job-name>.Backup.log). Just make sure to blur out any relevant info. Also, just be aware..we aren’t Veeam Support. We’re users just like you, but of course we’ll try and steer you in the right direction if we can.

There’s a 30K character limit apparently so I cannot past entire contents here. Link should have worked, I tested it in another browser and with another user. Link is https://we.tl/t-CHOg5OnEjz 


Is there a reason for doing Agent vs VM image-based?

To have a backup of the VM basically


Is there a reason for doing Agent vs VM image-based?

To have a backup of the VM basically

Sorry, that didn’t fully answer your question. At the time this was setup, we didn’t think of adding it to our VBR infrastructure. This is a system I inherited, and I’m trying to rebuild it, but I need to get this VM off Hyper-V and onto VMware like everything else.


If you are using an NFR is there any reason you cannot put the VBR server in TN not TX?  The issue based on my guess is traversing the WAN for the different components.

Also the logs if you are looking for Technical Support it is better to open a case and then post on the forums - https://forums.veeam.com

You can still create a case based on NFR licensing.

We just don’t have it setup there. The Hyper-V VM is being backed up using the Agent for Windows Free.

So, this is coming across the WAN or Internet from TN to TX.  It will take some time depending on the size of the disk.  Seems like it is probably the OS disk so that could play a part as well.  Secondary disks tend to run much faster with the Agent.


If you are using an NFR is there any reason you cannot put the VBR server in TN not TX?  The issue based on my guess is traversing the WAN for the different components.

Also the logs if you are looking for Technical Support it is better to open a case and then post on the forums - https://forums.veeam.com

You can still create a case based on NFR licensing.

We just don’t have it setup there. The Hyper-V VM is being backed up using the Agent for Windows Free.

So, this is coming across the WAN or Internet from TN to TX.  It will take some time depending on the size of the disk.  Seems like it is probably the OS disk so that could play a part as well.  Secondary disks tend to run much faster with the Agent.

This would be an additional backup on top of the agent running on the VM itself. Could the agent components be interfering with this? If the job is configured to use on-host as proxy and the repo is local to the VM, why would it traverse the WAN? At that point, the VBR server would just be the orchestrator?


If you are using an NFR is there any reason you cannot put the VBR server in TN not TX?  The issue based on my guess is traversing the WAN for the different components.

Also the logs if you are looking for Technical Support it is better to open a case and then post on the forums - https://forums.veeam.com

You can still create a case based on NFR licensing.

We just don’t have it setup there. The Hyper-V VM is being backed up using the Agent for Windows Free.

So, this is coming across the WAN or Internet from TN to TX.  It will take some time depending on the size of the disk.  Seems like it is probably the OS disk so that could play a part as well.  Secondary disks tend to run much faster with the Agent.

This would be an additional backup on top of the agent running on the VM itself. Could the agent components be interfering with this? If the job is configured to use on-host as proxy and the repo is local to the VM, why would it traverse the WAN? At that point, the VBR server would just be the orchestrator?

That is true and it should not.  VBR is the orchestrator but if you have VBR backing up the VM and Agent they could be in conflict.  This information is ideal to put in the initial posting when asking questions here as we are not technical support and try our best to help.  😁


I guess I still don’t understand why the need to use Agent vs VM image. With VM image, you have less complexity, and better central backup/restore mgmt via VBR.

That being said, Chris’s link may not be the best to use here since you’re using Agent. What you moreso need is the VAW Mgmt Guide.

https://helpcenter.veeam.com/docs/backup/agents/introduction.html?ver=120


I guess I still don’t understand why the need to use Agent vs VM image. With VM image, you have less complexity, and better central backup/restore mgmt via VBR.

That being said, Chris’s link may not be the best to use here since you’re using Agent. What you moreso need is the VAW Mgmt Guide.

https://helpcenter.veeam.com/docs/backup/agents/introduction.html?ver=120

I can try to remove the agent for this test but we need to use VBR in order to get the instant recovery to go from Hyper-V to VMware.  I’m also thinking of possibly using VMware Converter to bring this VM over to VMware-world. Only a test and time will tell.


Ah yes..that’s right. But, you don’t need to use the Agent to do that. It’s just adding more complexity than you need. Just do a backup of the VM. Simple job. Then, use this link from a Veeam SE on how to get that hyper-v backed up VM to VMware using IR

https://rhyshammond.com/converting-hyper-v-vms-to-vmware-using-veeam/

Hope that helps @jaceg23 


I guess I still don’t understand why the need to use Agent vs VM image. With VM image, you have less complexity, and better central backup/restore mgmt via VBR.

That being said, Chris’s link may not be the best to use here since you’re using Agent. What you moreso need is the VAW Mgmt Guide.

https://helpcenter.veeam.com/docs/backup/agents/introduction.html?ver=120

I can try to remove the agent for this test but we need to use VBR in order to get the instant recovery to go from Hyper-V to VMware.  I’m also thinking of possibly using VMware Converter to bring this VM over to VMware-world. Only a test and time will tell.

So then if you are using VBR to control the Agent on the VM to back it up then it will come across your WAN in some fashion.  If you have the Agent installed on the VM and send to a repository where the VBR server is, you can still do the instant recovery on that side like you want.  I think this is being made too complex and you need to go simple.


I had a look at the logs and it looks like a network problem to read the data because is traffic betweens different networks. Any network problem that happens between TN and TX would affect the backup process.

I see many errors like this one:
 

>13.02.2024 22:30:08.099] <  3716>          | ERR |Asynchronous data reader has failed.
.13.02.2024 22:30:08.099] <  3716>          | ERR |Shared memory connection was closed.
c13.02.2024 22:30:08.099] <  3716>          | >>  |--tr:Failed to read data from shared memory IO device.
 

This is always a network problem.


If you have a firewall between the localities make sure the traffic between VBR and the Hyper-V is not being analyzed by firewall filters.


I guess I still don’t understand why the need to use Agent vs VM image. With VM image, you have less complexity, and better central backup/restore mgmt via VBR.

That being said, Chris’s link may not be the best to use here since you’re using Agent. What you moreso need is the VAW Mgmt Guide.

https://helpcenter.veeam.com/docs/backup/agents/introduction.html?ver=120

I can try to remove the agent for this test but we need to use VBR in order to get the instant recovery to go from Hyper-V to VMware.  I’m also thinking of possibly using VMware Converter to bring this VM over to VMware-world. Only a test and time will tell.

So then if you are using VBR to control the Agent on the VM to back it up then it will come across your WAN in some fashion.  If you have the Agent installed on the VM and send to a repository where the VBR server is, you can still do the instant recovery on that side like you want.  I think this is being made too complex and you need to go simple.

Agent is not controlled by VBR, it’s stand alone, something that was already in place. I’ll try and remove the agent and try the job again perhaps.


I had a look at the logs and it looks like a network problem to read the data because is traffic betweens different networks. Any network problem that happens between TN and TX would affect the backup process.

I see many errors like this one:
 

>13.02.2024 22:30:08.099] <  3716>          | ERR |Asynchronous data reader has failed.
<13.02.2024 22:30:08.099] <  3716>          | ERR |Shared memory connection was closed.
l13.02.2024 22:30:08.099] <  3716>          | >>  |--tr:Failed to read data from shared memory IO device.
 

This is always a network problem.


If you have a firewall between the localities make sure the traffic between VBR and the Hyper-V is not being analyzed by firewall filters.

Any network identifiers in said logs you looked at? What were the logs you looked at and I’ll check too. 


Yeah.. I recommend doing just that. Agents don't use Proxies so hard telling why you were having problems. But, VM image backups do. I would make sure to explicitly assign your local Proxy to the job so data doesn't traverse the WAN. 


Yeah.. I recommend doing just that. Agents don't use Proxies so hard telling why you were having problems. But, VM image backups do. I would make sure to explicitly assign your local Proxy to the job so data doesn't traverse the WAN. 

The Hyper-V host was the proxy (on-host) which the VM is on. Had I used the VBR as the proxy then it would have been “off-host” and traffic would more than likely traverse the WAN. The repo is local to the Hyper-V host. After I got my initial backup, I was going to do backup copy to TX and then do instant recovery.


Yeah.. I recommend doing just that. Agents don't use Proxies so hard telling why you were having problems. But, VM image backups do. I would make sure to explicitly assign your local Proxy to the job so data doesn't traverse the WAN. 

The Hyper-V host was the proxy (on-host) which the VM is on. Had I used the VBR as the proxy then it would have been “off-host” and traffic would more than likely traverse the WAN. The repo is local to the Hyper-V host. After I got my initial backup, I was going to do backup copy to TX and then do instant recovery.

Well try without the Agent installed and see how it goes.  Let us know.


I had a look at the logs and it looks like a network problem to read the data because is traffic betweens different networks. Any network problem that happens between TN and TX would affect the backup process.

I see many errors like this one:
 

>13.02.2024 22:30:08.099] <  3716>          | ERR |Asynchronous data reader has failed.
b13.02.2024 22:30:08.099] <  3716>          | ERR |Shared memory connection was closed.
o13.02.2024 22:30:08.099] <  3716>          | >>  |--tr:Failed to read data from shared memory IO device.
 

This is always a network problem.


If you have a firewall between the localities make sure the traffic between VBR and the Hyper-V is not being analyzed by firewall filters.

Any network identifiers in said logs you looked at? What were the logs you looked at and I’ll check too. 

The logs will not show any identifier, but you can check this info on Hyper-V server, then job name, Agent logs with “target”: Agent.PIC-DC.PIC-DC.Target.log


Yeah.. I recommend doing just that. Agents don't use Proxies so hard telling why you were having problems. But, VM image backups do. I would make sure to explicitly assign your local Proxy to the job so data doesn't traverse the WAN. 

The Hyper-V host was the proxy (on-host) which the VM is on. Had I used the VBR as the proxy then it would have been “off-host” and traffic would more than likely traverse the WAN. The repo is local to the Hyper-V host. After I got my initial backup, I was going to do backup copy to TX and then do instant recovery.

You can use VBR as a off-host proxy, but you must enable Hyper-V feature on this server and it must be the same version of the Hyper-V where the VM being backup. Also it must be a physical server.


Ok, so in that case, they Hyper-V are different, the TN server is 2012R2 and the TX server is 2019. I’m still curious about the network layer.


Comment