HotAdd Transport Mode

In my experience, hotadd works as long as the proxy is in the same cluster as the vm being backed up. (In our environment all hosts in a cluster has access to the same storage resources).

I have not tested if it would work while in different clusters on a host that has access to the same storage. 

Hi ​@AndrePulia -

Yep, I did thorough testing of VM Proxies with Support when I had some performance issues with some Jobs which use that mode. That statement is incorrect. The Requirements for VM Proxies is they only need to be in the same Datacenter as the source VMs are in for the Job to use the VM Proxy.
See: https://helpcenter.veeam.com/docs/backup/vsphere/virtual_appliance.html?ver=120#requirements-for-the-virtual-appliance-mode (2nd bullet)

The BP Guide states “as close to source as possible”, not on same Host.
https://bp.veeam.com/vbr/2_Design_Structures/D_Veeam_Components/D_backup_proxies/vmware_proxies.html

Also, during my Job issues with HA, I found this KB which was great in helping me test:
https://www.veeam.com/kb1184

Hope that helps.

Best.

Nice refresher on HotAdd. Thanks.

This is the same thing that I have found is a proxy on each host in each cluster to get hotadd to work.  Otherwise it will use NBD.

Thank you all for your opinion, I agree with you all, but, since I saw the info on the course guide and the info is also mentioned here: https://techdocs.broadcom.com/us/en/vmware-cis/vsphere/vsphere-sdks-tools/8-0/virtual-disk-development-kit-programming-guide/virtual-disk-interfaces/virtual-disk-transport-methods/hotadd-transport.html  I thought to myself, did something change in v8? that is why I raised the question.

I use version 8 at home and have two proxies which do hotadd but I only have one cluster.  I will see how it works when we update our environments at work to v8.

No problem Andre...All good.

I think the User Guide bullet is accurate, and during my testing seemed to be the case.

At the very least, you can confirm with Support.

Best.

great Chris, thx!!! will be waiting !

Thank you Shane! I’ll try to get some support

Hi ​@Tommy O'Shea I agree with you, but this doubt I had is due to the fact that it is in a v8 course material and also in the link https://techdocs.broadcom.com/us/en/vmware-cis/vsphere/vsphere-sdks-tools/8-0/virtual-disk-development-kit-programming-guide/virtual-disk-interfaces/virtual-disk-transport-methods/hotadd-transport.html , could you validate if you are using  vSphere v8?,

Thank you!

Hi ​@Chris.Childerhose ,
From what I inderstood from your answer, you agree that a proxy must be installed on each host in the cluster for hotadd to happen, is my understanding correct?

Thank you!

Typically yes if you have multiple clusters. I have one so two proxies and both do hotadd.

This is not correct, you don’t have to have one proxy on every single host in the cluster. We have multiple clusters where only one proxy exists, and it processes backups via hotadd because the host it’s on has access to the same storage that the vm being backed up is on.

This has definitely been true on vsphere 7, I will validate later that it is also true for 8, since in my opinion that would be a big step backwards.

One per host is not a hard requirement but it is best practice to have one per.  Like I mentioned my cluster has four hosts and only two proxies and I get hotadd each time.

​@AndrePulia -

The Veeam User Guide info I shared is the most accurate (almost) practice. As I shared earlier...1 VM Proxy per Datacenter is what it states, which is incorrect. The correct statement should be 1 Proxy per Cluster. Where a Proxy per Host comes into play is when VMs are stored on an ESXi Local Storage.

Best.

I know conversation has gone on a bit here, but a few clarifying points:

1. Veeam doesn’t use VDDK to read/write with hotadd, native OS calls are used.
2. Hotadd from VDDK is used to _mount_ the disk in VMware, but reads are done via native OS calls
3. For hotadd to work, proxy must be*:
   1. In same DataCenter as the VMs you want to backup
   2. Be able to access vCenter and hosts of Vms you want to backup
   3. Be able to access storage of hosts/VMs you want to backup

* All of these points must be true; access for hotadding disks goes through vCenter/ESXi, and the vCenter/hosts must be able to communicate. The proxy simply having access to the shared storage is not enough as if the proxy isn’t in same data center, VDDK won’t know how to complete the call (you need to pass host id (moref), vm id (moref), and snapshot id to VDDK for virtually all calls -- if proxy and guest VM being backed up are not in same data-center, we have no way to supply these arguments to VDDK)

Only if you use VSAN do you need one host per node -- for normal VMFS, spreading the workload out across hosts/datastores makes sense, but this should be driven based on testing in your environment. There are some situations however (load balancing and network traffic out of hosts namely) where you might want more intelligent hotadd proxy placement; hotadd proxies should be light and designed to scale horizontally (read: more proxies instead of one big hotadd proxy) so don’t hesitate to destruct and deploy fresh hotadd proxies assuming your change management control allows for this.

The read part from point 1 is important as it means not all advices from VMWare will apply for hotadd as at a certain point we’re no longer doing VMware hotadd by the books.

Hello,

from by veeam kb and by vmware kb, on each host has to placed one proxy server, because speed is maximal and latency is minimal. What depends is datastore. If you are using one datastore, for example vSAN, you can useless proxy servers than esx hosts and you will get hotadd method for backup. This is valid usecase within cluster. So if you have multiple cluster, only proxy servers from that cluster could use hotadd method.

We have design with half of proxy servers than esx hosts with vSAN and with multiple clusters. So backed up VM on esx1 is backed up by proxy server on esx2 with hotadd method. We don't see any performance degradation of backups.

So if you are suing one stretched datastore between multiple locations whitin cluster, you can get hotadd method, even proxy is on another DC as backed up VM. 

It is important to note the distinction between required and recommended. At a minimum, it just requires 1 proxy on a single host in the cluster. That will allow hotadd to be used.

However to optimize hotadd in an cluster with multiple hosts, it certainly would be a good idea to have one proxy per host. You should still follow the Veeam best practices to determine how many proxy tasks you need for your workload in that cluster, and divide it equally amongst the all the hosts/proxies. the downside of this is more systems to manages, and unless you use linux proxies, more license to pay for.

The upside would be that you can take advantage of each hosts network stack more fully, whereas a single proxy on a single host could quickly see a bottleneck on the network, especially if you’ve only got a 1G uplink. 

With more proxies, you also avoid a single point of failure in the cluster.

Exactly → “required” vs “recommended”. Use the User Guide for the starting point, then go from there.

​@Tommy O'Shea I completely agree with you, however, the information I provided is from two different sources from VMware, which is why I am still talking about this topic.
Thank you Tommy, I'll wait for your tests.

Ok, I just checked. I have a cluster with multiple hosts on VMware ESXi, 8.0.3.

There are 3 proxies in that cluster (Anti-affinity rule in place to keep the proxies on separate hosts, for reasons mentioned in previous comments).

I checked the backup of a single VM with multiple disks:

As you can see, each of these proxies that were not on the same hosts as the VM being backed up were able to hotadd the disk.
So those Broadcom sources are not correct about them having to be on the same host, even on v8.

I’m pretty sure the Broadcom article is specifically talking about linked clones , I’m honestly not sure why they bother with that as that’s for a very specific configuration in VMware. I would just ignore it, hotadd can work across hosts, it’s about access to the hosts/vcenter and proxy being in same datacenter.