Veeam - Physical or Virtual that is the question?

@haslund

Yes, the BP guide recommends to use physical repository where possible. I would never recommend using a virtual repository :)

https://bp.veeam.com/vbr/2_Design_Structures/D_Veeam_Components/D_backup_repositories/

• Physical Repositories are recommended where possible (ideally combined with proxy role using backup from storage snapshots).

Physical or Virtual? 

In general, we recommend whenever possible to use physical machines as repositories, in order to maximize performance and have a clear separation between the production environment that needs to be protected and the backup storage. It is also recommended combining this with the proxy role if backup from Storage Snapshots is possible. This keeps overheads on the virtual environment and network to a minimum.

Lots of good discussion here already, maybe I can add some value. Here goes:

In security, the goal posts have always been “less secure” and “more secure”. There was a PoC less than a month ago that used SATA cables as an antenna to exfiltrate data in an air gapped environment 🤯 (Source: https://thehackernews.com/2022/07/new-air-gap-attack-uses-sata-cable-as.html).

So when designing architectures, I like to consider the way Apple tier their products “good”, “better”, “best”, except in backup designs we can certainly have “not good”, “terrible”, “resume generating event imminent”. 😆

Before the whole physical vs virtual question becomes a key question, the first question should be about the fault domain. If you’ve got a Veeam Cloud Connect environment as service provider, that’s a separate fault domain and it doesn’t warrant much discussion. As a compromise on a customer environment won’t USUALLY result in a compromise on your part. (Though on this point, if you’re using the same RMM tools to manage your platform and your customer, this is certainly possible. Source: https://www.reddit.com/r/msp/comments/cl3g23/continuum_used_to_deploy_ransomware_because/).

If we’re talking about backup infrastructure being implemented at a customer’s site, we need to create boundaries to strengthen isolation, as the probability of the fault domains (production/workloads and management/backup) overlapping increases.

Some of these exist as utilising a dedicated compute cluster with a separate vCenter and a management domain, with one directional trust such as an AD Bastion Forest. This would usually be strengthened with firewalls with IPS/IDS and restrictive controls, in these scenarios, sure virtual isn’t as much of a risk, because of the degrees of isolation. Our mission critical goal is to secure our backups, so VBR running virtually on this is fine and then we’d purchase some dedicated storage for a repository so at that design decision I’d always be advocating for a physical with hardened repo deployed vs a virtual machine accessing a SAN. I like this from a performance standpoint too as disk IO is handled via SAS or PCI-E vs iSCSI/FC, that’s more networking, more shared resources etc.

The truth is, building and maintaining such platforms takes time & expertise, something that is hard to quantify how much you need, but always easy to say you don’t have enough of. And this is where going physical shines. You don’t need a management domain to use a hardened repository. You can unplug any iLO/iDRAC and physically secure your infrastructure to prevent console access or remote control. So physical repository always makes sense.

Proxies are throwaway and whatever will get the best performance gets my vote on the physical/virtual.

As for VBR. I take the “it depends” stance. Because, how will you recover? If I’ve got a VMware environment that got destroyed, have I got another? If not what am I recovering to anyway? If I need to rebuild the entire cluster because I don’t have a DR cluster or even standalone host, my RTO is gonna suck anyway. So VBR being physical or virtual doesn’t matter. If I’ve got somewhere to recover to as a dedicated resource, I’d either be considering running VBR as a VM on that or running physically. I’d still be saying physical by default however.

Everyone decides for themselves how much risk they need to mitigate to get a good night sleep, and I see physical servers as a great step when done right, and if virtual, only in an isolated platform.

I don’t remember reading anything in the BP guide about suggesting a virtual backup repository.

Are you asking about repo? or which components?

To add my two cents, I have seen a design where the Veeam environment (VBR and repo, VONE, BEM) are in a separate vSphere cluster under a separate vCenter, using a separate shared storage infrastructure (or VSAN). This comes from the same reckoning as the architecture of management clusters for vSphere environments. The same risks about compromise of the VMware components exist, and other complications may ensue if a management domain is used (as recommended by Edwin), but this design has some interesting advantages in terms of deployment and management tools. In my day as a VMware engineer, I was taught (and held to it by my first boss) to only recommended and implement a physical vCenter server, for many of the same reasons put forth in this post and in the Forums. That “caution!” has been washed away by the sands of time, it seems.

No one size fits all, but as was stated in the forum thread, backup is all about risk management. You decide!

I think this is a classic case of “it depends”. 

I have SEVERAL clients with virtual backup servers.  The thing I don’t like about using virtual backup servers is that more often than not, it’s going to mean that I’m using a prosumer NAS for the backup repository, such as a QNAP or Synology.  No battery backed RAID controller, no controller redundancy, possible software tweaks to the software RAID that can introduce corruption, etc.

For my clients with physical servers, that means I have a Dell server in place with a battery-backed RAID card, enterprise-grade local drives with enterprise support agreements when a drive fails rather than having to figure out what the current model drive most closely matches the failed drive or figuring out if there is still a warranty from the drive manufacturer, and then finding a replacement while being down a drive for a week or more.  For those physical servers, there is the caveat of having some sort of on-host proxy server for VM access, or, my preference, using direct storage access via ISCSI or SAS/Direct Attach connections depending on what the primary storage is.  Makes for some really fast backups.  Also, if I have direct storage access, it makes it easier to segment the backup environment from the production environment.

Probably my favorite configuration is using a hybrid of the two….the backup server calling the shots resides at the recovery site and can be a VM.  The repository server at the primary site is a physical box with local disks.  The repository at the recovery site can be connected to a VM or a physical box, but again, if a VM, I’m probably using a NAS.  But I’m less concerned about corruption or slowness from a NAS at the recovery site that by copy jobs are targeting than at the primary site where the initial back data is landing.

So my preference is physical, but the sky is not falling if you use a VM, assuming that you think it through and take that into account.

Thanks for explaining.

I just thought, because the forum post was about using virtual machines as Veeam Backup repositories, you wanted discuss the same (VM as Repo) here in the community 👍

I‘m very sorry if I misinterpreted the topic of your post.

Some components can be virtual, of course. Just the repository shouldn‘t be virtual, if you want to follow our best practices. For example virtual proxies could be mandatory for some environments. 
 

Interesting topic, obviously physical for backup repository. I prefer to go on VM for VBR to enjoy the benefits of vsphere cluster (HA, DRS...). For me backup is tier 0 like AD and must be on dedicated virtualization infrastructure.

Let’s say we losted vsphere infra, well i have a backup configuration on repository or on tape. I can install temporary vbr roles on physical host with FC ports to restore mandatory objects.
I agree in this scenario you will need a physical server in spare or which is used in proxy SAN.

If you don’t have a physical server, go from scratch on vsphere infra. Be prepare to redeploy windows VM quickly with terraform or ansible code, you put outside of your infra then install vbr roles etc…

I think it's mostly a matter of prep what to do if a disaster strikes.

@JMeixner you can have tempest cable if you need or have datacenter room in faraday style :D

https://fr.wikipedia.org/wiki/TEMPEST

@dloseke & @dips on your points around SMB based setups, I had a call with Object First last week, and I honestly see that their platform could become a massive game changer for SMB/ROBO deployments especially.

I’m forseeing a scenario of:

• VBR as a VM
• Either dedicated virtual proxies or the role co-existing with VBR if the environment is small enough
• Either the virtual proxies or VBR server being the gateway server to communicate with Object First
• Object First node(s) being tier 1 backups locally, with immutability. For shorter term retention.
• Offload to public cloud object storage via vendor of preference, Wasabi/AWS/Azure/Backblaze etc, securing the longer term retention required.

A lot of SMBs won’t have dedicated DR sites or resources, they’ll likely find that the day to day recovery for non-environmental will be fulfilled by Object First, with any environmental disasters (fire/flood etc) being a restore to AWS/Azure/GCP approach until they can sort themselves out.

Well put, once physical servers reach capacity i.e RAM for example, it is hard to allocate more resources to it which means budgeting for a new physical server. On the virtual side of things, if there is some spare capacity available, it can easily be allocated. 

Yes I was just referencing the post from the forum to see what people thought but to that point repos “shouldn’t” be virtual is true but can they be virtual - yes.  I get the BP stuff and follow it to the letter most times but on that forum post was just expressing my opinion on the subject based on what I do daily that is all.   Good conversation as they say but I tend to shy away from the forums at times due to the way people can jump all over a comment made by someone expressing an opinion.

I am all good though as a good debate is always fun.  😁

​I have worked in places which have had physical Backup Servers and also Virtual Servers. As @dloseke  said, it depends. Going by my experience, in smaller organisations, there seems to be a preference for a dedicated physical backup server with a Synology or a QNAP for storage. In larger places, especially with those that have the Veeam Enterprise Plus License and storage integration, the backup server is a Virtual machine with a large number of proxies. 

This allows for quick recovery at the secondary site with the backup data already mirrored to the recovery storage array. One of the great things about Veeam is that it is possible to recover from a backup with just the Veeam B&R Console by importing the backup data. 

The other option is to have 2 Veeam instances running; one at the Primary and one at the Secondary site. So in case of failover, the data can get backed up when VMs are brought back online at the Secondary site. One less thing to worry about during a DR scenario. The secondary instance could also be connected to an Object Storage provider to recover from cloud, as required.  

​

I am truly liking the conversation here as it gives nice points from both sides which was the end goal.  Not about which is right or better, etc.  💪🏼

Now this does look interesting. Thanks for the heads up @MicoolPaul I will check them out