Veeam - Physical or Virtual that is the question?

I don’t remember reading anything in the BP guide about suggesting a virtual backup repository.

Are you asking about repo? or which components?

I was referring to all components.  I know what the guide says but just wanted to see what general opinion was from the community.  I know what those working at Veeam will say.  😉😋

Yes I was just referencing the post from the forum to see what people thought but to that point repos “shouldn’t” be virtual is true but can they be virtual - yes.  I get the BP stuff and follow it to the letter most times but on that forum post was just expressing my opinion on the subject based on what I do daily that is all.   Good conversation as they say but I tend to shy away from the forums at times due to the way people can jump all over a comment made by someone expressing an opinion.

I am all good though as a good debate is always fun.  😁

Great points Joe.  Now in our case the Veeam virtual infrastructure is separate from anything client related as we have our own management stack and cluster that is used whereas the clients use vCloud and have clusters allocated there.  So we do ensure security between things.

I see things from both sides and sometimes we need to do what the company requires even though we are the experts on the subject and follow best practices. 😂

Lots of good discussion here already, maybe I can add some value. Here goes:

In security, the goal posts have always been “less secure” and “more secure”. There was a PoC less than a month ago that used SATA cables as an antenna to exfiltrate data in an air gapped environment 🤯 (Source: https://thehackernews.com/2022/07/new-air-gap-attack-uses-sata-cable-as.html).

So when designing architectures, I like to consider the way Apple tier their products “good”, “better”, “best”, except in backup designs we can certainly have “not good”, “terrible”, “resume generating event imminent”. 😆

Before the whole physical vs virtual question becomes a key question, the first question should be about the fault domain. If you’ve got a Veeam Cloud Connect environment as service provider, that’s a separate fault domain and it doesn’t warrant much discussion. As a compromise on a customer environment won’t USUALLY result in a compromise on your part. (Though on this point, if you’re using the same RMM tools to manage your platform and your customer, this is certainly possible. Source: https://www.reddit.com/r/msp/comments/cl3g23/continuum_used_to_deploy_ransomware_because/).

If we’re talking about backup infrastructure being implemented at a customer’s site, we need to create boundaries to strengthen isolation, as the probability of the fault domains (production/workloads and management/backup) overlapping increases.

Some of these exist as utilising a dedicated compute cluster with a separate vCenter and a management domain, with one directional trust such as an AD Bastion Forest. This would usually be strengthened with firewalls with IPS/IDS and restrictive controls, in these scenarios, sure virtual isn’t as much of a risk, because of the degrees of isolation. Our mission critical goal is to secure our backups, so VBR running virtually on this is fine and then we’d purchase some dedicated storage for a repository so at that design decision I’d always be advocating for a physical with hardened repo deployed vs a virtual machine accessing a SAN. I like this from a performance standpoint too as disk IO is handled via SAS or PCI-E vs iSCSI/FC, that’s more networking, more shared resources etc.

The truth is, building and maintaining such platforms takes time & expertise, something that is hard to quantify how much you need, but always easy to say you don’t have enough of. And this is where going physical shines. You don’t need a management domain to use a hardened repository. You can unplug any iLO/iDRAC and physically secure your infrastructure to prevent console access or remote control. So physical repository always makes sense.

Proxies are throwaway and whatever will get the best performance gets my vote on the physical/virtual.

As for VBR. I take the “it depends” stance. Because, how will you recover? If I’ve got a VMware environment that got destroyed, have I got another? If not what am I recovering to anyway? If I need to rebuild the entire cluster because I don’t have a DR cluster or even standalone host, my RTO is gonna suck anyway. So VBR being physical or virtual doesn’t matter. If I’ve got somewhere to recover to as a dedicated resource, I’d either be considering running VBR as a VM on that or running physically. I’d still be saying physical by default however.

Everyone decides for themselves how much risk they need to mitigate to get a good night sleep, and I see physical servers as a great step when done right, and if virtual, only in an isolated platform.

Lots of great answers already 👍

In my opinion it depends on the requirements, the desired security and the implemented storage technology. You can achieve almost anything with anything if it's well thought.

• Are you using a separate virtual environment (with separate permissions)?
• Can you secure a virtual environment as much as a standalone physical system?
• How will you recover im case your virtual infrastructure goes down? Especially will you be able to meet your target RTO?
• Do you use immutable or offline storage?
• ...

Especially in smaller environments I like to use a dedicated physical server as it makes both securing and restoring much easier. In bigger environments with different locations, it's can make sense to virtualize the VBR server role; especially if you replicate it or maintain a cold standby instance.

With a decent immutable/offline solution I could think of going all virtual. The only requirement would be to use seperate virtual ressources; like not creating primary backups on production storage.

Interesting link. Thank you, Bertrand

I think this is a classic case of “it depends”. 

I have SEVERAL clients with virtual backup servers.  The thing I don’t like about using virtual backup servers is that more often than not, it’s going to mean that I’m using a prosumer NAS for the backup repository, such as a QNAP or Synology.  No battery backed RAID controller, no controller redundancy, possible software tweaks to the software RAID that can introduce corruption, etc.

For my clients with physical servers, that means I have a Dell server in place with a battery-backed RAID card, enterprise-grade local drives with enterprise support agreements when a drive fails rather than having to figure out what the current model drive most closely matches the failed drive or figuring out if there is still a warranty from the drive manufacturer, and then finding a replacement while being down a drive for a week or more.  For those physical servers, there is the caveat of having some sort of on-host proxy server for VM access, or, my preference, using direct storage access via ISCSI or SAS/Direct Attach connections depending on what the primary storage is.  Makes for some really fast backups.  Also, if I have direct storage access, it makes it easier to segment the backup environment from the production environment.

Probably my favorite configuration is using a hybrid of the two….the backup server calling the shots resides at the recovery site and can be a VM.  The repository server at the primary site is a physical box with local disks.  The repository at the recovery site can be connected to a VM or a physical box, but again, if a VM, I’m probably using a NAS.  But I’m less concerned about corruption or slowness from a NAS at the recovery site that by copy jobs are targeting than at the primary site where the initial back data is landing.

So my preference is physical, but the sky is not falling if you use a VM, assuming that you think it through and take that into account.

Yeah, I’ve noticed that as well except I usually see a virtual backup server if a QNAP or Synology NAS (or similar, I’ve seen NetGear ReadyNAS and AsuSTOR as well) is in use for the backing storage.  Physical backup servers with local storage can vary in size for sure and in smaller environments as a smaller repo can easily be put on a physical server, up to nearly 250TB depending on disk sizing and RAID type. For instance, on my larger physical boxes, I will use a PowerEdge T640 (in a rack configuration if a rack is available) where I can cram 18 3.5” drives….and with drives reaching 12/14/16TB, in a RAID 6, that nets around 230-260TB.  That’s not bad.  On the flip side, I’ve built a T340 with 4 local disks in a RAID 5 and had something like 12TB before as well for the more ROBO side of things.  It just depends on use case.

It seems that once you get much larger than that, it’s not as feasible to use a physical server...you either need to have more than one or you start looking at more of your enterprise-style storage such as using an actual SAN or enterprise NAS or deduplicating appliances or maybe something slightly less such as a large SMB NAS like a Synology with Expansion drawers, etc.  I live in the Dell world, so something like Isilon (PowerScale) or DataDomain, or in the HPE world, StoreOnce might be more common.  But performance suffers on the larger deduplicating appliance in my experience, and when you’re at that scale, you probably need more performance so that your backups aren’t piling on top of each other.

@dloseke & @dips on your points around SMB based setups, I had a call with Object First last week, and I honestly see that their platform could become a massive game changer for SMB/ROBO deployments especially.

I’m forseeing a scenario of:

• VBR as a VM
• Either dedicated virtual proxies or the role co-existing with VBR if the environment is small enough
• Either the virtual proxies or VBR server being the gateway server to communicate with Object First
• Object First node(s) being tier 1 backups locally, with immutability. For shorter term retention.
• Offload to public cloud object storage via vendor of preference, Wasabi/AWS/Azure/Backblaze etc, securing the longer term retention required.

A lot of SMBs won’t have dedicated DR sites or resources, they’ll likely find that the day to day recovery for non-environmental will be fulfilled by Object First, with any environmental disasters (fire/flood etc) being a restore to AWS/Azure/GCP approach until they can sort themselves out.

I’m very interested in seeing what Object First is doing…...the main issue I tend to have is that I want to have VBR running on a different environment than the performance workload, but want the data to reside on a separate server as well, and want to use Hardened Linux Immutability, which means if I’m using the same server, I’m looking at VM’s, so I need to either store the data in a VMDK which is not advised, or I need to use some sort of RDM which tends to be unsupported on local storage but can be cobbled together.  I’m trying to figure out how Object First would fit into that picture and while it doesn’t fix all of my concerns, it sure helps!

Now this does look interesting. Thanks for the heads up @MicoolPaul I will check them out

Well put, once physical servers reach capacity i.e RAM for example, it is hard to allocate more resources to it which means budgeting for a new physical server. On the virtual side of things, if there is some spare capacity available, it can easily be allocated.