Veeam - Physical or Virtual that is the question?


Userlevel 7
Badge +7

I wanted to ask the community on here to get a discussion going about whether using VMs for Veeam is good, bad or indifferent?  It is based on a question from the forum here - VM as repo - Veeam R&D Forums

I added my thoughts to this post as I disagree with some of the comments about having to use physical servers as the BP site mentions both physical and virtual.  I also work at an MSP and everything we build is virtual for Veeam other than a few tape servers we have which are physical boxes since they perform better this way with direct connect to the FC fabric.

Anyway just wanted to see the communities opinions here and not to put one or the other down just thoughts and what everyone is doing now.

And go…….  🤣


15 comments

Userlevel 7
Badge +5

I don’t remember reading anything in the BP guide about suggesting a virtual backup repository.

 

Are you asking about repo? or which components?

Userlevel 7
Badge +4

@haslund

Yes, the BP guide recommends to use physical repository where possible. I would never recommend using a virtual repository :)

 

https://bp.veeam.com/vbr/2_Design_Structures/D_Veeam_Components/D_backup_repositories/

 

  • Physical Repositories are recommended where possible (ideally combined with proxy role using backup from storage snapshots).

 

Physical or Virtual? 

In general, we recommend whenever possible to use physical machines as repositories, in order to maximize performance and have a clear separation between the production environment that needs to be protected and the backup storage. It is also recommended combining this with the proxy role if backup from Storage Snapshots is possible. This keeps overheads on the virtual environment and network to a minimum.

 

 

Userlevel 7
Badge +7

I was referring to all components.  I know what the guide says but just wanted to see what general opinion was from the community.  I know what those working at Veeam will say.  😉😋

Userlevel 7
Badge +4

Thanks for explaining.

I just thought, because the forum post was about using virtual machines as Veeam Backup repositories, you wanted discuss the same (VM as Repo) here in the community 👍

I‘m very sorry if I misinterpreted the topic of your post.

 

Some components can be virtual, of course. Just the repository shouldn‘t be virtual, if you want to follow our best practices. For example virtual proxies could be mandatory for some environments. 
 

Userlevel 7
Badge +7

Thanks for explaining.

I just thought, because the forum post was about using virtual machines as Veeam Backup repositories, that you wanted discuss the same here in the community 👍

I‘m very sorry if I misinterpreted the topic of your post.

 

Some components can be virtual, of course. Just the repository shouldn‘t be virtual, if you want to follow our best practices. For example virtual proxies could be mandatory for some environments. 
 

Yes I was just referencing the post from the forum to see what people thought but to that point repos “shouldn’t” be virtual is true but can they be virtual - yes.  I get the BP stuff and follow it to the letter most times but on that forum post was just expressing my opinion on the subject based on what I do daily that is all.   Good conversation as they say but I tend to shy away from the forums at times due to the way people can jump all over a comment made by someone expressing an opinion.

I am all good though as a good debate is always fun.  😁

Userlevel 7
Badge +7

In a normal enterprise production environment I prefer physical VBR servers and Repositories  to separate the backup environment from the virtualization clusters. I separate as much as I can.

If there is an attack on this environment and the vSphere Cluster is hacked tje backup environment should stay save.

 

As an MSP where my complete environment is for backup data from customers  I think I would use virtual components, too. Then I have to be able to scale my Veeam environment if there are new customers. But I think I would use physical Storage Systems or Servers for my repositories, too. I don’t want to get them deleted vie the vCenter or similar management parts.

 

Userlevel 7
Badge +7

In a normal enterprise production environment I prefer physical VBR servers and Repositories  to separate the backup environment from the virtualization clusters. I separate as much as I can.

If there is an attack on this environment and the vSphere Cluster is hacked tje backup environment should stay save.

 

As an MSP where my complete environment is for backup data from customers  I think I would use virtual components, too. Then I have to be able to scale my Veeam environment if there are new customers. But I think I would use physical Storage Systems or Servers for my repositories, too. I don’t want to get them deleted vie the vCenter or similar management parts.

 

Great points Joe.  Now in our case the Veeam virtual infrastructure is separate from anything client related as we have our own management stack and cluster that is used whereas the clients use vCloud and have clusters allocated there.  So we do ensure security between things.

I see things from both sides and sometimes we need to do what the company requires even though we are the experts on the subject and follow best practices. 😂

Userlevel 4
Badge

To add my two cents, I have seen a design where the Veeam environment (VBR and repo, VONE, BEM) are in a separate vSphere cluster under a separate vCenter, using a separate shared storage infrastructure (or VSAN). This comes from the same reckoning as the architecture of management clusters for vSphere environments. The same risks about compromise of the VMware components exist, and other complications may ensue if a management domain is used (as recommended by Edwin), but this design has some interesting advantages in terms of deployment and management tools. In my day as a VMware engineer, I was taught (and held to it by my first boss) to only recommended and implement a physical vCenter server, for many of the same reasons put forth in this post and in the Forums. That “caution!” has been washed away by the sands of time, it seems.

No one size fits all, but as was stated in the forum thread, backup is all about risk management. You decide!

Userlevel 7
Badge +8

Lots of good discussion here already, maybe I can add some value. Here goes:

 

In security, the goal posts have always been “less secure” and “more secure”. There was a PoC less than a month ago that used SATA cables as an antenna to exfiltrate data in an air gapped environment 🤯 (Source: https://thehackernews.com/2022/07/new-air-gap-attack-uses-sata-cable-as.html).

 

So when designing architectures, I like to consider the way Apple tier their products “good”, “better”, “best”, except in backup designs we can certainly have “not good”, “terrible”, “resume generating event imminent”. 😆

 

Before the whole physical vs virtual question becomes a key question, the first question should be about the fault domain. If you’ve got a Veeam Cloud Connect environment as service provider, that’s a separate fault domain and it doesn’t warrant much discussion. As a compromise on a customer environment won’t USUALLY result in a compromise on your part. (Though on this point, if you’re using the same RMM tools to manage your platform and your customer, this is certainly possible. Source: https://www.reddit.com/r/msp/comments/cl3g23/continuum_used_to_deploy_ransomware_because/).

 

If we’re talking about backup infrastructure being implemented at a customer’s site, we need to create boundaries to strengthen isolation, as the probability of the fault domains (production/workloads and management/backup) overlapping increases.

 

Some of these exist as utilising a dedicated compute cluster with a separate vCenter and a management domain, with one directional trust such as an AD Bastion Forest. This would usually be strengthened with firewalls with IPS/IDS and restrictive controls, in these scenarios, sure virtual isn’t as much of a risk, because of the degrees of isolation. Our mission critical goal is to secure our backups, so VBR running virtually on this is fine and then we’d purchase some dedicated storage for a repository so at that design decision I’d always be advocating for a physical with hardened repo deployed vs a virtual machine accessing a SAN. I like this from a performance standpoint too as disk IO is handled via SAS or PCI-E vs iSCSI/FC, that’s more networking, more shared resources etc.

 

The truth is, building and maintaining such platforms takes time & expertise, something that is hard to quantify how much you need, but always easy to say you don’t have enough of. And this is where going physical shines. You don’t need a management domain to use a hardened repository. You can unplug any iLO/iDRAC and physically secure your infrastructure to prevent console access or remote control. So physical repository always makes sense.

 

Proxies are throwaway and whatever will get the best performance gets my vote on the physical/virtual.

 

As for VBR. I take the “it depends” stance. Because, how will you recover? If I’ve got a VMware environment that got destroyed, have I got another? If not what am I recovering to anyway? If I need to rebuild the entire cluster because I don’t have a DR cluster or even standalone host, my RTO is gonna suck anyway. So VBR being physical or virtual doesn’t matter. If I’ve got somewhere to recover to as a dedicated resource, I’d either be considering running VBR as a VM on that or running physically. I’d still be saying physical by default however.

 

Everyone decides for themselves how much risk they need to mitigate to get a good night sleep, and I see physical servers as a great step when done right, and if virtual, only in an isolated platform.

Userlevel 7
Badge +7

Good points 👍🏼

I am astonished every time I hear from such attack vendors like “SATA cables as antennas to exfiltrate data" or “using radiation from screen cables to get your screen content through several walls".

The bad guys are very creative….

Userlevel 7
Badge +6

Lots of great answers already 👍

In my opinion it depends on the requirements, the desired security and the implemented storage technology. You can achieve almost anything with anything if it's well thought.

  • Are you using a separate virtual environment (with separate permissions)?
  • Can you secure a virtual environment as much as a standalone physical system?
  • How will you recover im case your virtual infrastructure goes down? Especially will you be able to meet your target RTO?
  • Do you use immutable or offline storage?
  • ...

Especially in smaller environments I like to use a dedicated physical server as it makes both securing and restoring much easier. In bigger environments with different locations, it's can make sense to virtualize the VBR server role; especially if you replicate it or maintain a cold standby instance.

With a decent immutable/offline solution I could think of going all virtual. The only requirement would be to use seperate virtual ressources; like not creating primary backups on production storage.

Userlevel 7
Badge +3

Interesting topic, obviously physical for backup repository. I prefer to go on VM for VBR to enjoy the benefits of vsphere cluster (HA, DRS...). For me backup is tier 0 like AD and must be on dedicated virtualization infrastructure.

Let’s say we losted vsphere infra, well i have a backup configuration on repository or on tape. I can install temporary vbr roles on physical host with FC ports to restore mandatory objects.
I agree in this scenario you will need a physical server in spare or which is used in proxy SAN.

If you don’t have a physical server, go from scratch on vsphere infra. Be prepare to redeploy windows VM quickly with terraform or ansible code, you put outside of your infra then install vbr roles etc…

I think it's mostly a matter of prep what to do if a disaster strikes.

@JMeixner you can have tempest cable if you need or have datacenter room in faraday style :D

https://fr.wikipedia.org/wiki/TEMPEST

Userlevel 7
Badge +7

@JMeixneryou can have tempest cable if you need or have datacenter room in faraday style :D

https://fr.wikipedia.org/wiki/TEMPEST

Interesting link. Thank you, Bertrand

Badge +2

Wow, hot topic over here,

In my personal experience, the Veeam Server as a Virtual Machine always give me the flexibility to have a backup of himself, so if something goes wrong, I have a plan B prepared just to recover the vm and re-scan the repos to start restoring.
Also before patching, snapshot, etc… all the great features of a production vm in my backup server.

Most of my deploys are 2 to 4 ESXi hosts, and most of the time a Prod Cluster and Dev Cluster, and my Veeam Server Replicated (Veeam Replicas) between both Clusters, so in case of disaster, I can run my Veeam Server, and then run Replicas from it, of restore what I need.
;)

Userlevel 6
Badge +1

I think this is a classic case of “it depends”. 

I have SEVERAL clients with virtual backup servers.  The thing I don’t like about using virtual backup servers is that more often than not, it’s going to mean that I’m using a prosumer NAS for the backup repository, such as a QNAP or Synology.  No battery backed RAID controller, no controller redundancy, possible software tweaks to the software RAID that can introduce corruption, etc.

For my clients with physical servers, that means I have a Dell server in place with a battery-backed RAID card, enterprise-grade local drives with enterprise support agreements when a drive fails rather than having to figure out what the current model drive most closely matches the failed drive or figuring out if there is still a warranty from the drive manufacturer, and then finding a replacement while being down a drive for a week or more.  For those physical servers, there is the caveat of having some sort of on-host proxy server for VM access, or, my preference, using direct storage access via ISCSI or SAS/Direct Attach connections depending on what the primary storage is.  Makes for some really fast backups.  Also, if I have direct storage access, it makes it easier to segment the backup environment from the production environment.

Probably my favorite configuration is using a hybrid of the two….the backup server calling the shots resides at the recovery site and can be a VM.  The repository server at the primary site is a physical box with local disks.  The repository at the recovery site can be connected to a VM or a physical box, but again, if a VM, I’m probably using a NAS.  But I’m less concerned about corruption or slowness from a NAS at the recovery site that by copy jobs are targeting than at the primary site where the initial back data is landing.

So my preference is physical, but the sky is not falling if you use a VM, assuming that you think it through and take that into account.

Comment