I don’t remember reading anything in the BP guide about suggesting a virtual backup repository.

Are you asking about repo? or which components?

@haslund

Yes, the BP guide recommends to use physical repository where possible. I would never recommend using a virtual repository :)

https://bp.veeam.com/vbr/2_Design_Structures/D_Veeam_Components/D_backup_repositories/

• Physical Repositories are recommended where possible (ideally combined with proxy role using backup from storage snapshots).

Physical or Virtual? 

In general, we recommend whenever possible to use physical machines as repositories, in order to maximize performance and have a clear separation between the production environment that needs to be protected and the backup storage. It is also recommended combining this with the proxy role if backup from Storage Snapshots is possible. This keeps overheads on the virtual environment and network to a minimum.

I was referring to all components.  I know what the guide says but just wanted to see what general opinion was from the community.  I know what those working at Veeam will say.  

Thanks for explaining.

I just thought, because the forum post was about using virtual machines as Veeam Backup repositories, you wanted discuss the same (VM as Repo) here in the community

I‘m very sorry if I misinterpreted the topic of your post.

Some components can be virtual, of course. Just the repository shouldn‘t be virtual, if you want to follow our best practices. For example virtual proxies could be mandatory for some environments. 
 

Yes I was just referencing the post from the forum to see what people thought but to that point repos “shouldn’t” be virtual is true but can they be virtual - yes.  I get the BP stuff and follow it to the letter most times but on that forum post was just expressing my opinion on the subject based on what I do daily that is all.   Good conversation as they say but I tend to shy away from the forums at times due to the way people can jump all over a comment made by someone expressing an opinion.

I am all good though as a good debate is always fun.  

In a normal enterprise production environment I prefer physical VBR servers and Repositories  to separate the backup environment from the virtualization clusters. I separate as much as I can.

If there is an attack on this environment and the vSphere Cluster is hacked tje backup environment should stay save.

As an MSP where my complete environment is for backup data from customers  I think I would use virtual components, too. Then I have to be able to scale my Veeam environment if there are new customers. But I think I would use physical Storage Systems or Servers for my repositories, too. I don’t want to get them deleted vie the vCenter or similar management parts.

Great points Joe.  Now in our case the Veeam virtual infrastructure is separate from anything client related as we have our own management stack and cluster that is used whereas the clients use vCloud and have clusters allocated there.  So we do ensure security between things.

I see things from both sides and sometimes we need to do what the company requires even though we are the experts on the subject and follow best practices. 

To add my two cents, I have seen a design where the Veeam environment (VBR and repo, VONE, BEM) are in a separate vSphere cluster under a separate vCenter, using a separate shared storage infrastructure (or VSAN). This comes from the same reckoning as the architecture of management clusters for vSphere environments. The same risks about compromise of the VMware components exist, and other complications may ensue if a management domain is used (as recommended by Edwin), but this design has some interesting advantages in terms of deployment and management tools. In my day as a VMware engineer, I was taught (and held to it by my first boss) to only recommended and implement a physical vCenter server, for many of the same reasons put forth in this post and in the Forums. That “caution!” has been washed away by the sands of time, it seems.

No one size fits all, but as was stated in the forum thread, backup is all about risk management. You decide!

Lots of good discussion here already, maybe I can add some value. Here goes:

In security, the goal posts have always been “less secure” and “more secure”. There was a PoC less than a month ago that used SATA cables as an antenna to exfiltrate data in an air gapped environment 🤯 (Source: https://thehackernews.com/2022/07/new-air-gap-attack-uses-sata-cable-as.html).

So when designing architectures, I like to consider the way Apple tier their products “good”, “better”, “best”, except in backup designs we can certainly have “not good”, “terrible”, “resume generating event imminent”.

Before the whole physical vs virtual question becomes a key question, the first question should be about the fault domain. If you’ve got a Veeam Cloud Connect environment as service provider, that’s a separate fault domain and it doesn’t warrant much discussion. As a compromise on a customer environment won’t USUALLY result in a compromise on your part. (Though on this point, if you’re using the same RMM tools to manage your platform and your customer, this is certainly possible. Source: https://www.reddit.com/r/msp/comments/cl3g23/continuum_used_to_deploy_ransomware_because/).

If we’re talking about backup infrastructure being implemented at a customer’s site, we need to create boundaries to strengthen isolation, as the probability of the fault domains (production/workloads and management/backup) overlapping increases.

Some of these exist as utilising a dedicated compute cluster with a separate vCenter and a management domain, with one directional trust such as an AD Bastion Forest. This would usually be strengthened with firewalls with IPS/IDS and restrictive controls, in these scenarios, sure virtual isn’t as much of a risk, because of the degrees of isolation. Our mission critical goal is to secure our backups, so VBR running virtually on this is fine and then we’d purchase some dedicated storage for a repository so at that design decision I’d always be advocating for a physical with hardened repo deployed vs a virtual machine accessing a SAN. I like this from a performance standpoint too as disk IO is handled via SAS or PCI-E vs iSCSI/FC, that’s more networking, more shared resources etc.

The truth is, building and maintaining such platforms takes time & expertise, something that is hard to quantify how much you need, but always easy to say you don’t have enough of. And this is where going physical shines. You don’t need a management domain to use a hardened repository. You can unplug any iLO/iDRAC and physically secure your infrastructure to prevent console access or remote control. So physical repository always makes sense.

Proxies are throwaway and whatever will get the best performance gets my vote on the physical/virtual.

As for VBR. I take the “it depends” stance. Because, how will you recover? If I’ve got a VMware environment that got destroyed, have I got another? If not what am I recovering to anyway? If I need to rebuild the entire cluster because I don’t have a DR cluster or even standalone host, my RTO is gonna suck anyway. So VBR being physical or virtual doesn’t matter. If I’ve got somewhere to recover to as a dedicated resource, I’d either be considering running VBR as a VM on that or running physically. I’d still be saying physical by default however.

Everyone decides for themselves how much risk they need to mitigate to get a good night sleep, and I see physical servers as a great step when done right, and if virtual, only in an isolated platform.

Good points

I am astonished every time I hear from such attack vendors like “SATA cables as antennas to exfiltrate data" or “using radiation from screen cables to get your screen content through several walls".

The bad guys are very creative….

Lots of great answers already

In my opinion it depends on the requirements, the desired security and the implemented storage technology. You can achieve almost anything with anything if it's well thought.

• Are you using a separate virtual environment (with separate permissions)?
• Can you secure a virtual environment as much as a standalone physical system?
• How will you recover im case your virtual infrastructure goes down? Especially will you be able to meet your target RTO?
• Do you use immutable or offline storage?
• ...

Especially in smaller environments I like to use a dedicated physical server as it makes both securing and restoring much easier. In bigger environments with different locations, it's can make sense to virtualize the VBR server role; especially if you replicate it or maintain a cold standby instance.

With a decent immutable/offline solution I could think of going all virtual. The only requirement would be to use seperate virtual ressources; like not creating primary backups on production storage.

Interesting topic, obviously physical for backup repository. I prefer to go on VM for VBR to enjoy the benefits of vsphere cluster (HA, DRS...). For me backup is tier 0 like AD and must be on dedicated virtualization infrastructure.

Let’s say we losted vsphere infra, well i have a backup configuration on repository or on tape. I can install temporary vbr roles on physical host with FC ports to restore mandatory objects.
I agree in this scenario you will need a physical server in spare or which is used in proxy SAN.

If you don’t have a physical server, go from scratch on vsphere infra. Be prepare to redeploy windows VM quickly with terraform or ansible code, you put outside of your infra then install vbr roles etc…

I think it's mostly a matter of prep what to do if a disaster strikes.

@JMeixner you can have tempest cable if you need or have datacenter room in faraday style :D

https://fr.wikipedia.org/wiki/TEMPEST

Interesting link. Thank you, Bertrand

Wow, hot topic over here,

In my personal experience, the Veeam Server as a Virtual Machine always give me the flexibility to have a backup of himself, so if something goes wrong, I have a plan B prepared just to recover the vm and re-scan the repos to start restoring.
Also before patching, snapshot, etc… all the great features of a production vm in my backup server.

Most of my deploys are 2 to 4 ESXi hosts, and most of the time a Prod Cluster and Dev Cluster, and my Veeam Server Replicated (Veeam Replicas) between both Clusters, so in case of disaster, I can run my Veeam Server, and then run Replicas from it, of restore what I need.
;)

I think this is a classic case of “it depends”. 

I have SEVERAL clients with virtual backup servers.  The thing I don’t like about using virtual backup servers is that more often than not, it’s going to mean that I’m using a prosumer NAS for the backup repository, such as a QNAP or Synology.  No battery backed RAID controller, no controller redundancy, possible software tweaks to the software RAID that can introduce corruption, etc.

For my clients with physical servers, that means I have a Dell server in place with a battery-backed RAID card, enterprise-grade local drives with enterprise support agreements when a drive fails rather than having to figure out what the current model drive most closely matches the failed drive or figuring out if there is still a warranty from the drive manufacturer, and then finding a replacement while being down a drive for a week or more.  For those physical servers, there is the caveat of having some sort of on-host proxy server for VM access, or, my preference, using direct storage access via ISCSI or SAS/Direct Attach connections depending on what the primary storage is.  Makes for some really fast backups.  Also, if I have direct storage access, it makes it easier to segment the backup environment from the production environment.

Probably my favorite configuration is using a hybrid of the two….the backup server calling the shots resides at the recovery site and can be a VM.  The repository server at the primary site is a physical box with local disks.  The repository at the recovery site can be connected to a VM or a physical box, but again, if a VM, I’m probably using a NAS.  But I’m less concerned about corruption or slowness from a NAS at the recovery site that by copy jobs are targeting than at the primary site where the initial back data is landing.

So my preference is physical, but the sky is not falling if you use a VM, assuming that you think it through and take that into account.

​I have worked in places which have had physical Backup Servers and also Virtual Servers. As @dloseke  said, it depends. Going by my experience, in smaller organisations, there seems to be a preference for a dedicated physical backup server with a Synology or a QNAP for storage. In larger places, especially with those that have the Veeam Enterprise Plus License and storage integration, the backup server is a Virtual machine with a large number of proxies. 

This allows for quick recovery at the secondary site with the backup data already mirrored to the recovery storage array. One of the great things about Veeam is that it is possible to recover from a backup with just the Veeam B&R Console by importing the backup data. 

The other option is to have 2 Veeam instances running; one at the Primary and one at the Secondary site. So in case of failover, the data can get backed up when VMs are brought back online at the Secondary site. One less thing to worry about during a DR scenario. The secondary instance could also be connected to an Object Storage provider to recover from cloud, as required.  

​

Yeah, I’ve noticed that as well except I usually see a virtual backup server if a QNAP or Synology NAS (or similar, I’ve seen NetGear ReadyNAS and AsuSTOR as well) is in use for the backing storage.  Physical backup servers with local storage can vary in size for sure and in smaller environments as a smaller repo can easily be put on a physical server, up to nearly 250TB depending on disk sizing and RAID type. For instance, on my larger physical boxes, I will use a PowerEdge T640 (in a rack configuration if a rack is available) where I can cram 18 3.5” drives….and with drives reaching 12/14/16TB, in a RAID 6, that nets around 230-260TB.  That’s not bad.  On the flip side, I’ve built a T340 with 4 local disks in a RAID 5 and had something like 12TB before as well for the more ROBO side of things.  It just depends on use case.

It seems that once you get much larger than that, it’s not as feasible to use a physical server...you either need to have more than one or you start looking at more of your enterprise-style storage such as using an actual SAN or enterprise NAS or deduplicating appliances or maybe something slightly less such as a large SMB NAS like a Synology with Expansion drawers, etc.  I live in the Dell world, so something like Isilon (PowerScale) or DataDomain, or in the HPE world, StoreOnce might be more common.  But performance suffers on the larger deduplicating appliance in my experience, and when you’re at that scale, you probably need more performance so that your backups aren’t piling on top of each other.

I am truly liking the conversation here as it gives nice points from both sides which was the end goal.  Not about which is right or better, etc.  

@dloseke & @dips on your points around SMB based setups, I had a call with Object First last week, and I honestly see that their platform could become a massive game changer for SMB/ROBO deployments especially.

I’m forseeing a scenario of:

• VBR as a VM
• Either dedicated virtual proxies or the role co-existing with VBR if the environment is small enough
• Either the virtual proxies or VBR server being the gateway server to communicate with Object First
• Object First node(s) being tier 1 backups locally, with immutability. For shorter term retention.
• Offload to public cloud object storage via vendor of preference, Wasabi/AWS/Azure/Backblaze etc, securing the longer term retention required.

A lot of SMBs won’t have dedicated DR sites or resources, they’ll likely find that the day to day recovery for non-environmental will be fulfilled by Object First, with any environmental disasters (fire/flood etc) being a restore to AWS/Azure/GCP approach until they can sort themselves out.

Oh, Object First is a /can be a local storage?

I see, I have to have a closer look at this…

I’m very interested in seeing what Object First is doing…...the main issue I tend to have is that I want to have VBR running on a different environment than the performance workload, but want the data to reside on a separate server as well, and want to use Hardened Linux Immutability, which means if I’m using the same server, I’m looking at VM’s, so I need to either store the data in a VMDK which is not advised, or I need to use some sort of RDM which tends to be unsupported on local storage but can be cobbled together.  I’m trying to figure out how Object First would fit into that picture and while it doesn’t fix all of my concerns, it sure helps!

Check this out:

and then check out their website https://objectfirst.com/ they’re doing demos every Friday

It’s a really clean deployment I’ve got to say, there’s some advanced features IMO that are missing, but those are self-imposed restrictions for them to focus on getting v1 GA.

They aren’t offering any hosted storage, so it’s purely on-premises object storage, and it shows, they’ve got speeds of up to 4GB/s! More than enough for an SMB.

Now this does look interesting. Thanks for the heads up @MicoolPaul I will check them out

Yeah I attended the webinar a few weeks ago.  Really want to look in to this as we are moving off Windows to Linux servers for the Immutability but would be great to have a cluster of these to use in specific use cases at our DCs or clients.  Looking forward to the GA release. 

Well put, once physical servers reach capacity i.e RAM for example, it is hard to allocate more resources to it which means budgeting for a new physical server. On the virtual side of things, if there is some spare capacity available, it can easily be allocated.