@haslund

Yes, the BP guide recommends to use physical repository where possible. I would never recommend using a virtual repository :)

https://bp.veeam.com/vbr/2_Design_Structures/D_Veeam_Components/D_backup_repositories/

• Physical Repositories are recommended where possible (ideally combined with proxy role using backup from storage snapshots).

Physical or Virtual? 

In general, we recommend whenever possible to use physical machines as repositories, in order to maximize performance and have a clear separation between the production environment that needs to be protected and the backup storage. It is also recommended combining this with the proxy role if backup from Storage Snapshots is possible. This keeps overheads on the virtual environment and network to a minimum.

In a normal enterprise production environment I prefer physical VBR servers and Repositories  to separate the backup environment from the virtualization clusters. I separate as much as I can.

If there is an attack on this environment and the vSphere Cluster is hacked tje backup environment should stay save.

As an MSP where my complete environment is for backup data from customers  I think I would use virtual components, too. Then I have to be able to scale my Veeam environment if there are new customers. But I think I would use physical Storage Systems or Servers for my repositories, too. I don’t want to get them deleted vie the vCenter or similar management parts.

Lots of good discussion here already, maybe I can add some value. Here goes:

In security, the goal posts have always been “less secure” and “more secure”. There was a PoC less than a month ago that used SATA cables as an antenna to exfiltrate data in an air gapped environment 🤯 (Source: https://thehackernews.com/2022/07/new-air-gap-attack-uses-sata-cable-as.html).

So when designing architectures, I like to consider the way Apple tier their products “good”, “better”, “best”, except in backup designs we can certainly have “not good”, “terrible”, “resume generating event imminent”.

Before the whole physical vs virtual question becomes a key question, the first question should be about the fault domain. If you’ve got a Veeam Cloud Connect environment as service provider, that’s a separate fault domain and it doesn’t warrant much discussion. As a compromise on a customer environment won’t USUALLY result in a compromise on your part. (Though on this point, if you’re using the same RMM tools to manage your platform and your customer, this is certainly possible. Source: https://www.reddit.com/r/msp/comments/cl3g23/continuum_used_to_deploy_ransomware_because/).

If we’re talking about backup infrastructure being implemented at a customer’s site, we need to create boundaries to strengthen isolation, as the probability of the fault domains (production/workloads and management/backup) overlapping increases.

Some of these exist as utilising a dedicated compute cluster with a separate vCenter and a management domain, with one directional trust such as an AD Bastion Forest. This would usually be strengthened with firewalls with IPS/IDS and restrictive controls, in these scenarios, sure virtual isn’t as much of a risk, because of the degrees of isolation. Our mission critical goal is to secure our backups, so VBR running virtually on this is fine and then we’d purchase some dedicated storage for a repository so at that design decision I’d always be advocating for a physical with hardened repo deployed vs a virtual machine accessing a SAN. I like this from a performance standpoint too as disk IO is handled via SAS or PCI-E vs iSCSI/FC, that’s more networking, more shared resources etc.

The truth is, building and maintaining such platforms takes time & expertise, something that is hard to quantify how much you need, but always easy to say you don’t have enough of. And this is where going physical shines. You don’t need a management domain to use a hardened repository. You can unplug any iLO/iDRAC and physically secure your infrastructure to prevent console access or remote control. So physical repository always makes sense.

Proxies are throwaway and whatever will get the best performance gets my vote on the physical/virtual.

As for VBR. I take the “it depends” stance. Because, how will you recover? If I’ve got a VMware environment that got destroyed, have I got another? If not what am I recovering to anyway? If I need to rebuild the entire cluster because I don’t have a DR cluster or even standalone host, my RTO is gonna suck anyway. So VBR being physical or virtual doesn’t matter. If I’ve got somewhere to recover to as a dedicated resource, I’d either be considering running VBR as a VM on that or running physically. I’d still be saying physical by default however.

Everyone decides for themselves how much risk they need to mitigate to get a good night sleep, and I see physical servers as a great step when done right, and if virtual, only in an isolated platform.

Lots of great answers already

In my opinion it depends on the requirements, the desired security and the implemented storage technology. You can achieve almost anything with anything if it's well thought.

• Are you using a separate virtual environment (with separate permissions)?
• Can you secure a virtual environment as much as a standalone physical system?
• How will you recover im case your virtual infrastructure goes down? Especially will you be able to meet your target RTO?
• Do you use immutable or offline storage?
• ...

Especially in smaller environments I like to use a dedicated physical server as it makes both securing and restoring much easier. In bigger environments with different locations, it's can make sense to virtualize the VBR server role; especially if you replicate it or maintain a cold standby instance.

With a decent immutable/offline solution I could think of going all virtual. The only requirement would be to use seperate virtual ressources; like not creating primary backups on production storage.

Check this out:

and then check out their website https://objectfirst.com/ they’re doing demos every Friday

It’s a really clean deployment I’ve got to say, there’s some advanced features IMO that are missing, but those are self-imposed restrictions for them to focus on getting v1 GA.

They aren’t offering any hosted storage, so it’s purely on-premises object storage, and it shows, they’ve got speeds of up to 4GB/s! More than enough for an SMB.

I don’t remember reading anything in the BP guide about suggesting a virtual backup repository.

Are you asking about repo? or which components?

I think this is a classic case of “it depends”. 

I have SEVERAL clients with virtual backup servers.  The thing I don’t like about using virtual backup servers is that more often than not, it’s going to mean that I’m using a prosumer NAS for the backup repository, such as a QNAP or Synology.  No battery backed RAID controller, no controller redundancy, possible software tweaks to the software RAID that can introduce corruption, etc.

For my clients with physical servers, that means I have a Dell server in place with a battery-backed RAID card, enterprise-grade local drives with enterprise support agreements when a drive fails rather than having to figure out what the current model drive most closely matches the failed drive or figuring out if there is still a warranty from the drive manufacturer, and then finding a replacement while being down a drive for a week or more.  For those physical servers, there is the caveat of having some sort of on-host proxy server for VM access, or, my preference, using direct storage access via ISCSI or SAS/Direct Attach connections depending on what the primary storage is.  Makes for some really fast backups.  Also, if I have direct storage access, it makes it easier to segment the backup environment from the production environment.

Probably my favorite configuration is using a hybrid of the two….the backup server calling the shots resides at the recovery site and can be a VM.  The repository server at the primary site is a physical box with local disks.  The repository at the recovery site can be connected to a VM or a physical box, but again, if a VM, I’m probably using a NAS.  But I’m less concerned about corruption or slowness from a NAS at the recovery site that by copy jobs are targeting than at the primary site where the initial back data is landing.

So my preference is physical, but the sky is not falling if you use a VM, assuming that you think it through and take that into account.

Interesting topic, obviously physical for backup repository. I prefer to go on VM for VBR to enjoy the benefits of vsphere cluster (HA, DRS...). For me backup is tier 0 like AD and must be on dedicated virtualization infrastructure.

Let’s say we losted vsphere infra, well i have a backup configuration on repository or on tape. I can install temporary vbr roles on physical host with FC ports to restore mandatory objects.
I agree in this scenario you will need a physical server in spare or which is used in proxy SAN.

If you don’t have a physical server, go from scratch on vsphere infra. Be prepare to redeploy windows VM quickly with terraform or ansible code, you put outside of your infra then install vbr roles etc…

I think it's mostly a matter of prep what to do if a disaster strikes.

@JMeixner you can have tempest cable if you need or have datacenter room in faraday style :D

https://fr.wikipedia.org/wiki/TEMPEST

Good points

I am astonished every time I hear from such attack vendors like “SATA cables as antennas to exfiltrate data" or “using radiation from screen cables to get your screen content through several walls".

The bad guys are very creative….

To add my two cents, I have seen a design where the Veeam environment (VBR and repo, VONE, BEM) are in a separate vSphere cluster under a separate vCenter, using a separate shared storage infrastructure (or VSAN). This comes from the same reckoning as the architecture of management clusters for vSphere environments. The same risks about compromise of the VMware components exist, and other complications may ensue if a management domain is used (as recommended by Edwin), but this design has some interesting advantages in terms of deployment and management tools. In my day as a VMware engineer, I was taught (and held to it by my first boss) to only recommended and implement a physical vCenter server, for many of the same reasons put forth in this post and in the Forums. That “caution!” has been washed away by the sands of time, it seems.

No one size fits all, but as was stated in the forum thread, backup is all about risk management. You decide!

I was referring to all components.  I know what the guide says but just wanted to see what general opinion was from the community.  I know what those working at Veeam will say.  

Great points Joe.  Now in our case the Veeam virtual infrastructure is separate from anything client related as we have our own management stack and cluster that is used whereas the clients use vCloud and have clusters allocated there.  So we do ensure security between things.

I see things from both sides and sometimes we need to do what the company requires even though we are the experts on the subject and follow best practices. 

Interesting link. Thank you, Bertrand

Thanks for explaining.

I just thought, because the forum post was about using virtual machines as Veeam Backup repositories, you wanted discuss the same (VM as Repo) here in the community

I‘m very sorry if I misinterpreted the topic of your post.

Some components can be virtual, of course. Just the repository shouldn‘t be virtual, if you want to follow our best practices. For example virtual proxies could be mandatory for some environments. 
 

I agree to you, @MicoolPaul 

I am using Dell PowerEdge R740XD (or R740 when less repository space is needed) often. Or now R750… My experiences with these servers are good. They are sufficient for many use cases.

I use “real” storage systems for customers with higher repository storage demand - 200 TB or more… These have redundant controller and can have much more disks or SSDs…

Sometimes I use two or more PowerEdge servers, if the growth expectation is not that big….

@dloseke & @dips on your points around SMB based setups, I had a call with Object First last week, and I honestly see that their platform could become a massive game changer for SMB/ROBO deployments especially.

I’m forseeing a scenario of:

• VBR as a VM
• Either dedicated virtual proxies or the role co-existing with VBR if the environment is small enough
• Either the virtual proxies or VBR server being the gateway server to communicate with Object First
• Object First node(s) being tier 1 backups locally, with immutability. For shorter term retention.
• Offload to public cloud object storage via vendor of preference, Wasabi/AWS/Azure/Backblaze etc, securing the longer term retention required.

A lot of SMBs won’t have dedicated DR sites or resources, they’ll likely find that the day to day recovery for non-environmental will be fulfilled by Object First, with any environmental disasters (fire/flood etc) being a restore to AWS/Azure/GCP approach until they can sort themselves out.

Well put, once physical servers reach capacity i.e RAM for example, it is hard to allocate more resources to it which means budgeting for a new physical server. On the virtual side of things, if there is some spare capacity available, it can easily be allocated. 

Yeah I attended the webinar a few weeks ago.  Really want to look in to this as we are moving off Windows to Linux servers for the Immutability but would be great to have a cluster of these to use in specific use cases at our DCs or clients.  Looking forward to the GA release. 

Agreed, unless you’ve got dedicated backup/management compute, then it doesn’t help fix that problem, as you can’t run any applications on those boxes.

But you could grab as cheap as possible a physical VBR server if required for some level of isolation, or just running the VBR + DB virtually.

Object First is dedicated physical hardware, from 1 to 4 nodes in scale, each node contributing either 96TB or 128TB of storage, backed by RAID6 + Hot spare, it’s spinny disk + NVMe cache (key note: cache is for backup performance only, to avoid wear and tear on the NVMe it’s not used for recovery). I thought this was a weird comment they made, as previously we’ve not been able to use object storage for performance tier, stuff like instant VM recovery and SureBackup wouldn’t be possible, but I’m suspecting that they are in v12, hence this comment.

As this is S3 compatible and supporting object lock / versioning out of the box, it’s certified for object + immutability with Veeam from day one, so that gives you the immutability protection locally instead of having to use a hardened repository.

@dlosekefunnily enough I spoke to Dell today with that exact same comment about their 15th gen, have been told “it’s in its way” but no eta! Handy as Intel are winding down their OCP NICs for 14th gen now.

 

Fingers crossed the R750XD (or whatever they call it) is available sooner rather than later or we’ll be on the 16th gen soon…

 

RE the 18 disk config for R740XD, if you speak to Dell they can still do this, it’s 12x bays on the front, then 4x bays in the mid-plane, then a further 2x bays at the rear.

 

R750xs, as in Extreme Scalability.  There’s also an “xa”, but I can’t for the life of me remember what the “a” stands for.  I know it’s focused towards the AI/ML/DL vertical for GPU intensive operations, but I doubt it stands for AI.  I’ll have to dig up my release notes if this one bothers me too much.

Thanks for pointing out that configuration on the “xd”….I had completely missed it when looking at their Solutions Configurator.  I’ll have to price out a xd with BOSS vs a T640 and see where it comes out….the rack space savings would be nice.

 

 

I’d seen them but shared your sentiment that they’re not true replacements, and Dell have said there will be an XD 15th gen but isn’t out yet, so that should plug the gap.

Agree, you get to a point where you’ve just got to start attaching dedicated storage shelves.

R750xs, as in Extreme Scalability.  There’s also an “xa”, but I can’t for the life of me remember what the “a” stands for.  I know it’s focused towards the AI/ML/DL vertical for GPU intensive operations, but I doubt it stands for AI.  I’ll have to dig up my release notes if this one bothers me too much.

Thanks for pointing out that configuration on the “xd”….I had completely missed it when looking at their Solutions Configurator.  I’ll have to price out a xd with BOSS vs a T640 and see where it comes out….the rack space savings would be nice.

Yeah it’d be the same as if you had a Data Domain, perform encryption at rest so it can dedupe, otherwise you’d just have to let Veeam do compression & deduplication and use it as nothing more than a fast repo, or disable encryption. But as you say, the money for it, you’d be using it as a limited retention tier 1 for backups, and then copying your backups out to something like an R740XD/T640 anyway for retention.

I still wish that Dell did the XE7100 server in a newer generation, 100x 3.5” disks in 5U!

It was great to see them at VeeamON too at their booth.  It will be a great solution when ready I think.

I’m very interested in seeing what Object First is doing…...the main issue I tend to have is that I want to have VBR running on a different environment than the performance workload, but want the data to reside on a separate server as well, and want to use Hardened Linux Immutability, which means if I’m using the same server, I’m looking at VM’s, so I need to either store the data in a VMDK which is not advised, or I need to use some sort of RDM which tends to be unsupported on local storage but can be cobbled together.  I’m trying to figure out how Object First would fit into that picture and while it doesn’t fix all of my concerns, it sure helps!

I am truly liking the conversation here as it gives nice points from both sides which was the end goal.  Not about which is right or better, etc.