What's Your Veeam B&R Design?

Hi Shane,

it depends, many unknown informations..but just for fun, this is my simple design 🙂:

• a VBR installed on physical Dell at DR site
• a Linux Hardened Repository with ISCSI LUN from NIMBLE installed on HP at Main site
• a Linux Hardened Repository with ISCSI LUN from NIMBLE installed on HP at DR site
• Use NIMBLE deduplication instead of Veeam (like a dedup appliance) → I tested it is much better
• one or more Virtual Proxy at Main site
• one or more Virtual Proxy at DR site
• Primary backup to Main site repo
• Copy job to DR site repo
• Replication job to DR site (no CDP) + failover plans

Be careful with replicas, if you have legacy hardware at DR site you have to pay attention to virtual hardware version! 😜

Going to have to make some assumptions, but high-level. 

• One HPE DL380p at each location, each connected to a Nimble via iSCSI to create a Linux Hardened Repo.  Backups to the primary site, copy jobs to the recovery site.
• Dell R740, load up ESXI and utilize as the Backup orchestration server running a Veeam for VBR to coordinate things. 
• Dell R740 If it has enough space, also have a VM in place as a proxy server for replica’s.  Perhaps tie it into the Nimble as well for the replica’s.  If it has a lot of local space, still replica’s.  
• Dell R740 - host the vCenter if possible or have it running at the recovery site either way, either on the production hosts or the Dell at that location.
• Dell R740 - host a VM running Orchestrator to facilitate orchestration for recovery.  This is why I have VBR and vCenter running at the recovery site as well.
• Linux proxy servers will run on each cluster at the primary and recovery sites.
• Newer hosts are going to need to run older virtual machine versions that are compatible with the recovery site assuming that they’re not running the same version of ESXI.  If they are, then this can be skipped.
• CDP replication for Core services and critical data.  I’m assuming that there’s no storage-based replication here.  Because vCenter and such are running at the recovery site, I’m going to have replica’s or at least backups running in reverse so that backups are stored at the recovery site, but are copied or replicated to the primary site in case the recovery site is taken offline and I need to spin up in what I’d loosely call a “reverse DR”.  CDP replication to traverse the dedicated storage link assuming that there’s not much other traffic there.
• Snapshot-based replication for the important and secondary data sets.  Can be set to replicate within the RPO’s defined, but can be replicated as separate jobs if desired.

Things I’d also consider:

• Implementing deduplication at the secondary site, or local object storage, such as via a virtual Quantum DXi v5000 appliance or MINIO.  I’ve not played with these, but they’re on the end of my very long list of things to check out and might be worth utilizing local storage or iSCSI storage back to the Nimble array.
• Assuming the four Nimbles are available for dedicated backups, I’d probably utilize two at each location either in a group to share the resources or as replica pairs, so that there’s extra redundancy at both locations.

I very often will go with a physical VBR server, but this is more tailored to when I have a “purpose-built Veeam appliance”.  That is, Veeam in Windows with a local disk repo.  But if I want to use that server or a multitude of servers, then it tends to be virtual.  When virtual, it’s going to generally be using an external storage subsystem like a NAS or SAN or a physical server, such as a Linux server for a LHR.  Also, most of my all-in-one boxes going forward have ISCSI or SAS connectivity with direct storage access to the array for faster backups.  If I’m going to be virtual, I don’t generally have direct storage access.  So for me, it’s really tailored to what environment I’m walking into, but in this case, I went virtual.

A couple of things I like about physical

• It survives a cluster failure - I had this happen where a SAN critically failed losing multiple disks (we warned the client that a double-digit-old Equallogic was a ticking time bomb, and I was proven correct a few months later) so to revive the server, I had to get working hardware in place, build a new Windows Server, install Veeam, attach to the VCC repo where I had the configuration database backed up, restore the config, THEN I was able to begin restoring VM’s.  It all worked great, but that added a couple of hours to the recovery process before I could even begin restoring VM’s, and this particular client was a small community hospital that had to divert incoming patients to other hospitals in neighboring communities while things were being restored.
• It can be portable - so if I need a crash kit to deploy to a client, or I need to temporarily put in a server if a new client that has no backups during onboarding, I have something I can more or less drop-in, configure network, accounts for access to vCenter, etc and I’m ready to go.
• I can utilize Direct Storage Access - now granted, I can do this with a physical proxy server as well, and I’m not limited to Windows in that case, but again, as an all-in-one solution, this works well.
• Most of the workload is taken off of the cluster, aside from a proxy server if you need one on-host.
• The Backup server is physically segmented from the compute cluster.  So if malware gets in and encrypts VM’s at the VM/host level, the backup server survives, and unpatched VMware hosts are definitely a big target the past few months (especially if you have them directly on the internet - stupid….)

There’s probably just as many arguments for using a virtual backup server, HA failover as you noted being one, but I really do like the posture that using a physical server gives me when I can do it.

Hi @coolsport00 if you can add another LAN/VLAN for management use you can install an Esxi host for Veeam using different vm for console, VBR, EM etc. Accessing only from this network. 

Yeah, in this case, the SAN wasn’t sending out alerts and I’m guessing that the backup battery on the main SAN controller had previously failed.  On this particular PS6100, it uses a “supercapacitor board” of which the caps like to leak and kill the board.  When this happened to the second board, it went down.  Turns out that when this board fails, it causes BSD (the underlying OS) to kernel panic and boot loop.  Ultimately, all backups had been successful except for one DC that had failed because of a space issue on the repository, and it turned out that the DC was also a file server for whatever reason.  Customer requested that I try to get the data off of that VM which I was successful in doing.  I had a previous client that had a PS4100 that failed hard down because they were running on very questionable hardware that I had of advised of the risk but their executives wouldn’t replace it while it was still working….I know…..  Anyhow, they decided to do power maintenance with the hardware online assuming their UPS would carry them through (spoiler alert:  it didn’t) and when the 12-disk array started back up (or tried), 5 of the disks had either a failed or unknown state resulting in total data loss, and they didn’t have any backups in place).  Anyhow, the owner had shelved the array and donated it to my cause upon my request and I was able to pull the supercap boards from the PS4100 controllers and install them on the PS6100 controllers and bring the array up, connect it to my lab environment, and extract the missing data from the VM in question.

While true, the general recommendation is that if you’re going to perform DirectSAN, physical is better by placing no workload on the virtual environment.  And in the case of a lot of my SMB clients, the SAN is connected via SAS/DAS, so it has to be virtual since there’s no ISCSI connectivity.  If ISCSI was in use, then yes, either works.

I have yet to provide what my actual design is, but Scott came pretty close to what I have set up. I'm on vacay for the next week, but will share what I have in place when I get back. 

Good feedback all 😊

That looks decent. Tape, Cloud and Disk. you are covered!

It really comes down to the requirements.

Things like data retention, amount of copies, and how big your production dataset is.. I am dealing with 100’s of TB’s, into PB’s of data.    So If I keep too many copies it gets VERY expensive, but also, with an insane retention policy of 100 Years for a lot of it, i also need to consider multiple copies for 100+ years incase something happens.

As it grows, I have to include this in time allocation for evacuating a tape library to a new library, man hours to migrate storage etc.  Getting a storage retention policy out early is key. 

That being said I may have just solved a HUGE pain point using tape, archives, and getting rid of 100’s of TB’s of old data that I’ll create a blog about if it works. 

Tape, Cloud, Disk, & Immutability. Yep....you should be covered 😊