Real-World Backup Speed Difference Between NBD and Direct SAN in Veeam

Hi ​@K Karthik -

To answer your questions:

1. Yes, it’s significant enough to implement, by typically 100s of MB/s But, if possible, if you have a supported Storage array, I would go with Backup From Storage Snapshots. Some of the config is like DirectSAN, but you don’t have the potential interaction of the Proxy with BfSS like you could with DirectSAN. Linux Proxies doesn’t pose as much a data risk as with Windows
2. See #1
3. You don’t present LUNs to VBR, but rather you grant access on your LUNs to the Veeam Proxy. In your case, your Proxy is the VBR server, but I wanted to be clear where the access is specifically given. And again...see #1
4. If you do go with DirectSAN, go with a Linux Proxy instead of using your VBR Server if you can. Linux removes most of the harm a Windows box poses for this scenario

So, my suggestions are:

• Use BfSS if you can
• Use a dedicated Linux Proxy if you can

Hi ​@Madi.Cristil ​@safiya ...could you please move this question to a more appropriate page, like Discussion Boards?

Thank you!

The answer to this will be “it depends” based on your architecture.  I have seen it where if you have a physical server with 10GB NICs you can get just as good of performance as Direct SAN sometimes, but still slightly lower.

To answer your questions -

1. In real-world environments, how much performance difference do you typically see between NBD (Network) mode and Direct SAN mode for backup and restore operations? -- Again as I mentioned it depends on architecture because if you have a physical environment for Veeam with 10+GB network you can do pretty good on NBD.  Direct SAN mode is great but this would depend on the type of arrays and connectivity too.

2. Is the performance improvement significant enough to justify implementing Direct SAN access? -- To me it would be as I have tested both but again you need to determine your architecture for best fit.

3. How risky is it to present production VMware datastore LUNs to the Veeam server for Direct SAN access? -- Normally you would install a plugin for your array type for direct SAN access as it uses snapshots from the array to do the backups and removes the pressure on the virtualization layer.  If you are presenting LUNs you typically are using HotAdd mode and Veeam attaches them in offline mode as to not mount them so they are RO only.

4. What best practices do you follow to prevent accidental modification or corruption of production datastore LUNs when they are presented to the Veeam server? -- You just allow Veeam to do it as it has the mechanisms in place to ensure this does not happen.

Check here - Direct SAN Access - Veeam Backup & Replication User Guide

Hi @ coolsport00,

Thanks for your response.

We are using HPE Alletra as our SAN storage and HPE StoreOnce as the backup repository. The Veeam Backup & Replication server is installed on a physical Windows server and currently backups are running in NBD mode.

At present, we are getting backup throughput of around 200–300 MB/s, and sometimes below 200 MB/s.

Considering our environment (HPE Alletra + HPE StoreOnce + FC SAN), would you recommend staying with NBD mode or moving to Direct SAN mode?

Has anyone implemented Direct SAN with a similar setup, and if so, what backup speed improvements did you observe compared to NBD?

Thank you.

​@K Karthik Just a small contribution, if your vSPhere env. is using cripted VMDK you can’t use DAS SAN as trasnport mode, only network or Virtual appliance as a proxies. 

​@K Karthik - well...200-300MB/s is pretty decent speed tbh. On my Incrementals, I run anywhere between that and 1GB/s (but mostly am around 200-500MB/s). The answer to your question “should you implement DirectSAN”?...well, do you need to even? Is there a need to increase your backup speed? If not, it’s wise not to incorporate more complexity into your VBR setup. And, how many VMs do you have? My guess due to your VBR setup is not many? (i.e. < 100?) At the very least, what you could do is add a couple VM Linux Proxies (Linux is free so no licensing cost) and run a Job or 2 you have using them and test out Job speeds using hotadd. I created a post a while back on how to get a Linux Proxy up and going. See below:

Hope that helps.

Thank you for your response.

In our environment, we are using HPE Alletra storage, HPE StoreOnce as the backup repository, and a physical Windows server running both Veeam Backup & Replication and the backup proxy. Currently, backups are running in NBD mode and we are typically getting around 200–300 MB/s.

You mentioned that NBD performance depends on the architecture and that a 10+ Gb network can provide good performance.

Could you please suggest what areas I should check or optimize to improve NBD backup performance in my environment? Also, at what backup throughput would you consider moving from NBD to Direct SAN mode?

We are currently using the same physical Windows server as both the VBR server and backup proxy.

Would it be a good approach to keep the VBR server on the physical Windows server and deploy a dedicated Linux proxy VM in the ESXi environment instead?

Are there any performance or security benefits to this design?

Thank you.

As long as the physical box is beefy enough to run the roles of backup and proxy you should be fine.  If anything I would keep the physical box as a proxy and move VBR to a VM, but that is just how I would do things.

The move from NBD to Direct SAN is subjective based on your environment.  You would need to do some benchmarking to see how things run for say a week on NBD then switch to Direct SAN and test for the same period.  Gather the data and compare it.

There is no right or wrong answer here it is just related to architecture and your requirements or wants as they say.

Thank you for your response.

Our ESXi hosts have 4 x 10GbE uplinks connected to TOR switches, and currently we are only getting around 200–300 MB/s in NBD mode. Considering the available network bandwidth, this throughput seems lower than expected.

Would you still consider this normal, or do you think there may be a bottleneck in our environment? What areas would you recommend checking to improve NBD performance before considering Direct SAN?

Also, I apologize if I am asking too many questions. I am relatively new to Veeam and am trying to understand the different backup transport modes, storage integrations, and best practices. Your insights and real-world experiences have been very helpful in my learning process.

Thank you for your patience and support.

​@K Karthik - no problem at all. Happy to help where we can and offer suggestions (and keep in mind...we offer suggesions, not official recommendations. You’ll have to make the best determinations for your own environment). Also, you’re not going to saturate all your whole network bandwidth. There will be overhead on it as well. Additionally, what transport mode you go with will be dependent on your RPO/RTO SLAs. Are you experiencing app latency at all? Etc. All these things and many more help *you* determine what transport mode is best for your environment. Read through the Veeam Best Practice Guide on Transport Modes:

https://bp.veeam.com/vbr/Support/S_Vmware/backup.html

As well as the User Guide:

https://helpcenter.veeam.com/docs/vbr/userguide/transport_modes.html?ver=13

Then give your environment another looking at and make the best decision. No decision will be “perfect”. And, it’s ok to try and tweak things here and there after you implement something.

​@K Karthik 

You might benefit from starting with the VMware FAQ from our RND forums

A lot of your questions are answered there with the pros and cons as well. While FAQ was written with Veeam 12.3 in mind, nothing significant has changed with regards to VMware VM backups (there have been some nominal improvements on hotadd performance but nothing too outstanding)

As others said, it’s about testing and your operational needs.

• Does your current backup window meet your needs or do you have requirement to decrease the backup window?
• Do you plan on adding more workloads to Veeam or the workload state will remain constant?

Basically, if it’s not broke don’t fix it IMO. Faster backups are cool, but DirectSAN is not free and has considerations, for example, right now you have no redundancy as the backup server is your only proxy; Veeam Support can help troubleshoot issues of course but troubleshooting takes time and that means no backups / restores if there’s issue with the backup server as a proxy; using other proxies (virtual ones) may be advisable.

The FAQ covers this but I will repeat it: there is a common misconception that VMware transport mode performance always goes DirectSAN > Hotadd > NBD > NBDSSL

But this is outdated model, and even when it was relevant, hotadd typically could match or exceed DirectSAN performance on both backups and restores. (also NBDSSL is performant for some versions now, so please use NBDSSL)

And don’t forget about restores -- a common “gotcha” moment I see with DirectSAN is customers implement it and see amazing backup speeds, and then try a restore and become disappointed because you 

• Must restore with thick disks to get good performance
  • Eager-zero typically is faster but this depends on your storage vendor
  • Eager-zero typically will have a long “waiting” phase depending on the size of the VM you need to restore, that allocation takes time
• Even with thick disks for restore, you still won’t necessarily get speeds close to your backup speeds, and this is purely VDDK stack limitation

Get the idea? DirectSAN can be very good, but you need to plan for it, and there should be a reason you are doing it. If you’re only backing up a few hundred VMs and < 250 TiB of data, I wouldn’t bother with DirectSAN. (numbers are purely whispers from devils in my ear, but at first blush seem appropriate)

I recommend start with the FAQ, and then write your Veeam Account Representative to discuss solution architecting -- your environment as I see it:

1. One physical backup server acting as
   1. Backup Server
   2. Proxy server
   3. Gateway server for StoreOnce
2. StoreOnce Repository (unclear if it’s also FC connected; current best practice is network whenever possible, it will be less complex, more stable, and comparable speeds, plus it means more flexibility with proxies)
3. Alletra production storage connected via Fiberchannel (as I got it, maybe I misunderstood)

Backup from Storage Snapshot (BfSS) may be the answer here, but you have an architectural consideration because of the need for a gateway server for your Storeonce, and right now that role falls on your Backup Server, and it’s unclear how your environment is sized.

I strongly recommend engage with a Veeam Account Representative to go over your available infrastructure and find the “best” solution. Off the cuff but not an official recommendation, if NBD can fit all your backups into your backup window, I’d shift to virtualized proxies with hotadd (failback to NBD) to remove some of the workload off the backup server and add redundancy to the backup / restore operations. 

Our ESXi hosts have 4 x 10GbE uplinks connected to TOR switches, and currently we are only getting around 200–300 MB/s in NBD mode. Considering the available network bandwidth, this throughput seems lower than expected.

Also want to comment on this -- NBD has a limit on max speed per connection (disk), usually ~ 1 Gbit/s (so 120MB - 128 MB /s). You may see it peak over that sometimes but it will level out for larger items.

the 10GbE uplinks are fine, the issue is NBD limitations, not your network here. 

I think I heard of that limitation in a VMCA course awhile ago David, but certainly forgot that. That’s good info on NDB. Thanks!