Quick performance tip for single repository (Windows) users with a single or spanned volume.

  • 20 December 2023
  • 6 comments
  • 85 views

Userlevel 7
Badge +8

I’m planning on making a better writeup with my settings and some photos, but for now this is on my mind after helping someone today. 

 

I often see Veeam setups with 1 large datastore / volume in Windows for their repository. Other times quite often it will be multiple disks spanned into a single logical Volume, lets call it D:\.

 

You can still get good performance like this, and the simplicity is quite nice, but can it be better? If you have some spare storage kicking around I encourage you to try this in your lab.

 

Run your benchmarks on this single volume (D:\) record your results and then then delete the logical and physical volume. Create 3 new equal sized volumes with matching logical volumes. Lets call them E:\ F:\ and G:\. Next run IOMeter or your favorite benchmarking tool.  Make sure you are pushing the queues to see the benefits of multiple physical/logical volumes. 

 

The next thing you can do is check and modify queue depth on the adapters and Windows server. With a large backup job, there is often many parallel tasks completing at once. Because you have fresh benchmarks, run them again and see the difference. Quite often SANs and Windows are designed for fairness, VMware is like this too. This is because there is usually many servers connecting at once where as a Veeam Repository is often the only server connected. Increasing the queue depth, and using multiple disks allows you to actually push the IOPS and throughput much harder.

 

In 15 minutes on a call with someone today, their benchmarks went from about 130k IOPS to 600k.  I didn’t even touch the queue depth and only created a few volumes to show them for an example. The Latency was still at 1ms as well.  I also had their proxy transferring at 40Gbps data as well which was much higher than they have ever seen.  They were shocked when their next Veeam backup job showed them 5GB/s in the console for the first time ever. 

 

It’s always good to monitor things like latency from within the server and on the SAN as well. Quite often the SAN is sitting at or below 1ms and Windows is swamped begging to chew up that disk. If you have a VMware environment, there is another level to the equation.  I will state that if you don’t know what you are doing or the workload, sometimes it’s best to avoid messing with queue depth in  VMware because things are created for fairness. In simple terms, it stops one VM from overloading the SAN, LUN, Adapter, etc. That being said, if you deal with SQL servers, more often than not you do want to prioritize them over other VM’s. This is where setting things per host, per datastore, per adapter come in to play. There is always a bottleneck, but you can choose where and for what. It’s amazing the information you can get from Resource Monitor, ESXtop, and storage. 

 

Even on a single SAN and repository setup, creating it as a SOBR with multiple logical/physical volumes can increase your performance significantly. Just make sure each volume has the space required for large VM’s and their datasets. Also make sure there are enough disks, or fast enough disks in the array.  These days, with large SSD pools, NVMe pools or large amounts of cache, sometimes things require some extra tweaking to get the full benefit out of them. 

 

When I get time to write some of this up, I’ll show some helpful commands for monitoring and modifying these things. I’ll also include a disclaimer 😄. No matter what you do, record your settings to know what you changed, and benchmark every step of the way. Know what’s running on your systems and what workload will be running as well. Quite often, if you are not pushing your disk, you will notice no change and can potentially make things worse off. 

 

 

 


6 comments

Userlevel 7
Badge +20

Some really great tips there Scott.  Thanks for sharing this.

Userlevel 7
Badge +17

Good stuff @Scott . And, with regard to SANs...do not forget Jumbo Frames from end-to-end. Looking forward to the more detailed post.

Userlevel 7
Badge +8

Good stuff @Scott . And, with regard to SANs...do not forget Jumbo Frames from end-to-end. Looking forward to the more detailed post.

 

I use Fiber in my environment, but that is very true for network/iSCSI/SAN Replicaiton/VMware environments . 

It’s easy to install a server/DB/etc. Tuning an environment to preform it’s best is a bit of an art. Choosing specific block sizes for the right workload, queue depth, jumboframes, to QOS rules. That’s not even touching the SQL or Server config / layout.

 

I used to hate the answer “It depends” when I was trying to learn, but 99% of the time it’s the right answer. What works better for one person may be the exact opposite setup another wants. 

 

 

Userlevel 7
Badge +17

I still don’t think I configure my storage right. Yet, things seem to be pretty stable and performant mostly, so I guess I’m doing something right 😂

Cheers Scott

Userlevel 7
Badge +6

Good stuff @Scott . And, with regard to SANs...do not forget Jumbo Frames from end-to-end. Looking forward to the more detailed post.

I’d be curious to see if there was a noticeable difference between using jumbo frames and not in backup performance.  Also, where that difference is felt...if it matters if using jumbo frames to your backup repository vs jumbo frame for SAN access, etc.  My suspicion is that it won’t make a huge difference, but I’m betting there would be something.

Userlevel 7
Badge +6

I’d never considered using a SOBR like this for performance.  My first dip into a SOBR was using SSD’s as a performance tier and spinning disks as a capacity tier.  This was in my lab and may actually still be configured this way.  With that said, my lab doesn’t have a lot of VM’s and is really less than idea currently, so I don’t know if I’d see much if a difference even there.  But this is interesting in what you’re doing and how this works…..something I hadn’t considered but should give some further thought to.  Thanks Scott!

Comment