Skip to main content
  • EN

Hey everyone,

 

Has anyone ever seen Veeam throttling bandwidth with the RMAN plugin?

 

I’m working with a really beefy Veeam environment at the moment. Source data is all on flash, 40GbE networking available, high performance RAID controller + 4x RAID 60 repositories in a SOBR.

 

Backups are flying for most workloads, VMware, Agent, NAS etc, but RMAN is letting me down.

 

For example, a Veeam Agent using an unoptimised network path via the firewall can still do over 200MBps (Firewall is the bottleneck here), but RMAN is doing 5-7MBps and it’s on an L2 network with no firewalls involved. Testing with iperf can get 6-8Gbps easily without tuning between the hosts and the backup repositories, one test with iperf between the backup repositories was giving 26Gbps, so the network really is moving quick.


When I’m going through the Veeam logs I’m finding things like this:

 

plugin_co|     u8b43] >> : Invoke: Network.UpdateThrottling\n{   (EInt32) KbpsReadBandwidth = 0\n  (EInt32) KbpsWriteBandwidth = 0\n }
plugin_co|     u8b43] out: Thread finished. Role: 'traffic throttler'.

OR

plugin_co|     ub876] >> : Invoke: Network.UpdateThrottling\n{   (EInt32) KbpsReadBandwidth = 7200\n  (EInt32) KbpsWriteBandwidth = 7200\n }
plugin_co|     ub876] out: Waiting till another side opens the pipe [/tmp/pipe{2a86813e-ae0c-4a1c-8dfb-a9285eaca677}] for reading

I typically see the throttling get set to zero at the end of a transfer, but during the transfer it’s being set to values such as 7200, or 5760 for read & write.

 

But the only network throttling I have configured is the default rule of ANY source to Internet, but this is private networking, RFC 1918 defined private subnets. I can’t help but feel this is related to the network bandwidth rule somehow, but I’ve got full object uploads running so I need to keep the bandwidth limitation in place.

 

I can’t find anything in Veeam’s documentation talking about this network throttling to know if this is my problem. The values don’t match with my networking throttling directly, and it raises more questions as it says KbpsReadBandwidth, not KBpsreadbandwidth, so is it capping at 7.2Mbps or 7.2MBps? as my backup jobs are typically running at 5-7MBps this would make sense if it’s actually KBps. My network throttling is set to 100Mbps, but as there’s an upload job it’s quite possible Veeam is splitting the pipe in half and granting roughly 50% resource to RMAN, thereby giving me the 5-7MBps range. As the workloads are distributed across the SOBR I can’t get these metrics so easily to see what is being uploaded. It would align with this however:https://helpcenter.veeam.com/archive/backup/110/vsphere/setting_network_traffic_throttling.html?zoom_highlight=bandwidth

 

When several jobs that fall into the same network rule run simultaneously, Veeam Backup & Replication equally splits the throttled traffic between them. For example, if two jobs run at a time, each job gets half of the throttled traffic. Once one job finishes, the other gets the entire bandwidth allowed by the rule.

 

Unfortunately network throttling doesn’t allow you to create superceding rules, just a rule of most restriction, so again, I can’t test without breaking my ANY to Internet rule.

 

I’ve opened a support case but I’m wondering if anyone in the community has seen a rogue setting that could be to blame here.

 

Thanks!

Mhh, could this be a setting in RMAN itself?

I have never seen a throttling option in the plugin configuration….

I wish, I’ve just done troubleshooting and I believe I’ve found a bug… Network Bandwidth for the ANY:Internet rule is impacting RMAN performance, doubling the throttle = double the throughput on my RMAN job, disabling the throttle made my job skyrocket with performance. 😩

 

I’m glad I’ve found a root cause though.

Ok… good finding. 👍🏼

I was not aware that this setting influences the plugin traffic...

I wish, I’ve just done troubleshooting and I believe I’ve found a bug… Network Bandwidth for the ANY:Internet rule is impacting RMAN performance, doubling the throttle = double the throughput on my RMAN job, disabling the throttle made my job skyrocket with performance. 😩

 

I’m glad I’ve found a root cause though.

Well, nice discover @MicoolPaul and thank you for sharing. I don’t have any throttle on our VBRs so i don’t have any perfomance issue.

Thank you for the info @MicoolPaul !

Update on this one. I tried to be clever and work around the issue, but the network traffic throttling behaviour is completely broken at this point.

 

Backup Repositories have 3x interfaces:

  • Production network, with a default gateway to upload to cloud
  • Backup network, for backup jobs, preferred networks makes Veeam use this first
  • Replication network, for backup copy jobs, preferred networks makes Veeam use this first

All IP ranges exist within /24 carvings of the 10.0.0.0/8 IP addressing space, but I won’t share the specific subnets here.

I’ve tried to deliberately rule out any possibility of Veeam making an incorrect judgement of my network by doing the following:

Disabled the default ANY:Internet rule

Created 7x “internet” rules, these are all different IP scopes to avoid private subnets, for example Rule 1’s target IP range is 1.0.0.1-9.255.255.255 and then Rule 2’s target IP range is 11.0.0.1-100.63.255.255, to prevent the 10.0.0.0/8 range being selected.

Each of these internet rules has a source IP range of ONLY explicitly the 2x IP addresses of the production network repositories (thank goodness they’re sequential). The backup network IP addresses aren’t referenced anywhere.

 

When the RMAN backup takes place, and the RMAN server, on its backup network IP address, communicates to the backup repository, on its backup network IP address, somehow the throttling rules still kick in!

Comment


Terms & Conditions