Signifcant Performance Issue Even Though No Changes Made

I guess the first question is what changed (if anything) on Friday?  Since you updated on Sunday and the problem was first detected Friday it seems like it is not the update.  Based on the bottleneck being the Target now (Repo) I would investigate there first to see.  Not sure what to suggest as everyone’s environment is different and hopefully support can get to the bottom of it with a log deep dive.  Best of luck and it will be interesting to know what the issue is.

Forgot to share one of the biggest things - no changes were even made! 😁 Interestingly, I did make a slight Repo storage and array change to another subset of jobs (1 job actually) which are working fine. But nothing was done to this perf problem group of Jobs 🤷🏻‍♂️ Thanks Chris.

Well as we tell others best to work with support at this point to narrow things down.  😋

Best of luck. 👍🏼

What is networking looking like underneath the repositories and proxies, do you see similar throughput during backup windows?

I haven’t yet checked network throughput during Windows. Going to run FIO test (Veeam KB) on the Repo (since my bottleneck since changed from the “source” to “target”). But, mgmt network is different than storage network. So my assumption is this FIO test will hopefully tell the story. Will update after I run it.

Sounds like my fiber SFP issue that I had a while ago. was very intermittent causing latency spikes. Only effected 1 database causing “Application issues” which were also intermittent and kept giving different results resembling anything but a storage/SFP issue. 

VeeamONE actually triggered some latency alerts finally and saved the day.

Back to your issue, did you have any custom settings or registry changes before the upgrade? Is it possible they have reverted back to the standard vanilla settings?

As far as the “subsets” are the networks the same for both Linux environments? It seems odd that only one would be effected. 

Does it happen when you run an active full or create a new job using that proxy/repo?

Could it potentially be a failing disk causing latency but not actually failed yet? Do you have any S.M.A.R.T data on the drives or something to look into hardware issues, latency etc? 

Additionally, I’d try running some benchmarks on both of your Linux repos and see how they respond. If it’s poor I’d start looking into disk/server type issues. It may not be the upgrade at all.

If they benchmark fine, I’d look at configs, networking or something further up the chain. 

Hi Scott...thanks for offering some suggestions. Yeah..the more I’m looking into this, the more I think some kind of network issue. I also had a similar issue like this yr or so ago. Veeam has a cool diskspd tool I used to determine latency on a Proxy/Repo combo box I had, but a 2nd combo box ran fine. It was a fiber issue (got either cut or bent, but regardless had to be replaced). This issue wouldn’t be that as the data is not going to a ‘local offsite’ array but rather an array not only in the same DC, but same rack 😂 So, what I’m thinking at this point (about to run a Linux FIO test) is maybe a bad twinax cable or something like that...either on the Repo host or somewhere there. Will update when I can. BTW..checked my array (good suggestion there..hadn’t thought of that)...but no bad disks there. The Nimble UI ‘Hardware’ tab would show if one is bad or not running properly. Thanks man.

Glad to see you resolved the issue and it was something simple like a cable.