Veeam Proxy Servers Maxing Out RAM – How to Identify Jobs Causing Spike?

Hi ​@fem -

In your Jobs history (double-click a Job), look at the Task list on the right and scroll down to the end. You’ll see a performance or “Load” row/line like below:

Start there and see if one shows your Proxy being the highest metric.

In order to see which job you need to check the running jobs at the time of the spike and in the details pane it will show you which VMs and disks are being processed by the proxy.   Typically proxy servers if VMs are 8 vCPU and 16GB of ram then you add more as needed.   If physical then it is different.

Check the status window to see and take it from there.

coolsport00 wrote:

Hi ​@fem -

In your Jobs history (double-click a Job), look at the Task list on the right and scroll down to the end. You’ll see a performance or “Load” row/line like below:

Start there and see if one shows your Proxy being the highest metric.

Thanks for getting back to me. What if the history doesn’t go that far back? Is there a way to get the details from the proxy or VBR logs? If so, could you guide me on how to do that?

fem wrote:

coolsport00 wrote:

Hi ​@fem -

In your Jobs history (double-click a Job), look at the Task list on the right and scroll down to the end. You’ll see a performance or “Load” row/line like below:

Start there and see if one shows your Proxy being the highest metric.

Thanks for getting back to me. What if the history doesn’t go that far back? Is there a way to get the details from the proxy or VBR logs? If so, could you guide me on how to do that?

You could try looking at the logs which are here - C:\ProgramData\Veeam\Backup\

But you may have a much harder time checking there than watching the line above or when the job runs and the status window.  The above line in the status window is a good indicator.

What mode are your Proxy servers set to?  Automatic?  Network?  Maybe it is the mode that is causing the issue possibly.  Transport Modes - User Guide for VMware vSphere

There should be logs on the Proxy. Is it a Windows Proxy? It’s been a while since I used Windows for my Proxy, but if memory serves, logs should be in:
C:\ProgramData\Veeam\Backup & Replication. I think for Linux they’re in /var/logs

Check there for Job logs.

Hi ​@fem 

the tips from ​@coolsport00  and ​@Chris.Childerhose  are great.

 It is best practices to plan for 1 physical core or 1 vCPU and 2 GB of RAM for each of the tasks. A task processes 1 VM disk at a time and CPU/RAM resources are used for inline data deduplication, compression, encryption etc.

Proxy servers require 2 GB RAM + 500 MB per task. Please consider these values as minimum requirements.

• To monitor the RAMmap-exe in real time and analyze it use rammap on the two proxies to check the processes that cause ram consumption

RAMMap - Sysinternals | Microsoft Learn

• you can also schedule a task to run this powershell script that writes a csv fiule with the report of the processes that consume the most ram on your proxies.
   

# Get Computer Object
 $CompObject =  Get-WmiObject -Class WIN32_OperatingSystem
 $Memory = ((($CompObject.TotalVisibleMemorySize - $CompObject.FreePhysicalMemory)*100)/ $CompObject.TotalVisibleMemorySize)

 Write-Host "Memory usage in Percentage:" $Memory

 # Top 5 process Memory Usage (MB)
 $processMemoryUsage = Get-WmiObject WIN32_PROCESS | Sort-Object -Property ws -Descending | Select-Object processname, @{Name="Mem Usage(MB)";Expression={[math]::round($_.ws / 1mb)}}, @{Name="Datetime";Expression={get-date}}
 $processMemoryUsage | export-csv 'c:\pathtofile.csv' -append

or

 Get-WmiObject WIN32_PROCESS | Sort-Object -Property ws -Descending | Select-Object -first 10 ProcessID,Name,WS

• Decrease concurrent jobs and check the proper sizing of proxies against the jobs and their respective VMs by calculating how many disks each VM has.  If you do not want to decrease concurrent jobs, you may as well not increase the resources of the existing proxies, but you can scale horizontally by adding proxies so as to distribute the daily workload equally and not exceed the desired backup window.

here are some interesting links for proper sizing of proxies;

Limitation of Concurrent Tasks - User Guide for VMware vSphere

vSphere Proxy | Veeam Backup & Replication Best Practice Guide

calculation of backup proxy need | Veeam Community Resource Hub

Calculating 🤓Required Proxy Tasks 💪 | Veeam Community Resource Hub

Limitation of Concurrent Tasks - User Guide for VMware vSphere

regards

It is worth checking the jobs running at the time when high memory consumption occurs, especially if more than one job is running simultaneously. Some tasks, like merge, can consume a significant amount of memory. Additionally, it's important to assess if there is any WAM Accelerator active, as it can also impact memory usage depending on the volume of data being processed. Identifying which tasks are being performed and how resources are allocated can help diagnose the issue.

It would be helpful to send us the logs, as requested by our colleagues, to facilitate understanding the environment and better identify the cause of the problem.

@Chris.Childerhose The proxy servers are set to Automatic Selection transport mode. The last time I had the spike was 2 days ago, so monitoring the line isn't feasible right now.

@coolsport00 They are Windows proxies. My issue is knowing which file or folder to focus on in C:\ProgramData\Veeam\Backup . Any suggestions?

Are you utilizing Veeam One in your environment? You may be able to see more job history there, using the Job Calendar, or other reports.

You also mentioned that you can’t see history going back 2 days. You can increase the session history by following this article, but be careful, I’ve seen instances where setting these too high will slow down the Veeam console quite a bit. Try bumping up the “Show only last XXX sessions” setting in increments until it shows the session you’re looking for.

​@fem -

You would start by looking through your Job History in the Veeam Console and see which Job, if any, have high Proxy “load” as I shared above. Then from there, you go into the job named folder. There are several “types” of logs in there. The “source” logs would focus on the connection of to where your source VMs reside; “transform” logs are for any merge process during the Job; “compact” should be for the maintenance task of compacting (shouldn’t be configured to do often..1x a week or so); and so forth. Those are just guesses as I’m not a Veeam log analyst 😊 ...but pretty sure that’s what they’re for. For specific log file analytics, it would be best for you to contact Veeam Support. They’ll know exactly what logs to look at. They have a fairly decent performance team. I’ve used them a few times the past couple yrs.

Best.

​@MicoolPaul This definitely helps. I will get the RAM increased on the VMs and see how it goes.

Amazing, if you want to supply more infrastructure design here we’ll happily try to input into what the specs should look like or other suggestions (e.g. above 8vCPU on a single VM is normally suboptimal based on NUMA alignment for VMs)

​@fem  - were you able to get an answer to your question?  Please update the thread so we can select the best answer for the post to close out the question.  If not answered please continue the discussion.

​@Chris.Childerhose Yes, I got the answer. The issue was resolved by adding RAM.

Great glad to hear that you resolved the issue and it was simple.

I was going to add, if your server isn’t busy all the time, just spread out the tasks more.

I didn’t see how many VM’s you had in that one job, or if you were running jobs concurrently, but you could create more backup jobs, and move the VM’s so less are running at a time.

Another option is to limit the concurrent tasks for that proxy

• Under Backup infrastructure
• Backup Proxies
• Right click the proxy, Properties
• Adjust the Max. concurrent tasks lower

Job will run for a bit longer, but won’t use as much CPU/Memory while it does. 

VeeamONE and Enterprise manager are both great tools to see job history. VeeamONE actually has a calendar type schedule of your jobs. 

I believe another free option would be use the Veeam HealthCheck tool to get more insight on your environemnt.

https://github.com/VeeamHub/veeam-healthcheck