Veeam Proxy Servers Maxing Out RAM – How to Identify Jobs Causing Spike?

Hi ​@fem -

In your Jobs history (double-click a Job), look at the Task list on the right and scroll down to the end. You’ll see a performance or “Load” row/line like below:

Start there and see if one shows your Proxy being the highest metric.

In order to see which job you need to check the running jobs at the time of the spike and in the details pane it will show you which VMs and disks are being processed by the proxy.   Typically proxy servers if VMs are 8 vCPU and 16GB of ram then you add more as needed.   If physical then it is different.

Check the status window to see and take it from there.

coolsport00 wrote:

Hi ​@fem -

In your Jobs history (double-click a Job), look at the Task list on the right and scroll down to the end. You’ll see a performance or “Load” row/line like below:

Start there and see if one shows your Proxy being the highest metric.

Thanks for getting back to me. What if the history doesn’t go that far back? Is there a way to get the details from the proxy or VBR logs? If so, could you guide me on how to do that?

fem wrote:

coolsport00 wrote:

Hi ​@fem -

In your Jobs history (double-click a Job), look at the Task list on the right and scroll down to the end. You’ll see a performance or “Load” row/line like below:

Start there and see if one shows your Proxy being the highest metric.

Thanks for getting back to me. What if the history doesn’t go that far back? Is there a way to get the details from the proxy or VBR logs? If so, could you guide me on how to do that?

You could try looking at the logs which are here - C:\ProgramData\Veeam\Backup\

But you may have a much harder time checking there than watching the line above or when the job runs and the status window.  The above line in the status window is a good indicator.

What mode are your Proxy servers set to?  Automatic?  Network?  Maybe it is the mode that is causing the issue possibly.  Transport Modes - User Guide for VMware vSphere

There should be logs on the Proxy. Is it a Windows Proxy? It’s been a while since I used Windows for my Proxy, but if memory serves, logs should be in:
C:\ProgramData\Veeam\Backup & Replication. I think for Linux they’re in /var/logs

Check there for Job logs.

Hi ​@fem 

the tips from ​@coolsport00  and ​@Chris.Childerhose  are great.

 It is best practices to plan for 1 physical core or 1 vCPU and 2 GB of RAM for each of the tasks. A task processes 1 VM disk at a time and CPU/RAM resources are used for inline data deduplication, compression, encryption etc.

Proxy servers require 2 GB RAM + 500 MB per task. Please consider these values as minimum requirements.

• To monitor the RAMmap-exe in real time and analyze it use rammap on the two proxies to check the processes that cause ram consumption

RAMMap - Sysinternals | Microsoft Learn

• you can also schedule a task to run this powershell script that writes a csv fiule with the report of the processes that consume the most ram on your proxies.
   

# Get Computer Object
 $CompObject =  Get-WmiObject -Class WIN32_OperatingSystem
 $Memory = ((($CompObject.TotalVisibleMemorySize - $CompObject.FreePhysicalMemory)*100)/ $CompObject.TotalVisibleMemorySize)

 Write-Host "Memory usage in Percentage:" $Memory

 # Top 5 process Memory Usage (MB)
 $processMemoryUsage = Get-WmiObject WIN32_PROCESS | Sort-Object -Property ws -Descending | Select-Object processname, @{Name="Mem Usage(MB)";Expression={[math]::round($_.ws / 1mb)}}, @{Name="Datetime";Expression={get-date}}
 $processMemoryUsage | export-csv 'c:\pathtofile.csv' -append

or

 Get-WmiObject WIN32_PROCESS | Sort-Object -Property ws -Descending | Select-Object -first 10 ProcessID,Name,WS

• Decrease concurrent jobs and check the proper sizing of proxies against the jobs and their respective VMs by calculating how many disks each VM has.  If you do not want to decrease concurrent jobs, you may as well not increase the resources of the existing proxies, but you can scale horizontally by adding proxies so as to distribute the daily workload equally and not exceed the desired backup window.

here are some interesting links for proper sizing of proxies;

Limitation of Concurrent Tasks - User Guide for VMware vSphere

vSphere Proxy | Veeam Backup & Replication Best Practice Guide

calculation of backup proxy need | Veeam Community Resource Hub

Calculating 🤓Required Proxy Tasks 💪 | Veeam Community Resource Hub

Limitation of Concurrent Tasks - User Guide for VMware vSphere

regards

It is worth checking the jobs running at the time when high memory consumption occurs, especially if more than one job is running simultaneously. Some tasks, like merge, can consume a significant amount of memory. Additionally, it's important to assess if there is any WAM Accelerator active, as it can also impact memory usage depending on the volume of data being processed. Identifying which tasks are being performed and how resources are allocated can help diagnose the issue.

It would be helpful to send us the logs, as requested by our colleagues, to facilitate understanding the environment and better identify the cause of the problem.

@Chris.Childerhose The proxy servers are set to Automatic Selection transport mode. The last time I had the spike was 2 days ago, so monitoring the line isn't feasible right now.

@coolsport00 They are Windows proxies. My issue is knowing which file or folder to focus on in C:\ProgramData\Veeam\Backup . Any suggestions?

Are you utilizing Veeam One in your environment? You may be able to see more job history there, using the Job Calendar, or other reports.

You also mentioned that you can’t see history going back 2 days. You can increase the session history by following this article, but be careful, I’ve seen instances where setting these too high will slow down the Veeam console quite a bit. Try bumping up the “Show only last XXX sessions” setting in increments until it shows the session you’re looking for.

​@fem -

You would start by looking through your Job History in the Veeam Console and see which Job, if any, have high Proxy “load” as I shared above. Then from there, you go into the job named folder. There are several “types” of logs in there. The “source” logs would focus on the connection of to where your source VMs reside; “transform” logs are for any merge process during the Job; “compact” should be for the maintenance task of compacting (shouldn’t be configured to do often..1x a week or so); and so forth. Those are just guesses as I’m not a Veeam log analyst 😊 ...but pretty sure that’s what they’re for. For specific log file analytics, it would be best for you to contact Veeam Support. They’ll know exactly what logs to look at. They have a fairly decent performance team. I’ve used them a few times the past couple yrs.

Best.

Hi ​@fem 

Unfortunately it seems your proxy server is underspec’d for what you’ve configured it to.

CPU: x86-64 processor with 2 cores (vCPUs) minimum, plus 1 core (vCPU) for each 2 additional concurrent tasks. Using faster processors improves data processing performance. For more information, see Limitation of Concurrent Tasks.

Memory: 2 GB RAM plus 1 GB for each concurrent task. The actual size of memory required may be larger and depends on the amount of data to back up, machine configuration, and job settings. Using faster memory improves data processing performance.

This comes from the v12 documentation of VBR.

You’ve specified 16 concurrent tasks, which means you need 10 vCPU allocated to the proxy as a MINIMUM for the proxy workload, plus 18GB of RAM MINIMUM for the proxy workload.

Consider that this is a minimum without any considerations to any bloat on the OS such as AV that might be causing further memory pressure or other auditing applications. Assuming this is Windows Server 2022 with GUI, the OS needs a minimum of 2GB of RAM and 4GB RAM is recommended. This still leaves you at a 2GB loss on minimum recommendations.

Also consider anything else running on this server will also impact RAM allocation, such as if the proxy is also acting as a repository etc.

Start by reducing the number of concurrent tasks and see how this impacts your RAM spikes, or recalculate the amount of RAM required and allocate as appropriate.

Hope this helps!

​@MicoolPaul This definitely helps. I will get the RAM increased on the VMs and see how it goes.

Amazing, if you want to supply more infrastructure design here we’ll happily try to input into what the specs should look like or other suggestions (e.g. above 8vCPU on a single VM is normally suboptimal based on NUMA alignment for VMs)