Just for your information: The retry1 of the hanging job was successfull without any problems. As you can see, the retry1 took place from 09:10 am to 12:56 pm. At this time the backed up servers have much more load than in the night.

Mhh, perhaps some connection problems?

I have one customer environment where a firewall gets overloaded in one night of the week and then some (not all) DNS queries do not get an answer in appropriate time. This brings some backup jobs or parts of the to fail… So, every single part in your environment can have influence on your backup jobs.

Hello @Michail  welcome to the community.
Can you attach all the parts with log errors of the respective job that randomly fails?

The Veeam job logs can be found in "C:\ProgramData\Veeam\Backup\"Job.<JOBNAME>.Backup.log" ".

Job/backup/replica task log
The tasks represent the virtual machines in the job itself.  
These logs will contain all errors of a virtual machine that fails during the job.

Tools required
- Notepad++ 
The only downside is that it cannot handle logs >200MB very well
- Vim 
For logs >200MB
- WinRAR 
For unpacking logs as well as being able to search text within the logs prior to unpacking
- BareTail 
BareTail mimics the 'tail' option typically found in Unix/Linux systems, where logs can be viewed in real-time

thanks

Well, my backup jobs still get stuck. some people here wrote that firewalls can cause stuck jobs. Well we have a Sophos XG 85 and Sophos Endpoint Agents installed on the hanging VM servers. Are there any known problems with Sophos firewalls or Endpoint Agents?

Can you provide the error logs as requested?
To make an anlaysis they are essential if you cannot understand the problem.

Regards.

Well those logs are huge… As an example I took out the logs for the following job, which got stuck and failed:

This the link to the Backup Job log:

https://ipblaw-my.sharepoint.com/:u:/g/personal/bachinger_ipblaw_at/EUaXgM5aG8RBovC0yU_rISQBhJnGNJiUlgD5nKkQn-ra3w?e=IZmyOW

This is the link to the Task log:

https://ipblaw-my.sharepoint.com/:u:/g/personal/bachinger_ipblaw_at/EaYA92AMsy9NiV1955iRhgsBLzOdFAPJ9V55CxAwLzj2ew?e=1vJegQ

Hi @Michail  there is to investigate these errors,

https://helpcenter.veeam.com/docs/backup/agents/used_ports.html?ver=120

Check whether all necessary TCp ports are open between infra.

It seems that the proxy? server AW1 does not have all the ports open

greetings

Thanks Link State, I learned a lot again I knew that specific ports are important, but i didnt know that Veeam doesnt open the needed ports by default… So I quickly checked ports on the proxy. The result is:

So what I read the above ports should be good enough? Or are there any ports I missed and should be opened on the proxy?

What I forgot. Its not a random problem only. The jobs get stuck out of nothing too. So one second the throughput speed is 100 MB/s+ and the next second it drops to zero and stays there for hours - until retry1 starts.

I am really thankfull for all advices here. I will take a closer look to the ports topic. 

Can any1 answer my above question, whether Sophos firewalls or Sophos Endpoint Agents might cause problems?

If you have the possibility try bypassing the firewall, disable antivirus or add exception

KB1999: Antivirus Exclusions for Veeam Backup & Replication

regards

Thanks again Link State, I am aware of and up to that. I have running Windows Defender and Sophos Endpoint Agent on the hanging VM. In a first step I added all exclusions to Defender. That didnt help. Next I will deactivate defender completely. If that fails too, I will deactivate Sophos Endpoint Agent.

Agree, as a test disable defender, open all ports between Veeam and the infrastructure and test. If it works, enable 1 thing at a time and find the issue. The list of ports in the Veeam documentation is a good place to start when creating firewall rules. 

Try a VM without the endpoint agenet. If you suspect it’s the agent can you disable it and test? can you clone the machine and rename it, remove the enpoint and test if it’s a production machine?   

Hi Andanet,

Any speciality on the setup with Datadomain? I am experiencing drastic performance degradation for backups due to the fact of “Resource not ready: gateway server” 

Hello @Kamson, 

deduplication appliance needs to have a gateway… normally is a proxy that have the role of bridging backup server and repository. 

https://helpcenter.veeam.com/docs/backup/vsphere/gateway_server.html?ver=120

Here you can find all infos needed. 

Consider a small thing. Best practice is to assign a dedicated gateway server to every dedup appliance to have best performance. It all depends from your infrastructure. If it’s no a large environmnet you can use a proxy server as a gateway BUT compute resource (RAM and CPU) will be shared with natural proxy role (data transport). 

How many concurrent tasks is right to set? 

Check your datadomain streams (model based) relate to your concurrent backup/copy job and you RPO. after this you can calculate right number of tasks. 

Another info…. Veeam talking about DD used as performance tier with some limits: https://helpcenter.veeam.com/docs/backup/vsphere/performance_tier_limitations.html?ver=120 

Cheers