VeeamON 2024 - Use Code "COMMUNITY10" for 10% Off!
Thanks again Link State, I am aware of and up to that. I have running Windows Defender and Sophos Endpoint Agent on the hanging VM. In a first step I added all exclusions to Defender. That didnt help. Next I will deactivate defender completely. If that fails too, I will deactivate Sophos Endpoint Agent.
What I forgot. Its not a random problem only. The jobs get stuck out of nothing too. So one second the throughput speed is 100 MB/s+ and the next second it drops to zero and stays there for hours - until retry1 starts.I am really thankfull for all advices here. I will take a closer look to the ports topic. Can any1 answer my above question, whether Sophos firewalls or Sophos Endpoint Agents might cause problems?
Thanks Link State, I learned a lot again :) I knew that specific ports are important, but i didnt know that Veeam doesnt open the needed ports by default… So I quickly checked ports on the proxy. The result is: Port 135 (TCP) Microsoft Windows RPC Port 139 (TCP) Microsoft Windows netbios-ssn Port 445 (TCP) Port 2500 (TCP) Port 2501 (TCP) Port 2502 (TCP) Port 2503 (TCP) Port 2504 (TCP) Port 3389 (TCP) Tunnel is Microsoft SChannel TLS: unknown service Port 5040 (TCP) Port 6160 (TCP) Port 6162 (TCP) Port 6190 (TCP) Port 6290 (TCP) Port 11731 (TCP) Port 19500 (TCP) Tunnel is Microsoft SChannel TLS
Well those logs are huge… As an example I took out the logs for the following job, which got stuck and failed:This the link to the Backup Job log:https://ipblaw-my.sharepoint.com/:u:/g/personal/bachinger_ipblaw_at/EUaXgM5aG8RBovC0yU_rISQBhJnGNJiUlgD5nKkQn-ra3w?e=IZmyOWThis is the link to the Task log:https://ipblaw-my.sharepoint.com/:u:/g/personal/bachinger_ipblaw_at/EaYA92AMsy9NiV1955iRhgsBLzOdFAPJ9V55CxAwLzj2ew?e=1vJegQ
Well, my backup jobs still get stuck. some people here wrote that firewalls can cause stuck jobs. Well we have a Sophos XG 85 and Sophos Endpoint Agents installed on the hanging VM servers. Are there any known problems with Sophos firewalls or Endpoint Agents?
Just for your information: The retry1 of the hanging job was successfull without any problems. As you can see, the retry1 took place from 09:10 am to 12:56 pm. At this time the backed up servers have much more load than in the night.
Unfortunatly support immediatly closes all opend cases due to personell shortage….
Unfortunatly I didnt have to wait until sunday… Tonight the backup job hung up again. Its a bit frustrating, because its not recognizable, why the job gets stuck. It happens absolutly randomly, so I am really clueless what to do.New today is the error message:I belive the message occured because I killed the “Veeam Agent for Microsoft Windows” service. I dont really need the Agent. So is it advisable to uninnstall the Agent?The need of continuously manual interventions is really frustrating. I run a small business. So its no big deal, but how do large companies handle such problems?Is there a (built in) way to shorten the duration of hanging jobs? Like fall into retry1 after 15 minutes of zero data processed or transported?Is there a detailed log of the reason why a job gets stuck?
Thanks again Adanet, the first try in network transport mode was successfull. The backup job was done even faster than in virtual appliance mode. I will report after saturday, when next full backup job to linux NAS will have been done.
Dear Andanet,I doubt that the SQL Installation on the Win 2016 Server interferes the backup job, because the backup is run during night (starting at 1:30 am). The database, the guest machine and the host are in idle mode at that time. No users or other machines access the server (except the backup server ofc).Could you tell me how to identify unused files in the datastore folder? There are several log-files in it. The other files seem to belong to the VM.
Thanks for your quick reply Andanet. Due to several best practice advices, I setup a single backup job for two VMs. Both VMs run on the same physical host and one virtual disk (2,73 TB). One is a Win Server 2012 R2 with an Exchange 2013 Server installed on it (300GB). The other is a Win Server 2016 with SQL installed on it (2,3 TB). Only one VM gets stuck. In most cases the Exchange VM finishes successfully and the other gets stuck a little later. But I also had the opposite case already.I checked the VSS writers and they all show “last error: no error”. But i will keep an eye on them (thanks for the avice).I checked the datastore folder, but unfortunatly I dont know how to identify unused files…Queiesence has never vbeen checked.CBT has been check by default, i never changed that.For the next try, I changed the transport mode from “virtal appliance” to “network”.
Since I installed the new version 12.0.0.1420, I get this error message: “Processing xyserveryx Error: Transmission pipeline hanged, aborting process”. This message appears almost 5 hours after the process hung up… So is there at least a possibility to shorten the time until the defective job aborts and falls into retry? I cant find any setting options...
Unfortunatly I cant close this topic yet. I updated Veeam to version 12.0.0.1420 recently, because I read that the new version solves some issues. Well, it didnt solve mine… I still have the problem that Veeam backup jobs gets stuck completely randomly. The new version seems to make the problem even worse, because the stuck job doesnt fall into retry. I really optimized the whole infrastructure. So I upgraded the memory of the backup server and host to be backed up. The backup job is executed at night (1:30 am) so nobody interferes the job. Still it gets stuck… Are there any good advices out there? The Veeam support doesnt do anything and closes all cases immediatly...
In retry3 the backup job now got stuck in exactly the same completion position as in retry2 (processed 573,3 GB). Thats exactly the same processed data as in retry2. Could that mean that the data I try to backup is kind of corrupted or something like that? As I mentioned above, yesterday and the day before all backups ran smooth and successfull….
After two days of all successfull backup jobs, tonight one backup job got stuck at 20%, then at 21% in retry1 and now at 24% in retry2. All three atempts together took 12 hours so far… The job itself is an incremental backup job of a fileserver with an SQL server installed (nothing more). Data on the server have been almost unchanged from yesterday, so the throughput shows almost read only at an average 114 MB/s. Then througput speed suddenly drops from 185 MB/s to 0KB/s… I am a bit clueless...
Hi Andanet, I use a very simple and basic setup. I run Veeam on a virtual Windows 2022 Server, which is installed on an ESXi 6.5 Host. The only Software on the Server is Veeam B&R 12. This backup server is my standard gateway also. There is no duplication or deduplication.I have no storage infrastructure on Veeam. I just have backup repositories, one proxy and jobs. I run 2 subsequent backup jobs, one subsequent backup copy job and one subsequent RDX full backup job. All jobs are scheduled one after the other.
No snapshots or replications around. I have a subsequent backup copy job, but this job doesnt start until the restore point of the failed job appeared...
Hello Moustafa, good to see you again :) I use transport mode “Virtual Appliance” and I added all Veaam sources to exclusions. The bottleneck on the failed job was: Source 30% > Proxy 8% > Network 10% > Target 6%. The bottleneck on the successfull retry was: Load: Source 0% > Proxy 7% > Network 0% > Target 3%
For example log file for the hanging job says: Resource not ready: gateway server Processing finished with errors at 20.03.2023 21:17:37 This would match witch my perception, that the “backup service” got completly stuck.
I went through some log files but couldnt find anything unusal. The logs basically show that I tried to stop hanging jobs and then restarted the backup server… because of the tons of log information, could some1 advice me, which log file(s) is/are most important?
Well I doubt its caused by a lack of CPU power, because some days all jobs run smooth and success without retry and within an abslolut acceptable time (incremental as well as full). The failures I am facing are absolutly random. But I will collect logs and post them. Tankes in advance :)
Done!
Well I marked the wrong answer accidently… I dont find a way to change the “best answer”. Is there a way?
Thankes Chris, your advice was golden. I know there should be a more differentiated user and user-rights strategy, but I simply tried to change the Veeam Services from “local system” to “domain admin” account and the Pre-Job script works like charming. I might create an own user for the Veeam Services (having sufficient rights), but for now I am happy that I found out the reason :) Thankes again!
Thankes guys, but i went through all those documentations already. Problem is, that Veeam doesnt tell me anything about the reason or lack of rights or something like that. The Pre-Job scripts simply doesnt launch. The whole backup job gets stuck by that. When I manually stop the job, I get “ Error running pre-job script: Operation was canceled by user” or a simple “timed out”.Meanwhile I found out by myself, that the Veeam Console is run by either local or domain Adminstrator account, while the Veeam Services are run by the local system account. So… can any1 please explain, where in Windows I can assign what rights and privileges to local system account, that Pre-Job Scripts are launching? In my understanding the “local system account” should have highest privileges anyway, but I might be wrong...
Already have an account? Login
Enter your username or e-mail address. We'll send you an e-mail with instructions to reset your password.
Sorry, we're still checking this file's contents to make sure it's safe to download. Please try again in a few minutes.
Sorry, our virus scanner detected that this file isn't safe to download.
