Veeam agent based backup is failing, SQL Appaware is not enabled, but backup is failing due to SQLServer writer in non-retryable error. Has anyone came accross this?
Found this in event application logs:
Write on "{1ABD5CD5-0679-4CAC-8E24-9D5844B071D7}64" failed: 995(failed to retrieve text for this error. Reason: 15105)
+17
Hi
which Version of MS SQL and Windows is installed on this machine?
Hello
+17
Ok, that’s rather old, isn’t it. Both versions are not supported anymore by Microsoft.
I found some hints that MS SQL 2008 had this error with several backup application, but I have not found a concrete solution up to now.
Yes, they are. Usually we used to take a reboot when the failure occurs and backup used to run fine post the reboot for a few days. But, this time, post multiple reboot too backup is failing.
Had logged a case with support, didn’t find much solution on this.
Case # 05850789
+17
May I ask which Veeam Version you are using?
The newest agent supports Windows Server 2008 R2 SP1 and and MS SQL 2008 R2 SP3
Veeam B&R 11 is used in our environment.
Agent version - 5.0.3.4708
+20
Veeam B&R 11 is used in our environment.
Agent version - 5.0.3.4708
Those should support the versions that you have.
It does support, YES.
+20
I would check the logs to see if there is anything that you can find otherwise would contact support again.
Logs are here - C:\ProgramData\Veeam\Backup
Hello
[27.03.2023 21:50:12] <26> Info Offhost processing spec: ''
[27.03.2023 21:50:12] <26> Info Checking remote agent is alive...
[27.03.2023 21:52:12] <26> Info Checking remote agent is alive...
[27.03.2023 21:54:12] <26> Info Checking remote agent is alive...
[27.03.2023 21:56:12] <26> Error Sub-task [On-host backup] failed
[27.03.2023 21:56:12] <26> Error Exception of type 'Veeam.Backup.Core.CAmTaskSessionFailedException' was thrown. (Veeam.Backup.Core.CAmTaskSessionFailedException)
[27.03.2023 21:56:12] <26> Error at Veeam.Backup.Core.AgentManagement.COnHostBackupSubTask.WaitComplete(IStopSessionSync stopSessionSync)
[27.03.2023 21:56:12] <26> Error at Veeam.Backup.Core.CEpSubTasksCoordinator.ExecuteSubTask(CSubTaskContext subTaskContext, LogStorage logStorage)
[27.03.2023 21:56:13] <45> Info [CAmSubTasksCoordinator] All sub-tasks have been completed
[27.03.2023 21:56:14] <45> Info [AgentsSession] Disposing...
[27.03.2023 21:56:14] <45> Info [AgentsSession] Removing agents connection
[27.03.2023 21:56:15] <45> Info Cannot create a shadow copy of the volumes containing writer's data.
[27.03.2023 21:56:15] <45> Info A VSS critical writer has failed. Writer name: [SqlServerWriter]. Class ID: [{a65faa63-5ea8-4ebc-9dbd-a0c4db26912a}]. Instance ID: [{075cee4b-bf3a-4c35-adee-c401bf2eb314}]. Writer's state: [VSS_WS_FAILED_AT_PREPARE_SNAPSHOT]. Error code: [0x800423f4].
[27.03.2023 21:56:15] <45> Info [CEpAgentAuxData] Update LastUpdateTime '27/03/2023 20:56:15', ExcludeDate ''
[27.03.2023 21:56:15] <45> Info [DB] Sync agent [08578a1c-7129-47f2-948d-24af02bd0f61] lastOnlineTime [27/03/2023 20:56:15]
[27.03.2023 22:00:27] <39> Info [ReconnectableSocket][StopCondition] Stop confirmation was received on [6c8bb952-392b-48bd-962c-be02076d0d00].
[27.03.2023 22:00:27] <31> Info [ReconnectableSocket] Stop confirmation was sent on [6c8bb952-392b-48bd-962c-be02076d0d00].
[27.03.2023 22:00:27] <31> Info [ReconnectableSocket][StopCondition] Stop request was received on [6c8bb952-392b-48bd-962c-be02076d0d00].
[27.03.2023 22:00:27] <39> Info [ReconnectableSocket][StopCondition] Stop confirmation was received on [24136b23-a8b8-46a7-af4e-bdc31afa0231].
[27.03.2023 22:00:27] <31> Info [ReconnectableSocket] Stop confirmation was sent on [24136b23-a8b8-46a7-af4e-bdc31afa0231].
[27.03.2023 22:00:27] <31> Info [ReconnectableSocket][StopCondition] Stop request was received on [24136b23-a8b8-46a7-af4e-bdc31afa0231].
+20
Based on the lines below it seems there is an issue with the VSS writer for SQL.
[27.03.2023 21:56:15] <45> Info Cannot create a shadow copy of the volumes containing writer's data.
[27.03.2023 21:56:15] <45> Info A VSS critical writer has failed. Writer name: [SqlServerWriter]. Class ID: [{a65faa63-5ea8-4ebc-9dbd-a0c4db26912a}]. Instance ID: [{075cee4b-bf3a-4c35-adee-c401bf2eb314}]. Writer's state: [VSS_WS_FAILED_AT_PREPARE_SNAPSHOT]. Error code: [0x800423f4].
I would run on the SQL server - vssadmin list writers --- this will check the state. You might need to try restarting the SQL VSS Writer service to fix this and then try the backup.
The other option when it comes to SQL Server is to use a built-in task to create the backup to a drive which creates the native BAK files for SQL. Then back up that drive with Veeam. Another potential workaround if you cannot solve this with Support.
vssadmin list writers, lists out that sqlserver writer will be in non-retryable error. After getting the server restarted, writer will be in normal state.
When backup starts, same writer will change to non-retryable error again.
Here, backup is not app aware, normal agent based.
+20
vssadmin list writers, lists out that sqlserver writer will be in non-retryable error. After getting the server restarted, writer will be in normal state.
When backup starts, same writer will change to non-retryable error again.
Here, backup is not app aware, normal agent based.
Unsure what to suggest at this point other than a new support case to try and sort this out. Also possibly upgrading SQL to a more supported version will help but may not be feasible for you.
I don’t think upgrading to v12 will make any difference and also the supported version list gets smaller.
Also, I don’t understand the last statement to make a comment about it - “Here, backup is not app aware, normal agent based.”
Hello
“Here, backup is not app aware, normal agent based.”
+20
Hello
“Here, backup is not app aware, normal agent based.”
So if App Aware is not enabled then the backups should run without having to truncate logs, etc. I thought the errors were from App Aware backup.
Yes, it should be running. But it is not.
+20
Yes, it should be running. But it is not.
Then I would suggest at this point to submit a support case again and work with them. There is not much else we can do here on the community to help other than giving suggestions and tips that we know.
Im able to resolve it by restarting “SQL Server VSS Writer” services, but before that do check the writer services before doing the restart.
*Rebooting server doesn't resolve, solution below is not require reboot
Step1: Run “CMD” as administrator
Step2: Prompt “vssadmin list writers”
Step3: Indicate writer name: ‘SqlServerWriter’ State: [8] Failed Last error: Non-retryable error
Step4: Launch services.msc > locate ‘SQL Server VSS Writer’ services > Restart
Step5: Repeat step 2, until you see ‘SqlServerWriter’ State: [1] Stable Last error: No error
Step6: Retry backup job, if fail again repeat step1
Maximum i’v encounter were restarting two times ‘SQL Server VSS Writer’ services, only the backup able to perform snapshot and complete the backup.
Hi,
So if I understand correctly, you are using a standalone Veeam Agent for Windows (server) and disable app aware processing in this menu: https://helpcenter.veeam.com/docs/agentforwindows/userguide/backup_job_vss.html?ver=60
If so, then
Enter your username or e-mail address. We'll send you an e-mail with instructions to reset your password.
