Sometimes, we need to remember some basic concepts to understand what happens "under the hood" in operations we perform on a day-to-day basis, such as performing an application-consistent backup and how to troubleshoot it.
So, what is Microsoft VSS?
Microsoft Volume Shadow Copy Service (VSS) is a volume backup framework in Microsoft Windows Operating Systems.
It integrates with many applications and backup solutions, like Veeam Backup & Replication, to provide application-consistent backups.
Why is VSS used?
Some enterprise applications maintain transactional interdependencies between data files, logs, metadata, and in-memory data.
For these applications, an application-consistent backup ensures that all these transactional elements are synchronized when captured by the backup request
Here, we show examples of applications integration with Microsoft VSS for consistent backups:
- Databases: SQL Server, Oracle, MySQL;
- Email Server: Microsoft Exchange;
- Enterprise Applications: SAP, Microsoft Dynamics;
- Hypervisors: Microsoft Hyper-V, VMware ESXi (via VMware Tools), Nutanix AHV (via NGT), etc;
- File Systems: Windows File Server and Windows Distributed File Systems.
VSS Components
A VSS backup solution consists of the following components:
- The VSS Service is part of the Windows Operational System and is responsible for the communication between all other components
- The VSS Requestor is typically a backup application that requests snapshots. It can be, for example, a Windows Agent from Veeam Backup & Replication.
- The VSS Writer is an application component that ensures a consistent data set before taking the snapshot.
- The VSS Provider is in charge of creating and maintaining the snapshots. A provider can be based on software or hardware.
How does VSS work?
The diagram at the top of the page shows the interaction between all VSS components when processing a backup request.
Summarizing the processes, we have:
- Veeam Backup & Replication (or any VSS Requestor) initiates a backup request for the VSS Service.
- After the backup initialization step is concluded, the writer enters a Quiescence phase, ensuring that all pending transactions are completed and all in-memory data is synchronized with the data on disk.
- After that, VBR requests the snapshot, and the Writers enter a Freeze state. The application writes are held, and the caches are flushed to disk.
- After the Provider commits the snapshot, the Writer "thaws" the application to write I/O requests. Observation: the application freeze cannot take longer than sixty seconds.
- Veeam VBR imports the snapshot, and the backup operation is completed.
- Veeam VBR triggers VSS Service to remove the persistent VSS snapshot, and the VSS backup process is finished.
If a problem occurs in any part of this process, a troubleshooting action will be necessary.
Troubleshooting Microsoft VSS
First step: get familiar with the VSS Framework
Microsoft VSS:
https://learn.microsoft.com/en-us/windows-server/storage/file-server/volume-shadow-copy-service
Application-Aware Processing:
https://helpcenter.veeam.com/docs/backup/vsphere/application_aware_processing.html?ver=120
The second step is to check for Veeam Support Knowledge Base (KBs)
Verify if it is a Know Challenge and apply the Know Solution. If the issue is not listed in KBs and persists, proceed with the third step (manual troubleshooting).
KB 1789: How to Locate and Collect VSS/VIX Log Files From Guest OS
KB 316: How to Use VSSTrace to Collect VSS Diagnostic Logging
KB 1980: Using the Diskshadow Utility to Manually Test VSS Operations
KB 2041: Windows VSS Writers and Corresponding Service Names
KB 2095: SQL VSS Writer is missing: databases will be backed up in crash-consistent state and transaction log processing will be skipped
KB1855: Hyper-V Guest processing skipped (check guest OS VSS state and integration components version)
KB 1794: Insufficient Storage for VSS Error "0x8004231F or Error: VSSControl: -2147467259"
KB 1697: VSS Errors Related to the 'NTDS' VSS Writer
KB 1680: VSS Timeout when backing up Exchange VM
https://www.veeam.com/kb1680
KB 1377: VSS Freeze Wait Timeout
KB 1785: 0x8004230F or Error: VSSControl: -2147212529 (VSS_E_UNEXPECTED_PROVIDER_ERROR)
KB 4333: Veeam's VSS Operations With SMB 3.0 Impacted by Windows June 14, 2022 Update
KB 4573: Backup fails with: "The system cannot find the file specified." or "The device is not ready."
Third step: manual troubleshooting
Verify the prerequisites:
- Windows VSS Services should be running;
- Volume with shadow storage has enough free space;
- Enable the VSS Provider on Veeam VBR
- Enable the VSS backup job (Application Aware Processing job); enable the advanced options options "Require successful processing" and "Process transaction logs with this job”.
- Ensure that VSS writers are enabled and in a consistent state;
- At least one NTFS volume is present on Windows XP or older Windows systems.
Verify for Veeam VBR errors and verify if any previous KB addresses it.
If not, you can try to verify the Windows errors:
- Run “eventvwr.msc” command;
- Get Windows Logs for Applications and check errors associated with VSS. Search for solutions in Microsoft Support articles; for example, see this Microsoft TechNet Article;
- Get Windows Logs System and check errors associated with VSS. Verify if it is a Know Chall and apply the Know Solution from Veeam KBs.
If it does not work, try to verify VSS Errors:
- Run "vssadmin list writers" command;
- If a writer is in "retryable" error, restart the writer and retry the backup;
- If the writer is still in an error state, validate it using Diskshadow as a VSS Provider. If a writer fails during this operation or creates an incomplete snapshot, troubleshoot the writer.
If all previous steps fail, collect all logs and open a case with the Veeam Support team to get help.
I hope this article was helpful.
You can also access this article on my blog, www.cloudnroll.com:
https://cloudnroll.com/2024/09/15/microsoft-vss-framework-recap-and-troubleshooting/