Don’t you just hate it when something is working and then it breaks?
I was working with a customer recently that had their Azure Storage configured as a backup repository within Veeam (Direct to Object Storage) and all was working great. However, after a few months Veeam was attempting to perform lifecycle operations such as creating synthetic full backups and purging old restore chains, and then suddenly a backup job started to fail.
After diving into the logs I could see the following error was happening with every backup attempt:
Details: HTTP code: 403, Content Length: 246, Code: AuthorizationFailure,
<DATE TIME> :: Error: This request is not authorized to perform this operation.
Failed to copy azure blob.
Source path: [{ContainerName}//Veeam/Backup/{Folder Name}/Clients/{GUID}/{GUID}/Metadata/Checkpoint/1.
Version: [{File Version Date}].],
Target path: [{ContainerName}//Veeam/Backup/{Folder Name}/Clients/{GUID}/{GUID}/Metadata/Metadata/CheckpointToDelete/1]
Details: HTTP code: 403, Content Length: 246, Code: AuthorizationFailure,
Error: This request is not authorized to perform this operation.
RequestId:{GUID}
Time:{Request Time}
Agent failed to process method {Cloud.DeleteCheckpointsNoSession}.Now, Veeam was using Azure Storage Account Shared Keys as expected and they hadn’t been rotated, there weren’t any read-only resource locks assigned, the IP address being used to communicate with the Azure Storage account hadn’t changed, nor had the firewall rules on the Azure Storage account permitting the access. What had changed?
The answer to this was, nothing! But there was a setting causing this behaviour and it’s not labelled as you’d expect. This is called:
Permitted Scope for copy operations (preview)
This setting has three possible options, these are:
- From any storage account
- From storage accounts in the same Microsoft Entra tenant
- From storage accounts that have a private endpoint to the same virtual network
Now, this organisation was using the most restrictive setting, ‘From storage accounts that have a private endpoint to the same virtual network’, however this wasn’t seen as a concern because any copies are taking place within the same storage account. We can see from the error message that Veeam was attempting to copy a blob to another folder within the same storage account.
This argument is backed up by Microsoft’s own documentation:
Private Link: Permits copying only from storage accounts that have private links to the same virtual network as the destination account.
https://learn.microsoft.com/en-us/azure/storage/common/security-restrict-copy-operations – 21st November 2023
You would expect that copying to/from the same storage account would mean that the copying would be successful, but this isn’t the case. I’ve never seen this issue previously, but without getting heavily into the testing of whether private endpoints would impact this, or the placement of Veeam in relation to the storage account, this clearly didn’t work anymore for the customer.
Now, after some discussions with Veeam it was agreed that as it’s a preview setting, the best approach to avoid any unexpected changes from Microsoft is to utilise the ‘default’ from before this setting was exposed, which is ‘From any storage account’. However, from my testing I did find that ‘From storage accounts in the same Microsoft Entra tenant’ was sufficient to resolve this issue, but should you find similar issues in the future, you should definitely set it to ‘From any storage account’ to rule out if this preview setting is breaking anything.
