• EN
Solved

hardened repository - Hyper-V backups fail yet Windows Agent Backups complete successfully?

A
Userlevel 5
Badge +1

Hello, Thanks,

For the linux hardened repository, Windows Agent Backups (volume and file) complete successfully.
However, Hyper-V fails.

Windows.Server.DataCenter.Evaluation.Edition.2022.21H2
VBR server - 12.0.0.1420 P20230178

The backup repository is linux hardened, located over the internet, not local.
Note: for this linux machine, I made sure ping from repo -> VBR work ;wink
The backup repository location is a mounted SMB share.

Given the error is `Failed to make the backup file immutable: Inappropriate ioctl for device`,
my small brain thinks ALL immutable backups would fail, not just Hyper-V?

Please advise?

 

Queued for processing at 11/5/2023 12:25:15 PM  
Required backup infrastructure resources have been assigned  
VM processing started at 11/5/2023 12:25:20 PM  
VM size: 127 GB (5.4 GB used)  
VM is now in the required state for processing  
Preparing to create snapshot  
Guest processing skipped (VM is powered off)  
Creating VM recovery checkpoint (mode: Crash consistent)  
Storage initialized  
Network traffic will be encrypted  
Using source proxy vserver03 (onhost)  
Saving config.wmi  
VMCX configuration file (58.8 KB) 58.8 KB read at 59 KB/s
VMRS configuration file (44 KB) 44 KB read at 44 KB/s
VMGS configuration file (4 MB) 4 MB read at 4 MB/s
Hard disk 1 (127 GB) 0 B read at 0 KB/s [CBT]
Finalizing  
Failed to make the backup file immutable: Inappropriate ioctl for device
Failed to set flags for file.
Failed to invoke rpc command
Busy: Source 0% > Proxy 5% > Network 0% > Target 15%  
Primary bottleneck: Target  
Network traffic verification detected no corrupted blocks  
Processing finished with errors at 11/5/2023 12:27:26 PM  

 

Info         [HardenedImmutability] Created immutability proxy for host id:cf17ce51-c672-423b-aac4-3314d3ebafee type:Linux
Info         [HardenedImmutability] Set immutability for backup immutability provider.
Info         [HardenedImmutability] Set immutability for Immutability request: 8fbb70eb-01ba-4395-9e9b-7eb72161d215, c3b24449-dcba-41a3-be20-7c56f9a05cd7, d484a68a-9510-48a0-b7b4-48e690fd8789
Info         [CProxyVcpInvoker] [ProxyVcpInvoker] Call 'ImmutabilityApplyLocks' method without timeout.
Info         [CProxyVcpInvoker] [ProxyVcpInvoker] 'Transport.DoRpc' error: Inappropriate ioctl for device
Info         Failed to set flags for file.
Info         Failed to invoke rpc command
Error        Inappropriate ioctl for device (Veeam.Backup.Common.CCppComponentException)
Error        Failed to set flags for file. (Veeam.Backup.Common.CCppComponentException)
Error           in c++: Failed to execute method [0] for class [immurepolib::IImmutableRepoService].
Error        Failed to invoke rpc command (Veeam.Backup.Common.CCppComponentException)
Error           at Veeam.Backup.Common.CVcpInvoker.Invoke(CVcpStreamBase stream, String command, CVcpCommandArgs inArgs, Boolean noLog)
Error           at Veeam.Backup.ProxyProvider.CVcpInvokerHolder.Invoke(String command, CVcpCommandArgs inArgs, Boolean noLog)
Error           at Veeam.Backup.ProxyProvider.CProxyVcpInvoker.CallImpl(String methodName, CProxyInvokeInputArg inputArg, Boolean secureCall, Nullable`1 timeout)
Error           at Veeam.Backup.ProxyProvider.CProxyVcpInvoker.Call(String methodName, CProxyInvokeInputArg inputArg)
Error           at Veeam.Backup.ProxyProvider.CTransportService.ImmutabilityApplyLocks(String[] storageLinks, TimeSpan lockPeriod, String& error, String[]& notProcessedStorageLinks, Boolean finalize)
Error           at Veeam.Backup.Core.CProxyTransport.ImmutabilityApplyLocks(String[] storageLinks, DateTimeUtc immutableTill, String& error, String[]& notProcessedStorageLinks)
Error           at Veeam.Backup.RepositoryImmutability.CSetImmutabilityCommand.Execute(IImmutableRepositoryProxy proxy, IImmutabilityPeriodProvider periodProvider)
Error           at Veeam.Backup.RepositoryImmutability.CImmutabilityLockPeriodManager.SetImmutability(ISetImmutabilityCommand command, IImmutabilityPeriodProvider periodProvider)
Error           at Veeam.Backup.RepositoryImmutability.CImmutabilityStorageCommander.SetLock(Guid backupId, IReadOnlyCollection`1 storageIds, Boolean finalizeLocks)
Error           at Veeam.Backup.RepositoryImmutability.CImmutabilityStorageCommander.SetLock(SetImmutabilityLockRequest request)
Error           at Veeam.Backup.Core.CRepositoryImmutabilityProcessingAlgorithm.SetLock(IRepositoryImmutabilityValidator repositoryImmutabilityValidator, SetImmutabilityLockRequest setImmutabilityLockRequest)
Error           at Veeam.Backup.Core.CRepositoryImmutabilityProcessingAlgorithm.SetLockSafe(IRepositoryImmutabilityValidator repositoryImmutabilityValidator, SetImmutabilityLockRequest setImmutabilityLockRequest)
Error        Inappropriate ioctl for device (Veeam.Backup.Common.CCppComponentException)
Error        Failed to set flags for file. (Veeam.Backup.Common.CCppComponentException)
Error           in c++: Failed to execute method [0] for class [immurepolib::IImmutableRepoService].
Error        Failed to invoke rpc command (Veeam.Backup.Common.CCppComponentException)
Error           at Veeam.Backup.Common.CVcpInvoker.Invoke(CVcpStreamBase stream, String command, CVcpCommandArgs inArgs, Boolean noLog)
Error           at Veeam.Backup.ProxyProvider.CVcpInvokerHolder.Invoke(String command, CVcpCommandArgs inArgs, Boolean noLog)
Error           at Veeam.Backup.ProxyProvider.CProxyVcpInvoker.CallImpl(String methodName, CProxyInvokeInputArg inputArg, Boolean secureCall, Nullable`1 timeout)
Error           at Veeam.Backup.ProxyProvider.CProxyVcpInvoker.Call(String methodName, CProxyInvokeInputArg inputArg)
Error           at Veeam.Backup.ProxyProvider.CTransportService.ImmutabilityApplyLocks(String[] storageLinks, TimeSpan lockPeriod, String& error, String[]& notProcessedStorageLinks, Boolean finalize)
Error           at Veeam.Backup.Core.CProxyTransport.ImmutabilityApplyLocks(String[] storageLinks, DateTimeUtc immutableTill, String& error, String[]& notProcessedStorageLinks)
Error           at Veeam.Backup.RepositoryImmutability.CSetImmutabilityCommand.Execute(IImmutableRepositoryProxy proxy, IImmutabilityPeriodProvider periodProvider)
Error           at Veeam.Backup.RepositoryImmutability.CImmutabilityLockPeriodManager.SetImmutability(ISetImmutabilityCommand command, IImmutabilityPeriodProvider periodProvider)
Error           at Veeam.Backup.RepositoryImmutability.CImmutabilityStorageCommander.SetLock(Guid backupId, IReadOnlyCollection`1 storageIds, Boolean finalizeLocks)
Error           at Veeam.Backup.RepositoryImmutability.CImmutabilityStorageCommander.SetLock(SetImmutabilityLockRequest request)
Error           at Veeam.Backup.Core.CRepositoryImmutabilityProcessingAlgorithm.SetLock(IRepositoryImmutabilityValidator repositoryImmutabilityValidator, SetImmutabilityLockRequest setImmutabilityLockRequest)
Error           at Veeam.Backup.Core.CRepositoryImmutabilityProcessingAlgorithm.SetLockSafe(IRepositoryImmutabilityValidator repositoryImmutabilityValidator, SetImmutabilityLockRequest setImmutabilityLockRequest)

icon

Best answer by regnor 6 November 2023, 07:13

View original

13 comments

Chris.Childerhose
Userlevel 7
Badge +20

Are the correct permissions assigned via the SMB mount?  Not sure I would use this method for the repo let alone over the internet.

Chris Childerhose - VMCA2022 | VMCE2023 | Veeam Vanguard 7* | Veeam Legend 3* | vExpert 5* | VCAP-DCV/VCP-DCV | Object First Ace | Cisco Champion | Twitter: @cchilderhose | Blog Site – https://just-virtualization.tech
regnor
Userlevel 7
Badge +14

Just to make sure that I get it right. Your hardened repository server is using a SMB share? If so, this won't work with immutability as the immutable flag is only available in Linux file systems.

Maybe your agent backups aren't immutable at all?

Max M. | Systems Engineer @Veeam | VMCE/VMCA | former Legend & Vanguard
A
Userlevel 5
Badge +1

@Chris.ChilderhoseThanks,

“Are the correct permissions assigned via the SMB mount?”
Maybe, as it works for Windows Agent Backups, using the same exact hardened BR.
FWIW, been using that mount SMB share for over a month, tho non-hardened repo.

“Not sure I would use this method for the repo let alone over the internet.“
Good point, let me clarify.
The linux server and SMB share are in the same datacenter, never routed/exposed to internet

 

A
Userlevel 5
Badge +1

@regnor, Maybe your agent backups aren't immutable at all?

I think you are correct, and now that I am learning to look into the VBR .log file, i think i know what is going on.

Maybe this is a bug with VBR, as it explains what is going on.
Should I report it?

Windows Backup Agent shows success when it should fail in the same exact way as Hyper-V backup.

With Windows Agent Backups, the VBR console is falsely showing success.
In fact, as shown below, the corresponding log shows the same exact errors as with Hyper-V.

here is the console output showing success.

 

This is the VBR log file, showing the exact same errors as backup log for the Hyper-V.

and it has the exact same errors as the log from the Hyper-V backup.

[05.11.2023 14:47:03.181] <18> Info         [CProxyVcpInvoker] [ProxyVcpInvoker] 'Transport.DoRpc' error: Inappropriate ioctl for device
[05.11.2023 14:47:03.181] <18> Info         Failed to set flags for file.
[05.11.2023 14:47:03.181] <18> Info         Failed to invoke rpc command
[05.11.2023 14:47:03.181] <18> Error        Inappropriate ioctl for device (Veeam.Backup.Common.CCppComponentException)
[05.11.2023 14:47:03.181] <18> Error        Failed to set flags for file. (Veeam.Backup.Common.CCppComponentException)
[05.11.2023 14:47:03.181] <18> Error           in c++: Failed to execute method [0] for class [immurepolib::IImmutableRepoService].
[05.11.2023 14:47:03.181] <18> Error        Failed to invoke rpc command (Veeam.Backup.Common.CCppComponentException)
[05.11.2023 14:47:03.181] <18> Error           at Veeam.Backup.Common.CVcpInvoker.Invoke(CVcpStreamBase stream, String command, CVcpCommandArgs inArgs, Boolean noLog)
[05.11.2023 14:47:03.181] <18> Error           at Veeam.Backup.ProxyProvider.CVcpInvokerHolder.Invoke(String command, CVcpCommandArgs inArgs, Boolean noLog)
[05.11.2023 14:47:03.181] <18> Error           at Veeam.Backup.ProxyProvider.CProxyVcpInvoker.CallImpl(String methodName, CProxyInvokeInputArg inputArg, Boolean secureCall, Nullable`1 timeout)
[05.11.2023 14:47:03.181] <18> Error           at Veeam.Backup.ProxyProvider.CProxyVcpInvoker.Call(String methodName, CProxyInvokeInputArg inputArg)
[05.11.2023 14:47:03.181] <18> Error           at Veeam.Backup.ProxyProvider.CTransportService.ImmutabilityApplyLocks(String[] storageLinks, TimeSpan lockPeriod, String& error, String[]& notProcessedStorageLinks, Boolean finalize)
[05.11.2023 14:47:03.181] <18> Error           at Veeam.Backup.Core.CProxyTransport.ImmutabilityApplyLocks(String[] storageLinks, DateTimeUtc immutableTill, String& error, String[]& notProcessedStorageLinks)
[05.11.2023 14:47:03.181] <18> Error           at Veeam.Backup.RepositoryImmutability.CSetImmutabilityCommand.Execute(IImmutableRepositoryProxy proxy, IImmutabilityPeriodProvider periodProvider)
[05.11.2023 14:47:03.181] <18> Error           at Veeam.Backup.RepositoryImmutability.CImmutabilityLockPeriodManager.SetImmutability(ISetImmutabilityCommand command, IImmutabilityPeriodProvider periodProvider)
[05.11.2023 14:47:03.181] <18> Error           at Veeam.Backup.RepositoryImmutability.CImmutabilityStorageCommander.SetLock(Guid backupId, IReadOnlyCollection`1 storageIds, Boolean finalizeLocks)
[05.11.2023 14:47:03.181] <18> Error           at Veeam.Backup.RepositoryImmutability.CImmutabilityStorageCommander.SetLock(SetImmutabilityLockRequest request)
[05.11.2023 14:47:03.181] <18> Error           at Veeam.Backup.Core.CRepositoryImmutabilityProcessingAlgorithm.SetLock(IRepositoryImmutabilityValidator repositoryImmutabilityValidator, SetImmutabilityLockRequest setImmutabilityLockRequest)
[05.11.2023 14:47:03.181] <18> Error           at Veeam.Backup.Core.CRepositoryImmutabilityProcessingAlgorithm.SetLockSafe(IRepositoryImmutabilityValidator repositoryImmutabilityValidator, SetImmutabilityLockRequest setImmutabilityLockRequest)
[05.11.2023 14:47:03.196] <18> Error        Inappropriate ioctl for device (Veeam.Backup.Common.CCppComponentException)
[05.11.2023 14:47:03.196] <18> Error        Failed to set flags for file. (Veeam.Backup.Common.CCppComponentException)
[05.11.2023 14:47:03.196] <18> Error           in c++: Failed to execute method [0] for class [immurepolib::IImmutableRepoService].
[05.11.2023 14:47:03.196] <18> Error        Failed to invoke rpc command (Veeam.Backup.Common.CCppComponentException)
[05.11.2023 14:47:03.196] <18> Error           at Veeam.Backup.Common.CVcpInvoker.Invoke(CVcpStreamBase stream, String command, CVcpCommandArgs inArgs, Boolean noLog)
[05.11.2023 14:47:03.196] <18> Error           at Veeam.Backup.ProxyProvider.CVcpInvokerHolder.Invoke(String command, CVcpCommandArgs inArgs, Boolean noLog)
[05.11.2023 14:47:03.196] <18> Error           at Veeam.Backup.ProxyProvider.CProxyVcpInvoker.CallImpl(String methodName, CProxyInvokeInputArg inputArg, Boolean secureCall, Nullable`1 timeout)
[05.11.2023 14:47:03.196] <18> Error           at Veeam.Backup.ProxyProvider.CProxyVcpInvoker.Call(String methodName, CProxyInvokeInputArg inputArg)
[05.11.2023 14:47:03.196] <18> Error           at Veeam.Backup.ProxyProvider.CTransportService.ImmutabilityApplyLocks(String[] storageLinks, TimeSpan lockPeriod, String& error, String[]& notProcessedStorageLinks, Boolean finalize)
[05.11.2023 14:47:03.196] <18> Error           at Veeam.Backup.Core.CProxyTransport.ImmutabilityApplyLocks(String[] storageLinks, DateTimeUtc immutableTill, String& error, String[]& notProcessedStorageLinks)
[05.11.2023 14:47:03.196] <18> Error           at Veeam.Backup.RepositoryImmutability.CSetImmutabilityCommand.Execute(IImmutableRepositoryProxy proxy, IImmutabilityPeriodProvider periodProvider)
[05.11.2023 14:47:03.196] <18> Error           at Veeam.Backup.RepositoryImmutability.CImmutabilityLockPeriodManager.SetImmutability(ISetImmutabilityCommand command, IImmutabilityPeriodProvider periodProvider)
[05.11.2023 14:47:03.196] <18> Error           at Veeam.Backup.RepositoryImmutability.CImmutabilityStorageCommander.SetLock(Guid backupId, IReadOnlyCollection`1 storageIds, Boolean finalizeLocks)
[05.11.2023 14:47:03.196] <18> Error           at Veeam.Backup.RepositoryImmutability.CImmutabilityStorageCommander.SetLock(SetImmutabilityLockRequest request)
[05.11.2023 14:47:03.196] <18> Error           at Veeam.Backup.Core.CRepositoryImmutabilityProcessingAlgorithm.SetLock(IRepositoryImmutabilityValidator repositoryImmutabilityValidator, SetImmutabilityLockRequest setImmutabilityLockRequest)
[05.11.2023 14:47:03.196] <18> Error           at Veeam.Backup.Core.CRepositoryImmutabilityProcessingAlgorithm.SetLockSafe(IRepositoryImmutabilityValidator repositoryImmutabilityValidator, SetImmutabilityLockRequest setImmutabilityLockRequest)

 

Chris.Childerhose
Userlevel 7
Badge +20

@Chris.ChilderhoseThanks,

“Are the correct permissions assigned via the SMB mount?”
Maybe, as it works for Windows Agent Backups, using the same exact hardened BR.
FWIW, been using that mount SMB share for over a month, tho non-hardened repo.

“Not sure I would use this method for the repo let alone over the internet.“
Good point, let me clarify.
The linux server and SMB share are in the same datacenter, never routed/exposed to internet

 

Have you tried adding the server to Veeam without SMB and going direct to it?  Just wondering if that would produce different results or not.

Chris Childerhose - VMCA2022 | VMCE2023 | Veeam Vanguard 7* | Veeam Legend 3* | vExpert 5* | VCAP-DCV/VCP-DCV | Object First Ace | Cisco Champion | Twitter: @cchilderhose | Blog Site – https://just-virtualization.tech
A
Userlevel 5
Badge +1

@Chris.Childerhose, again, thanks!

Have you tried adding the server to Veeam without SMB and going direct to it?  Just wondering if that would produce different results or not.

I am more than willing to test anything you wonder about. I have been learning so much in this forum.
but small brain not understanding exactly what you want me to try?

That i should install SAMBA on the linux server, create a SMB share, mount the share and test that for the hardened BR location?

Chris.Childerhose
Userlevel 7
Badge +20

@Chris.Childerhose, again, thanks!

Have you tried adding the server to Veeam without SMB and going direct to it?  Just wondering if that would produce different results or not.

I am more than willing to test anything you wonder about. I have been learning so much in this forum.
but small brain not understanding exactly what you want me to try?

That i should install SAMBA on the linux server, create a SMB share, mount the share and test that for the hardened BR location?

No I mean adding the server directly to Veeam as a Linux Hardened repository.  If you do that will it produce a different result.

Chris Childerhose - VMCA2022 | VMCE2023 | Veeam Vanguard 7* | Veeam Legend 3* | vExpert 5* | VCAP-DCV/VCP-DCV | Object First Ace | Cisco Champion | Twitter: @cchilderhose | Blog Site – https://just-virtualization.tech
regnor
Userlevel 7
Badge +14

@asdffdsa6132 It could be a UI bug with the agent Jobs but in General immutability won't work in combination with SMB shares. Only disks/volumes with a Linux filesystem will be able to set the immutable flag. 

Max M. | Systems Engineer @Veeam | VMCE/VMCA | former Legend & Vanguard
A
Userlevel 5
Badge +1

No I mean adding the server directly to Veeam as a Linux Hardened repository.  If you do that will it produce a different result.

I think i understand your confusion, sorry about that.
In this older topic, I did connect a SMB share on the internet to VBR as an network attached storage repo.
https://community.veeam.com/discussion-boards-66/license-question-can-use-smb-share-hosted-in-cloud-as-backup-repository-5931

I have since moved on from that tho it worked fine.

However, in https://community.veeam.com/discussion-boards-66/vm-non-instant-restore-works-vm-instant-restore-fails-5938 and now, in this topic, only use linux repo server directly connected to the VBR.

 

 

 

A
Userlevel 5
Badge +1

@regnorIt could be a UI bug with the agent Job

Yes, that is my point. Should I report it, what is the correct way to do that?

Sorry, a little too excited that I found a bug, if only a UI bug 🕵🏻

A
Userlevel 5
Badge +1

@regnor, Thanks. the immutable flag is only available in Linux file systems.

The mounted SMB share supports `xattr`and Veeam sets`user.immutable.until`to be set correctly, 7 days from today

mount -t cifs -o gid=1000,uid=1000 -o credentials=~/.credentials //xxx.your-storagebox.de/xxx-sub2 /hard01
xattr -l /hard01/backups/ABJ_X:_FOLDER-_HARD01/192.168.62.233/'ABJ_X:_FOLDER-_HARD01 - 192.168.62.233D2023-11-05T115841_300A.vbk'
user.immutable.until: 2023-11-12 19:47:02

the problem is that Veeam .log does not list the flag it cannot set.
Is there another flag Veeam tries to set and that fails?

regnor
Userlevel 7
Badge +14

Besides the technical details, Veeam explicitly states that it requires block storage. NFS and SMB cannot be used:

  • As the hardened repository requires the block storage, you cannot use the following storage types:
    • NFS share or a Linux machine with the mounted NFS volume
    • ⁠A Linux machine with the mounted SMB (CIFS) volume

https://helpcenter.veeam.com/docs/backup/vsphere/hardened_repository_limitations.html?ver=120

Regarding the agent job not displaying a warning/error, it would be best to open a support case. Support will be able to confirm if it's a bug and forward it to development.

Max M. | Systems Engineer @Veeam | VMCE/VMCA | former Legend & Vanguard
A
Userlevel 5
Badge +1

@regnor“ A Linux machine with the mounted SMB (CIFS) volume“

I did see that in the docs but since mounted SMB works for non-hardened repo, i tried hardened repo.

I do not use block copy, fast rclone, XFS, etc.
with hardened repo, despite the flag error, the backup is valid, health check works, restore works and even instant restore works.

 

https://helpcenter.veeam.com/docs/backup/hyperv/linux_server.html?ver=120
“You can add a Linux server with local, directly attached storage or mounted NFS as a backup repository. The storage can be a local disk, directly attached disk-based storage (such as a USB hard drive), NFS share”

Interesting, that for non-hardened repo, the docs mention mounted NFS but not mounted SMB.
But as I have posted, for non-hardened repo, backups work, backup copy works, health checks work, veeam.validator works, instant restore works.

so far, no reason why mount SMB cannot be supported.

I will open a support case about the possible UI bug and a support case about using mounted SMB share.
I will update the forum about that.

Once again, thanks for the great forum and taking your time to help.

Comment


Terms & Conditions