Solved

Veeam B&R Suddenly loses connection to NFS repository


Userlevel 2

I’m a user of Veeam B&R Community Build 11.0.1.1261, and I’ve just started experiencing a problem a second time that I thought was fixed.

Some time ago (a couple of weeks to a month), I started experiencing an issue with Veeam backups. When it would try backing up, it would quickly error out with the following error:

Warning    Failed to synchronize [Backup Repository] Details: Failed to call RPC function 'NfsIsExist': Failed to reset previous connection exception: Permission denied.

 

The first time that happened, I assumed it was because I was disabling Veeam services that I thought were unnecessary. Despite re-enabling them, the error persisted. I eventually realized my installation of B&R was out of date, so I installed an updated version and that seemed to fix the issue for the time being.

Fast forward to today, and I’m seeing the aforementioned error again when trying to run backups. The repository is a Synology DiskStation, and I can verify that the share is accessible from a Linux host. I’m searching through the Veeam logs for something more helpful than the error above, but all I can find is a Task.Repository.log file which is actually less helpful (to me):

[02.12.2022 13:35:25] <14> Info         [CProxyRpcInvoker] RpcInvoker [29525276] has been created. Host: [host:6160]
[02.12.2022 13:35:25] <14> Info         [NFS] Setting share: Share nfs3://[Repository host]:/volume1/Backup
[02.12.2022 13:35:25] <14> Info         [NFS] Checking supported NFS protocol versions for [Repository host]
[02.12.2022 13:35:25] <14> Info         [CProxyRpcInvoker] RpcInvoker [29525276] was disposed
[02.12.2022 13:35:25] <14> Info         [CRepositoryScanProcessor] Checking container scan actuality. ContainerId: 0621ecc2-da41-42a4-9b51-6add65cf0ab3, repositoryId: 442f5777-7196-4750-b152-1a0dfa873b44
[02.12.2022 13:35:25] <14> Info         Rescanning container 0621ecc2-da41-42a4-9b51-6add65cf0ab3
[02.12.2022 13:35:25] <14> Info         [CProxyRpcInvoker] RpcInvoker [52590851] has been created. Host: [host:6160]
[02.12.2022 13:35:25] <14> Info         [NFS] Setting share: Share nfs3://[Repository host]:/volume1/Backup
[02.12.2022 13:35:25] <14> Info         [NFS] Checking if NFS mount accessible on link nfs3://[Repository host]:/volume1/Backup
[02.12.2022 13:35:25] <14> Info         [CProxyRpcInvoker] RpcInvoker [52590851] was disposed
[02.12.2022 13:35:25] <14> Error        Failed to investigate free space and total size for repository '[Repository]'
[02.12.2022 13:35:25] <14> Error        Failed to access NFS datastore (System.Exception)
[02.12.2022 13:35:25] <14> Error           at Veeam.Backup.ProxyProvider.CProxyNfsFileCommander.GetStorageInfo(CNfsFcPath nfsPath)
[02.12.2022 13:35:25] <14> Error           at Veeam.Backup.Core.CNfsFileCommander.FindDirInfo(String link)
[02.12.2022 13:35:25] <14> Error           at Veeam.Backup.Core.Repository.CDefaultRepositoryInvestigator.Investigate()
[02.12.2022 13:35:25] <14> Error        Unable to investigate free space:
[02.12.2022 13:35:25] <14> Error        System.Exception: Failed to access NFS datastore
[02.12.2022 13:35:25] <14> Error           at Veeam.Backup.Core.Repository.CDefaultRepositoryInvestigator.Investigate()
[02.12.2022 13:35:25] <14> Error           at Veeam.Backup.ResourceScanner.CRepositoryContainerResourceProvider.RescanContainer(CRepositoryContainerResource containerResource)
[02.12.2022 13:35:25] <14> Error           at Veeam.Backup.ResourceScanner.CRepositoryContainerResourceProvider.GetActualContainerResource(Boolean forcedScan)
[02.12.2022 13:35:25] <14> Error           at Veeam.Backup.ResourceScanner.CRepositoryScanProcessor.GetActualContainerResource(Boolean forcedScan)
[02.12.2022 13:35:25] <14> Error           at Veeam.Backup.ResourceScanner.CRepositoryScanProcessor.Execute(Boolean forcedScan)
[02.12.2022 13:35:25] <14> Info         Setting storage availability bit RepositoryUnavailable for repository 442f5777-7196-4750-b152-1a0dfa873b44
[02.12.2022 13:35:25] <14> Info         [ResourceScanner] Update resource [Repository] result: av. Unavailable, options
[02.12.2022 13:35:25] <14> Info         Prev result was Unavailable, options
[02.12.2022 13:35:25] <14> Info         [ResourceScanner] Scan is finished for resource [Repository]
[02.12.2022 13:35:25] <14> Info         [ResourceScanner] Resourse scan period is 00:15:00 for repository [Repository].

 

I’m at a loss for where to go forward from here. Thanks in advance!

icon

Best answer by MicoolPaul 5 December 2022, 18:20

View original

10 comments

Userlevel 7
Badge +14

I suppose you have already rebooted the Veeam Backup  and it's components?

Also if possible restart the Synology. I once had a case of hanging sessions on such a NAS.

Userlevel 7
Badge +3

I’m a user of Veeam B&R Community Build 11.0.1.1261, and I’ve just started experiencing a problem a second time that I thought was fixed.

Some time ago (a couple of weeks to a month), I started experiencing an issue with Veeam backups. When it would try backing up, it would quickly error out with the following error:

Warning    Failed to synchronize [Backup Repository] Details: Failed to call RPC function 'NfsIsExist': Failed to reset previous connection exception: Permission denied.

 

The first time that happened, I assumed it was because I was disabling Veeam services that I thought were unnecessary. Despite re-enabling them, the error persisted. I eventually realized my installation of B&R was out of date, so I installed an updated version and that seemed to fix the issue for the time being.

Fast forward to today, and I’m seeing the aforementioned error again when trying to run backups. The repository is a Synology DiskStation, and I can verify that the share is accessible from a Linux host. I’m searching through the Veeam logs for something more helpful than the error above, but all I can find is a Task.Repository.log file which is actually less helpful (to me):

[02.12.2022 13:35:25] <14> Info         [CProxyRpcInvoker] RpcInvoker [29525276] has been created. Host: [host:6160]
[02.12.2022 13:35:25] <14> Info         [NFS] Setting share: Share nfs3://[Repository host]:/volume1/Backup
[02.12.2022 13:35:25] <14> Info         [NFS] Checking supported NFS protocol versions for [Repository host]
[02.12.2022 13:35:25] <14> Info         [CProxyRpcInvoker] RpcInvoker [29525276] was disposed
[02.12.2022 13:35:25] <14> Info         [CRepositoryScanProcessor] Checking container scan actuality. ContainerId: 0621ecc2-da41-42a4-9b51-6add65cf0ab3, repositoryId: 442f5777-7196-4750-b152-1a0dfa873b44
[02.12.2022 13:35:25] <14> Info         Rescanning container 0621ecc2-da41-42a4-9b51-6add65cf0ab3
[02.12.2022 13:35:25] <14> Info         [CProxyRpcInvoker] RpcInvoker [52590851] has been created. Host: [host:6160]
[02.12.2022 13:35:25] <14> Info         [NFS] Setting share: Share nfs3://[Repository host]:/volume1/Backup
[02.12.2022 13:35:25] <14> Info         [NFS] Checking if NFS mount accessible on link nfs3://[Repository host]:/volume1/Backup
[02.12.2022 13:35:25] <14> Info         [CProxyRpcInvoker] RpcInvoker [52590851] was disposed
[02.12.2022 13:35:25] <14> Error        Failed to investigate free space and total size for repository '[Repository]'
[02.12.2022 13:35:25] <14> Error        Failed to access NFS datastore (System.Exception)
[02.12.2022 13:35:25] <14> Error           at Veeam.Backup.ProxyProvider.CProxyNfsFileCommander.GetStorageInfo(CNfsFcPath nfsPath)
[02.12.2022 13:35:25] <14> Error           at Veeam.Backup.Core.CNfsFileCommander.FindDirInfo(String link)
[02.12.2022 13:35:25] <14> Error           at Veeam.Backup.Core.Repository.CDefaultRepositoryInvestigator.Investigate()
[02.12.2022 13:35:25] <14> Error        Unable to investigate free space:
[02.12.2022 13:35:25] <14> Error        System.Exception: Failed to access NFS datastore
[02.12.2022 13:35:25] <14> Error           at Veeam.Backup.Core.Repository.CDefaultRepositoryInvestigator.Investigate()
[02.12.2022 13:35:25] <14> Error           at Veeam.Backup.ResourceScanner.CRepositoryContainerResourceProvider.RescanContainer(CRepositoryContainerResource containerResource)
[02.12.2022 13:35:25] <14> Error           at Veeam.Backup.ResourceScanner.CRepositoryContainerResourceProvider.GetActualContainerResource(Boolean forcedScan)
[02.12.2022 13:35:25] <14> Error           at Veeam.Backup.ResourceScanner.CRepositoryScanProcessor.GetActualContainerResource(Boolean forcedScan)
[02.12.2022 13:35:25] <14> Error           at Veeam.Backup.ResourceScanner.CRepositoryScanProcessor.Execute(Boolean forcedScan)
[02.12.2022 13:35:25] <14> Info         Setting storage availability bit RepositoryUnavailable for repository 442f5777-7196-4750-b152-1a0dfa873b44
[02.12.2022 13:35:25] <14> Info         [ResourceScanner] Update resource [Repository] result: av. Unavailable, options
[02.12.2022 13:35:25] <14> Info         Prev result was Unavailable, options
[02.12.2022 13:35:25] <14> Info         [ResourceScanner] Scan is finished for resource [Repository]
[02.12.2022 13:35:25] <14> Info         [ResourceScanner] Resourse scan period is 00:15:00 for repository [Repository].

 

I’m at a loss for where to go forward from here. Thanks in advance!

I had been having NFS issues with one of my clients as well. Just randomly started happening. Didn’t really reach a solution with support. They seemed to want to point the finger at VMware...I’m not sure if it’s a Veeam issue or not, at this point. It comes and goes and it just started for no reason. There’s not really any consistency to it, but it was happening fairly regularly.

Userlevel 7
Badge +22

@bp4JC if you’ve got some troubleshooting time, I’d take a packet capture of what’s happening. Sometimes it’s a lookup failure for DNS, but often it’s the NFS Server services not responding in a timely manner with stuff like Synology & QNAP. Unfortunately I find their solutions work like 98% of the time, creating lovely headaches like this!

Userlevel 7
Badge +3

@bp4JC if you’ve got some troubleshooting time, I’d take a packet capture of what’s happening. Sometimes it’s a lookup failure for DNS, but often it’s the NFS Server services not responding in a timely manner with stuff like Synology & QNAP. Unfortunately I find their solutions work like 98% of the time, creating lovely headaches like this!

That’s a great idea but it’s just so random that I am not sure how I can determine when to run the packet capture. It was happening for about two or three weeks, and then it suddenly stopped. To be fair, I know that my CTO was doing some work on the VMware environment around the time it stopped and I don’t think I saw the issue pop up at all last week, but it could very well start again. I’ll have to look into trying that if it starts again. Thanks for chiming in on this!

Userlevel 7
Badge +22

I’d also add are you targeting it via DNS or IP address? Is the NFS server in the same LAN or at a remote site to the gateway server?

Userlevel 7
Badge +22

@bp4JC if you’ve got some troubleshooting time, I’d take a packet capture of what’s happening. Sometimes it’s a lookup failure for DNS, but often it’s the NFS Server services not responding in a timely manner with stuff like Synology & QNAP. Unfortunately I find their solutions work like 98% of the time, creating lovely headaches like this!

That’s a great idea but it’s just so random that I am not sure how I can determine when to run the packet capture. It was happening for about two or three weeks, and then it suddenly stopped. To be fair, I know that my CTO was doing some work on the VMware environment around the time it stopped and I don’t think I saw the issue pop up at all last week, but it could very well start again. I’ll have to look into trying that if it starts again. Thanks for chiming in on this!

Drop me a DM if I can help when it’s flaring up, if I’m about I’ll try!

Userlevel 2

@bp4JC if you’ve got some troubleshooting time, I’d take a packet capture of what’s happening. Sometimes it’s a lookup failure for DNS, but often it’s the NFS Server services not responding in a timely manner with stuff like Synology & QNAP. Unfortunately I find their solutions work like 98% of the time, creating lovely headaches like this!

That’s a great idea but it’s just so random that I am not sure how I can determine when to run the packet capture. It was happening for about two or three weeks, and then it suddenly stopped. To be fair, I know that my CTO was doing some work on the VMware environment around the time it stopped and I don’t think I saw the issue pop up at all last week, but it could very well start again. I’ll have to look into trying that if it starts again. Thanks for chiming in on this!

If you don’t mind me asking, is this VMWare providing the NFS service? Trying to find some thread tying my DiskStation to your situation.

Userlevel 2

Just want to update everyone. I started experiencing this issue again, right around when a power outage occurred and my DiskStation went through it’s UPS shutdown procedure. I’ve executed a full reboot on the DiskStation, and after coming back up, backups are working fine now. This is with the DiskStation Manager software version DSM 6.2.4-25556 Update 6 (latest available as of now.) I haven’t upgraded to 7 yet so I probably need to do that.

Userlevel 7
Badge +8

@Mildur 

Userlevel 2

I suppose you have already rebooted the Veeam Backup  and it's components?

Also if possible restart the Synology. I once had a case of hanging sessions on such a NAS.

I have rebooted the host running Veeam B&R, correct. I did not think to reboot the NAS however!

I’d also add are you targeting it via DNS or IP address? Is the NFS server in the same LAN or at a remote site to the gateway server?

I am actually targeting the repository by DNS, and also authenticating the B&R host in NFS by DNS as well. Both hosts are on the same LAN, including DNS server.

 

I got an email this morning that a backup completed successfully. Sure enough, the repository is marked as accessible in Veeam too. There’s absolutely nothing I’ve changed since I gave up troubleshooting yesterday. Unfortunately the problem remains a mystery, although I like where @regnor is going in thinking that the DiskStation may have needed a reboot. Probably couldn’t hurt to update it to the latest DSM version, too.

Going forward it would be helpful if Veeam could be more clear about NFS errors, but for now all is well. Thank you all for the replies, I really appreciate it.

Comment