Skip to main content
Solved

Veeam B&R Suddenly loses connection to NFS repository

D

I’m a user of Veeam B&R Community Build 11.0.1.1261, and I’ve just started experiencing a problem a second time that I thought was fixed.

Some time ago (a couple of weeks to a month), I started experiencing an issue with Veeam backups. When it would try backing up, it would quickly error out with the following error:

Warning    Failed to synchronize [Backup Repository] Details: Failed to call RPC function 'NfsIsExist': Failed to reset previous connection exception: Permission denied.

 

The first time that happened, I assumed it was because I was disabling Veeam services that I thought were unnecessary. Despite re-enabling them, the error persisted. I eventually realized my installation of B&R was out of date, so I installed an updated version and that seemed to fix the issue for the time being.

Fast forward to today, and I’m seeing the aforementioned error again when trying to run backups. The repository is a Synology DiskStation, and I can verify that the share is accessible from a Linux host. I’m searching through the Veeam logs for something more helpful than the error above, but all I can find is a Task.Repository.log file which is actually less helpful (to me):

[02.12.2022 13:35:25] <14> Info         [CProxyRpcInvoker] RpcInvoker [29525276] has been created. Host: [host:6160]
[02.12.2022 13:35:25] <14> Info         [NFS] Setting share: Share nfs3://[Repository host]:/volume1/Backup
[02.12.2022 13:35:25] <14> Info         [NFS] Checking supported NFS protocol versions for [Repository host]
[02.12.2022 13:35:25] <14> Info         [CProxyRpcInvoker] RpcInvoker [29525276] was disposed
[02.12.2022 13:35:25] <14> Info         [CRepositoryScanProcessor] Checking container scan actuality. ContainerId: 0621ecc2-da41-42a4-9b51-6add65cf0ab3, repositoryId: 442f5777-7196-4750-b152-1a0dfa873b44
[02.12.2022 13:35:25] <14> Info         Rescanning container 0621ecc2-da41-42a4-9b51-6add65cf0ab3
[02.12.2022 13:35:25] <14> Info         [CProxyRpcInvoker] RpcInvoker [52590851] has been created. Host: [host:6160]
[02.12.2022 13:35:25] <14> Info         [NFS] Setting share: Share nfs3://[Repository host]:/volume1/Backup
[02.12.2022 13:35:25] <14> Info         [NFS] Checking if NFS mount accessible on link nfs3://[Repository host]:/volume1/Backup
[02.12.2022 13:35:25] <14> Info         [CProxyRpcInvoker] RpcInvoker [52590851] was disposed
[02.12.2022 13:35:25] <14> Error        Failed to investigate free space and total size for repository '[Repository]'
[02.12.2022 13:35:25] <14> Error        Failed to access NFS datastore (System.Exception)
[02.12.2022 13:35:25] <14> Error           at Veeam.Backup.ProxyProvider.CProxyNfsFileCommander.GetStorageInfo(CNfsFcPath nfsPath)
[02.12.2022 13:35:25] <14> Error           at Veeam.Backup.Core.CNfsFileCommander.FindDirInfo(String link)
[02.12.2022 13:35:25] <14> Error           at Veeam.Backup.Core.Repository.CDefaultRepositoryInvestigator.Investigate()
[02.12.2022 13:35:25] <14> Error        Unable to investigate free space:
[02.12.2022 13:35:25] <14> Error        System.Exception: Failed to access NFS datastore
[02.12.2022 13:35:25] <14> Error           at Veeam.Backup.Core.Repository.CDefaultRepositoryInvestigator.Investigate()
[02.12.2022 13:35:25] <14> Error           at Veeam.Backup.ResourceScanner.CRepositoryContainerResourceProvider.RescanContainer(CRepositoryContainerResource containerResource)
[02.12.2022 13:35:25] <14> Error           at Veeam.Backup.ResourceScanner.CRepositoryContainerResourceProvider.GetActualContainerResource(Boolean forcedScan)
[02.12.2022 13:35:25] <14> Error           at Veeam.Backup.ResourceScanner.CRepositoryScanProcessor.GetActualContainerResource(Boolean forcedScan)
[02.12.2022 13:35:25] <14> Error           at Veeam.Backup.ResourceScanner.CRepositoryScanProcessor.Execute(Boolean forcedScan)
[02.12.2022 13:35:25] <14> Info         Setting storage availability bit RepositoryUnavailable for repository 442f5777-7196-4750-b152-1a0dfa873b44
[02.12.2022 13:35:25] <14> Info         [ResourceScanner] Update resource [Repository] result: av. Unavailable, options
[02.12.2022 13:35:25] <14> Info         Prev result was Unavailable, options
[02.12.2022 13:35:25] <14> Info         [ResourceScanner] Scan is finished for resource [Repository]
[02.12.2022 13:35:25] <14> Info         [ResourceScanner] Resourse scan period is 00:15:00 for repository [Repository].

 

I’m at a loss for where to go forward from here. Thanks in advance!

Best answer by MicoolPaul

@bp4JC if you’ve got some troubleshooting time, I’d take a packet capture of what’s happening. Sometimes it’s a lookup failure for DNS, but often it’s the NFS Server services not responding in a timely manner with stuff like Synology & QNAP. Unfortunately I find their solutions work like 98% of the time, creating lovely headaches like this!

View original
Did this topic help you find an answer to your question?

10 comments

Madi.Cristil
Forum|alt.badge.img+8
  • Madi.Cristil
  • Community Manager
  • 617 comments
  • December 3, 2022

@Mildur 

Madi Cristil
regnor
Forum|alt.badge.img+14
  • regnor
  • Veeam MVP
  • 1352 comments
  • December 3, 2022

I suppose you have already rebooted the Veeam Backup  and it's components?

Also if possible restart the Synology. I once had a case of hanging sessions on such a NAS.

Max M. | Systems Engineer @Veeam | VMCE/VMCA | VeeaMVP 2023 | former Legend & Vanguard
Madi.Cristil
D
2 people like this
  • Madi.Cristil
    Madi.Cristil
  • D
    dataduper
MicoolPaul
Forum|alt.badge.img+23
  • MicoolPaul
  • 2361 comments
  • December 3, 2022

I’d also add are you targeting it via DNS or IP address? Is the NFS server in the same LAN or at a remote site to the gateway server?

Michael Paul - Opinions are my own and do not necessarily reflect the opinion of Veeam | https://micoolpaul.com | Mastodon: @micoolpaul@masto.nu | Bluesky: @micoolpaul.com
D
1 person likes this
  • D
    dataduper
D
  • dataduper
  • Author
  • Comes here often
  • 5 comments
  • December 3, 2022
regnor wrote:

I suppose you have already rebooted the Veeam Backup  and it's components?

Also if possible restart the Synology. I once had a case of hanging sessions on such a NAS.

I have rebooted the host running Veeam B&R, correct. I did not think to reboot the NAS however!

MicoolPaul wrote:

I’d also add are you targeting it via DNS or IP address? Is the NFS server in the same LAN or at a remote site to the gateway server?

I am actually targeting the repository by DNS, and also authenticating the B&R host in NFS by DNS as well. Both hosts are on the same LAN, including DNS server.

 

I got an email this morning that a backup completed successfully. Sure enough, the repository is marked as accessible in Veeam too. There’s absolutely nothing I’ve changed since I gave up troubleshooting yesterday. Unfortunately the problem remains a mystery, although I like where @regnor is going in thinking that the DiskStation may have needed a reboot. Probably couldn’t hurt to update it to the latest DSM version, too.

Going forward it would be helpful if Veeam could be more clear about NFS errors, but for now all is well. Thank you all for the replies, I really appreciate it.

bp4JC
Forum|alt.badge.img+3
  • bp4JC
  • Influencer
  • 136 comments
  • December 5, 2022
dataduper wrote:

I’m a user of Veeam B&R Community Build 11.0.1.1261, and I’ve just started experiencing a problem a second time that I thought was fixed.

Some time ago (a couple of weeks to a month), I started experiencing an issue with Veeam backups. When it would try backing up, it would quickly error out with the following error:

Warning    Failed to synchronize [Backup Repository] Details: Failed to call RPC function 'NfsIsExist': Failed to reset previous connection exception: Permission denied.

 

The first time that happened, I assumed it was because I was disabling Veeam services that I thought were unnecessary. Despite re-enabling them, the error persisted. I eventually realized my installation of B&R was out of date, so I installed an updated version and that seemed to fix the issue for the time being.

Fast forward to today, and I’m seeing the aforementioned error again when trying to run backups. The repository is a Synology DiskStation, and I can verify that the share is accessible from a Linux host. I’m searching through the Veeam logs for something more helpful than the error above, but all I can find is a Task.Repository.log file which is actually less helpful (to me):

[02.12.2022 13:35:25] <14> Info         [CProxyRpcInvoker] RpcInvoker [29525276] has been created. Host: [host:6160]
[02.12.2022 13:35:25] <14> Info         [NFS] Setting share: Share nfs3://[Repository host]:/volume1/Backup
[02.12.2022 13:35:25] <14> Info         [NFS] Checking supported NFS protocol versions for [Repository host]
[02.12.2022 13:35:25] <14> Info         [CProxyRpcInvoker] RpcInvoker [29525276] was disposed
[02.12.2022 13:35:25] <14> Info         [CRepositoryScanProcessor] Checking container scan actuality. ContainerId: 0621ecc2-da41-42a4-9b51-6add65cf0ab3, repositoryId: 442f5777-7196-4750-b152-1a0dfa873b44
[02.12.2022 13:35:25] <14> Info         Rescanning container 0621ecc2-da41-42a4-9b51-6add65cf0ab3
[02.12.2022 13:35:25] <14> Info         [CProxyRpcInvoker] RpcInvoker [52590851] has been created. Host: [host:6160]
[02.12.2022 13:35:25] <14> Info         [NFS] Setting share: Share nfs3://[Repository host]:/volume1/Backup
[02.12.2022 13:35:25] <14> Info         [NFS] Checking if NFS mount accessible on link nfs3://[Repository host]:/volume1/Backup
[02.12.2022 13:35:25] <14> Info         [CProxyRpcInvoker] RpcInvoker [52590851] was disposed
[02.12.2022 13:35:25] <14> Error        Failed to investigate free space and total size for repository '[Repository]'
[02.12.2022 13:35:25] <14> Error        Failed to access NFS datastore (System.Exception)
[02.12.2022 13:35:25] <14> Error           at Veeam.Backup.ProxyProvider.CProxyNfsFileCommander.GetStorageInfo(CNfsFcPath nfsPath)
[02.12.2022 13:35:25] <14> Error           at Veeam.Backup.Core.CNfsFileCommander.FindDirInfo(String link)
[02.12.2022 13:35:25] <14> Error           at Veeam.Backup.Core.Repository.CDefaultRepositoryInvestigator.Investigate()
[02.12.2022 13:35:25] <14> Error        Unable to investigate free space:
[02.12.2022 13:35:25] <14> Error        System.Exception: Failed to access NFS datastore
[02.12.2022 13:35:25] <14> Error           at Veeam.Backup.Core.Repository.CDefaultRepositoryInvestigator.Investigate()
[02.12.2022 13:35:25] <14> Error           at Veeam.Backup.ResourceScanner.CRepositoryContainerResourceProvider.RescanContainer(CRepositoryContainerResource containerResource)
[02.12.2022 13:35:25] <14> Error           at Veeam.Backup.ResourceScanner.CRepositoryContainerResourceProvider.GetActualContainerResource(Boolean forcedScan)
[02.12.2022 13:35:25] <14> Error           at Veeam.Backup.ResourceScanner.CRepositoryScanProcessor.GetActualContainerResource(Boolean forcedScan)
[02.12.2022 13:35:25] <14> Error           at Veeam.Backup.ResourceScanner.CRepositoryScanProcessor.Execute(Boolean forcedScan)
[02.12.2022 13:35:25] <14> Info         Setting storage availability bit RepositoryUnavailable for repository 442f5777-7196-4750-b152-1a0dfa873b44
[02.12.2022 13:35:25] <14> Info         [ResourceScanner] Update resource [Repository] result: av. Unavailable, options
[02.12.2022 13:35:25] <14> Info         Prev result was Unavailable, options
[02.12.2022 13:35:25] <14> Info         [ResourceScanner] Scan is finished for resource [Repository]
[02.12.2022 13:35:25] <14> Info         [ResourceScanner] Resourse scan period is 00:15:00 for repository [Repository].

 

I’m at a loss for where to go forward from here. Thanks in advance!

I had been having NFS issues with one of my clients as well. Just randomly started happening. Didn’t really reach a solution with support. They seemed to want to point the finger at VMware...I’m not sure if it’s a Veeam issue or not, at this point. It comes and goes and it just started for no reason. There’s not really any consistency to it, but it was happening fairly regularly.

D
1 person likes this
  • D
    dataduper
MicoolPaul
Forum|alt.badge.img+23
  • MicoolPaul
  • 2361 comments
  • Answer
  • December 5, 2022

@bp4JC if you’ve got some troubleshooting time, I’d take a packet capture of what’s happening. Sometimes it’s a lookup failure for DNS, but often it’s the NFS Server services not responding in a timely manner with stuff like Synology & QNAP. Unfortunately I find their solutions work like 98% of the time, creating lovely headaches like this!

Michael Paul - Opinions are my own and do not necessarily reflect the opinion of Veeam | https://micoolpaul.com | Mastodon: @micoolpaul@masto.nu | Bluesky: @micoolpaul.com
D
1 person likes this
  • D
    dataduper
bp4JC
Forum|alt.badge.img+3
  • bp4JC
  • Influencer
  • 136 comments
  • December 5, 2022
MicoolPaul wrote:

@bp4JC if you’ve got some troubleshooting time, I’d take a packet capture of what’s happening. Sometimes it’s a lookup failure for DNS, but often it’s the NFS Server services not responding in a timely manner with stuff like Synology & QNAP. Unfortunately I find their solutions work like 98% of the time, creating lovely headaches like this!

That’s a great idea but it’s just so random that I am not sure how I can determine when to run the packet capture. It was happening for about two or three weeks, and then it suddenly stopped. To be fair, I know that my CTO was doing some work on the VMware environment around the time it stopped and I don’t think I saw the issue pop up at all last week, but it could very well start again. I’ll have to look into trying that if it starts again. Thanks for chiming in on this!

MicoolPaul
1 person likes this
  • MicoolPaul
    MicoolPaul
MicoolPaul
Forum|alt.badge.img+23
  • MicoolPaul
  • 2361 comments
  • December 5, 2022
bp4JC wrote:
MicoolPaul wrote:

@bp4JC if you’ve got some troubleshooting time, I’d take a packet capture of what’s happening. Sometimes it’s a lookup failure for DNS, but often it’s the NFS Server services not responding in a timely manner with stuff like Synology & QNAP. Unfortunately I find their solutions work like 98% of the time, creating lovely headaches like this!

That’s a great idea but it’s just so random that I am not sure how I can determine when to run the packet capture. It was happening for about two or three weeks, and then it suddenly stopped. To be fair, I know that my CTO was doing some work on the VMware environment around the time it stopped and I don’t think I saw the issue pop up at all last week, but it could very well start again. I’ll have to look into trying that if it starts again. Thanks for chiming in on this!

Drop me a DM if I can help when it’s flaring up, if I’m about I’ll try!

Michael Paul - Opinions are my own and do not necessarily reflect the opinion of Veeam | https://micoolpaul.com | Mastodon: @micoolpaul@masto.nu | Bluesky: @micoolpaul.com
D
  • dataduper
  • Author
  • Comes here often
  • 5 comments
  • December 5, 2022
bp4JC wrote:
MicoolPaul wrote:

@bp4JC if you’ve got some troubleshooting time, I’d take a packet capture of what’s happening. Sometimes it’s a lookup failure for DNS, but often it’s the NFS Server services not responding in a timely manner with stuff like Synology & QNAP. Unfortunately I find their solutions work like 98% of the time, creating lovely headaches like this!

That’s a great idea but it’s just so random that I am not sure how I can determine when to run the packet capture. It was happening for about two or three weeks, and then it suddenly stopped. To be fair, I know that my CTO was doing some work on the VMware environment around the time it stopped and I don’t think I saw the issue pop up at all last week, but it could very well start again. I’ll have to look into trying that if it starts again. Thanks for chiming in on this!

If you don’t mind me asking, is this VMWare providing the NFS service? Trying to find some thread tying my DiskStation to your situation.

D
  • dataduper
  • Author
  • Comes here often
  • 5 comments
  • December 30, 2022

Just want to update everyone. I started experiencing this issue again, right around when a power outage occurred and my DiskStation went through it’s UPS shutdown procedure. I’ve executed a full reboot on the DiskStation, and after coming back up, backups are working fine now. This is with the DiskStation Manager software version DSM 6.2.4-25556 Update 6 (latest available as of now.) I haven’t upgraded to 7 yet so I probably need to do that.

Comment


Terms & Conditions