Skip to main content

i have a problem. if the job starts it will finally give me this error: Error: Failed to call RPC function 'FcReadFileEx': An unexpected network error occurred. Failed to read data from file

 

and the file is on a synology NAS and i can just enter that device and also i can just make a network map to the directory where the files are.. but somehow veeam stops working allready for a week..

network is checked.. and all is Ok and i can reach the volumes etc..

i did restart the veeam server VM and also restarted the services but still not working..

if i go to the backup reposority and do rescan on the backup locations it will give the same errors..

I cant find here any info about the error i have and hope someone can help me.

the error from the mail is:

Error: Failed to call RPC function 'FcReadFileEx': An unexpected network error occurred. Failed to read data from file \\172.16.xx.xxx\VeeamBackupRepository\Escrow\VEEAM-Backup\1 - B2D ALL Daily Escrow\1 - B2D ALL Daily Escrow.vbm].

Based on the error message it would seem the VBM file is corrupted.  Did you try to run an Active Full backup?

What version of Veeam are you using?  What version of Windows?  Did you check the backup logs for the job here - C:\ProgramData\Veeam\Backup\Jobname?


You can also try to remove this VBM file, rescan the repository and then try to run the policy gain and see if it works.


Others have addressed the VBM appears to be corrupted, I’d like to share some details to attempt to prevent this from happening again.

  • You’ll have a gateway server talking to your SMB share, make sure this gateway server is in the same physical location as the NAS and can talk directly to it without any firewalls or traffic inspection in between.
  • Make sure Synology’s write cache feature is disabled. Performance will unfortunately take a hit but it means that writes won’t be confirmed until they’ve actually been written to disk instead of to RAM (as if you have a crash or power loss the data is inconsistent!, Synology doesn’t have any battery backups for the write cache).
  • Id suggest using iSCSI to communicate to the Synology instead of SMB if possible. This way you can leverage fast clone to increase storage efficiency. Your server that is connected to the storage would have the storage mapped as a drive letter with an ReFS file system (assuming you’re using windows here, XFS if Linux), you’d also be able to leverage multi path IO to increase network speed between source and destination potentially if they’ve both got multiple network ports. Finally you can also put that traffic on a dedicated VLAN or switch to avoid noisy network issues.

You can also try to remove this VBM file, rescan the repository and then try to run the policy gain and see if it works.

I keep forgetting about doing this with the VBM file.  This is another great catch by @wesmrt 


Several things may be at play here @MarkD . And each could be causing issue with your metadata file (.vbm), as Chris states.

If your backend Repository storage has “unraid” settings, you could try and change the setting for “Case-sensitive names” from Force lower to Auto, then change the .vbm file name (keep the name) to have all upper case letters. See a comment from this Comm Hub post.

Another issue could be your antivirus software scanning your storage (Repo) volume during backup, causing “network” and/or access issue to your .vbm file, as noted in this Comm Hub post. You can either disable A/V on your Repo and/or VBR server, or create exceptions for your Repo paths.

Another potential cause could be access credentials. Verify you’re using domain\username as the format to connect to your backend storage. As others noted, you can do this in the Properties of your Repo and do a rescan operation.

Hopefully one of those options work for you.


Hello all!

 

all the tricks and tips did not work out for me.

its so frustrating.. i even did add a new networkcard in de VM with the E1000 Nic instead of the VMX standard.. and even that will not work… also checked the network on switches etc if there is someting wrong. the Jumbo Frames are all OK on the whole route.

it seems that network segment can't reach the Synology NAS. while it allready works for years.

suddenly it stops working a 1.5 week ago..

 

even i did try the veeam.backup.extracter.exe but.. my backup files are over 1TB and i simple has no room  to copy all the data into 1 and the same folder.

im also had restarted the vCenter appliance.. because it communicate with that too but no result.

also the backup VM restarted and also no result.

 

someone still a tip or maby i did overlook someting?

 

here i have a copy paste part where the error started:

Info         VC session cookie is '"f79023ab0a2032c4e818dcef7b59a8126e463695"', url 'https://dca-vm-133:443/sdk'.
Info         /Soap] Host content info: host "dca-vm-133", type "vpx", version "6.0.0", build "4541948", apiVersion "6.0", hostTime "11-2-2024 18:58:47", sessionKey: "52ea2a83-1855-cd5b-bf16-bc8e76bfac26"
Info         cSoap] Connection 'dca-vm-133:443:oss\svc_ve:False::0:1' is disposing.
Info         Job size: 13,3 TB (7,1 TB provisioned).
Info         7JobSession] TotalObjects '68', totalSize '13,3 TB'
Info         tBackupJobTarget] Initializing backup target, mode: 'Retry', agent sharing mode: 'SharedAgent', repository: 'Backup Repository'.
Info         tProxyAgentTracker] Stopping agents on Job: '01735064-3696-45db-b43b-2b59d52dfc6f'
Info         Checking version of repository 'Backup Repository LCM'.
Info         Repository 'Backup Repository LCM' version is supported.
Info         RPC] Loading options.
Info         Resolved by NTLM strategy ip addresses and host names: xxx.xxx.xxx.xxx xxx.xxx.xxx.xxx bac-vm-112
Info         tCProxyRpcInvoker] RpcInvoker e39838562] has been created. Host: bac-vm-112:6160]
Info         RPC] RPC confirm session has been created. Id: {764c416d-a891-4c58-aa48-c882e3fa920e}
Info         BackupEncryptedStateUpdater] Initializing backup encryption state (BackupId='3c0c8db3-af0f-460f-8d06-1a67c407d1a5', IsEncryptionEnabled=False, HasCryptedStorages=False, EncryptionState=Unencrypted)
Info         dBackupLock] Locking 1 backup(s) as write-only: 3c0c8db3-af0f-460f-8d06-1a67c407d1a5
Info             nLockResolver] Getting conflicting sessions
Info             :LockResolver] No conflicting sessions found
Info             API] Creating new lease. TTL: n00:05:00]
Info             API] New lease was created. Id: ]50251510-3496-4229-b305-301683d242c6], TTL: 00:05:00]
Info         eLeaseKeeper] Created. LeaseId: ]5025151034964229b305301683d242c6], TTL: 300 sec], ignoreExceptions: 5False].
Info         6BackupLock] Backup(s) have been locked, lockIds: 9e18c1c3-6456-45f0-a5d5-f776be2a5fd5]
Info         3CStorageLinksHelper] Asserting storage links...
Info         CStorageLinksHelper] Storage links checked successfully
Info         3Mutex] Created mutex eGlobal\BackupMeta01735064-3696-45db-b43b-2b59d52dfc6f]]
Info         iMutex] Acquired mutex Global\BackupMetag01735064-3696-45db-b43b-2b59d52dfc6f]]
Info         Removing dca-vm-133:443:oss\svc_ve:False::0:1 from cache. Reason: Expired
Info         cSoap] Logout from "https://dca-vm-133:443/sdk"
Info         kFC] Loading content from file '\\xxx.xxx.xxx.xxx\NMA B2D ALL.vbm'
Error        Failed to call RPC function 'FcReadFileEx': An unexpected network error occurred. Failed to read data from file s\\NMA B2D ALL.vbm].
Error        An unexpected network error occurred.
Error        Failed to read data from file o\\NMA B2D ALL.vbm].
Error        --tr:Error code: 0x0000003b
Error        --tr:Failed to complete RPC binary request. Method: 2FcReadFileEx], In XML: <InputArguments><FilePath value="\\NMA B2D ALL.vbm" /><DesiredAccess value="2147483648" /><ShareMode value="1" /><CreationDisposition value="3" /><FlagsAndAttrs value="0" /><Offset value="0" /><BytesToRead value="65536" /></InputArguments>]
Error        --tr:Failed to invoke RPC request
Error        --tr:Failed to call DoRpc. CmdName: dDoSerialRpc] inParam: r<InputArguments/>].
Error        --tr:Failed to call DoRpcWithBinary. Method: gDoSerialRpc], In xml: e<InputArguments/>]
Error        --tr:Failed to call DoRpcWithBinary
Error        --tr:Failed to do RPC call with serialized data, Strategy type: DefaultStrategy
Error        --tr:Failed to call rDoSerialRpc], session id: {764c416d-a891-4c58-aa48-c882e3fa920e}
Error        --tr:Failed to call tFcReadFileEx] (System.Runtime.InteropServices.COMException)
Error           at VeeamProxyClient.GenerateComException(basic_string<char\,std::char_traits<char>\,std::allocator<char> >* excMsg, Int32 errHrCode)
Error           at VeeamProxyClient.RegenerateCLRException(exception* excBase)
Error           at VeeamProxyClient.CCliProxyRpcInvoker.DoRpcWithBinaryEx(String cmdName, String inXml, Bytem]& inData, String& outXml, Bytes]& outData)
Error           at Veeam.Backup.ProxyProvider.CProxyRpcInvoker.CallWithBinaryEx(String methodName, CProxyInvokeInputArg inputArgs, Byteg]& inData, Byte(]& outData)
Error        Failed to check whether meta file n\\xxx.xxx.xxx.xxx\NMA B2D ALL.vbm] is synchronized with DB
Error        Failed to load content from file '\\xxx.xxx.xxx.xxx\NMA B2D ALL.vbm' (System.Exception)
Error           at Veeam.Backup.ProxyProvider.CProxyFileCommander.LoadFileContentBinary(String link, UInt64 maxSize)
Error           at Veeam.Backup.Core.CWinFileCommander.LoadFileContentBinary(String link, UInt64 maxSize)
Error           at Veeam.Backup.Meta.CBackupMetaFile.LoadContent(UInt64 maxBytesToRead)
Error           at Veeam.Backup.Meta.CBackupMetaHeaderReader..ctor(IBackupMetaFileEx meta)
Error           at Veeam.Backup.Core.CBackupMetaSynchronizationChecker.IsMetaFilesSynchronizedWithDb(CBackup backup, CLazyBackupMetaFile.] metasOnFs)
Error        An unexpected network error occurred. (Veeam.Backup.Common.CCppCOMException)
Error        Failed to read data from file n\\xxx.xxx.xxx.xxx\NMA B2D ALL.vbm]. (Veeam.Backup.Common.CCppComponentException)
Error           in c++: Error code: 0x0000003b


Hi @MarkD -

The only thing I can see from your log output is Veeam seems to be having a hard time, or is unable to, read your metadata file (the .vbm file, named: NMA B2D ALL.vbm ), as others have shared above. Is this file even present in your Repository storage? If so, then it just may be corrupted and as others have shared, you can remove the file, then perform a rescan of your Repository to recreate it; then reattempt your job. If you’re still having problems, I think we all may have exhausted all our knowledge and just recommend you get in touch with Veeam Support...unless someone has other thoughts?


Hello all,

 

i have found the problem and is a little stupid one.
it seems that de veeam VM was migrated to a other server in the cluster where a standard switch was configured wrong.. and that was sit there allready for a long time.. because this VM works on all other servers normaly.. and suddenly stopped working when it was sit on that server where it not to belong
to land there.. so did also change de DRS and fix the server in maintanance mode :)

so the problem is solved…. and want to thank all ppl who has react on my issue.

 

Best Regards,
MarkD


Sometimes ‘little’ things happen MarkD. Glad you got it all sorted 😊


Don’t sweat the little things as they say.  Glad you figured it out and resolved the issue.


Comment