• EN
Solved

Error: Failed to call RPC function 'FcReadFileEx': An unexpected network error occurred. Failed to read data from file

M
Userlevel 1
  • MarkD
  • Not a newbie anymore
  • 2 comments

i have a problem. if the job starts it will finally give me this error: Error: Failed to call RPC function 'FcReadFileEx': An unexpected network error occurred. Failed to read data from file

 

and the file is on a synology NAS and i can just enter that device and also i can just make a network map to the directory where the files are.. but somehow veeam stops working allready for a week..

network is checked.. and all is Ok and i can reach the volumes etc..

i did restart the veeam server VM and also restarted the services but still not working..

if i go to the backup reposority and do rescan on the backup locations it will give the same errors..

I cant find here any info about the error i have and hope someone can help me.

the error from the mail is:

Error: Failed to call RPC function 'FcReadFileEx': An unexpected network error occurred. Failed to read data from file [\\172.16.xx.xxx\VeeamBackupRepository\Escrow\VEEAM-Backup\1 - B2D ALL Daily Escrow\1 - B2D ALL Daily Escrow.vbm].

icon

Best answer by MarkD 21 February 2024, 14:52

View original

10 comments

Chris.Childerhose
Userlevel 7
Badge +20

Based on the error message it would seem the VBM file is corrupted.  Did you try to run an Active Full backup?

What version of Veeam are you using?  What version of Windows?  Did you check the backup logs for the job here - C:\ProgramData\Veeam\Backup\Jobname?

Chris Childerhose - VMCA2022 | VMCE2023 | Veeam Vanguard 7* | Veeam Legend 3* | vExpert 5* | VCAP-DCV/VCP-DCV | Object First Ace | Cisco Champion | Twitter: @cchilderhose | Blog Site – https://just-virtualization.tech
wesmrt
Userlevel 7
Badge +4

You can also try to remove this VBM file, rescan the repository and then try to run the policy gain and see if it works.

Wesley Martins @ Specialist Enginner at Veeam Technical Support | My blog: https://itproland.com.br (PT-BR)
MicoolPaul
Userlevel 7
Badge +20

Others have addressed the VBM appears to be corrupted, I’d like to share some details to attempt to prevent this from happening again.

  • You’ll have a gateway server talking to your SMB share, make sure this gateway server is in the same physical location as the NAS and can talk directly to it without any firewalls or traffic inspection in between.
  • Make sure Synology’s write cache feature is disabled. Performance will unfortunately take a hit but it means that writes won’t be confirmed until they’ve actually been written to disk instead of to RAM (as if you have a crash or power loss the data is inconsistent!, Synology doesn’t have any battery backups for the write cache).
  • Id suggest using iSCSI to communicate to the Synology instead of SMB if possible. This way you can leverage fast clone to increase storage efficiency. Your server that is connected to the storage would have the storage mapped as a drive letter with an ReFS file system (assuming you’re using windows here, XFS if Linux), you’d also be able to leverage multi path IO to increase network speed between source and destination potentially if they’ve both got multiple network ports. Finally you can also put that traffic on a dedicated VLAN or switch to avoid noisy network issues.
Michael Paul - #VeeamLegend | #VeeamVanguard | Veeam Certified Architect | VMware vExpert | https://micoolpaul.com | Twitter: @MicoolPaul | Mastodon: @micoolpaul@masto.nu | Bluesky: @micoolpaul.bsky.social
Chris.Childerhose
Userlevel 7
Badge +20

You can also try to remove this VBM file, rescan the repository and then try to run the policy gain and see if it works.

I keep forgetting about doing this with the VBM file.  This is another great catch by @wesmrt 

Chris Childerhose - VMCA2022 | VMCE2023 | Veeam Vanguard 7* | Veeam Legend 3* | vExpert 5* | VCAP-DCV/VCP-DCV | Object First Ace | Cisco Champion | Twitter: @cchilderhose | Blog Site – https://just-virtualization.tech
coolsport00
Userlevel 7
Badge +17

Several things may be at play here @MarkD . And each could be causing issue with your metadata file (.vbm), as Chris states.

If your backend Repository storage has “unraid” settings, you could try and change the setting for “Case-sensitive names” from Force lower to Auto, then change the .vbm file name (keep the name) to have all upper case letters. See a comment from this Comm Hub post.

Another issue could be your antivirus software scanning your storage (Repo) volume during backup, causing “network” and/or access issue to your .vbm file, as noted in this Comm Hub post. You can either disable A/V on your Repo and/or VBR server, or create exceptions for your Repo paths.

Another potential cause could be access credentials. Verify you’re using domain\username as the format to connect to your backend storage. As others noted, you can do this in the Properties of your Repo and do a rescan operation.

Hopefully one of those options work for you.

Shane Williford - Veeam VMCA/VMCE | Veeam Legend | VUG Leader | VCP-DCV | Twitter: @coolsport00
M
Userlevel 1

Hello all!

 

all the tricks and tips did not work out for me.

its so frustrating.. i even did add a new networkcard in de VM with the E1000 Nic instead of the VMX standard.. and even that will not work… also checked the network on switches etc if there is someting wrong. the Jumbo Frames are all OK on the whole route.

it seems that network segment can't reach the Synology NAS. while it allready works for years.

suddenly it stops working a 1.5 week ago..

 

even i did try the veeam.backup.extracter.exe but.. my backup files are over 1TB and i simple has no room  to copy all the data into 1 and the same folder.

im also had restarted the vCenter appliance.. because it communicate with that too but no result.

also the backup VM restarted and also no result.

 

someone still a tip or maby i did overlook someting?

 

here i have a copy paste part where the error started:

Info         VC session cookie is '"f79023ab0a2032c4e818dcef7b59a8126e463695"', url 'https://dca-vm-133:443/sdk'.
Info         [Soap] Host content info: host "dca-vm-133", type "vpx", version "6.0.0", build "4541948", apiVersion "6.0", hostTime "11-2-2024 18:58:47", sessionKey: "52ea2a83-1855-cd5b-bf16-bc8e76bfac26"
Info         [Soap] Connection 'dca-vm-133:443:oss\svc_ve:False::0:1' is disposing.
Info         Job size: 13,3 TB (7,1 TB provisioned).
Info         [JobSession] TotalObjects '68', totalSize '13,3 TB'
Info         [BackupJobTarget] Initializing backup target, mode: 'Retry', agent sharing mode: 'SharedAgent', repository: 'Backup Repository'.
Info         [ProxyAgentTracker] Stopping agents on Job: '01735064-3696-45db-b43b-2b59d52dfc6f'
Info         Checking version of repository 'Backup Repository LCM'.
Info         Repository 'Backup Repository LCM' version is supported.
Info         [RPC] Loading options.
Info         Resolved by NTLM strategy ip addresses and host names: xxx.xxx.xxx.xxx xxx.xxx.xxx.xxx bac-vm-112
Info         [CProxyRpcInvoker] RpcInvoker [39838562] has been created. Host: [bac-vm-112:6160]
Info         [RPC] RPC confirm session has been created. Id: {764c416d-a891-4c58-aa48-c882e3fa920e}
Info         [BackupEncryptedStateUpdater] Initializing backup encryption state (BackupId='3c0c8db3-af0f-460f-8d06-1a67c407d1a5', IsEncryptionEnabled=False, HasCryptedStorages=False, EncryptionState=Unencrypted)
Info         [BackupLock] Locking 1 backup(s) as write-only: 3c0c8db3-af0f-460f-8d06-1a67c407d1a5
Info             [LockResolver] Getting conflicting sessions
Info             [LockResolver] No conflicting sessions found
Info             [API] Creating new lease. TTL: [00:05:00]
Info             [API] New lease was created. Id: [50251510-3496-4229-b305-301683d242c6], TTL: [00:05:00]
Info         [LeaseKeeper] Created. LeaseId: [5025151034964229b305301683d242c6], TTL: [300 sec], ignoreExceptions: [False].
Info         [BackupLock] Backup(s) have been locked, lockIds: [9e18c1c3-6456-45f0-a5d5-f776be2a5fd5]
Info         [CStorageLinksHelper] Asserting storage links...
Info         [CStorageLinksHelper] Storage links checked successfully
Info         [Mutex] Created mutex [Global\BackupMeta[01735064-3696-45db-b43b-2b59d52dfc6f]]
Info         [Mutex] Acquired mutex [Global\BackupMeta[01735064-3696-45db-b43b-2b59d52dfc6f]]
Info         Removing dca-vm-133:443:oss\svc_ve:False::0:1 from cache. Reason: Expired
Info         [Soap] Logout from "https://dca-vm-133:443/sdk"
Info         [FC] Loading content from file '\\xxx.xxx.xxx.xxx\NMA B2D ALL.vbm'
Error        Failed to call RPC function 'FcReadFileEx': An unexpected network error occurred. Failed to read data from file [\\NMA B2D ALL.vbm].
Error        An unexpected network error occurred.
Error        Failed to read data from file [\\NMA B2D ALL.vbm].
Error        --tr:Error code: 0x0000003b
Error        --tr:Failed to complete RPC binary request. Method: [FcReadFileEx], In XML: [<InputArguments><FilePath value="\\NMA B2D ALL.vbm" /><DesiredAccess value="2147483648" /><ShareMode value="1" /><CreationDisposition value="3" /><FlagsAndAttrs value="0" /><Offset value="0" /><BytesToRead value="65536" /></InputArguments>]
Error        --tr:Failed to invoke RPC request
Error        --tr:Failed to call DoRpc. CmdName: [DoSerialRpc] inParam: [<InputArguments/>].
Error        --tr:Failed to call DoRpcWithBinary. Method: [DoSerialRpc], In xml: [<InputArguments/>]
Error        --tr:Failed to call DoRpcWithBinary
Error        --tr:Failed to do RPC call with serialized data, Strategy type: DefaultStrategy
Error        --tr:Failed to call [DoSerialRpc], session id: {764c416d-a891-4c58-aa48-c882e3fa920e}
Error        --tr:Failed to call [FcReadFileEx] (System.Runtime.InteropServices.COMException)
Error           at VeeamProxyClient.GenerateComException(basic_string<char\,std::char_traits<char>\,std::allocator<char> >* excMsg, Int32 errHrCode)
Error           at VeeamProxyClient.RegenerateCLRException(exception* excBase)
Error           at VeeamProxyClient.CCliProxyRpcInvoker.DoRpcWithBinaryEx(String cmdName, String inXml, Byte[]& inData, String& outXml, Byte[]& outData)
Error           at Veeam.Backup.ProxyProvider.CProxyRpcInvoker.CallWithBinaryEx(String methodName, CProxyInvokeInputArg inputArgs, Byte[]& inData, Byte[]& outData)
Error        Failed to check whether meta file [\\xxx.xxx.xxx.xxx\NMA B2D ALL.vbm] is synchronized with DB
Error        Failed to load content from file '\\xxx.xxx.xxx.xxx\NMA B2D ALL.vbm' (System.Exception)
Error           at Veeam.Backup.ProxyProvider.CProxyFileCommander.LoadFileContentBinary(String link, UInt64 maxSize)
Error           at Veeam.Backup.Core.CWinFileCommander.LoadFileContentBinary(String link, UInt64 maxSize)
Error           at Veeam.Backup.Meta.CBackupMetaFile.LoadContent(UInt64 maxBytesToRead)
Error           at Veeam.Backup.Meta.CBackupMetaHeaderReader..ctor(IBackupMetaFileEx meta)
Error           at Veeam.Backup.Core.CBackupMetaSynchronizationChecker.IsMetaFilesSynchronizedWithDb(CBackup backup, CLazyBackupMetaFile[] metasOnFs)
Error        An unexpected network error occurred. (Veeam.Backup.Common.CCppCOMException)
Error        Failed to read data from file [\\xxx.xxx.xxx.xxx\NMA B2D ALL.vbm]. (Veeam.Backup.Common.CCppComponentException)
Error           in c++: Error code: 0x0000003b

coolsport00
Userlevel 7
Badge +17

Hi @MarkD -

The only thing I can see from your log output is Veeam seems to be having a hard time, or is unable to, read your metadata file (the .vbm file, named: NMA B2D ALL.vbm ), as others have shared above. Is this file even present in your Repository storage? If so, then it just may be corrupted and as others have shared, you can remove the file, then perform a rescan of your Repository to recreate it; then reattempt your job. If you’re still having problems, I think we all may have exhausted all our knowledge and just recommend you get in touch with Veeam Support...unless someone has other thoughts?

Shane Williford - Veeam VMCA/VMCE | Veeam Legend | VUG Leader | VCP-DCV | Twitter: @coolsport00
M
Userlevel 1

Hello all,

 

i have found the problem and is a little stupid one.
it seems that de veeam VM was migrated to a other server in the cluster where a standard switch was configured wrong.. and that was sit there allready for a long time.. because this VM works on all other servers normaly.. and suddenly stopped working when it was sit on that server where it not to belong
to land there.. so did also change de DRS and fix the server in maintanance mode :)

so the problem is solved…. and want to thank all ppl who has react on my issue.

 

Best Regards,
MarkD

coolsport00
Userlevel 7
Badge +17

Sometimes ‘little’ things happen MarkD. Glad you got it all sorted 😊

Shane Williford - Veeam VMCA/VMCE | Veeam Legend | VUG Leader | VCP-DCV | Twitter: @coolsport00
Chris.Childerhose
Userlevel 7
Badge +20

Don’t sweat the little things as they say.  Glad you figured it out and resolved the issue.

Chris Childerhose - VMCA2022 | VMCE2023 | Veeam Vanguard 7* | Veeam Legend 3* | vExpert 5* | VCAP-DCV/VCP-DCV | Object First Ace | Cisco Champion | Twitter: @cchilderhose | Blog Site – https://just-virtualization.tech

Comment


Terms & Conditions