Restore failed to load metadata bank

Hi ​@delumerlino,

How exactly did you delete the file? This sounds like a mismatch between the Veeam database and the repository data to me.

What happens when you delete a file outside of Veeam (e.g. by a file explorer or else):

The file gets deleted but the Veeam database might not get the information so Veeam might think that the file is still present.

When you now manually restore the file (which an option from the repository provider like Backblaze) the checksum might chance because on a logical level it might be a different file but carries all visible attributes identically to the old (deleted) one which is the cause of the mismatch.

Depending on the chain and data format Veeam is using this can lead to errors if only certain files (e.g. incremental backup files from a forward chain) within a “set” or a chain get deleted.

My suggestion is to restore all depending files from the same chain and try to import it once again, otherwise start a new full backup (or a new full backup copy in case you are using a backup copy job) to clean things up.

Hope that helps!

Lukas

Hi Lukas,

thanks for your answer.

The backup file has been deleted from VBR console (from backups/disk).

Bear in mind that I use Backblaze personal cloud backup, which trasfer in chunks the files from local disk to BB.

The restore from BB is by zip file, then extracted and imported (without any problem) from VBR import backup.

I have already restored all the chain, but also the full backup fails...

So when I get you right you have a local disk backup with Veeam and you backup these files with Backblaze personal cloud backup to the BB cloud?

Yep, you are right

Got you!

Basically it comes to the same: The restored file from the Veeam Repo basically has changed or you can say is different as the original file (which has been deleted, but on the correct way via Veeam GUI).

This basically causes the CRC mismatch and this might cause the issue. It should work when you restore the whole chain but please keep in mind that you not only need to restore the vbk (full backup) and vib (incremental backup) files but also the vbm (metadata file). You have to restore every element of the chain to let this work: Veeam File Extensions | Veeam Community Resource Hub

If it still doesn’t work I suggest to try opening a Veeam support case (not sure if that works without a valid license so on the community edition, but worth a try).

Besides the current issue I recommend not to “combine” different backup solutions in that way. In case you want to offload or copy backup data to a different location (e.g. cloud) - which is definately reommended - please give the primary software (which is Veeam in your case) the full control and handling.

That means: Suggest creating a backup copy job to a cloud object repository (Backblaze should offer that as far as I know) using Veeam Backup Copy and try not to “manipulate” (from a Veeam perspective) the Veeam properitary files that lay on the primary repository by using an isolated solution to simply copy the block files. Unfortunately it isn’t as easy as it seems because of the Veeam database and multiple dependencies.

How are you using the BB service to get the backup files from Veeam?  Using a copy job or something else?  Based on the message you got about CRC there is a data issue with the files you restored.  But not knowing how you send files to BB makes it hard to guess.

BB have its own client that sends big files in chunks to BB storage.

Basically, I have only a backup storage that is synced with BB client to BB cloud

Then this will never work properly for a restore.  You need to get Veeam to send to the BB service otherwise there is metadata missing that Veeam cannot sync properly to ensure the backup chains are complete.

Why if BB copies the entire folder where Veeam store the backups?

What if I zip before sending?

Anyway, I have already used with Veeam Agent and it worked...

You cannot use it with VBR like the Agent for Veeam.  There are certain things stored within the backup chain and in the database that Veeam needs to properly read and restore from a backup chain.  So Agent is fine as that you can point directly to BB without issue but when it comes to VBR that is a different story as it works differently than the agent.  There is no way around it as you will probably run in to this problem every time.

Maybe I still don’t understant what is different between a local folder and a 1:1 copy of the same folder on a cloud storage and then restored.

Everything is copied as the original folder, what’s changed?

Test for Lukas (has problems posting)

Edit: can be deleted

Thanks a lot for your time!

Not a problem.  Let us know if you get it working.

I have finished restoring from vbk after a veeam.backup.validate check, so something is still possible 😁

That is good to hear.  Yeah the Veeam validator is a good tool as well and maybe the solution for your use case.

just some input ​@delumerlino, while i agree with the topic consensus that it’s not great to use a sync tool for 3-2-1 rule (especially when Veeam can do backup copies to backblaze as a normal feature), some light to shed on the topic:

• CRC Error: Veeam read a datablock, but the CRC value of the read block was not the same as what the backup file recorded for this data block. Typically means bit rot (read: data corruption), but in essence the issue is a failed CRC check
• Syncing the backups with a 3rd party tool is not ideal - it can work, but it’s prone to errors (lock conflicts) and potential missed data. I’ve not used the BackBlaze sync tool, but if it’s really chunking out the data, i would personally be nervous as now another application is transforming the backup data for its own purposes, basically another layer of abstraction over your backup data that can have issues preventing restores
  • Similarly, it’s unclear from a quick look how good the integrity checking for the BackBlaze sync tool is. That is, does it do similar CRC checks, how does it handled lost data, etc. Complete mystery and without a way to check it prior to restore, it’s adding a big risk IMO as testing your backups regularly is essential. Bitrot happens, as do bugs in software, and without a way to check it on demand, you’re at the mercy of probability when it comes to restoring the data
• That the restore worked after the validation likely is coincidence; the good news is that it doesn’t sound like data corruption, so likely there was an issue with reading the data blocks that self-resolved.

Long term, I would add BackBlaze as a repository directly in Veeam and use Backup Copies to create a copy of your data on BackBlaze. You’ll be able to do Health Checks, SureBackup, many more checks to ensure the data is still usable.

You’ve already gotten your answer, but just for clarity for anyone else that may read this thread, backing up your Veeam Backup server with another backup service is not recommended nor supported and may result in corruption of your backup data in the local copy, your cloud copy, or both. I recall Anton Gostev mentioning this elsewhere in recent history as well when someone raised the same question, curiously also to Backblaze personal in that post as well.

In order to use cloud storage, the proper method is to utilize a S3 or S3 Compatible object storage (Amazon S3, Wasabi, Backblaze B2, Azure Blob, etc) and then utilize either a SOBR or Copy to Object.  Note that these options are not available on Community Edition.