Troubleshooting a very weird issue right now

I believe they’re on a QNAP.

Is the copy job configured to use a different source than you’re expecting?

I wouldn’t expect the copy job to be talking about VMDKs, but the VBR/VIB files

Just double-checked the target and it’s correct. It’s our offsite datastore.

Sorry I meant the source for the BCJ 🙂

I checked the job settings and it’s pulling from the backup job for this server and it’s moving to our offsite datastore. Nothing has changed. This copyjob has been running fine for over a year. I’m not sure what’s happening. Is there a specific place in the BCJ’s settings I should check?

That’s correct, it’s only software RAID. I had a scheduled power outage at a DC, took 4x of them down gracefully, 2x of them showed RAW file systems upon power up, all the data was gone. Last time I used them!

Few inputs here.

@bp4JC, I’d strongly recommend review this on a support case. It’s probable the backups indeed have some corruption, but let the support Engineers guide you through this confirmation. If you want to try it yourself, use the Veeam Validator with the /backup flag (/file will try to make a connection on its own to the target you send it, so if it’s a UNC path for a network share, it tries to start its own session instead of using what’s in VBR; this can work, but very often causes some headaches). Support can help you with checking which Restore Point in particular is suspect and give more targeted commands

@MicoolPaul Backup Copy will get deep inside the backup file to get at blocks, it doesn’t parse on just the top level storage container. Backup Copy is pretty “intelligent” in that it’s not just copying restore points, it opens the source backup file, parses its contents, and moves truly only the relevant data; this is why you’ll see it references FIBs (File in Backup) like VMDK, VHDX, etc.

There is a slight chance Failed to call CryptDecrypt can be an interruption between the gateway/source share, but I want to be clear it can equally mean that indeed there is file level corruption.

(

Imagine the call stack of Gateway => NAS looking like

Gateway: “Hey NAS friend, I need to read this offset from a file for CryptDecrypt”,

NAS:“here you go, one block coming right up!”,

cosmic rays or something malform the packet/payload and the CryptDecrypt gets this malformed payload

Gateway: “wtf m8?”

)

But I think it’s best to check it out with Support to confirm what you’re dealing with.

Just an extra comment on this, depending on whether you’re going “bargain basement” QNAP level, or decent spec, you might find a low-end Dell or HPE server to be far more cost effective. I had a quote the other year for a Synology 10GbE model end up more expensive than a Dell NX series server, the only thing I disliked about those was that they used 2x 600GB disks in RAID1 as their boot OS. Whereas you could use a Dell PowerEdge R740XD/XD2 to get a lot of spindles. (And if you REALLY need a lot of spindles, the HPE Apollos are great).

3-2-1 Rule.