Numerous Backup Cache Issues

So I’m am basically a day or two away from taking the time and going through and disabling the backup cache across all of my customers, it causes way too many problems. But thought I’d try asking here as a last resort since Veeam’s support seems useless as usual and my R&D forum posts aren’t getting any responses. 

I have had various cache related issues, especially since the ridiculously buggy V12 update, but here this past week it’s been especially bad. I have about 20 computers, that I know of right now, which are failing to upload the backup cache to the VCC server. As such these computers have not been backing up. At first I thought maybe they just needed some time to catch up or something, but it’s been over a week now and they’re not even modifying the files in the VCC repository. (Side note, yes, I did go look at the files, because it’s useful, even though Veeam says there’s no reason I should ever need to look at the files.)

Seems there’s numerous different errors and behaviors across the affected agents, some show status in the VSPC, some don’t, some show data transferred in the VCC console, some don’t, some show files being modified, some don’t (none show an actual increase in file size, so I don’t know what’s changing, but I’m thinking the file contents aren’t actually being modified), some show errors in the VSPC, some don’t.

My latest one I’ve looked more closely at does the following:

Shows the backup cache sync job running in the VCC console, shows the backup job running in the VSPC, updates (continually) the modified time for the relevant VIB file in the repository, the file size has stayed the same for several days, the VBM file modified time is also being updated. The job in the VCC console shows data being transferred. The job ends after a short time. When not running the VSPC shows a “cache is full” error. Which I’m assuming is why new backups are not going to the cache, and has nothing to do with the cache getting uploaded to the server.

I’ve also had other issues on other agents here at the same time, one which showed no activity in VSPC and no data transferred in the VCC console, though it did show a job running for seconds at a time, I wouldn’t have noticed it except for the bug in the console that causes completed jobs to not get removed from the “Running” view. This job said there was a corrupt file somewhere, in the logs, not in an error message actually reported to anything. Apparently Veeam can’t tell from the error message or the log files where the corrupt file is.

That scenario doesn’t make sense to me though, if the corrupt files in the repository, why did manually disabling the cache (to empty it) fix the problem? If the issue was a corrupt file in the cache, how did a VCC issue cause that? 

I am convinced there was some sort of VCC issue that caused all this, because there’s too may different customers affected at the same time for it to be anything else that I can think of. But no idea what sort of issue could be the problem, other customers are working just fine, it’s just the cache syncing jobs that are broken. 

Most of the issues have been fixed by simply disabling the cache, restarting the agent, and running the job. But I can’t come up with anything that could be caused by the VCC infrastructure having a problem, but could be fixed by the cache being cleared on the agent side.

So, some questions I’d like answered if anyone knows:

Does the backup cache syncing process literally just copy the files from the cache to the repository? Or does it do something funny and unnecessarily complicated like when you do a backup copy job? 

Anyone else seen similar issues with cache syncing that were resolved by disabling the cache?

Anyone have any general ideas? I’m still working with support, but thus far no one’s found what the problem is.