Fun Friday #45: Strange Things on Backup Environment

Veeam agent during backup go stuck but doesn't stop and after more than 60+ hours breaks definitely vss of that machine 😫

What I have seen : the backups were written to a USB-disk as a primary repository. The disk was connected always of course, no copy job or so to a secondary repository 🤔. A matter of time when backups were all gone or broken 😅. I like a USB-disk as an offline solution (second or third backup copy), but NEVER as a primary and only solution !!!

@wolff.mateus  : VBM365 is using a jet database, so block cloning and so is not being used or necessary

The whole VB365 with ReFS thing is just about performance, by default ReFS retains data integrity features that reduce performance, and these data integrity features don’t exist in NTFS. If you’ve deployed ReFS but not disabled this you can restore performance to NTFS levels by running in PowerShell:

Set-FileIntegrity <DriveLetter>:\ -Enable $False

Full details here: https://bp.veeam.com/vb365/guide/buildconfig/proxy-repo.html

Set-FileIntegrity <DriveLetter>:\ -Enable $False

Very cool I might just do this.

At the risk of spamming I need to follow this up with a story of a customer that I remembered after writing this.

We had a completely new client, no prior engagement, want to talk to us about Veeam, as they suspected their current MSP might not be best protecting them.

I had a call with them and it was clear we could help, and we agreed I’d carry out a day’s discovery as to how it all works, and provide a write up.

That one day produced near 30 pages of varying severity issues, around half were critical.

Of particular note was their replicas were all being remapped for VM port groups, but the VM port group in their DR site they were going to, had no NICs attached to it! So it would NEVER work for a functional DR.

The unfortunate customer also told me their MSP had suggested they’d be better off moving from 1Gbps network mode to 10Gbps Direct SAN Access & Storage Snapshots on their NetApp, sold them a brand new, beefy server for Direct SAN Access, all the NetApp & Veeam licenses as well for Storage Snapshot functionality. Then, on day 1 of implementation, did a complete reversal and said actually it doesn’t make sense to do it and advised them to remain with 1Gbps Network mode, because “it’s easier & safer”. 🤯

Not testing or having test environments is something I see often.  I went to a new company and they had Veeam, SRM, and a bunch of other great DR software with a fully mirrored site for hardware, Flash SANS, the works.

There was no test groups, and no tests preformed. I created some test SAN volumes, VM’s and SRM jobs and not only did it fail when run, it really did a number on it. Turns out there were some significant networking issues that were never realized.  It took 2 minutes to find the issue, and a while to resolve, but it’s so much easier when it’s not a site down disaster. 

They were planning to do a full scale test as well which would have also been a nightmare involving 100’s of TB of stuck data at the DR site with volumes no longer syncing.

This goes for testing backups and documentation for everything as well. It is so much easier to not figure this stuff out at 4 AM when everyone is in a panic.