Fun Friday #45: Strange Things on Backup Environment


Userlevel 7
Badge +11

Here we go, another Friday, another Fun Friday chapter!

As an architect solution, we normally develop, deploy and maintain a vast number of data protection scenarios. We often take care many backup environment and occasionally, we have contact with many technologies and professionals.

So, what was the stranger thing that you saw in a backup infrastructure?

It could be anything:

  • Strange kind of deployment scenarios.
  • Bad behavior during backup executions.
  • People doing strange things during the project.

Starting now....

 


14 comments

Userlevel 7
Badge +21

One of our VBM365 installs done using ReFS and not NTFS like you are supposed to. 🤯

Userlevel 7
Badge +13

Veeam agent during backup go stuck but doesn't stop and after more than 60+ hours breaks definitely vss of that machine 😫

Userlevel 7
Badge +11

One of our VBM365 installs done using ReFS and not NTFS like you are supposed to. 🤯

Saw this also multiple times. Why? Because they are often installing VBM365 on the same server that is already being used for VBR. I don’t like it. I never instasll VBM365 on the same server where VBR is installed? Why? No only because you have to need repo’s : one in REFS for VBR and one in NTFS for VBM365 but you are dependent on the versions of both. Extra take care when upgrading and especially : the same resources (memory and CPU) is being used on the same server, not knowing from each other what is bein occupied. I do not recommend it to install both on the same server.

Userlevel 7
Badge +11

What I have seen : the backups were written to a USB-disk as a primary repository. The disk was connected always of course, no copy job or so to a secondary repository 🤔. A matter of time when backups were all gone or broken 😅. I like a USB-disk as an offline solution (second or third backup copy), but NEVER as a primary and only solution !!!

Userlevel 7
Badge +11

One of our VBM365 installs done using ReFS and not NTFS like you are supposed to. 🤯

Saw this also multiple times. Why? Because they are often installing VBM365 on the same server that is already being used for VBR. I don’t like it. I never instasll VBM365 on the same server where VBR is installed? Why? No only because you have to need repo’s : one in REFS for VBR and one in NTFS for VBM365 but you are dependent on the versions of both. Extra take care when upgrading and especially : the same resources (memory and CPU) is being used on the same server, not knowing from each other what is bein occupied. I do not recommend it to install both on the same server.

I have never use or implement VBM365.

So, sorry for my question… Why cannot we use ReFS with that?

Userlevel 7
Badge +11

@wolff.mateus  : VBM365 is using a jet database, so block cloning and so is not being used or necessary

Userlevel 7
Badge +11

@wolff.mateus  : VBM365 is using a jet database, so block cloning and so is not being used or necessary

A nice to know! I notice that for repo ReFS and NTFS are allow.

Thank for that Nico!

Userlevel 7
Badge +20

The whole VB365 with ReFS thing is just about performance, by default ReFS retains data integrity features that reduce performance, and these data integrity features don’t exist in NTFS. If you’ve deployed ReFS but not disabled this you can restore performance to NTFS levels by running in PowerShell:

Set-FileIntegrity <DriveLetter>:\ -Enable $False

Full details here: https://bp.veeam.com/vb365/guide/buildconfig/proxy-repo.html

Userlevel 7
Badge +20

As for me, the strangest behaviour I’ve witnessed is people having a very “defeatist” attitude towards issues in Veeam.

 

They have VSS issues in the VM? They disable application aware processing for the VM or set it to try and ignore failures (bonus points when this is one of the most mission critical servers).

Getting issues with VMware Tools services not starting in SureBackup? Disable the test for this!

 

My absolute favourite: Don’t understand how SureBackup works? Don’t read about it, just don’t implement it 😩

 

Ive had customers amazed that Veeam could do stuff like instant VM recovery as they’ve had an MSP forcing them to do entire VM restores, quick rollback will not help when your blocks have nearly all been changed by ransomware!

Userlevel 7
Badge +21

The whole VB365 with ReFS thing is just about performance, by default ReFS retains data integrity features that reduce performance, and these data integrity features don’t exist in NTFS. If you’ve deployed ReFS but not disabled this you can restore performance to NTFS levels by running in PowerShell:

Set-FileIntegrity <DriveLetter>:\ -Enable $False

Full details here: https://bp.veeam.com/vb365/guide/buildconfig/proxy-repo.html

Very cool I might just do this.

Userlevel 7
Badge +17

My favorite are people complaining that Veeam does not support a new vSphere version from day 1on….

Userlevel 7
Badge +20

At the risk of spamming I need to follow this up with a story of a customer that I remembered after writing this.

 

We had a completely new client, no prior engagement, want to talk to us about Veeam, as they suspected their current MSP might not be best protecting them.

 

I had a call with them and it was clear we could help, and we agreed I’d carry out a day’s discovery as to how it all works, and provide a write up.

 

That one day produced near 30 pages of varying severity issues, around half were critical.

 

Of particular note was their replicas were all being remapped for VM port groups, but the VM port group in their DR site they were going to, had no NICs attached to it! So it would NEVER work for a functional DR.

 

The unfortunate customer also told me their MSP had suggested they’d be better off moving from 1Gbps network mode to 10Gbps Direct SAN Access & Storage Snapshots on their NetApp, sold them a brand new, beefy server for Direct SAN Access, all the NetApp & Veeam licenses as well for Storage Snapshot functionality. Then, on day 1 of implementation, did a complete reversal and said actually it doesn’t make sense to do it and advised them to remain with 1Gbps Network mode, because “it’s easier & safer”. 🤯

Userlevel 7
Badge +11

As for me, the strangest behaviour I’ve witnessed is people having a very “defeatist” attitude towards issues in Veeam.

 

They have VSS issues in the VM? They disable application aware processing for the VM or set it to try and ignore failures (bonus points when this is one of the most mission critical servers).

Getting issues with VMware Tools services not starting in SureBackup? Disable the test for this!

 

My absolute favourite: Don’t understand how SureBackup works? Don’t read about it, just don’t implement it 😩

 

Ive had customers amazed that Veeam could do stuff like instant VM recovery as they’ve had an MSP forcing them to do entire VM restores, quick rollback will not help when your blocks have nearly all been changed by ransomware!

Excellent colocations Michael!

Userlevel 7
Badge +8

Not testing or having test environments is something I see often.  I went to a new company and they had Veeam, SRM, and a bunch of other great DR software with a fully mirrored site for hardware, Flash SANS, the works.

 

There was no test groups, and no tests preformed. I created some test SAN volumes, VM’s and SRM jobs and not only did it fail when run, it really did a number on it. Turns out there were some significant networking issues that were never realized.  It took 2 minutes to find the issue, and a while to resolve, but it’s so much easier when it’s not a site down disaster. 

 

They were planning to do a full scale test as well which would have also been a nightmare involving 100’s of TB of stuck data at the DR site with volumes no longer syncing.

 

This goes for testing backups and documentation for everything as well. It is so much easier to not figure this stuff out at 4 AM when everyone is in a panic. 

Comment