Hardened (Immutable) extent in Scale Out Repository


Userlevel 7
Badge +6
  • Veeam Legend, Veeam Vanguard
  • 694 comments

If you do no know much about Hardened Repository in Veeam VBR v11, read here to learn more about:

I was asked, if a hardened repository can be a extent of a Scale Out Repository (SOBR). Good news: yes it can! But you have to take some facts into account.

  1. Hardened repositories can be Performance Tiers of SOBR.
  2. A SOBR can contain a mix of repositories: ReFS, XFS, Immutable and mutable.
  3. Even in mixed SOBR, Performance Placement Policy can be set. That leads to, for example, immutable increments and mutable fulls.
  4. When you mix, for example, ReFS with immutable extents, you can use Evacuate backups to free an extent.

     

I would not recommend any points 1 - 3!

In my opinion the point 4 is the most important. With evacuating an extent you can easily migrate from ReFS to XFS with immutability.

Notice:

  • If you evacuate an immutable repository, Veeam performs a copy, not a move operation! Which makes perfect sense!
  • When backups are evacuated to a hardened repository, files will be immutable as long as defined in repository.
  • During evacuation operation backups may fail because backup files are locked by the operation.

24 comments

Userlevel 7
Badge +5

From my experience recently we had an issue when mixing ReFS and XFS together and were actually told by Veeam that it is not a good idea.  We have since removed XFS to put in its own SOBR away from ReFS and things operate much better.

Userlevel 7
Badge +6

From my experience recently we had an issue when mixing ReFS and XFS together and were actually told by Veeam that it is not a good idea.  We have since removed XFS to put in its own SOBR away from ReFS and things operate much better.

Yes, you commented already here:

This is one of the reasons, “I would not recommend any points 1 - 3!”

Userlevel 7
Badge +5

Ah gotcha.  My bad not seeing that one. :stuck_out_tongue_winking_eye:

Userlevel 7
Badge +3

Thanks for the feedback of your test @vNote42 , it’s good to know!

Userlevel 1
Badge

I have been doing some testing on mixed ReFS and XFS immutable repositories  myself over the last few days and whilst it does work, its good to know that just because you can, doesn't mean you should  :grinning:

Userlevel 6
Badge +2

@vNote42 

Hi, you have a list of tasks on how to perform a best practice hardening on an O.S. Linux distro Ubutu LTS dedicated to the immutabiliy repo? I need to add the new featurte to my infrastructure, so any technical advice on security is welcome. Thank you in advance.

Userlevel 7
Badge +6

I have been doing some testing on mixed ReFS and XFS immutable repositories  myself over the last few days and whilst it does work, its good to know that just because you can, doesn't mean you should  :grinning:

I think there is an issue also due to the fact that ReFS normally for Veeam is formatted as 64K whereas XFS is 4K.

 

Userlevel 7
Badge +6

I have been doing some testing on mixed ReFS and XFS immutable repositories  myself over the last few days and whilst it does work, its good to know that just because you can, doesn't mean you should  :grinning:

I think there is an issue also due to the fact that ReFS normally for Veeam is formatted as 64K whereas XFS is 4K.

 

Do you think? For my understanding, we need 64k for ReFS because of the memory usage with 4k in Windows Server. But this could be outdated.

Userlevel 7
Badge +6

@vNote42

Hi, you have a list of tasks on how to perform a best practice hardening on an O.S. Linux distro Ubutu LTS dedicated to the immutabiliy repo? I need to add the new featurte to my infrastructure, so any technical advice on security is welcome. Thank you in advance.

Still working on it :grin:

Userlevel 7
Badge +6

I have been doing some testing on mixed ReFS and XFS immutable repositories  myself over the last few days and whilst it does work, its good to know that just because you can, doesn't mean you should  :grinning:

I think there is an issue also due to the fact that ReFS normally for Veeam is formatted as 64K whereas XFS is 4K.

 

Do you think? For my understanding, we need 64k for ReFS because of the memory usage with 4k in Windows Server. But this could be outdated.

Yeah I remember when REFS3 came out and there was even a huge forum topic about data loss due to 4K. It is probably still there. Then we had a veeam support case and one of the cloud engineers said that the mixing was problematic due to the block size difference among other things.

Userlevel 7
Badge +6

I have been doing some testing on mixed ReFS and XFS immutable repositories  myself over the last few days and whilst it does work, its good to know that just because you can, doesn't mean you should  :grinning:

I think there is an issue also due to the fact that ReFS normally for Veeam is formatted as 64K whereas XFS is 4K.

 

Do you think? For my understanding, we need 64k for ReFS because of the memory usage with 4k in Windows Server. But this could be outdated.

Yeah I remember when REFS3 came out and there was even a huge forum topic about data loss due to 4K. It is probably still there. Then we had a veeam support case and one of the cloud engineers said that the mixing was problematic due to the block size difference among other things.

Thanks for this information! It seems I was successful in suppressing the beginning of Block Cloning in ReFS :joy:

Userlevel 6
Badge +2

@AndyPilk

I need to add Windows repo server (Extent ReFS) to SOBR with XFS repo for immutability. Should I expect bad news? This afternoon I proceed to upgrade v11 and install linux server repo. stay tuned

Userlevel 6
Badge +2

I have been doing some testing on mixed ReFS and XFS immutable repositories  myself over the last few days and whilst it does work, its good to know that just because you can, doesn't mean you should  :grinning:

I think there is an issue also due to the fact that ReFS normally for Veeam is formatted as 64K whereas XFS is 4K.

 

Do you think? For my understanding, we need 64k for ReFS because of the memory usage with 4k in Windows Server. But this could be outdated.

Yeah I remember when REFS3 came out and there was even a huge forum topic about data loss due to 4K. It is probably still there. Then we had a veeam support case and one of the cloud engineers said that the mixing was problematic due to the block size difference among other things.

I have a sobr with 4 ReFS 64k formatted extents.
If I want to add an additional extent dedicated to immutability.
Do I have to format it to 64k to make it homogeneous?
Is this problem fixed with v11?

Userlevel 6
Badge +2

everyone on vacation to eat easter eggs? :joy::rofl:

 

V11 upgrade success.

Ubuntu VM created and configured, configured raw lun repo xfs format.
I'm doing the hardening

Userlevel 7
Badge +6

everyone on vacation to eat easter eggs? :joy::rofl:

 

V11 upgrade success.

Ubuntu VM created and configured, configured raw lun repo xfs format.
I'm doing the hardening

I hope, your Ubunto VM is not for production Repo?

Userlevel 7
Badge +6

I have been doing some testing on mixed ReFS and XFS immutable repositories  myself over the last few days and whilst it does work, its good to know that just because you can, doesn't mean you should  :grinning:

I think there is an issue also due to the fact that ReFS normally for Veeam is formatted as 64K whereas XFS is 4K.

 

Do you think? For my understanding, we need 64k for ReFS because of the memory usage with 4k in Windows Server. But this could be outdated.

Yeah I remember when REFS3 came out and there was even a huge forum topic about data loss due to 4K. It is probably still there. Then we had a veeam support case and one of the cloud engineers said that the mixing was problematic due to the block size difference among other things.

I have a sobr with 4 ReFS 64k formatted extents.
If I want to add an additional extent dedicated to immutability.
Do I have to format it to 64k to make it homogeneous?
Is this problem fixed with v11?

I did not have any issues when adding a XFS Repo (4k formatted) to a SOBR with ReFS (64k formatted) extends. 

Userlevel 7
Badge +6

@AndyPilk

I need to add Windows repo server (Extent ReFS) to SOBR with XFS repo for immutability. Should I expect bad news? This afternoon I proceed to upgrade v11 and install linux server repo. stay tuned

How have you been so far. I guess everything worked fine?

Userlevel 6
Badge +2

Before I installed Ubunto repo to test immutability, everything was fine. Formatted 4k and created BCJ.

Security parameters set on ssh:

• Reject empty passwords (PermitEmptyPasswords no)

• Reject root login (PermitRootLogin no) • Set a session time out limit for SSH sessions (ClientAliveINterval 300) • Limit the number of authentication attempts (MaxAuthTries 3)

• Limit the maximum number of unauthenticated connections (MaxStartups 3: 50: XX) • Reduce the maximum amount of time allowed to successfully login (LoginGraceTime 60)

• Limit the users on the server and those allowed to login via SSH ((AllowUsers User1 User2) All

• Consider installing failban and whitelisting the know IP addresses that may log in.

 

Userlevel 2
Badge

everyone on vacation to eat easter eggs? :joy::rofl:

 

V11 upgrade success.

Ubuntu VM created and configured, configured raw lun repo xfs format.
I'm doing the hardening

I hope, your Ubunto VM is not for production Repo?

What do you mean, it’s better to use it as a secondary repo (for copy jobs) and not as a primary?:confused:

Userlevel 7
Badge +6

everyone on vacation to eat easter eggs? :joy::rofl:

 

V11 upgrade success.

Ubuntu VM created and configured, configured raw lun repo xfs format.
I'm doing the hardening

I hope, your Ubunto VM is not for production Repo?

What do you mean, it’s better to use it as a secondary repo (for copy jobs) and not as a primary?:confused:

If I understand your question right, my answer is: Do not use VMs for Hardened Repositories!! Hardened Repositories works fine for both: primary and secondary target.

Userlevel 2
Badge

yes, my question was about using vm hardened repo..so if i don't have a physical linux server it is not recommended? can I ask you the main reasons?

Userlevel 7
Badge +6

Yes it is NOT recommended at all!

The/one reason is simple: If a hacker gets access to your hypervisor, he is able to delete the VM with all restore-points. Then it does not matter if files within VM are immutable or not. With a physicals server you are able to harden Linux and hardware, that way, this cannot happens.

I hope this answers your question? If not, please ask.

Userlevel 2
Badge

Yes thanks, it makes sense ..

What about iSCSI LUN from remote NAS ? by deleting the vm, shouldn't the LUN be safe anyway?

Userlevel 7
Badge +6

Yes thanks, it makes sense ..

What about iSCSI LUN from remote NAS ? by deleting the vm, shouldn't the LUN be safe anyway?

Right, LUNs are not deleted with VM. But you should choose NAS device very carefully. Devices like these from Synology offer a lot of features and protocols. From a security perspective, this is not recommendable. To much possibilities for hackers. In my opinion, a physical server with internal disks is a very good option. It is much easier to harden just one box instead of multiple.

Comment