• EN

Experience with XFS

vNote42
Userlevel 7
Badge +13

Because of the new v11-feature of immutable backups on XFS filesystem, I am looking for real-world experiences with XFS as Repository.

I just read through this forum entry:

https://forums.veeam.com/veeam-backup-replication-f2/v10-xfs-all-there-is-to-know-t65222-30.html

and the closing experience of user ferrus:

https://forums.veeam.com/veeam-backup-replication-f2/v10-xfs-all-there-is-to-know-t65222-30.html#p388579

 

Sounds very good to me so far. I can remember first implementations of ReFS Block Cloning. Lets say, it does not work that fine at the beginning.

Would you share some of your experiences with XFS as Veeam Repository? I would be interested in:

  • stability 
  • performance (over time)
  • needs for troubleshooting
  • administrative effort

Thanks!

Wolfgang | vnote42.net | @vNote42

38 comments

vNote42
Userlevel 7
Badge +13

For all those are interested in this topic as well, check out the in-depth video about XFS as a repository here: https://community.veeam.com/blogs-and-podcasts-57/building-secure-linux-repositories-168

 

Wolfgang | vnote42.net | @vNote42
randyweis
Userlevel 5
Badge +4

Not yet, but I have a few customers VERY interested and will be testing soon with them. I will update here.

Randy Weis, VMCE, VMCA
BertrandFR
Userlevel 7
Badge +8

No experiences on large scale just some tests on old storage array, works pretty well.

I’m excited to build a large (many PB+ ) SOF xfs based, 2021 will be fun! :nerd:

Thanks for sharing interesting links @vNote42 , from my pov XFS/Reflink is more old and stable than reFs when it appears.

gtelnet
Userlevel 3
Badge

We’ve had ~30 Ubuntu 20.04 repo’s in production since v10 launch in 2020, all with reflink enabled; the majority being at client sites and then a few ~500TB extents in sobr’s at our data centers; they all have been rock solid, which is to be expected from linux.

  • stability - no issues at all
  • performance (over time) - hasn’t changed in a year; only limited by the disk iops available from your hardware
  • needs for troubleshooting - no troubleshooting over the past year
  • administrative effort - linux experience or being a fast learner is important for the initial design and deployment; hardware monitoring/testing to determine the number of concurrent jobs that your hardware can handle requires some effort in the beginning but that applies to any OS; after that, cron jobs take care of updates and we schedule a manual reboot of the repo’s whenever an update notifies us that it requires it
jdw
Userlevel 4
Badge

I have installed a SOBR with three extents of XFS in our largest VCC datacenter.  It works extremely well and have not had issues.  We did mix it with ReFS but that was a bad idea and created a new SOBR just for XFS alone and no mixing.

We also try to keep SOBRs at 2-3 extents maximum and then create new ones as needed.

Thanks Chris!

What was the problem(s) when mixing XFS with ReFS in a SOBR?

One thing to note is the block size for ReFS is recommended on Veeam to be 64K and XFS is 4K, so I believe that alone would be a problem if trying to mix in one SOBR.

Also keep in mind that Microsoft only supports Trim/UNMAP for ReFS on Storage Spaces. ReFS shouldn’t be used with SAN storage or you could get some really weird space accounting. Anton also calls this out in the R&D Forum. 

 

VMCA | vExpert | Twitter @jdwallace | Blog https://jdwallace.com
gulzarshaikhveeam
Userlevel 6
Badge +1

I had tried to set up XFS as a Veeam repository and failed. No real help from the support either and I had to settle with NFS based repository. But this was with v10.

Veeam Legend | CEH | Azure(Expert, Sec) | VMCE | VCP | VERITAS VIP | Sec+ | MCSE | ITIL | NCIE | NCDA | +++
gulzarshaikhveeam
Userlevel 6
Badge +1

I had tried to set up XFS as a Veeam repository and failed. No real help from the support either and I had to settle with NFS based repository. But this was with v10.

Thanks for the information @gulzarshaikhveeam. What did not work?

It used to fail to get added as a repository. The support tried various steps but it did not work ATT. Case # 04450930

Veeam Legend | CEH | Azure(Expert, Sec) | VMCE | VCP | VERITAS VIP | Sec+ | MCSE | ITIL | NCIE | NCDA | +++
Link State
Userlevel 7
Badge +7

No experiences on large scale just some tests on old storage array, works pretty well.

I’m excited to build a large (many PB+ ) SOF xfs based, 2021 will be fun! :nerd:

Thanks for sharing interesting links @vNote42 , from my pov XFS/Reflink is more old and stable than reFs when it appears.

It is true for a while the refs driver was buggy :(

Veeam: VMCA | VMCE | VMXP | Veeam Legend - Microsoft: MCITP | MCP | MCSA | 2008 R2 | 2012R2 | 2016 | MCSE Core Infrastructure | MCSE Cloud Platform - Azure: AZ900 | AZ104 - VMWare: VCP-DCV Vsphere 7.x - Cisco: CCNA (Expired)
Chris.Childerhose
Userlevel 7
Badge +20

I have installed a SOBR with three extents of XFS in our largest VCC datacenter.  It works extremely well and have not had issues.  We did mix it with ReFS but that was a bad idea and created a new SOBR just for XFS alone and no mixing.

We also try to keep SOBRs at 2-3 extents maximum and then create new ones as needed.

Chris Childerhose - VMCA2022 | VMCE2023 | Veeam Vanguard 7* | Veeam Legend 3* | vExpert 5* | VCAP-DCV/VCP-DCV | Object First Ace | Cisco Champion | Twitter: @cchilderhose | Blog Site – https://just-virtualization.tech
MichaelSchroeder
Userlevel 1
Badge

I have successfully tested XFS repos in a POC.

One of my customers wants to exchange Windows repos with Linux.

Immutability flag is the most convincing argument.

vExpert | Trainer (VCI) | VCIX | VMCE | https://www.elasticsky.de
Chris.Childerhose
Userlevel 7
Badge +20

I have installed a SOBR with three extents of XFS in our largest VCC datacenter.  It works extremely well and have not had issues.  We did mix it with ReFS but that was a bad idea and created a new SOBR just for XFS alone and no mixing.

We also try to keep SOBRs at 2-3 extents maximum and then create new ones as needed.

Thanks Chris!

What was the problem(s) when mixing XFS with ReFS in a SOBR?

Just that XFS and ReFS did not play well together so it is best to have them separate.  Been rock solid since.

Chris Childerhose - VMCA2022 | VMCE2023 | Veeam Vanguard 7* | Veeam Legend 3* | vExpert 5* | VCAP-DCV/VCP-DCV | Object First Ace | Cisco Champion | Twitter: @cchilderhose | Blog Site – https://just-virtualization.tech
LostInEther
Userlevel 3

I have two XFS based repos each with a ~445TB extent. Excellent performance and stability. This is my preferred filesystem.

dmartin
Chris.Childerhose
Userlevel 7
Badge +20

Hi @StefanZi - this issues we saw was when a job ran if it was on XFS to start and then for some reason decided to write over to ReFS it seemed to cause major issues on the VCC server and repos.  I cannot explain it fully but when we separated the XFS from ReFS everything seems to work great at that point.

I know the documentation says you can mix them but from the people I spoke with at Veeam it was recommended not to.  So that is the approach we have been taking lately.

XFS has been great and looking forward to the Immutable storage now on it as well as the Linux Proxies. :sunglasses:

Chris Childerhose - VMCA2022 | VMCE2023 | Veeam Vanguard 7* | Veeam Legend 3* | vExpert 5* | VCAP-DCV/VCP-DCV | Object First Ace | Cisco Champion | Twitter: @cchilderhose | Blog Site – https://just-virtualization.tech
vNote42
Userlevel 7
Badge +13

Would like to warm up an old topic here again 😁

Some time ago a had to delete some huge backup-file on different ReFS volumes. During the deletion the Windows server hung completely. Furthermore it needed quite some time to delete the file. Today I did some similar on a RHEL on a XFS volume. To keep it short: No server hang, duration was - according to my feeling - like the delete of un-pointed files.

Great work, XFS!

 

Wolfgang | vnote42.net | @vNote42
vNote42
Userlevel 7
Badge +13

Good to know @vNote42 , thanks! How was fast the “storage space reclamation”? was it instant if you are doing a “df -h”?

yes, “df -h”  showed reclaimed space directly afterwards

Wolfgang | vnote42.net | @vNote42
vNote42
Userlevel 7
Badge +13

actually no one has any experiences with XFS as Veeam repository to share?

Wolfgang | vnote42.net | @vNote42
vNote42
Userlevel 7
Badge +13

I have installed a SOBR with three extents of XFS in our largest VCC datacenter.  It works extremely well and have not had issues.  We did mix it with ReFS but that was a bad idea and created a new SOBR just for XFS alone and no mixing.

We also try to keep SOBRs at 2-3 extents maximum and then create new ones as needed.

Thanks Chris!

What was the problem(s) when mixing XFS with ReFS in a SOBR?

Wolfgang | vnote42.net | @vNote42
vNote42
Userlevel 7
Badge +13

We’ve had ~30 Ubuntu 20.04 repo’s in production since v10 launch in 2020, all with reflink enabled; the majority being at client sites and then a few ~500TB extents in sobr’s at our data centers; they all have been rock solid, which is to be expected from linux.

  • stability - no issues at all
  • performance (over time) - hasn’t changed in a year; only limited by the disk iops available from your hardware
  • needs for troubleshooting - no troubleshooting over the past year
  • administrative effort - linux experience or being a fast learner is important for the initial design and deployment; hardware monitoring/testing to determine the number of concurrent jobs that your hardware can handle requires some effort in the beginning but that applies to any OS; after that, cron jobs take care of updates and we schedule a manual reboot of the repo’s whenever an update notifies us that it requires it

Thank you @gtelnet for your very detailed answer! I am very relieved, feedback about XFS was that positive. So v11 with hardened repos can come! 

Wolfgang | vnote42.net | @vNote42
jdw
Userlevel 4
Badge

I’ve done both XFS an EXT4 with Hardened Linux Repo, both worked fine. If you’re using a physical repo server with JBOD storage I would probably lean toward XFS so you can get the block clone features, especially since you have to use Forward Incremental with periodic fulls with HLR. I was using SAN storage with really good built in native dedup, so in that case I decided to stick with EXT4 since I didn’t really need the block clone benefits.

VMCA | vExpert | Twitter @jdwallace | Blog https://jdwallace.com
Geoff Burke
Userlevel 7
Badge +22

I have installed a SOBR with three extents of XFS in our largest VCC datacenter.  It works extremely well and have not had issues.  We did mix it with ReFS but that was a bad idea and created a new SOBR just for XFS alone and no mixing.

We also try to keep SOBRs at 2-3 extents maximum and then create new ones as needed.

Thanks Chris!

What was the problem(s) when mixing XFS with ReFS in a SOBR?

One thing to note is the block size for ReFS is recommended on Veeam to be 64K and XFS is 4K, so I believe that alone would be a problem if trying to mix in one SOBR.

VMCA2022, VMCE2023, CKA, CKAD, KCNA, LFCS, PCA,TARS
HenkRuis
Userlevel 2

Just for my interpretation, if you want XFS and have an REFS environment, you have to got a greenfield for that right?. How do existing customers wit SOBRS handle this (or what is VEEAM point of view in this. Especially in a window only environment>

StefanZi
Userlevel 6
Badge +3

I have installed a SOBR with three extents of XFS in our largest VCC datacenter.  It works extremely well and have not had issues.  We did mix it with ReFS but that was a bad idea and created a new SOBR just for XFS alone and no mixing.

We also try to keep SOBRs at 2-3 extents maximum and then create new ones as needed.

Thanks Chris!

What was the problem(s) when mixing XFS with ReFS in a SOBR?

One thing to note is the block size for ReFS is recommended on Veeam to be 64K and XFS is 4K, so I believe that alone would be a problem if trying to mix in one SOBR.

Continuing this - I’d really also like to know which issues you saw when with mixing XFS and ReFS in one SOBR?! @Chris.Childerhose 
I can’t think of any tbh.
@vNote42 The block cloning block size is completely irrelevant in SOBRs - it’s just something how the file is stored on the OS. You can also mix a dedup appliance and an ReFS repo and that works fine.
When moving files out of one extent to another you’ll loose the block cloning anyways.

So @HenkRuis: As said I don’t see a reason for not mixing XFS and ReFS repos, so I’d add XFS extents to your SOBR and seal the ReFS extents, so that the backups age out and you do not loose your ReFS block cloning space savings by evacuating the extent.
You’ll have to create a new active full to start with block cloning on XFS, but that’s it.

BertrandFR
Userlevel 7
Badge +8

Good to know @vNote42 , thanks! How was fast the “storage space reclamation”? was it instant if you are doing a “df -h”?

BertrandFR
Userlevel 7
Badge +8

Good to know @vNote42 , thanks! How was fast the “storage space reclamation”? was it instant if you are doing a “df -h”?

yes, “df -h”  showed reclaimed space directly afterwards

Amazing! It was such a pain with dedup appliance, zzz waiting recycle jobs to have the space available.

Chris.Childerhose
Userlevel 7
Badge +20

Would like to warm up an old topic here again 😁

Some time ago a had to delete some huge backup-file on different ReFS volumes. During the deletion the Windows server hung completely. Furthermore it needed quite some time to delete the file. Today I did some similar on a RHEL on a XFS volume. To keep it short: No server hang, duration was - according to my feeling - like the delete of un-pointed files.

Great work, XFS!

 

Yeah deleting on XFS even with block cloning seems way faster than wonderful Windows. 😋😂

Chris Childerhose - VMCA2022 | VMCE2023 | Veeam Vanguard 7* | Veeam Legend 3* | vExpert 5* | VCAP-DCV/VCP-DCV | Object First Ace | Cisco Champion | Twitter: @cchilderhose | Blog Site – https://just-virtualization.tech

Comment


Terms & Conditions