ReFS vs NTFS (or other)

I’m sitting on over PB of ReFS and have no issues, the benefits outweigh everything when it comes to Veeam.  You just can’t slap ReFS everywhere in the environment and hope it will be a benefit though.

I may pull the trigger on XFS and hardened repos next upgrade, or stick to my windows infrastructure. I need time to test XFS before I go that route though. 

We are definitely looking at XFS more now especially with the ISO for building the hardened repos.  That reminds me time to start testing and documenting things with this as we want to use it for our appliances, etc. 😎

I think I've been lucky as been using ReFS for the last few years without any major dramas although we only have approx 500TB stored across our repo’s

Just for the space savings and the time for your synthetic it would be better to switch. Win2022 ReFS has been great for me and no further issues like previous releases.

The best way to do it is a new Win2022 box with ReFS and migrate to it.  😋🤣

For us it was always a Windows Update that broke it.  We eventually excluded the servers with repos that were ReFS until we could migrate them to new ones.

We had disasters with ReFS, where entire volume became RAW. It was on Windows Server 2016 and 2019. However, ReFS-based storage wasn’t used as a Veeam repo. We haven’t faced such issues with Veeam repos, but we migrated to NTFS after disasters. I’ve heard that ReFS is more stable on 2022.

We now have XFS on our hardened repos and it works great.

You raise a good point.  The issue is that it’s a rabbit hole for me to go down...I end up killing day or two when I have other things that are a higher priority.  ADHD hyperfocus here plays a pretty pivotal role.  But!  There is a time where taking the time to out of the day to learn or script or whatever becomes a time-saver in the long run, so finding where that exists is important. 

It’s like scripting and automating - sure, it may take 4 hours for me to script something, and if that something takes me a hour or two each month, then it’s probably worth it in time saved, but for less visible things like the time it takes to circle back around and get refocused (my ADHD plays a role here too, but this is true for everyone).  But if it saves me 5 minutes once a month, is it worth it?  For repetitive tasks, absolutely, but for the occasional items?  Maybe not.

So is going down the rabbit hole of Linux worth it.  Sure is.  Same is to be said about the Azure training an certifications that are on my list of things to do as well!  Oh how I wish I had a time machine, or a cloning machine where I could create two or three of myself and then we can reconverge our gained knowledge back into the main person.  Wait….did I just refer to myself in both the first and third person at the same time?

Thanks @vmJoe @coolsport00.  I took some basic Linux courses about 20 years ago in college an have been using it on and off since then.  It was something I wanted to dig into for a long time, but as a Windows administrator, I never had much opportunity. It continues to become more and more prevalent so someday I’ll have to deep dive.  Until then, things like vCenter, ESXI and the LHR’s will be the places I continue to get the most exposure.  I’ll look into those courses as well - I don’t think I have a Pluralsight membership anymore, but I’ll have to check into it for sure.

@dloseke - All great points! I think you’ll enjoy Linux after you use it!  Also, there are plenty of Linux learning resources available to help you gain some familiarity with the OS!

Hey @vmJoe - honestly, it was the first thing that came to mind tbh. And the previous issues then crossed my mind as well. I do have a hardened repo already. Obviously I'm most comfortable with Windows, but more comfortable with Linux the past few months, so I'll probably indeed just go with Linux XFS. Thanks bud. 

@MicoolPaul hit the nail on the head here, but I’ll give my feelings as well. 

ReFS and I have a love-hate relationship.  I really liked it and then a peer noted that he had an issue in which we lost over 100TB of data and forced me really think critically about when and how I was using ReFS.  Gostev’s talks about ReFS and using storage that utilizes software RAID (such as Synology and QNAP NAS’s) made me think more critically as well.  My general rule is that ReFS is okay on “normal” sized volumes as long as you have hardware RAID with a batter backed cache.  I used to, and still have out in the wild ReFS running on iSCSI RDM’s backed by Synology or QNAP NAS’s, but I would not recommend doing this going forward and actively avoid that solution now - that is now relegated to NTFS when I have to use that hardware.  My preference is a server with local storage, or even DAS/SAN when economically possible.  However, there’s a lot of object-storage appliances on the market that are making me re-evaluate yet again.  If I have to use a NAS, it’s going to be relegated to NFS/SMB traffic which isn’t great, but going to be more reliable.  I liked the idea of ReFS on a RDM because synthetic operation will run better there than with NTFS, but I just don’t trust it very much used in this fashion.

Now, I’ll tell you that ReFS on a physical server with local storage is not foolproof.  I have a client who was having issues with his PERC on a server that had a 70ish TB RAID 10 volume.  In order to boot the machine at one point, I had to purge the data from the controller cache, and that corrupted the ReFS volume.  So I now have first-hand experience with this sort of issue that my peer had as well.  In this case, it was camera footage as this server was running a VMS system for several camera’s around a small town.  Wasn’t the end of the world to lose that data, but would have been nice to not have lost it as well.  When I formatted the volume to start over, I did use NTFS because it’s rock solid and we weren’t really taking advantage of ReFS, and in my case, it didn’t prove to be very resilient. 

As Michael noted, it’s as good as MS will let it be.  Older versions of ReFS were not as good as current versions by a long shot.  It should also be noted that if you have a ReFS volume on Server 2016 (and older versions I believe), and you migrate that volume to 2019 or 2022 server, there’s an upgrade that happens on mount - and depending on the backing storage and the size of that volume, that upgrade can take a long time, and doesn’t really give you a status as to what it’s doing.  And as I understand it, it is not backward compatible so you can’t move back to the old server if things don’t work out. 

And as Michael noted, I had several issues with Windows updates causing ReFS volumes to show up as RAW - some of that was resolved by adding a VMware advanced setting to the VM because in my case, Windows was seeing the RDM disk hosting the ReFS volume as removable storage, so the option had to be added to disable that from showing as removable.  But there were several updates that led up to that conclusion that could be uninstalled and the volume would show normally.

In the end, I’ll still use ReFS, but again, only on physical RAID controllers with a batter backed cache.  But my choice pick, and I say this as pretty heavy Windows user, and fairly light in the linux realm, will be XFS as that has proven time and time again to be solid in the industry.  Both have their place, but if you go into it knowing what the caveats are with ReFS, it can certainly be a useful tool given some of the advantages it has.  If it had native immutability, it would be even more competition to XFS, but for now, XFS wins out when applicable to the solution needed.

@Chris.Childerhose - I have phys boxes running 2019 currently. Interestingly, I have to replace those boxes too because they’re old!..like, 8yrs or so old (Proliant DL380p 😳 😂 ). Thankfully though, they’ve been running stable since I implemented them 4-5yrs ago or so. So, I’ll probably for sure look to get my server replacements (refurb’d) on 2022 & use ReFS

@Geoff Burke - yeah, I hadn’t really seen any issues recently either. Gostev used to share often in his weekly emails issues experienced with ReFS. Issue reporting died down even before he took a break from writing those, which is encouraging. I actually may just go ahead and bite a real big bullet and just quit using Windows for my repos and instead use Linux and XFS/immutable. Now that I’m more comfortable with Linux, and have a Hardened Repo already set up, I mean, why not? Only additional config I’ll need to work through is getting iSCSI setup on them. YOLO though, right?? 😂

Thanks for the input gents!