ReFS vs NTFS (or other)

I have implemented ReFS on Win2022 as it now seems like it is in a stable state compared to previous versions although there have been updates to Win2019 as well to address the issues that had.  I have not seen restores take any longer from ReFS but that is all dependent on other factors and the size of data you are restoring.

Nice to see new Nimbles as I would love to play in that realm again as they were amazing.  Best of luck but if these are new and you are worried just deploy Win2022 with an ReFS drive and run it through its paces before moving to PROD.  QA testing. 😎

Hi @coolsport00,

I’ve ridden the ReFS wave since the beginning with Veeam and my views are, it’s as good as Microsoft lets it be 🤷‍♂️

If you do ReFS, then Windows Server 2019 needs to be your minimum as 2016 was a mess. ReFS versions are tied to the OS and not backwards compatible once mounted Read & Write. Microsoft touted some operations were up to 50x faster on Server 2022 vs 2019, but as you’re only leveraging ReFS features at specific points (fast clone and reading from referenced blocks on restores for example) you won’t be seeing a dramatic difference. There’s arguments of a few % performance or efficiency here and there between ReFS and XFS, but they’re both pretty damn good at their jobs.

I say it’s as good as Microsoft let it be because one Windows update made all ReFS volumes appear as RAW, uninstalling the update made this read normally again, but I’m sure at least one person would’ve had a panic attack or worse seeing that!

With regards to performance, I’ll be honest, I haven’t benchmarked restores, but they’ve felt just as fast to me, but it does change the IO pattern as IO becomes more “random” during a restore, depending on if your Nimble is all flash or Hybrid will impact whether that’ll hurt your recovery times.

A big consideration though is that ReFS is RAM hungry. IIRC the guidance is 0.5GB of RAM per 1TB of ReFS file system. This is better in 2019/2022 vs 2016 and the BP guidance does state no need to go above 256GB RAM but you’ll certainly need some RAM to feed these things.

If any of these things put you off there’s no reason why you couldn’t just have a non-immutable XFS repository if not 🙂 if you feel the architecture won’t support immutability in the best possible ways.

Hope that helped!

@Chris.Childerhose - I have phys boxes running 2019 currently. Interestingly, I have to replace those boxes too because they’re old!..like, 8yrs or so old (Proliant DL380p 😳 😂 ). Thankfully though, they’ve been running stable since I implemented them 4-5yrs ago or so. So, I’ll probably for sure look to get my server replacements (refurb’d) on 2022 & use ReFS

@Geoff Burke - yeah, I hadn’t really seen any issues recently either. Gostev used to share often in his weekly emails issues experienced with ReFS. Issue reporting died down even before he took a break from writing those, which is encouraging. I actually may just go ahead and bite a real big bullet and just quit using Windows for my repos and instead use Linux and XFS/immutable. Now that I’m more comfortable with Linux, and have a Hardened Repo already set up, I mean, why not? Only additional config I’ll need to work through is getting iSCSI setup on them. YOLO though, right?? 😂

Thanks for the input gents!

@MicoolPaul hit the nail on the head here, but I’ll give my feelings as well. 

ReFS and I have a love-hate relationship.  I really liked it and then a peer noted that he had an issue in which we lost over 100TB of data and forced me really think critically about when and how I was using ReFS.  Gostev’s talks about ReFS and using storage that utilizes software RAID (such as Synology and QNAP NAS’s) made me think more critically as well.  My general rule is that ReFS is okay on “normal” sized volumes as long as you have hardware RAID with a batter backed cache.  I used to, and still have out in the wild ReFS running on iSCSI RDM’s backed by Synology or QNAP NAS’s, but I would not recommend doing this going forward and actively avoid that solution now - that is now relegated to NTFS when I have to use that hardware.  My preference is a server with local storage, or even DAS/SAN when economically possible.  However, there’s a lot of object-storage appliances on the market that are making me re-evaluate yet again.  If I have to use a NAS, it’s going to be relegated to NFS/SMB traffic which isn’t great, but going to be more reliable.  I liked the idea of ReFS on a RDM because synthetic operation will run better there than with NTFS, but I just don’t trust it very much used in this fashion.

Now, I’ll tell you that ReFS on a physical server with local storage is not foolproof.  I have a client who was having issues with his PERC on a server that had a 70ish TB RAID 10 volume.  In order to boot the machine at one point, I had to purge the data from the controller cache, and that corrupted the ReFS volume.  So I now have first-hand experience with this sort of issue that my peer had as well.  In this case, it was camera footage as this server was running a VMS system for several camera’s around a small town.  Wasn’t the end of the world to lose that data, but would have been nice to not have lost it as well.  When I formatted the volume to start over, I did use NTFS because it’s rock solid and we weren’t really taking advantage of ReFS, and in my case, it didn’t prove to be very resilient. 

As Michael noted, it’s as good as MS will let it be.  Older versions of ReFS were not as good as current versions by a long shot.  It should also be noted that if you have a ReFS volume on Server 2016 (and older versions I believe), and you migrate that volume to 2019 or 2022 server, there’s an upgrade that happens on mount - and depending on the backing storage and the size of that volume, that upgrade can take a long time, and doesn’t really give you a status as to what it’s doing.  And as I understand it, it is not backward compatible so you can’t move back to the old server if things don’t work out. 

And as Michael noted, I had several issues with Windows updates causing ReFS volumes to show up as RAW - some of that was resolved by adding a VMware advanced setting to the VM because in my case, Windows was seeing the RDM disk hosting the ReFS volume as removable storage, so the option had to be added to disable that from showing as removable.  But there were several updates that led up to that conclusion that could be uninstalled and the volume would show normally.

In the end, I’ll still use ReFS, but again, only on physical RAID controllers with a batter backed cache.  But my choice pick, and I say this as pretty heavy Windows user, and fairly light in the linux realm, will be XFS as that has proven time and time again to be solid in the industry.  Both have their place, but if you go into it knowing what the caveats are with ReFS, it can certainly be a useful tool given some of the advantages it has.  If it had native immutability, it would be even more competition to XFS, but for now, XFS wins out when applicable to the solution needed.

Hey @vmJoe - honestly, it was the first thing that came to mind tbh. And the previous issues then crossed my mind as well. I do have a hardened repo already. Obviously I'm most comfortable with Windows, but more comfortable with Linux the past few months, so I'll probably indeed just go with Linux XFS. Thanks bud. 

@dloseke - All great points! I think you’ll enjoy Linux after you use it!  Also, there are plenty of Linux learning resources available to help you gain some familiarity with the OS!

Thanks @vmJoe @coolsport00.  I took some basic Linux courses about 20 years ago in college an have been using it on and off since then.  It was something I wanted to dig into for a long time, but as a Windows administrator, I never had much opportunity. It continues to become more and more prevalent so someday I’ll have to deep dive.  Until then, things like vCenter, ESXI and the LHR’s will be the places I continue to get the most exposure.  I’ll look into those courses as well - I don’t think I have a Pluralsight membership anymore, but I’ll have to check into it for sure.

You raise a good point.  The issue is that it’s a rabbit hole for me to go down...I end up killing day or two when I have other things that are a higher priority.  ADHD hyperfocus here plays a pretty pivotal role.  But!  There is a time where taking the time to out of the day to learn or script or whatever becomes a time-saver in the long run, so finding where that exists is important. 

It’s like scripting and automating - sure, it may take 4 hours for me to script something, and if that something takes me a hour or two each month, then it’s probably worth it in time saved, but for less visible things like the time it takes to circle back around and get refocused (my ADHD plays a role here too, but this is true for everyone).  But if it saves me 5 minutes once a month, is it worth it?  For repetitive tasks, absolutely, but for the occasional items?  Maybe not.

So is going down the rabbit hole of Linux worth it.  Sure is.  Same is to be said about the Azure training an certifications that are on my list of things to do as well!  Oh how I wish I had a time machine, or a cloning machine where I could create two or three of myself and then we can reconverge our gained knowledge back into the main person.  Wait….did I just refer to myself in both the first and third person at the same time?

We had disasters with ReFS, where entire volume became RAW. It was on Windows Server 2016 and 2019. However, ReFS-based storage wasn’t used as a Veeam repo. We haven’t faced such issues with Veeam repos, but we migrated to NTFS after disasters. I’ve heard that ReFS is more stable on 2022.

We now have XFS on our hardened repos and it works great.

For us it was always a Windows Update that broke it.  We eventually excluded the servers with repos that were ReFS until we could migrate them to new ones.

The best way to do it is a new Win2022 box with ReFS and migrate to it.  😋🤣

Just for the space savings and the time for your synthetic it would be better to switch. Win2022 ReFS has been great for me and no further issues like previous releases.

I think I've been lucky as been using ReFS for the last few years without any major dramas although we only have approx 500TB stored across our repo’s