ReFS issues with latest Windows Server Updates (KB5009624, KB5009557, KB5009555)

Never heard of this, let’s see what community have to say about it.

Hello all, just wanted to throw my 2 cents in. Please delete or otherwise if not ok.

This is a thread i’ve seen as having an active conversation about these issues.

I don’t use Veeam, but MABS v3.

My server is a Dell R720 with one RAID array, containing 3 volumes; OS (NTFS), Recovery (NTFS), Data (ReFS). Windows Server 2019 Standard.

Up until the 20th Jan this server was operating fine (at least as far as I could tell, I was using it).

It installed KB5009557 automatically on 21st @ 4am, and since then, it started BSOD during boot, with:

SYSTEM THREAD EXCEPTION NOT HANDLED

ReFS.SYS

Checking the MEMORY.DMP file, tells this error to have code 0x0000007e.

What was stranger was I couldn’t boot from a Server 2019 ISO, as this also BSOD.

Used a Server 2016 ISO to boot, and rename refs.sys to refs.sys.bak, which allowed me to boot windows, without ReFS support, so the volume showed as RAW.

Created a support case with Microsoft, more on that later.

Found this article, and discovered the OOB update KB5010791.

Tricky to install it, as I needed refs.sys in place to be successful, but had to rename it back to boot windows. It did update the file though; 10.0.17763.2458.

But still BSOD.

Used refsutil.exe to assess the volume. Looked all fine:

PS C:\windows\system32> refsutil salvage -QS F: C:\refsutil
Microsoft ReFS Salvage [Version 10.0.11070]
Copyright (c) 2015 Microsoft Corp.

Local time: 1/24/2022 9:53:14

ReFS version: 3.4
Boot sector checked.
Cluster Size: 4096 (0x1000).
Cluster Count: 3265642496 (0xc2a5c000).
Superblocks checked.
Checkpoints checked.
4558 container table entry pages processed (0 invalid page(s))
1 container index table entry pages processed (0 invalid page(s)).
Container Table checked.

Processing 1403 of 1404 object table pages (99%)...

Object Table checked.

Examining identified metadata disk data for versioning and consistency.
134777 disk clusters analyzed (100%)...

Examining volume with signature 45246377 for salvageable files.
4558 container table entry pages processed (0 invalid page(s)).
1 container index table entry pages processed (0 invalid page(s)).
Validating discovered table roots on volume with signature 45246377.

36926 table roots validated (100%).
Enumerating files from discovered tables on volume with signature 45246377.

36926 tables enumerated (100%).
Command Complete.

Run time = 611 seconds.

Tried uninstalling both KBs.

I also tried manually copying a previous version of refs.sys. 10.0.17763.2452, 10.0.17763.2330. Still BSOD.

After waiting for Microsoft support to give me something useful, for a week since creating the case, finally took things more into my own hands.

Used refsutil to copy anything that might be useful to keep, deleted the ReFS volume, reinstated refs.sys, rebooted, and the BSOD went away.

Updated fully with both KB5009557 + KB5010791. Still no BSOD.

Recreated the ReFS volume, configured MABS to use it again. Still no BSOD.

Microsoft support have been about as helpful as a wet towel. Giving me basic instructions that I’ve already tried myself (sfc, dism, uninstall kb). They did get my MEMORY.DMP and claimed to be analysing it since, but so far have only given me the stop code 0x7e, which I’ve also got myself. Slow to respond to emails, and even ‘strongly suggest’ I contact Microsoft Premier Support Services. No, not spending more money on MS. Tried to get them to confirm if ReFS and Fixed Disks were a known issue they were working on; no word on that.

So this leaves me wondering.

Did something happen to my volume, a corruption maybe, that then broke refs.sys?

Or did their update, break something in my volume, which then broke refs.sys?

I was using this volume for other things (a window share), so I’ve now split those workloads out to separate NTFS volumes, but surely that can’t be it.

I read that ReFS when they get up in space used it can fall over. Is that true, or still true?

When you copy the data from volume to volume you will loose all block cloning savings….

My understanding is that only older v3.x versions update to 3.y   eg 3.1 to 3.4   or even to 3.7 now
New volumes get the relevant 3.y version
Older disks that may have been set up when originally attached to WS2012R2 will remain v1.2  and can still be used in the later OS (the OS has a separate refsv1 driver to handle them).
But some customers say they are not using old/existing disks and yet they are v1.2 somehow, and not sure how that can be.


 

The 1.2 is default version if formatted by Windows 8.1, Windows 10 up to v1607, Windows Server 2012 R2 and only if specified ReFSv1 on Windows Server 2016. I think it was the case.

In theory once a ReFS volume is mounted on a device that supports a newer ReFS version, it will be automatically upgraded to last possibly version. 

WARNING: after update, you can’t go back.

It should be at least 3.4 accordingly to

https://gist.github.com/0xbadfca11/da0598e47dd643d933dc

This setting indeed solves the issue for us on a Windows Server 2019 machine. Now the ReFS volume is accessible again after installing the updates. However, other VMs which still have accessible ReFS volumes after installing the patches don't have this configuration setting and they also have the eject option on the taskbar for the virtual disk. So, although the devices.hotplug setting fixes the issue, it's not the whole story.

Thanks @stephc_msft ! Questionable though why MS Support isn't aware of this setting.

Tonight Microsoft start releasing “fix for everything”:

https://www.catalog.update.microsoft.com/Search.aspx?q=%092022-01%20Cumulative%20Update%20Preview

Nice share of infos guys, this is the power of this community.

Yep

We are running vSphere 7 and have the issue on two VMs. This drive is shown as fixed. The VM runs Windows server 2022.

Agreed, this engineer is definitely overstepping by presenting their own personal views in such a way. Likely out of frustration as Microsoft will have been the ones stating to just uninstall the patch.

When Microsoft promised us singular “cumulative” updates to improve their QA, this is a trade off we now are experiencing, that you get all or nothing with their patches, production breaking updates vs exposure to zero day exploits is now the choice to make.

The ReFS comment makes me feel uneasy also, but this is why we need to plan for file system issues in our 3-2-1-1-0, as this patch, whilst bad, has the ability to be reversed. What happens next time when the patch corrupts the partitions beyond repair?

We need to treat this as an important lesson, and be glad that Veeam have been helping us break free of Microsoft’s chains with the use of Linux operating systems.

If I’m allowed to dream, I’d love to see XFS on What nodes, giving Microsoft some actual competition.

“In any case if you use NTFS I personally recommend using NTFS instead of ReFS, as ReFS is still immature as a file system as has several bugs.”

Wow.

So @Franc if you uninstall that update you can access to data?