ReFS issues with latest Windows Server Updates (KB5009624, KB5009557, KB5009555)


Userlevel 7
Badge +12
  • On the path to Greatness
  • 1273 comments

Just a quick post; the latest Windows Server updates for 2012R2, 2019 and 2022 (haven’t seen 2016) can cause ReFS issues. After the installation, ReFS volumes are shown as RAW and are no longer accessible; so if you’re using ReFS for your repositories, then take be aware of this when installating the update. Besides that, those updates can also cause bootloops for domain controllers and break Hyper-V.

Depending on your Windows Server version one of the following updates could have caused the issue: KB5009624, KB5009557, KB5009555

If you’re affected then removing the mentioned update should solve the issue. Don’t (!) try to repair the ReFS volume, because this could cause a dataloss.

This should remind everyone why 3-2-1 for Backups is so important. :wink:

Further information:

https://www.bleepingcomputer.com/news/microsoft/new-windows-server-updates-cause-dc-boot-loops-break-hyper-v/

https://forums.veeam.com/veeam-backup-replication-f2/beware-possible-raw-refs-volumes-after-installing-january-updates-t78634.html

Update #1:

Microsoft has pulled the updates, thanks @Mildur 

https://www.bleepingcomputer.com/news/microsoft/microsoft-pulls-new-windows-server-updates-due-to-critical-bugs/amp/

Update #2:

Microsoft has released the updates again and it looks like they didn't fix them. (Thanks @MicoolPaul )

https://www.bleepingcomputer.com/news/microsoft/microsoft-resumes-rollout-of-january-windows-server-updates/

Update #3:

Microsoft has released an out-of-band update, which can possibly resolve the issues:

https://www.bleepingcomputer.com/news/microsoft/microsoft-releases-emergency-fixes-for-windows-server-vpn-bugs/

 


106 comments

Userlevel 7
Badge +17

Oh.my.god…. 🤯🤯🤯

Userlevel 7
Badge +12

All they’ve done is add the problems as known issues… oh Microsoft…

What else should they do...fix the update?🤣

That’s crazy talk! What next, QA testing?

You say it. We get the updates for free, so the least we can do is giving something back to Microsoft; fixing bugs, QA, …

 

Out of curiosity, is anyone getting failed to create snapshot on Hyper-V after the Jan 11 patch?

Boot, login and VMs are fine, but I can’t get any backups to work post-update.

Userlevel 7
Badge +13

All they’ve done is add the problems as known issues… oh Microsoft…

the bottom in quality management does not seem to have been reached yet:rage:

Just had the ReFS issue on Server 2016 with KB5009546.  After install, reboot, target USB repository was RAW.  Uninstalled update and drive is visible again with all data.

 

UGH.  This has been a known issue for a week, why hasn’t MS pulled it?

Userlevel 2

Unfortunately just finding out about this. Entire ReFS volume appears to be shot even after uninstalling the update. Anyone else run into uninstalling not resolving the issue?

 

***Update - after uninstalling BOTH KB5009624 and KB5009595 our ReFS volumes are back. Running Server 2012 R2 Datacenter

Userlevel 7
Badge +12

Thanks for the feedback @Asmith; great that you were able to solve it.

@Bommer I haven't heard of this one so far. You could try to uninstall the updates to see if they were the cause.

Userlevel 7
Badge +12

Microsoft has released an out-of-band which should resolve all the problems: https://support.microsoft.com/en-au/topic/january-17-2022-non-security-update-kb5010796-out-of-band-e79a633f-e876-4268-a21e-de6a9ca52da7

According to @Franc it didn't solve his ReFS problem, so please still be careful.

Userlevel 7
Badge +13

Guys here’s updated solution from Microsoft for Windows Server 2022:

https://support.microsoft.com/en-us/topic/january-17-2022-kb5010796-os-build-20348-473-out-of-band-2e0408ba-10d4-4c68-9b3d-cc5fb8d6f4a8

@regnor your link’s for Azure Stack HCI, version 21H2

Userlevel 5
Badge

As @regnor stated, this update doesn’t solve the RAW issue on fixed ReFS drives for us. I’ll open a support case with Microsoft for it. Although I don’t get my hopes up too high, since my experience with MS Support lately is not that great to say the least.

Userlevel 7
Badge +12

@marcofabbri: Thanks! I just copied the link from the R&D forums, but the one from bleepingcomputer.com does include all possible updates.

@Franc I wish you good luck with that. Finding/Solving bugs with Microsoft support isn’t fun at all.

Userlevel 5
Badge

@regnor , well that was quick with MS support. They confirmed it's also an issue with fixed drives, but the patch from yesterday was only for external drives. They are still working on a fix for fixed drives and he will inform me once the patch is available. He couldn't explain though why Microsoft doesn't mention the issue for fixed drives anywhere, but only for external drives. He confirmed he already had multiple cases where other customers also experience the issue with fixed drives.

Userlevel 7
Badge +7

Just a quick post; the latest Windows Server updates for 2012R2, 2019 and 2022 (haven’t seen 2016) can cause ReFS issues. After the installation, ReFS volumes are shown as RAW and are no longer accessible; so if you’re using ReFS for your repositories, then take be aware of this when installating the update. Besides that, those updates can also cause bootloops for domain controllers and break Hyper-V.

Depending on your Windows Server version one of the following updates could have caused the issue: KB5009624, KB5009557, KB5009555

If you’re affected then removing the mentioned update should solve the issue. Don’t (!) try to repair the ReFS volume, because this could cause a dataloss.

This should remind everyone why 3-2-1 for Backups is so important. :wink:

Further information:

https://www.bleepingcomputer.com/news/microsoft/new-windows-server-updates-cause-dc-boot-loops-break-hyper-v/

https://forums.veeam.com/veeam-backup-replication-f2/beware-possible-raw-refs-volumes-after-installing-january-updates-t78634.html

Update #1:

Microsoft has pulled the updates, thanks @Mildur 

https://www.bleepingcomputer.com/news/microsoft/microsoft-pulls-new-windows-server-updates-due-to-critical-bugs/amp/

Update #2:

Microsoft has released the updates again and it looks like they didn't fix them. (Thanks @MicoolPaul )

https://www.bleepingcomputer.com/news/microsoft/microsoft-resumes-rollout-of-january-windows-server-updates/

Update #3:

Microsoft has released an out-of-band update, which can possibly resolve the issues:

https://www.bleepingcomputer.com/news/microsoft/microsoft-releases-emergency-fixes-for-windows-server-vpn-bugs/

 

Thanks for your update again.

Userlevel 7
Badge +12

Now that was quick @Franc! Lets hope they’ll also be able to solve the fixed disk problem soon.

Userlevel 7
Badge +13

@regnor , well that was quick with MS support. They confirmed it's also an issue with fixed drives, but the patch from yesterday was only for external drives. They are still working on a fix for fixed drives and he will inform me once the patch is available. He couldn't explain though why Microsoft doesn't mention the issue for fixed drives anywhere, but only for external drives. He confirmed he already had multiple cases where other customers also experience the issue with fixed drives.

No good Microsoft, no good...

Userlevel 2

FYI the out of bound update, KB5010794, is still breaking ReFS for 2012r2.

Userlevel 7
Badge +20

FYI the out of bound update, KB5010794, is still breaking ReFS for 2012r2.

Thanks for the update as we still are moving some Veeam 2012R2 repo servers to newer OS.  Will ensure this patch is not applied.

Userlevel 7
Badge +13

FYI the out of bound update, KB5010794, is still breaking ReFS for 2012r2.

That’s strange, multiple blogs online and relative comments declare that fix ReFS on 2012 r2 and storage can be mounted again:
 

https://www.borncity.com/blog/2022/01/18/sonderupdates-fr-windows-mit-fixes-fr-jan-2020-patchday-probleme-17-1-2022/#comment-120775

Userlevel 2

So possibly it’s the combination of installing the past ‘bad’ update and KB5010794 together that is needed? We blocked the original so we had KB5010794 install by itself. From your link:
 

So, I pulled it through today (I informed myself and resisted for a long time).
Thank you to everyone here who helped share information.
The new patches were used: KB5010791 (2019 server), KB5010790 (2016 server) and KB5010794 (server 2012R2)
1x 2012R2 server (DC)
4x 2019 server (DCs)
4x 2016 server (RODCs)
All servers had the latest patch status the Dec. Patchday updates.
First the 4x 2019 servers with KB5010791 (because it's a cumulative update, pulled from the update catalog), then the 2012R2 via Windows Update, here I selected the 'bad' January update from 01/11/22 TOGETHER with the optional KB5010794 and installed both at once.
Finally the 4x 2016s with KB5010790 from the Update Catalog.
Everything is running - lucky :)

Userlevel 7
Badge +13

So possibly it’s the combination of installing the past ‘bad’ update and KB5010794 together that is needed? We blocked the original so we had KB5010794 install by itself. From your link:
 

So, I pulled it through today (I informed myself and resisted for a long time).
Thank you to everyone here who helped share information.
The new patches were used: KB5010791 (2019 server), KB5010790 (2016 server) and KB5010794 (server 2012R2)
1x 2012R2 server (DC)
4x 2019 server (DCs)
4x 2016 server (RODCs)
All servers had the latest patch status the Dec. Patchday updates.
First the 4x 2019 servers with KB5010791 (because it's a cumulative update, pulled from the update catalog), then the 2012R2 via Windows Update, here I selected the 'bad' January update from 01/11/22 TOGETHER with the optional KB5010794 and installed both at once.
Finally the 4x 2016s with KB5010790 from the Update Catalog.
Everything is running - lucky :)

Maybe that’s it :thumbsup_tone1:

Userlevel 7
Badge +12

FYI the out of bound update, KB5010794, is still breaking ReFS for 2012r2.

Probably you really need to install both updates, one after the other. Although it’s also strange that the out of band update also introduces the issue…

By the way, just for myself, why did you decide to go with ReFS on 2012R2?

Userlevel 7
Badge +20

FYI the out of bound update, KB5010794, is still breaking ReFS for 2012r2.

Probably you really need to install both updates, one after the other. Although it’s also strange that the out of band update also introduces the issue…

By the way, just for myself, why did you decide to go with ReFS on 2012R2?

I’m not the person you asked the question to but I thought I’d jump in with my experience. The reason I jumped on the bandwagon early was because Microsoft literally named it “Resilient File System”, they touted automatic detection and repair of corruption as a major reason to go with it, plus the scaling side of ReFS for maximum volume and file size limits seemed better aligned to the constant marketing of data growth explosions.

 

Anything that keeps my backups healthy sounds good to me!

Userlevel 7
Badge +13

FYI the out of bound update, KB5010794, is still breaking ReFS for 2012r2.

Probably you really need to install both updates, one after the other. Although it’s also strange that the out of band update also introduces the issue…

By the way, just for myself, why did you decide to go with ReFS on 2012R2?

I’m not the person you asked the question to but I thought I’d jump in with my experience. The reason I jumped on the bandwagon early was because Microsoft literally named it “Resilient File System”, they touted automatic detection and repair of corruption as a major reason to go with it, plus the scaling side of ReFS for maximum volume and file size limits seemed better aligned to the constant marketing of data growth explosions.

 

Anything that keeps my backups healthy sounds good to me!

Back in 2014 was nearly like early adoption the ReFS file system, but what you say @MicoolPaul is right.

This is a 2019 article, but so actual: https://www.itprotoday.com/storage/it-finally-time-adopt-microsofts-resilient-file-system

Userlevel 2

FYI the out of bound update, KB5010794, is still breaking ReFS for 2012r2.

Probably you really need to install both updates, one after the other. Although it’s also strange that the out of band update also introduces the issue…

By the way, just for myself, why did you decide to go with ReFS on 2012R2?

A year or so ago we were in the process of rebuilding our storage infrastructure and creating a new scale-out repository in prep for Wasabi’s immutable functionality. We figured we would take advantage of the features of ReFS while we were at it.

Userlevel 7
Badge +12

Thanks @MicoolPaul and @marcofabbri ! For me Windows Server 2016 was the first contact with ReFS as Veeam started to integrated it. With 2012R2 I felt that it wasn’t mature enough for production. You’re right that the integrity part is/was an advantage, but without storage spaces you only had the corroption detection.

Comment