ReFS issues with latest Windows Server Updates (KB5009624, KB5009557, KB5009555)

regnor
Userlevel 7
Badge +11
  • regnor
  • Veeam Legend, Veeam Vanguard
  • 1177 comments

Just a quick post; the latest Windows Server updates for 2012R2, 2019 and 2022 (haven’t seen 2016) can cause ReFS issues. After the installation, ReFS volumes are shown as RAW and are no longer accessible; so if you’re using ReFS for your repositories, then take be aware of this when installating the update. Besides that, those updates can also cause bootloops for domain controllers and break Hyper-V.

Depending on your Windows Server version one of the following updates could have caused the issue: KB5009624, KB5009557, KB5009555

If you’re affected then removing the mentioned update should solve the issue. Don’t (!) try to repair the ReFS volume, because this could cause a dataloss.

This should remind everyone why 3-2-1 for Backups is so important. :wink:

Further information:

https://www.bleepingcomputer.com/news/microsoft/new-windows-server-updates-cause-dc-boot-loops-break-hyper-v/

https://forums.veeam.com/veeam-backup-replication-f2/beware-possible-raw-refs-volumes-after-installing-january-updates-t78634.html

Update #1:

Microsoft has pulled the updates, thanks @Mildur 

https://www.bleepingcomputer.com/news/microsoft/microsoft-pulls-new-windows-server-updates-due-to-critical-bugs/amp/

Update #2:

Microsoft has released the updates again and it looks like they didn't fix them. (Thanks @MicoolPaul )

https://www.bleepingcomputer.com/news/microsoft/microsoft-resumes-rollout-of-january-windows-server-updates/

Update #3:

Microsoft has released an out-of-band update, which can possibly resolve the issues:

https://www.bleepingcomputer.com/news/microsoft/microsoft-releases-emergency-fixes-for-windows-server-vpn-bugs/

 

Max M. - Veeam Legend/Vanguard/VMCE/VMCA

106 comments

marcofabbri
Userlevel 7
Badge +11

ReFS issues and DC bootloop and broken Hyper-v

Just talking about this some minutes ago.

More info here: https://www.bleepingcomputer.com/news/microsoft/new-windows-server-updates-cause-dc-boot-loops-break-hyper-v/

 

Backups are like pizza, I love them. | Linkedin: @marco-fabbri-it
Mildur
Userlevel 7
Badge +12

Microsoft has pulled now the updates from the update servers:

https://www.bleepingcomputer.com/news/microsoft/microsoft-pulls-new-windows-server-updates-due-to-critical-bugs/amp/

Product Management Analyst @ Veeam Software
vergoted
Userlevel 4
Badge +1

We decided to decline the updates from our WSUS after seeing the thread on Reddit before MS pulled them. 
https://www.reddit.com/r/sysadmin/comments/s1jcue/patch_tuesday_megathread_20220112/
 

The servers that got patched don’t show any errors.

Dries Vergote | VMCE 2021
Mildur
Userlevel 7
Badge +12

It’s a good start in 2022. To many things go sideways. :)

Product Management Analyst @ Veeam Software
Mildur
Userlevel 7
Badge +12

@Mildur But so far we havn’t experienced any good things since 12/31/2021 from them; except you count everything which is not broken or doesn’t have issues as positive :wink:

 

True. I was talking more general. Not only in 2022, and not only from Microsoft.

Bad things are more discussed than good things :)

 

Product Management Analyst @ Veeam Software
MicoolPaul
Userlevel 7
Badge +20

@Mildur But so far we havn’t experienced any good things since 12/31/2021 from them; except you count everything which is not broken or doesn’t have issues as positive :wink:

 

True. I was talking more general. Not only in 2022, and not only from Microsoft.

Bad things are more discussed than good things :)

 

Nope that’s it, Microsoft took it too far this morning! Decided my headset and webcam were untrustworthy (or maybe they meant the person on them, aka me!) 😆

Michael Paul - #VeeamLegend | #VeeamVanguard | Veeam Certified Architect | VMware vExpert | https://micoolpaul.com | Twitter: @MicoolPaul | Mastodon: @micoolpaul@masto.nu | Bluesky: @micoolpaul.bsky.social
MicoolPaul
Userlevel 7
Badge +20

Microsoft has pulled now the updates from the update servers:

https://www.bleepingcomputer.com/news/microsoft/microsoft-pulls-new-windows-server-updates-due-to-critical-bugs/amp/

Think it’s for the best… 😬

Michael Paul - #VeeamLegend | #VeeamVanguard | Veeam Certified Architect | VMware vExpert | https://micoolpaul.com | Twitter: @MicoolPaul | Mastodon: @micoolpaul@masto.nu | Bluesky: @micoolpaul.bsky.social
F
Userlevel 5
Badge

KB5009555 for Windows 2022 doesn’t seem to have been pulled. My test server still downloads this update from Windows Update. However, this is the faulty update which causes the raw ReFS issue.

MicoolPaul
Userlevel 7
Badge +20

All they’ve done is add the problems as known issues… oh Microsoft…

Michael Paul - #VeeamLegend | #VeeamVanguard | Veeam Certified Architect | VMware vExpert | https://micoolpaul.com | Twitter: @MicoolPaul | Mastodon: @micoolpaul@masto.nu | Bluesky: @micoolpaul.bsky.social
A
Userlevel 2

FYI the out of bound update, KB5010794, is still breaking ReFS for 2012r2.

A
Userlevel 2

Unfortunately just finding out about this. Entire ReFS volume appears to be shot even after uninstalling the update. Anyone else run into uninstalling not resolving the issue?

 

***Update - after uninstalling BOTH KB5009624 and KB5009595 our ReFS volumes are back. Running Server 2012 R2 Datacenter

A
Userlevel 2

So possibly it’s the combination of installing the past ‘bad’ update and KB5010794 together that is needed? We blocked the original so we had KB5010794 install by itself. From your link:
 

So, I pulled it through today (I informed myself and resisted for a long time).
Thank you to everyone here who helped share information.
The new patches were used: KB5010791 (2019 server), KB5010790 (2016 server) and KB5010794 (server 2012R2)
1x 2012R2 server (DC)
4x 2019 server (DCs)
4x 2016 server (RODCs)
All servers had the latest patch status the Dec. Patchday updates.
First the 4x 2019 servers with KB5010791 (because it's a cumulative update, pulled from the update catalog), then the 2012R2 via Windows Update, here I selected the 'bad' January update from 01/11/22 TOGETHER with the optional KB5010794 and installed both at once.
Finally the 4x 2016s with KB5010790 from the Update Catalog.
Everything is running - lucky :)

MicoolPaul
Userlevel 7
Badge +20

Latest update from MS Support: case is being archived with no resolution at the moment. Advice currently is, if the issue occurs after installing the cumulative patch, you have two options:

- uninstall the patch and copy the data to an NTFS volume and re-install the patch

- refrain from installing the patch entirely

The MS Support engineer even made a personal note:

In any case if you use NTFS I personally recommend using NTFS instead of ReFS, as ReFS is still immature as a file system as has several bugs.


I did let them know my frustration about this issue, since they broke it and still don't have a solution or even a cause after several weeks.

This response is… wow :scream: . It makes a very uncomfortable feeling at the moment.

In this case they should drop ReFS completely.

Agreed, this engineer is definitely overstepping by presenting their own personal views in such a way. Likely out of frustration as Microsoft will have been the ones stating to just uninstall the patch.

 

When Microsoft promised us singular “cumulative” updates to improve their QA, this is a trade off we now are experiencing, that you get all or nothing with their patches, production breaking updates vs exposure to zero day exploits is now the choice to make.

 

The ReFS comment makes me feel uneasy also, but this is why we need to plan for file system issues in our 3-2-1-1-0, as this patch, whilst bad, has the ability to be reversed. What happens next time when the patch corrupts the partitions beyond repair?

 

We need to treat this as an important lesson, and be glad that Veeam have been helping us break free of Microsoft’s chains with the use of Linux operating systems.

 

If I’m allowed to dream, I’d love to see XFS on What nodes, giving Microsoft some actual competition.

Michael Paul - #VeeamLegend | #VeeamVanguard | Veeam Certified Architect | VMware vExpert | https://micoolpaul.com | Twitter: @MicoolPaul | Mastodon: @micoolpaul@masto.nu | Bluesky: @micoolpaul.bsky.social
S
Userlevel 2

Can people on here please confirm if they are using Refs on VMWare VM’s   (where the hotplug feature makes the drives appear as removable)
Vmware fix to overcome that is https://kb.vmware.com/s/article/1012225?lang=en_us

Or are some of your system really using removable drives?

For WS2012R2 using refsv1  the OOB fix will not and never will overcome the issue on removable drives
The Microsoft OOB fix is only for refs 3.x on removable drives in ws2016 and later
NOte - some systems running later OS’s may be using old disk formatted with refs v1  and so those refsv1 disks will be affected if they are considered removable

fsutil fsinfo refsinfo x:   will show the refs version
although of course cant run taht if the volume is already showing RAW!

KB5010691: ReFS-formatted removable media may fail to mount or mounts as RAW after installing the January 11, 2022 Windows updates (microsoft.com)
Mentions the OOB fix but unfortunately doesnt link to them
Mentions  applicable to refs v2  (aka 3.x)
Unfortunately doesnt mention the vmware aspect  or the 2012r2/refsv1 aspect

S
Userlevel 2

The only solution for 2012R2 (running on VMWare) is the 
devices.hotplug with a value of false

mentioned in Disabling the HotAdd/HotPlug capability in virtual machines (1012225) (vmware.com)

 

T
Userlevel 4

hi guys, i have very bad experiece with ReFS on WS2016, one of my customer has around 60TB+ of backups on ReFS volume and after some windows update and reboot, the volume was “RAW”. from this time i not using ReFS on 2012R2,2016,2019… i cannot belive it. maybe on WS2022 the support is better. In the past i wrote some post, that ReFS is not suported for storage arrays, which are connected via iSCSI. its true with 2022 ? 

tom.

regnor
Userlevel 7
Badge +11

@TKA The storage or RAID controller needs to be certified by Microsoft or listed in the Windows Server catalog. In general you should at least have a battery backed RAID controller in place. Without it a powerless could corrupt your ReFS volume. I'm not sure about iSCSI but would say if the storage is descent it shouldn't be an issue.

For your case; what storage do you have? Is it a NAS?

Max M. - Veeam Legend/Vanguard/VMCE/VMCA
F
Userlevel 5
Badge

The only solution for 2012R2 (running on VMWare) is the 
devices.hotplug with a value of false

mentioned in Disabling the HotAdd/HotPlug capability in virtual machines (1012225) (vmware.com)

 

This setting indeed solves the issue for us on a Windows Server 2019 machine. Now the ReFS volume is accessible again after installing the updates. However, other VMs which still have accessible ReFS volumes after installing the patches don't have this configuration setting and they also have the eject option on the taskbar for the virtual disk. So, although the devices.hotplug setting fixes the issue, it's not the whole story.

 

Thanks @stephc_msft ! Questionable though why MS Support isn't aware of this setting.

marcofabbri
Userlevel 7
Badge +11

In Microsoft must change devs, they've been doing everything wrong lately.

Backups are like pizza, I love them. | Linkedin: @marco-fabbri-it
F
Userlevel 5
Badge

Can people on here please confirm if they are using Refs on VMWare VM’s   (where the hotplug feature makes the drives appear as removable)
Vmware fix to overcome that is https://kb.vmware.com/s/article/1012225?lang=en_us

Or are some of your system really using removable drives?

For WS2012R2 using refsv1  the OOB fix will not and never will overcome the issue on removable drives
The Microsoft OOB fix is only for refs 3.x on removable drives in ws2016 and later
NOte - some systems running later OS’s may be using old disk formatted with refs v1  and so those refsv1 disks will be affected if they are considered removable

 

@stephc_msft  When running the fsutil command on our affected server it returns:

 

REFS Volume Serial Number :       0xb660b96e60b935c9
REFS Version   :                  1.2
Number Sectors :                  0x0000000004fe0000
Total Clusters :                  0x000000000009fc00
Free Clusters  :                  0x0000000000049906
Total Reserved :                  0x0000000000000000
Bytes Per Sector  :               512
Bytes Per Physical Sector :       512
Bytes Per Cluster :               65536
Checksum Type:                    CHECKSUM_TYPE_NONE

So it looks like we have the old ReFS version although the system is running Windows Server 2019. Why is this volume still on the old version, shouldn't it be upgraded automatically? Other Windows 2019 machines which don't have the issue have ReFS version 3.4.

vNote42
Userlevel 7
Badge +12

Thanks for sharing @regnor! Really bad bugs that shouldn't exist! 

These days a customer told me he was suffering from regular reboots of his new 2022 test server. The guess was it had to do with trial-version. No its a feature … eh … bug

Wolfgang | vnote42.net | @vNote42
regnor
Userlevel 7
Badge +11

@Mildur But so far we havn’t experienced any good things since 12/31/2021 from them; except you count everything which is not broken or doesn’t have issues as positive :wink:

 

True. I was talking more general. Not only in 2022, and not only from Microsoft.

Bad things are more discussed than good things :)

 

 You’re definetly right with that :wink:

@Mildur But so far we havn’t experienced any good things since 12/31/2021 from them; except you count everything which is not broken or doesn’t have issues as positive :wink:

 

True. I was talking more general. Not only in 2022, and not only from Microsoft.

Bad things are more discussed than good things :)

 

Nope that’s it, Microsoft took it too far this morning! Decided my headset and webcam were untrustworthy (or maybe they meant the person on them, aka me!) 😆

Better check with a different person, just to be sure :wink:

But perhaps it’s a security feature? Microphones and webcams can spy on you, so better throw it all away :laughing:

@Iams3leI’m sure this won’t be the last time we see such issues; just hope it won’t be in January…

@vNote42Reboots tend to solve problems and make the system more stable; so you’re spot on with “it’s a feature” :nerd:

Max M. - Veeam Legend/Vanguard/VMCE/VMCA
F
Userlevel 5
Badge

Okay, so the only way to get to the new ReFS version is to create a new disk and copy the data from the 1.2 to the 3.x volume? There’s no way to manually upgrade to the new ReFS version?

Scott
Userlevel 7
Badge +8

You guys scared me bringing back this post. I just saw the title haha

Link State
Userlevel 7
Badge +7

aww bad patch MS

//VPN /IPSec Issue: Certain IPSEC connections might fail

 

//Domain Controller rebooting issue

 

//Hyper V issue

Veeam: VMCA 2022 | VMCE 2020 | VMXP | Veeam Legend - Microsoft: MCITP | MCP | MCSA | 2008 R2 | 2012R2 | 2016 | MCSE Core Infrastructure | MCSE Cloud Platform - Azure: AZ900 | AZ104 - VMWare: VCP-DCV Vsphere 7.x 2022 - Cisco: CCNA (Expired)

Comment


Terms & Conditions