ReFS issues with latest Windows Server Updates (KB5009624, KB5009557, KB5009555)



Show first post

106 comments

Userlevel 7
Badge +17

Latest update from MS Support: case is being archived with no resolution at the moment. Advice currently is, if the issue occurs after installing the cumulative patch, you have two options:

- uninstall the patch and copy the data to an NTFS volume and re-install the patch

- refrain from installing the patch entirely

The MS Support engineer even made a personal note:

In any case if you use NTFS I personally recommend using NTFS instead of ReFS, as ReFS is still immature as a file system as has several bugs.


I did let them know my frustration about this issue, since they broke it and still don't have a solution or even a cause after several weeks.

This response is… wow :scream: . It makes a very uncomfortable feeling at the moment.

In this case they should drop ReFS completely.

Userlevel 7
Badge +20

Latest update from MS Support: case is being archived with no resolution at the moment. Advice currently is, if the issue occurs after installing the cumulative patch, you have two options:

- uninstall the patch and copy the data to an NTFS volume and re-install the patch

- refrain from installing the patch entirely

The MS Support engineer even made a personal note:

In any case if you use NTFS I personally recommend using NTFS instead of ReFS, as ReFS is still immature as a file system as has several bugs.


I did let them know my frustration about this issue, since they broke it and still don't have a solution or even a cause after several weeks.

Wow that is quite the response.  Passing the buck.  Sheesh :rolling_eyes:

Userlevel 5
Badge

Latest update from MS Support: case is being archived with no resolution at the moment. Advice currently is, if the issue occurs after installing the cumulative patch, you have two options:

- uninstall the patch and copy the data to an NTFS volume and re-install the patch

- refrain from installing the patch entirely

The MS Support engineer even made a personal note:

In any case if you use NTFS I personally recommend using NTFS instead of ReFS, as ReFS is still immature as a file system as has several bugs.


I did let them know my frustration about this issue, since they broke it and still don't have a solution or even a cause after several weeks.

Userlevel 7
Badge +20

Thanks @MicoolPaul and @marcofabbri ! For me Windows Server 2016 was the first contact with ReFS as Veeam started to integrated it. With 2012R2 I felt that it wasn’t mature enough for production. You’re right that the integrity part is/was an advantage, but without storage spaces you only had the corroption detection.

Yeah we never looked at it until 2016 as well since it was just new in 2012R2.  Also by the time they came out with 2016 the product had matured and was much better.  Did not like anything below 2016 other than maybe 2008R2.  :joy:

Userlevel 7
Badge +13

Thanks @MicoolPaul and @marcofabbri ! For me Windows Server 2016 was the first contact with ReFS as Veeam started to integrated it. With 2012R2 I felt that it wasn’t mature enough for production. You’re right that the integrity part is/was an advantage, but without storage spaces you only had the corroption detection.

It was same for me @regnor 

But I hope that @Asmith can resolve his issue..

Userlevel 7
Badge +13

FYI the out of bound update, KB5010794, is still breaking ReFS for 2012r2.

Probably you really need to install both updates, one after the other. Although it’s also strange that the out of band update also introduces the issue…

By the way, just for myself, why did you decide to go with ReFS on 2012R2?

A year or so ago we were in the process of rebuilding our storage infrastructure and creating a new scale-out repository in prep for Wasabi’s immutable functionality. We figured we would take advantage of the features of ReFS while we were at it.

If you were rebuilding the backup repositories, did you think about upgrading to 2016 or 2019? This would have given you the advanced ReFS integration.

Userlevel 7
Badge +13

Thanks @MicoolPaul and @marcofabbri ! For me Windows Server 2016 was the first contact with ReFS as Veeam started to integrated it. With 2012R2 I felt that it wasn’t mature enough for production. You’re right that the integrity part is/was an advantage, but without storage spaces you only had the corroption detection.

Userlevel 2

FYI the out of bound update, KB5010794, is still breaking ReFS for 2012r2.

Probably you really need to install both updates, one after the other. Although it’s also strange that the out of band update also introduces the issue…

By the way, just for myself, why did you decide to go with ReFS on 2012R2?

A year or so ago we were in the process of rebuilding our storage infrastructure and creating a new scale-out repository in prep for Wasabi’s immutable functionality. We figured we would take advantage of the features of ReFS while we were at it.

Userlevel 7
Badge +13

FYI the out of bound update, KB5010794, is still breaking ReFS for 2012r2.

Probably you really need to install both updates, one after the other. Although it’s also strange that the out of band update also introduces the issue…

By the way, just for myself, why did you decide to go with ReFS on 2012R2?

I’m not the person you asked the question to but I thought I’d jump in with my experience. The reason I jumped on the bandwagon early was because Microsoft literally named it “Resilient File System”, they touted automatic detection and repair of corruption as a major reason to go with it, plus the scaling side of ReFS for maximum volume and file size limits seemed better aligned to the constant marketing of data growth explosions.

 

Anything that keeps my backups healthy sounds good to me!

Back in 2014 was nearly like early adoption the ReFS file system, but what you say @MicoolPaul is right.

This is a 2019 article, but so actual: https://www.itprotoday.com/storage/it-finally-time-adopt-microsofts-resilient-file-system

Userlevel 7
Badge +20

FYI the out of bound update, KB5010794, is still breaking ReFS for 2012r2.

Probably you really need to install both updates, one after the other. Although it’s also strange that the out of band update also introduces the issue…

By the way, just for myself, why did you decide to go with ReFS on 2012R2?

I’m not the person you asked the question to but I thought I’d jump in with my experience. The reason I jumped on the bandwagon early was because Microsoft literally named it “Resilient File System”, they touted automatic detection and repair of corruption as a major reason to go with it, plus the scaling side of ReFS for maximum volume and file size limits seemed better aligned to the constant marketing of data growth explosions.

 

Anything that keeps my backups healthy sounds good to me!

Userlevel 7
Badge +13

FYI the out of bound update, KB5010794, is still breaking ReFS for 2012r2.

Probably you really need to install both updates, one after the other. Although it’s also strange that the out of band update also introduces the issue…

By the way, just for myself, why did you decide to go with ReFS on 2012R2?

Userlevel 7
Badge +13

So possibly it’s the combination of installing the past ‘bad’ update and KB5010794 together that is needed? We blocked the original so we had KB5010794 install by itself. From your link:
 

So, I pulled it through today (I informed myself and resisted for a long time).
Thank you to everyone here who helped share information.
The new patches were used: KB5010791 (2019 server), KB5010790 (2016 server) and KB5010794 (server 2012R2)
1x 2012R2 server (DC)
4x 2019 server (DCs)
4x 2016 server (RODCs)
All servers had the latest patch status the Dec. Patchday updates.
First the 4x 2019 servers with KB5010791 (because it's a cumulative update, pulled from the update catalog), then the 2012R2 via Windows Update, here I selected the 'bad' January update from 01/11/22 TOGETHER with the optional KB5010794 and installed both at once.
Finally the 4x 2016s with KB5010790 from the Update Catalog.
Everything is running - lucky :)

Maybe that’s it :thumbsup_tone1:

Userlevel 2

So possibly it’s the combination of installing the past ‘bad’ update and KB5010794 together that is needed? We blocked the original so we had KB5010794 install by itself. From your link:
 

So, I pulled it through today (I informed myself and resisted for a long time).
Thank you to everyone here who helped share information.
The new patches were used: KB5010791 (2019 server), KB5010790 (2016 server) and KB5010794 (server 2012R2)
1x 2012R2 server (DC)
4x 2019 server (DCs)
4x 2016 server (RODCs)
All servers had the latest patch status the Dec. Patchday updates.
First the 4x 2019 servers with KB5010791 (because it's a cumulative update, pulled from the update catalog), then the 2012R2 via Windows Update, here I selected the 'bad' January update from 01/11/22 TOGETHER with the optional KB5010794 and installed both at once.
Finally the 4x 2016s with KB5010790 from the Update Catalog.
Everything is running - lucky :)

Userlevel 7
Badge +13

FYI the out of bound update, KB5010794, is still breaking ReFS for 2012r2.

That’s strange, multiple blogs online and relative comments declare that fix ReFS on 2012 r2 and storage can be mounted again:
 

https://www.borncity.com/blog/2022/01/18/sonderupdates-fr-windows-mit-fixes-fr-jan-2020-patchday-probleme-17-1-2022/#comment-120775

Userlevel 7
Badge +20

FYI the out of bound update, KB5010794, is still breaking ReFS for 2012r2.

Thanks for the update as we still are moving some Veeam 2012R2 repo servers to newer OS.  Will ensure this patch is not applied.

Userlevel 2

FYI the out of bound update, KB5010794, is still breaking ReFS for 2012r2.

Userlevel 7
Badge +13

@regnor , well that was quick with MS support. They confirmed it's also an issue with fixed drives, but the patch from yesterday was only for external drives. They are still working on a fix for fixed drives and he will inform me once the patch is available. He couldn't explain though why Microsoft doesn't mention the issue for fixed drives anywhere, but only for external drives. He confirmed he already had multiple cases where other customers also experience the issue with fixed drives.

No good Microsoft, no good...

Userlevel 7
Badge +13

Now that was quick @Franc! Lets hope they’ll also be able to solve the fixed disk problem soon.

Userlevel 7
Badge +7

Just a quick post; the latest Windows Server updates for 2012R2, 2019 and 2022 (haven’t seen 2016) can cause ReFS issues. After the installation, ReFS volumes are shown as RAW and are no longer accessible; so if you’re using ReFS for your repositories, then take be aware of this when installating the update. Besides that, those updates can also cause bootloops for domain controllers and break Hyper-V.

Depending on your Windows Server version one of the following updates could have caused the issue: KB5009624, KB5009557, KB5009555

If you’re affected then removing the mentioned update should solve the issue. Don’t (!) try to repair the ReFS volume, because this could cause a dataloss.

This should remind everyone why 3-2-1 for Backups is so important. :wink:

Further information:

https://www.bleepingcomputer.com/news/microsoft/new-windows-server-updates-cause-dc-boot-loops-break-hyper-v/

https://forums.veeam.com/veeam-backup-replication-f2/beware-possible-raw-refs-volumes-after-installing-january-updates-t78634.html

Update #1:

Microsoft has pulled the updates, thanks @Mildur 

https://www.bleepingcomputer.com/news/microsoft/microsoft-pulls-new-windows-server-updates-due-to-critical-bugs/amp/

Update #2:

Microsoft has released the updates again and it looks like they didn't fix them. (Thanks @MicoolPaul )

https://www.bleepingcomputer.com/news/microsoft/microsoft-resumes-rollout-of-january-windows-server-updates/

Update #3:

Microsoft has released an out-of-band update, which can possibly resolve the issues:

https://www.bleepingcomputer.com/news/microsoft/microsoft-releases-emergency-fixes-for-windows-server-vpn-bugs/

 

Thanks for your update again.

Userlevel 5
Badge

@regnor , well that was quick with MS support. They confirmed it's also an issue with fixed drives, but the patch from yesterday was only for external drives. They are still working on a fix for fixed drives and he will inform me once the patch is available. He couldn't explain though why Microsoft doesn't mention the issue for fixed drives anywhere, but only for external drives. He confirmed he already had multiple cases where other customers also experience the issue with fixed drives.

Userlevel 7
Badge +13

@marcofabbri: Thanks! I just copied the link from the R&D forums, but the one from bleepingcomputer.com does include all possible updates.

@Franc I wish you good luck with that. Finding/Solving bugs with Microsoft support isn’t fun at all.

Userlevel 5
Badge

As @regnor stated, this update doesn’t solve the RAW issue on fixed ReFS drives for us. I’ll open a support case with Microsoft for it. Although I don’t get my hopes up too high, since my experience with MS Support lately is not that great to say the least.

Userlevel 7
Badge +13

Guys here’s updated solution from Microsoft for Windows Server 2022:

https://support.microsoft.com/en-us/topic/january-17-2022-kb5010796-os-build-20348-473-out-of-band-2e0408ba-10d4-4c68-9b3d-cc5fb8d6f4a8

@regnor your link’s for Azure Stack HCI, version 21H2

Userlevel 7
Badge +13

Microsoft has released an out-of-band which should resolve all the problems: https://support.microsoft.com/en-au/topic/january-17-2022-non-security-update-kb5010796-out-of-band-e79a633f-e876-4268-a21e-de6a9ca52da7

According to @Franc it didn't solve his ReFS problem, so please still be careful.

Userlevel 7
Badge +13

Thanks for the feedback @Asmith; great that you were able to solve it.

@Bommer I haven't heard of this one so far. You could try to uninstall the updates to see if they were the cause.

Comment