ReFS issues and DC bootloop and broken Hyper-v

Just talking about this some minutes ago.

More info here: https://www.bleepingcomputer.com/news/microsoft/new-windows-server-updates-cause-dc-boot-loops-break-hyper-v/

Microsoft has pulled now the updates from the update servers:

https://www.bleepingcomputer.com/news/microsoft/microsoft-pulls-new-windows-server-updates-due-to-critical-bugs/amp/

We decided to decline the updates from our WSUS after seeing the thread on Reddit before MS pulled them. 
https://www.reddit.com/r/sysadmin/comments/s1jcue/patch_tuesday_megathread_20220112/
 

The servers that got patched don’t show any errors.

It’s a good start in 2022. To many things go sideways. :)

True. I was talking more general. Not only in 2022, and not only from Microsoft.

Bad things are more discussed than good things :)

Nope that’s it, Microsoft took it too far this morning! Decided my headset and webcam were untrustworthy (or maybe they meant the person on them, aka me!)

Think it’s for the best…

KB5009555 for Windows 2022 doesn’t seem to have been pulled. My test server still downloads this update from Windows Update. However, this is the faulty update which causes the raw ReFS issue.

All they’ve done is add the problems as known issues… oh Microsoft…

Unfortunately just finding out about this. Entire ReFS volume appears to be shot even after uninstalling the update. Anyone else run into uninstalling not resolving the issue?

***Update - after uninstalling BOTH KB5009624 and KB5009595 our ReFS volumes are back. Running Server 2012 R2 Datacenter

FYI the out of bound update, KB5010794, is still breaking ReFS for 2012r2.

So possibly it’s the combination of installing the past ‘bad’ update and KB5010794 together that is needed? We blocked the original so we had KB5010794 install by itself. From your link:
 

So, I pulled it through today (I informed myself and resisted for a long time).
Thank you to everyone here who helped share information.
The new patches were used: KB5010791 (2019 server), KB5010790 (2016 server) and KB5010794 (server 2012R2)
1x 2012R2 server (DC)
4x 2019 server (DCs)
4x 2016 server (RODCs)
All servers had the latest patch status the Dec. Patchday updates.
First the 4x 2019 servers with KB5010791 (because it's a cumulative update, pulled from the update catalog), then the 2012R2 via Windows Update, here I selected the 'bad' January update from 01/11/22 TOGETHER with the optional KB5010794 and installed both at once.
Finally the 4x 2016s with KB5010790 from the Update Catalog.
Everything is running - lucky :)

Agreed, this engineer is definitely overstepping by presenting their own personal views in such a way. Likely out of frustration as Microsoft will have been the ones stating to just uninstall the patch.

When Microsoft promised us singular “cumulative” updates to improve their QA, this is a trade off we now are experiencing, that you get all or nothing with their patches, production breaking updates vs exposure to zero day exploits is now the choice to make.

The ReFS comment makes me feel uneasy also, but this is why we need to plan for file system issues in our 3-2-1-1-0, as this patch, whilst bad, has the ability to be reversed. What happens next time when the patch corrupts the partitions beyond repair?

We need to treat this as an important lesson, and be glad that Veeam have been helping us break free of Microsoft’s chains with the use of Linux operating systems.

If I’m allowed to dream, I’d love to see XFS on What nodes, giving Microsoft some actual competition.

In Microsoft must change devs, they've been doing everything wrong lately.

Thanks for sharing @regnor! Really bad bugs that shouldn't exist! 

These days a customer told me he was suffering from regular reboots of his new 2022 test server. The guess was it had to do with trial-version. No its a feature … eh … bug

 You’re definetly right with that

Better check with a different person, just to be sure

But perhaps it’s a security feature? Microphones and webcams can spy on you, so better throw it all away

@Iams3leI’m sure this won’t be the last time we see such issues; just hope it won’t be in January…

@vNote42Reboots tend to solve problems and make the system more stable; so you’re spot on with “it’s a feature”

aww bad patch MS

//VPN /IPSec Issue: Certain IPSEC connections might fail

• https://docs.microsoft.com/en-us/windows/release-health/status-windows-11-21h2#2773msgdesc
• https://docs.microsoft.com/en-us/windows/release-health/status-windows-10-21h2#2773msgdesc
• https://docs.microsoft.com/en-us/windows/release-health/status-windows-10-21h1#2773msgdesc
• https://docs.microsoft.com/en-us/windows/release-health/status-windows-10-20h2#2773msgdesc
• https://docs.microsoft.com/en-us/windows/release-health/status-windows-10-1909#2773msgdesc
• https://docs.microsoft.com/en-us/windows/release-health/status-windows-10-1809-and-windows-server-2019#2773msgdesc
• https://docs.microsoft.com/en-us/windows/release-health/status-windows-10-1607-and-windows-server-2016#2773msgdesc
• https://docs.microsoft.com/en-us/windows/release-health/status-windows-10-1507#2773msgdesc
• https://docs.microsoft.com/en-us/windows/release-health/status-windows-server-2022#2773msgdesc

//Domain Controller rebooting issue

• https://docs.microsoft.com/en-us/windows/release-health/stat...
• https://docs.microsoft.com/en-us/windows/release-health/stat...

//Hyper V issue

• https://docs.microsoft.com/en-us/windows/release-health/stat...
• https://docs.microsoft.com/en-us/windows/release-health/stat...

That’s because Microsoft released them AGAIN on a FRIDAY 🤯

https://www.bleepingcomputer.com/news/microsoft/microsoft-resumes-rollout-of-january-windows-server-updates/

That’s crazy talk! What next, QA testing?

You say it. We get the updates for free, so the least we can do is giving something back to Microsoft; fixing bugs, QA, …

Microsoft has released an out-of-band which should resolve all the problems: https://support.microsoft.com/en-au/topic/january-17-2022-non-security-update-kb5010796-out-of-band-e79a633f-e876-4268-a21e-de6a9ca52da7

According to @Franc it didn't solve his ReFS problem, so please still be careful.

As @regnor stated, this update doesn’t solve the RAW issue on fixed ReFS drives for us. I’ll open a support case with Microsoft for it. Although I don’t get my hopes up too high, since my experience with MS Support lately is not that great to say the least.

@regnor , well that was quick with MS support. They confirmed it's also an issue with fixed drives, but the patch from yesterday was only for external drives. They are still working on a fix for fixed drives and he will inform me once the patch is available. He couldn't explain though why Microsoft doesn't mention the issue for fixed drives anywhere, but only for external drives. He confirmed he already had multiple cases where other customers also experience the issue with fixed drives.

Probably you really need to install both updates, one after the other. Although it’s also strange that the out of band update also introduces the issue…

By the way, just for myself, why did you decide to go with ReFS on 2012R2?

I’m not the person you asked the question to but I thought I’d jump in with my experience. The reason I jumped on the bandwagon early was because Microsoft literally named it “Resilient File System”, they touted automatic detection and repair of corruption as a major reason to go with it, plus the scaling side of ReFS for maximum volume and file size limits seemed better aligned to the constant marketing of data growth explosions.

Anything that keeps my backups healthy sounds good to me!