• EN

ReFS issues with latest Windows Server Updates (KB5009624, KB5009557, KB5009555)

Show first post

106 comments

Nico Losschaert
Userlevel 7
Badge +11

 

@dlosekeAre those virtual disks or physical volumes?

 

Excellent question!  In both cases, these are RDM disks and the underlying storage is an ISCSI volume presented to the ESXI hosts from a Synology NAS.

Try to use always iSCSI volumes inside the Windows VM and not to the ESXi host and use then RDM disks. It functions better in my experience and less dependent of things like VMware.

#Veeam Legend 5* | VMCA 2* | VMCE 3* | VMTSP+ 2* | VMSP 2* | DCIE 2* | Loves Best Practices | Twitter : @NLosschaert
Geoff Burke
Userlevel 7
Badge +22

Windows ah windows… Well apart from the this REFS fun (remember the REFS issues when it first came out and those poor souls who ventured into the 4k block settings), I believe there was some completely data wipeouts there. Huge thread in the forums if I remember correctly. That aside can you believe that you need to do this to get NFS write permissions on a nfs share? Or have I really gotten confused :)

“Browse to HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\ClientForNFS\CurrentVersion\Default.
Create a new New DWORD (32-bit) Value inside the Default folder named AnonymousUid and assign the UID found on the Linux directory as shared by the NFS system. This is the UID of the user that has the write access to that directory on Linux system.
Create a new New DWORD (32-bit) Value inside the Default folder named AnonymousGid and assign the GID found on the Linux directory as shared by the NFS system. This is the GID of the group that has the write access on the directory on Linux system.Windows 10: Regedit NFS AnonymousUid and AnonymousGid
Restart the NFS client or reboot the machine to apply the changes.
Now run the mount command and you will get the write access.”

VMCA2022, VMCE2023, CKA, CKAD, KCNA, LFCS, PCA,TARS
S
Userlevel 2

What OS?
If 2012R2 and if on a system like vmware where the disks have characteristics of removable, you will alway have issues.
With the patches removed, and the volume visible, can you check the refs version number
fsutil fsinfo refsinfo x:
If its v1.2   then that is the old version which will never work on ‘removable’ drives now
v3.x should be ok with latest updates (since Feb)

If this is a vmware system, and refs v1.2 disks, then the only option is a vmware configuartion chnage to disable hotplug

Disabling the HotAdd/HotPlug capability in virtual machines (vmware.com)


If none of the above, please see if the ReFS event log (under applicationas and services logs) says anything   eg about the version number
eg if somehow the v3.x refs disk have been attached to a later OS and updated to a later v3.x version, then they will no longer be readable if put them back on the earlier OS
 

MicoolPaul
Userlevel 7
Badge +20

Welcome to the Veeam Community @ianv. The community is open for everyone, even if your not a Veeam user :wink:

It’s hard to say what happend in your case and if you did experience something different. I think BSOD weren’t caused by this update, “only” the ReFS volumes turned inaccessible (RAW).

‘strongly suggest’ I contact Microsoft Premier Support Services.

I really love their support for such suggestions. They introduce an issue but won’t help you in anyway or admit that it could be a bug, without going through their premier support, which is by the way, not easily accessible at all.

I‘ve actually never had a successful resolution via Microsoft Premier Support, everything has always been out of scope or no root cause found, and I’ve always been given my credits back! 😆 I must need to get Microsoft Super Premier Justice League Support for that…

Michael Paul - #VeeamLegend | #VeeamVanguard | Veeam Certified Architect | VMware vExpert | https://micoolpaul.com | Twitter: @MicoolPaul | Mastodon: @micoolpaul@masto.nu | Bluesky: @micoolpaul.bsky.social
Iams3le
Userlevel 7
Badge +9

All they’ve done is add the problems as known issues… oh Microsoft…

You are right: https://support.microsoft.com/en-us/topic/january-11-2022-kb5009557-os-build-17763-2452-c3ee4073-1e7f-488b-86c9-d050672437ae

[Microsoft MVP | Veeam Legend 5* | VMware vExpert 4* | Cisco Champion 3* | Object First Ace]
regnor
Userlevel 7
Badge +14

All they’ve done is add the problems as known issues… oh Microsoft…

What else should they do...fix the update?🤣

Max M. | Systems Engineer @Veeam | VMCE/VMCA | former Legend & Vanguard
JMeixner
Userlevel 7
Badge +17

Oh.my.god…. 🤯🤯🤯

Jochen (Joe) Meixner | Veeam Vanguard 2024 | Veeam Legend 2021 - 2023 | Veeam Certified Architect (VMCA) 2022 | Twitter: @JoMeix | BlueSky: @jmeixner.bsky.social
B

Out of curiosity, is anyone getting failed to create snapshot on Hyper-V after the Jan 11 patch?

Boot, login and VMs are fine, but I can’t get any backups to work post-update.

vNote42
Userlevel 7
Badge +13

All they’ve done is add the problems as known issues… oh Microsoft…

the bottom in quality management does not seem to have been reached yet:rage:

Wolfgang | vnote42.net | @vNote42
D

Just had the ReFS issue on Server 2016 with KB5009546.  After install, reboot, target USB repository was RAW.  Uninstalled update and drive is visible again with all data.

 

UGH.  This has been a known issue for a week, why hasn’t MS pulled it?

Chris.Childerhose
Userlevel 7
Badge +20

FYI the out of bound update, KB5010794, is still breaking ReFS for 2012r2.

Thanks for the update as we still are moving some Veeam 2012R2 repo servers to newer OS.  Will ensure this patch is not applied.

Chris Childerhose - VMCA2022 | VMCE2023 | Veeam Vanguard 7* | Veeam Legend 3* | vExpert 5* | VCAP-DCV/VCP-DCV | Object First Ace | Cisco Champion | Twitter: @cchilderhose | Blog Site – https://just-virtualization.tech
marcofabbri
Userlevel 7
Badge +13

FYI the out of bound update, KB5010794, is still breaking ReFS for 2012r2.

That’s strange, multiple blogs online and relative comments declare that fix ReFS on 2012 r2 and storage can be mounted again:
 

https://www.borncity.com/blog/2022/01/18/sonderupdates-fr-windows-mit-fixes-fr-jan-2020-patchday-probleme-17-1-2022/#comment-120775

Backups are like pizza, I love them. | Linkedin: @marco-fabbri-it
marcofabbri
Userlevel 7
Badge +13

Guys here’s updated solution from Microsoft for Windows Server 2022:

https://support.microsoft.com/en-us/topic/january-17-2022-kb5010796-os-build-20348-473-out-of-band-2e0408ba-10d4-4c68-9b3d-cc5fb8d6f4a8

@regnor your link’s for Azure Stack HCI, version 21H2

Backups are like pizza, I love them. | Linkedin: @marco-fabbri-it
regnor
Userlevel 7
Badge +14

@marcofabbri: Thanks! I just copied the link from the R&D forums, but the one from bleepingcomputer.com does include all possible updates.

@Franc I wish you good luck with that. Finding/Solving bugs with Microsoft support isn’t fun at all.

Max M. | Systems Engineer @Veeam | VMCE/VMCA | former Legend & Vanguard
marcofabbri
Userlevel 7
Badge +13

FYI the out of bound update, KB5010794, is still breaking ReFS for 2012r2.

Probably you really need to install both updates, one after the other. Although it’s also strange that the out of band update also introduces the issue…

By the way, just for myself, why did you decide to go with ReFS on 2012R2?

I’m not the person you asked the question to but I thought I’d jump in with my experience. The reason I jumped on the bandwagon early was because Microsoft literally named it “Resilient File System”, they touted automatic detection and repair of corruption as a major reason to go with it, plus the scaling side of ReFS for maximum volume and file size limits seemed better aligned to the constant marketing of data growth explosions.

 

Anything that keeps my backups healthy sounds good to me!

Back in 2014 was nearly like early adoption the ReFS file system, but what you say @MicoolPaul is right.

This is a 2019 article, but so actual: https://www.itprotoday.com/storage/it-finally-time-adopt-microsofts-resilient-file-system

Backups are like pizza, I love them. | Linkedin: @marco-fabbri-it
A
Userlevel 2

FYI the out of bound update, KB5010794, is still breaking ReFS for 2012r2.

Probably you really need to install both updates, one after the other. Although it’s also strange that the out of band update also introduces the issue…

By the way, just for myself, why did you decide to go with ReFS on 2012R2?

A year or so ago we were in the process of rebuilding our storage infrastructure and creating a new scale-out repository in prep for Wasabi’s immutable functionality. We figured we would take advantage of the features of ReFS while we were at it.

regnor
Userlevel 7
Badge +14

Thanks @MicoolPaul and @marcofabbri ! For me Windows Server 2016 was the first contact with ReFS as Veeam started to integrated it. With 2012R2 I felt that it wasn’t mature enough for production. You’re right that the integrity part is/was an advantage, but without storage spaces you only had the corroption detection.

Max M. | Systems Engineer @Veeam | VMCE/VMCA | former Legend & Vanguard
regnor
Userlevel 7
Badge +14

FYI the out of bound update, KB5010794, is still breaking ReFS for 2012r2.

Probably you really need to install both updates, one after the other. Although it’s also strange that the out of band update also introduces the issue…

By the way, just for myself, why did you decide to go with ReFS on 2012R2?

A year or so ago we were in the process of rebuilding our storage infrastructure and creating a new scale-out repository in prep for Wasabi’s immutable functionality. We figured we would take advantage of the features of ReFS while we were at it.

If you were rebuilding the backup repositories, did you think about upgrading to 2016 or 2019? This would have given you the advanced ReFS integration.

Max M. | Systems Engineer @Veeam | VMCE/VMCA | former Legend & Vanguard
JMeixner
Userlevel 7
Badge +17

Latest update from MS Support: case is being archived with no resolution at the moment. Advice currently is, if the issue occurs after installing the cumulative patch, you have two options:

- uninstall the patch and copy the data to an NTFS volume and re-install the patch

- refrain from installing the patch entirely

The MS Support engineer even made a personal note:

In any case if you use NTFS I personally recommend using NTFS instead of ReFS, as ReFS is still immature as a file system as has several bugs.


I did let them know my frustration about this issue, since they broke it and still don't have a solution or even a cause after several weeks.

This response is… wow :scream: . It makes a very uncomfortable feeling at the moment.

In this case they should drop ReFS completely.

Jochen (Joe) Meixner | Veeam Vanguard 2024 | Veeam Legend 2021 - 2023 | Veeam Certified Architect (VMCA) 2022 | Twitter: @JoMeix | BlueSky: @jmeixner.bsky.social
F
Userlevel 5
Badge

Yep, uninstalling the patch restores access. That specific machine is running Windows Server 2022.

marcofabbri
Userlevel 7
Badge +13

Nice share of infos guys, this is the power of this community.

Backups are like pizza, I love them. | Linkedin: @marco-fabbri-it
marcofabbri
Userlevel 7
Badge +13

Tonight Microsoft start releasing “fix for everything”:


https://www.catalog.update.microsoft.com/Search.aspx?q=%092022-01%20Cumulative%20Update%20Preview

Backups are like pizza, I love them. | Linkedin: @marco-fabbri-it
bp4JC
Userlevel 7
Badge +3

@regnor thank you for the heads up on this!!!

T
Userlevel 4

@TKAThe storage or RAID controller needs to be certified by Microsoft or listed in the Windows Server catalog. In general you should at least have a battery backed RAID controller in place. Without it a powerless could corrupt your ReFS volume. I'm not sure about iSCSI but would say if the storage is descent it shouldn't be an issue.

For your case; what storage do you have? Is it a NAS?

Now the customer has a “new backup server” with WS2019 on DELL R720 with 10 GbE ethernets. The backup target is Storage Array - Infotrend DS1024 (dual controller, 4 GB RAM per ctl with BBU). But from my bad experience with Refs and lose around 60TB + of backup, we are still on NTFS :) now we ordered a new array with capacity 180 TB, so maybe is time to change to ReFS for “performance tier” 60 TB and 120 TB for ObjectStorage volume as “archive” .) 

regnor
Userlevel 7
Badge +14

Thanks @stephc_msft for jumping into the discussion. I’m wondering if disabling hotadd fixes the issue. Does this mean for a virtualized Windows it looks like the disk is external, with hotadd enabled?

@Everyone: If you set this configuration with a virtual backup server, probably hotadd transport mode will no longer work ?

Max M. | Systems Engineer @Veeam | VMCE/VMCA | former Legend & Vanguard

Comment


Terms & Conditions