VeeamON 2024 - Use Code "COMMUNITY10" for 10% Off!
no I didn’t. 3 engineers picked up the case after each other and they all went down the same road. So I was quite done with it. We have 23 backup jobs and 45 copy jobs running and this single job had one crc error. The server (Proliant DL380 gen10 plus) has ECC memory, so a RAM issue could also be ruled out I guess. So I really doubt a hardware issue could be the problem or else we would have seen it on much more jobs. Health checks on all jobs also pass without issues. After creating a new active full I haven’t had the issue again so I treat it as a one time occurrence for now. I’m glad Veeam does checksum checks again when reading the files, so that’s a positive outcome. ;-) My point merely is that support pointed right to filesystem corruption, because that’s the most probable cause of this error. However, they refused to look at other potential causes even though I provided evidence that rules out files system corruption.
Unfortunately the support case was closed without finding the solution. Support insisted that it was file system corruption and I’m convinced that can’t be the case since integrity streams are enabled on the backup files and no corruption is being reported in the eventlogs. I can copy the backup files fine, even the one which has the crc error. When there’s an integrity stream violation, Windows would throw an error during this copy operation. So to me this looks like the crc error occurred during the creation of the backup file, but I wasn’t able to convince support on that and they insisted it was file system corruption. Hence, I decided to close the case since it was leading nowhere,
If you enter the ‘normal’ command, the system reboots again. Right after pressing enter after typing the normal command press ESC multiple times until you see the grub menu.see here:https://askubuntu.com/questions/381613/how-to-return-from-grub-prompt-to-the-grub-menu
On EFI systems you have to use the ESC key to get into the grub menu. There is an issue though, if you press it too frequent you’ll end up in the grub command line. I haven’t figured out yet how you can edit the entry to enter single user mode from there.edit: just found on google that you have use the ‘normal’ command and press the ESC key again to return to the grub menu (this time it doesn't enter the CLI if you press it too many times). Just tested it and it works.
Hello, could you maybe tell us what server you used? Just to know what works… the more hardware information you can provide, the better it is. I think we should show the grub menu longer in a future version to avoid issues like that you cannot see it. That’s useful feedback! https://askubuntu.com/questions/1264119/how-to-change-grub-timeout-in-ubuntu-20-04 should work if you edit it before the system locks down itself during the second boot. Some settings are applied before the first boot, some (like the sudo stuff) are applied at second boot. The script does not do anything around grub, because it requires manual user interaction which we wanted to avoid Best regards, Hannes Hi,it’s an HPE Proliant DL360 Gen10 Plus server
Hi, I'm currently playing with the VHR ISO on a physical server. Works perfect, but I also want to test single user mode if for some reason we need to do some troubleshooting. However, the GRUB menu doesn't show up (it boots directly into the OS), so I'm unable to edit the boot entry. Reading the blogpost this is a requirement by the hardening script.Following Hannes blog post (Ubuntu Linux Defense: Secure Boot & Single User Mode (veeam.com) we need to set a password for grub in order to be able to edit the boot entries. We need to configure this before the second reboot after VHR deployment, since after that the system is locked down.I'm wondering if we have a chicken and egg situation here. Is the hardening script executed during the second reboot? If so, doesn't the script then disable GRUB again so we still won't be able to access the GRUB menu to configure single user mode?
I would love to have a screenshot of the window displaying this error for education purposes, would it be possible to send it to Rasmus -at- Veeam.com, please? I’ll make sure to anonymize any identifying factors. On it’s way.
Maybe also use PowerShell to confirm integrity stream really is enabled for that particular backup file? There is an example here: https://learn.microsoft.com/en-us/windows-server/storage/refs/integrity-streams just search for Get-FileIntegrity Did that, integrity streams are enabled and enforced.
I forgot to ask, you checked the system log already for any notifications about refs corruption? Yes, no corruption reported.
it’s a local repository on the Veeam server itself. Physical server DL380 Gen10 plus with 12x 8TB SAS drives.
Great list! Install AV on the Veeam server? Built-in Windows Defender or third-party? We’ve installed Sophos Intercept X Advanced for Server with XDR
VMware released a patch yesterday to fix this issue: https://docs.vmware.com/en/VMware-vSphere/7.0/rn/vsphere-esxi-70u3k-release-notes.html Edit: sorry - already posted by @regnor
What V12 capability are you most anticipating? MFA & bugs we encountered which are fixed in V12 How do you plan upgrades to Veeam Backup & Replication? Releasenotes, forums, blogposts SNARK - Do you tell your boss you are upgrading? Yep How long do you wait before upgrading? I would like to install it asap within 2 to 4 weeks, but we can't since our cloudconnect provider isn't ready for V12 yet and they don't give an estimate when they are. It would be nice to have a new version to be compatible the with n-1 version, so we don't have to wait for our cloud connect provider. Do you set a test lab for upgrades? Yes
In my lab I install new versions right away (even beta’s), I’m an early adopter. Test and play with the new features to see what I’ll use in production. For production use I’m limited by our service provider where we replicate our DR environment to using Veeam replication. They are too slow in updating to the latest versions in my opinion.
Interesting thread, We have the issue that after starting our DR site (which is built using a Veeam replication job from backup copies), the sysvol share is not replicating anymore (we have 3 DC’s). I can only get it to replicate again by adding the following regkeys to the DC which holds the FSMO roles: reg add "HKLM\System\CurrentControlSet\Services\DFSR\Restore" /v SYSVOL /t REG_SZ /d "authoritative"reg add "HKLM\SYSTEM\CurrentControlSet\Control\BackupRestore\SystemStateRestore" /v LastRestoreId /t REG_SZ /d "10000000-0000-0000-0000-000000000000" After a reboot of the DC I added these keys to sysvol is replicating again.Is this also a known issue?
Our environment was already at 7.0u3b before they pulled it, but we don't have any issues with it.I've just upgraded my lab to 7.0u3c. First strange thing already happened. It reported that at least one host was not compatible (it's running 7.03a) due to the duplicate intel driver issue and I should upgrade to 7.03c using the iso first before upgrading vCenter. However, after starting the vcenter upgrade for a second time, it let me proceed fine without throwing the error and updated fine.
Yep, but this specific volume fortunately doesn’t contain any Veeam backup data.
Okay, so the only way to get to the new ReFS version is to create a new disk and copy the data from the 1.2 to the 3.x volume? There’s no way to manually upgrade to the new ReFS version?
Yeah, but we for sure didn't format it with ReFSv1. How can I convert it to a higher version so that we don't have the issue any more after installing the patch? I'm going to detach the disk from this specific Win 2019 server and attach it to another Win 2019 server and see if it's upgraded. EDIT: sigh, attaching the disk to another Windows 2019 VM doesn't work. The volume isn't automatically upgraded to a ReFS version higher than 1.2. Adding the devices.hotplug parameter is the only solution for now.
Yep, but question is why isn't it upgraded to the latest version automatically? I didn't format it with the ReFSv1 paramter as specified on the github article. Is there a way to manually upgrade it to the latest version?
Can people on here please confirm if they are using Refs on VMWare VM’s (where the hotplug feature makes the drives appear as removable) Vmware fix to overcome that is https://kb.vmware.com/s/article/1012225?lang=en_us Or are some of your system really using removable drives? For WS2012R2 using refsv1 the OOB fix will not and never will overcome the issue on removable drives The Microsoft OOB fix is only for refs 3.x on removable drives in ws2016 and later NOte - some systems running later OS’s may be using old disk formatted with refs v1 and so those refsv1 disks will be affected if they are considered removable @stephc_msft When running the fsutil command on our affected server it returns: REFS Volume Serial Number : 0xb660b96e60b935c9REFS Version : 1.2Number Sectors : 0x0000000004fe0000Total Clusters : 0x000000000009fc00Free Clusters : 0x0000000000049906Total Reserved : 0x0000000000
The only solution for 2012R2 (running on VMWare) is the devices.hotplug with a value of false mentioned in Disabling the HotAdd/HotPlug capability in virtual machines (1012225) (vmware.com) This setting indeed solves the issue for us on a Windows Server 2019 machine. Now the ReFS volume is accessible again after installing the updates. However, other VMs which still have accessible ReFS volumes after installing the patches don't have this configuration setting and they also have the eject option on the taskbar for the virtual disk. So, although the devices.hotplug setting fixes the issue, it's not the whole story. Thanks @stephc_msft ! Questionable though why MS Support isn't aware of this setting.
Yep
We are running vSphere 7 and have the issue on two VMs. This drive is shown as fixed. The VM runs Windows server 2022.
Yep, uninstalling the patch restores access. That specific machine is running Windows Server 2022.
Already have an account? Login
Enter your username or e-mail address. We'll send you an e-mail with instructions to reset your password.
Sorry, we're still checking this file's contents to make sure it's safe to download. Please try again in a few minutes.
Sorry, our virus scanner detected that this file isn't safe to download.
