Restoring Exchange is always a bit special, in particular after a failed CU. Many settings are stored in AD, and those don't get changed back during a restore.

Did you do a full VM restore or an Instant VM recovery?

This feels like disk IO bottlenecking so @regnor’s comment around instant recovery makes sense. I know you mentioned a VeeamZIP backup, but it was slightly less clear what feature you used to restore.

I’d also check the hypervisor’s resources, just in case it’s over provisioned, if you restored a new copy of the VM and left the original running without network or something.

Thanks for the replies. It was a full restore by right-clicking the machine under “Inventory” and selecting restore from VeeamZIP, overwriting the failed machine. So, only one instance of the server after the process. It’s also the only one of the 10 that does this pause before boot, all the others begin the boot process with spinning dots pretty much immediately.  I’ve never seen a VM do this.

The host hardware is a Dell PowerEdge R710 with 6 Hitachi Ultrastar Enterprise drives in a RAID 6 (I realize that’s not everyone’s favorite layout but it works great) and 96GB memory. I don’t think I’m oversubscribed on any hardware in the VM settings. I just did a shutdown of this server while watching disk activity on the host, let it go to zero activity and monitored as it booted up. Disk activity associated with the Exch Svr rises quickly to a level that’s in-line with other running servers, but not to a level causing concern, all while the black screen/Hyper-V logo is there for about 2 & a half minutes then it boots normally. At least it’s consistent lol… Nothing odd in system or application logs in either host or guest.

As you've just restored it, there are probably no checkpoints/snapshots active on that VM? Any errors/warnings in the Hyper-V eventlog?

That does bring up an interesting point. Short answer is I can’t retrieve the logs anymore because I’ve since rebuilt the Veeam VM from the OS upward. Here’s why:

About a year ago I had purpose-built a Windows Server 2016 VM to house B&R. Shortly prior to this backup of the Exchange VM, I updated Veeam from 11 to 11A. The update went flawlessly as did the backup of Exchange (and as well, my Domain Controller, File Server, etc.). When things went south with the Exchange CU, and subsequently the lackluster (but booting up, at least) outcome of the restored Exchange (the troubleshooting I mentioned in my first post), I thought perhaps the Exchange CU had kludged my Domain Controller during ADprep/ForestPrep so I attempted restoring the DC as well - it failed twice with an “access denied” error.

If you’ve read this far, thank you, and I probably owe you a beer… At that point things weren’t making much sense, so I tried a repair install of Veeam, which reached 100% but had the red stopsign FAILURE. Because, of course…  So to land this plane, I’ve stuck with the existing DC believing it’s fine, and repaired the functions on Exchange the hard way by solving issues one by one. It now receives inbound mail, connects properly to Outlook clients, provides OWA, all the things I expect it to do. Except the mysterious 2 ½ minute pause on the Hyper-V logo. (SFC was clear btw) And of course I nuked Veeam and reinstalled it from scratch. I kept the backup repo and imported it.

Normally I try to repair an Exchange server and only use the backup as a last resort. While a restore is possible, it still requires some additional work; as you've seen

Next time you reboot the Exchange, scroll through all of the Hyper-V logs and check if there's anything suspicious.

Applications and Services -> Microsoft -> Windows → Hyper-V

If not, then I'm out of guesses for the moment. As long as the restored virtual disks are where they should be, without any checkpoints, I would not see another reason for your problem.

Thank you. Will check that in about 10 hours from now.

Nothing unusual in the Hyper-V host logs, that I can tell. All “Information” events in Hyper-V Worker, Admin log. It shows ID 12597 (“SynthNic” connecting to the virtual network immediately on machine start), then ID 18514 a split second later (“ExchSrvr was reset by the guest operating system...”),  finally 2:45 minutes later the machine is logged as successfully booted.

So far I've only found this thread, but I'm not sure if it's related to your problem. You could check the size of the VM's configuration file (vmcx).

https://forums.veeam.com/microsoft-hyper-v-f25/guest-os-starting-up-is-very-slow-on-hyper-v-2016-t50984-30.html

If it's OK then I'm not sure what else you could check. Perhaps someone with more Hyper-V experience will be able to help you further. Or you contact Veeam support and check if they have any ideas.

Wow, very interesting read, THANKS! I see I have some background reading to do beyond that as well. All my VMCX files are tiny. But the Exchange VMRS file is about 19GB, while the DC and File Server are each in the 4GB neighborhood. And I have a few rarely used ones that are in the KBs. That’s a quite a range.

Out of curiosity...

If you restore that VeeamZIP elsewhere, do you still get the same boot time?

And if you create a new VM and attach the VHDX’s to it, do you still get the same boot time?

Nice idea. I was actually considering doing an export & import of the VM to create a “new” machine to see if that solved it. Your suggestion might be cleaner. Will give it a shot.

Now that I’ve been “reminded” (lol) of what the VMRS files are, those are all in perfect agreement based on the varying amounts or RAM assigned to the machines, so that part makes sense now.

My next step this evening will be to keep the VHDX file, zap the rest of the folder contents, and build a new VM & finally attach the existing VHDX file & see what happens.

Keep us posted intrigued on this one

Also looking forward to your results

So after the VM deletion/recreation it is doing basically the same thing. The first time it booted it had a completely black screen for almost 3 minutes, followed by the “Hyper-V” logo for a short moment, then the usual “Getting Windows Ready” as it redetected devices. I needed to re-do my static IP info on the “new” NIC, and after several reboots it now consistently takes about 3:45 minutes to boot which is maybe slightly longer than before, but not by too much. Exchange is well behaved so I really can’t complain, but it would be nice to solve this little mystery. In the meantime, nightly backups from B&R CE are scheduled at 4am, so I’ll see if that fires off successfully.

Have you checked the Hyper-V VM settings of your exchange for any unused devices? Like multiple DVD drives? Also is the disk on the first place in the boot order?

Besides that I would check if Windows shows any orphaned devices in the device manager.

sometimes this is a bit tricky.
In my experience with Hyper-V, it happened some times, and with a registry cleanup / optimization, or restoring the machine full again, it worked well again.

With vSphere is not that often, some time because we left an iso file attached, or a confusion in destiny data store or cache.

as I said, for me, this is sometimes like “magic” and tricky!

Cheers,

Thanks for these suggestions. There was the “boot from file” entry that came back after recreating the VM, which I removed with my new favorite PowerShell gem:

Get-VMFirmware -VMName "YourVM_Name" |ForEach-Object {Set-VMFirmware -BootOrder ($_.Bootorder | Where-Object {$_.BootType -ne 'File'}) $_ }

Also an orphaned Microsoft Hyper-V Network Adapter #2, which I uninstalled from Device Manager.

Other than that, it looked lean & mean to me: no DVD, no other VHDs or ISOs. Auto-Startup set to 0 seconds delay.

I don’t generally use registry optimizers, but if someone has a legit one to suggest, I’ll have a look...

Maybe I should do an export from the Hyper-V panel and import it?

Did you manage to create a new Vm and attach the VHDX? If so, how did that go?

And finally, did you have another server you could restore the VM to and confirm if the boot time takes unexpectedly long still?

Thanks for the thoughts, yes I sure did. Details of that are in my post above (Dec 15th). TL;DR:No real change. I don’t readily have another server, that would take some doing, though not impossible. I might do an export/import to this same server if I get some time...