For the Agent backup are you turning on Application Aware processing for the job?  That is the recommended setting especially for recovery of servers like AD.

Short answer, yes. More detail, on the original tests with the old software, I don’t think there was a check box for Application Aware processing. That version actually did work twice. But as I upgraded versions I checked that box when it became available.

Hi @clabrown , I had this issue a few times, and it was some times a bit random, due to different esxi versions, application awareness on/off, databases…. etc.

What we did at the end was simple, but effective, 

Due to you are moving vms from one host to another, I suggest you to make yourself with a trial key for your old ESXi, license it properly, and perform the backups with Veeam correctly.

then, you will be able to easily restore them into the new Infra.

Another possibility will be to use the vm Quick Migration option, which works great, but as I said, the ESXis must be licenses.

cheers.

interesting … not sure I can get a license key for a host from 2014 “properly” from vmware. Also not sure what you mean by performing the backups “properly” since it seems that, that’s what I was doing? Please explain? we’re low budget and not gonna be able to purchase licensees from vmware for their free products.?

Thanks!

I’ve looked at the VMWare site, and don’t see anyway to “make a trial license key” for my old version 5.5 of ESXi. Please let me know if you know a “trick” to it. I see some links to end of life products with selections for versions 6 through 4 but only the 6 and 4 versions work and they take me to a download page. Just my luck, none of the 5.0, 5.5 etc versions work, they just take me back up to the beginning of the selection process.

The ESXi host does have a license installed, which it says never expires, but I think it’s just the “free” license for evaluation, and it’s past the eval period by quite a few years (LOL)

Product: VMware vSphere 5 Hypervisor Licensed for 1 physical CPUs (unlimited cores per CPU)
License Key: (I deleted the license text before posting this)
Expires: Never

Product Features:
    Up to 8-way virtual SMP

Using 5.5 for VMware is more than likely the issue if you are using a much newer version of Veeam CE.  Check the support matrix for Veeam - KB2443: Veeam Backup & Replication support for VMware vSphere

Regarding your original issue. You have a 2012 essentials server which is running the Active Directory Domain services? And after restoring it, does it only boot into safe mode? 

If so, this is expected. Veeam does restore domain controllers in nonauthoritative mode. So afterwards there will be some additional steps you need to take: https://www.veeam.com/kb2119

Would that matter in this case? Since I’m using the free versions of both Veeam and VMWare I’m unable to backup the VM as a VM from the inside the Virtual Infrastructure section of Veeam B&R. Instead I’m backing it up as a manually added machine inside Physical Infrastructure. My understanding is that backups in the Virtual Infrastructure section backup the Virtual machine settings in addition to the Operating System and Files of the machine, where as in the Physical Infrastructure section, it only backs up at the Operating System level and doesn’t concern itself with the underlying VM or PC hardware.

Well both the agent and VM based backup do process domain controllers in the same way, regardless if the underlying platform. Although I'm not sure how the free agent behaves without application aware processing. That's why I'm asking how the restored server behaved. Did it go to AD recovery mode? Or did you see an error while booting?

Well, I started by brute force copying the files from the VM’s folder to the new server, ?registering? the VM and that worked fine. Next, with the old version of Veeam Agent for Windows that had been backing up the server for years (no Backup and Replication server, just the agent), I did 2 successful bare metal restores, the server booted normally. Next I did a 3rd restore in the exact same way and it booted, sort of presented the login screen, but then said there was no login authority available, (I think that’s how it was phrased) Then  said it was going to fix that, churned for quite a while, rebooted once or twice and and came up in Recovery Mode. I tried a couple more times with new backups thinking maybe that backup was corrupt, with the same results. Then I started upgrading Veeam Agent, and eventually installed the B&R console on a different VM.

The Bare Metal Restore from the latest version was a little different. The status window said the restore was 100% complete with a nice green check, but then registered a couple of failed lines … sadly those error messages were too long for the status window, I was unable to enlarge or scroll the status window to actually read the meat of the error messages, and I was unable to find a separate log file to read the error within the Bare Metal recovery environment. At that point it as late on a Sunday night, and I didn’t try booting that version of the restore, and just reconfigured the systems so the old server was back online running normally for Monday Morning, so I don’t know how that system would have booted.

Somewhere in the process of all this I installed a new 2022 Standard server, joined it to the old server’s domain and configured it as an additional domain controller for that domain. I was able to add test users on each of the servers and see them show up on the other server, so that seemed successful. With that second domain controller on the network maybe when the restored server boots it will just update it’s database and work?

Sorry for being long winded, just not sure what detail might be important.

Thank you.

If it was the only domain controller for your environment, you would need to take some additional steps afterwards. But now, with your second DC online, it should resolve by itself.

In this scenario, a simple non-authoritative restore is sufficient to recover.

Restore the Domain Controller. The steps taken by Application-Aware Processing during the backup process will cause the Domain Controller to be restored in a non-authoritative state. The restored Domain Controller will first boot into Directory Services Restore Mode (DSRM) mode, then automatically reboot into a normal state. The Domain Controller will know that it has been recovered from backup and will allow the other DC's in the environment to update it.

regnor:

Thanks for explaining the DC ramifications. You were exactly right. With the second DC available it did exactly that, rebooted on it’s own 2 or 3 times then presented a login prompt and worked normally. My level of relief at that moment was vast, Thank YOU!. So that part of the migration, moving the old server VM to the new ESXi Host (fingers crossed etc) seems to be functioning properly. The workstations all automatically reconnected to the Shares and the Legacy Database app when it finished booting up.

This doesn’t explain why the first couple restores I did with the very old Veeam Windows Agents (maybe V3 series?) did work without issue, and then why with no changes they stopped working.

Also it would be nice for people like me who don’t spend our days restoring Domain Controllers, if the Veeam Bare Metal Restore would explicitly state that it was going to demote the Domain Controller, or give the option to just do it anyway. Coming up in a “broken” non-usable state with only cryptic error messages was NOT helpful. Searching those error messages led me not to “Demoted Domain Controller” but into Domain Data Base Corruption articles, which made me think the back or restore process had corrupted the AD database… or that there were significant issues lurking in the old server preventing a good backup of the AD database.

REALLY appreciate the insight.

Thanks for your feedback, I’m glad that it worked in the end @clabrown 

Maybe initially you didn’t see this behaviour because the Veeam agent didn’t prepare the domain controller for restore; maybe application aware processing was disabled.

Veeam restores domain contollers in that way to prevent any issues in AD from restoring old states.  I do understand that this can cause some trouble, if you’re not aware of it, but on the other hand it’s the safest way of restoring a DC.

As a follow on question. In Veeam B&R I now have 2 computers in the inventory with the same OLD Server name. The Backup job in question only has 1 computer listed, the name of the OLD Server. When the backup job runs it fails because it apparently remembers the UUIDs of both the OLD Server on the OLD Host and the “new” OLD Server on the NEW Host and tries to back up both of them separately even though they both have the same IP and there is only 1 computer in the Backup Job Configuration. The Old version is disconnected from the network. It does successfully backup the new one, but I have to dig into the the report to see that because the job is listed as failed since it does not backup the old disconnected one:

12/13/2023 12:00:12 AM :: Building hosts list  
12/13/2023 12:00:15 AM :: Processing GBRSVR12 Error: Failed to process GBRSVR12: BIOS UUID has been modified, please rescan the host  
12/13/2023 12:00:25 AM :: Processing GBRSVR12  

I have rescanned both copies of that server in the inventory but rescanning doesn’t seem to do anything. I have run individual “Quick Backups” on each copy of that server in the inventory and think I know which one is the old one. I can’t remove that copy from the inventory because it says it’s used in a job.

I could probably just delete the job and start over, but don’t want to loose my existing backup chain, or force a full backup during the middle of the week. Any suggestions to get Veeam to realize that there is only 1 computer with that name and IP on the network?

Thanks.

Are you able to create a new backup job and map the backup to the existing backup repository chain?  I’m a little fuzzy on the details here, but generally you should be able to map the backup job.  If you were using a vCenter, then there is an ID that would likely have changed, but since you’re not using a vCenter since you’re on the ESXI free licenses, I’d check into the mapping.  The Copy job info goes a bit deeper into mapping, but number 3 on the second link below addresses mapping in backup jobs.

https://helpcenter.veeam.com/docs/backup/vsphere/backup_copy_mapping_file_job.html?ver=120

https://helpcenter.veeam.com/docs/backup/vsphere/backup_job_storage_vm.html?ver=120

I fear you'll have to create a new full backup after restoring/migrating the server. For Veeam it is a new system and you cannot continue the existing backup chain.