As much as many of us would love to help and figure this one out I am going to suggest that a Support ticket be opened to determine what the issue is.

Have you tried to edit the Proxy or Repo server in Veeam to go through the wizard or does the error you posted prevent that due to the OS not booting?

I am not a Linux expert but know enough to be dangerous.  Typically updating the OS should not cause an issue with Veeam so you will need to sort that out before Veeam will work again.  In the case of the Proxy servers you could deploy new ones with the updates already installed.  The Repos will be a harder thing to replace.

I’ve not seen that error before @gmattbrown . If only the issue was Proxy-related rather than Repo… 

I know you may not want to do so..but you could detach your repo Volumes in your Lnx Repo, recreate a Lnx VM with patches, then re-add them and add a new one to Veeam. Of course, this would mean having to disable your Jobs and removing your Repo and Lnx Server from Veeam initially. Kind of a pain for sure...but about the only thing I can think of since Support isn’t really willing to assist.

@gmattbrown 

Hi, Have a look at this post https://askubuntu.com/questions/1439112/failed-to-start-default-target-transaction-for-graphical-target-start-is-destru, looks the same message, perhaps you can fix your issue.

Andre Pulia

I was just about to say, check mounts and ip tables. 

Update on my situation.

I was able to modify the /etc/fstab file to get the repo VM to boot without it screaming at me and going into emergency mode.

The problem now - I think - is that there *was* a drive mounted that pointed to the external NetApp array we have that should be holding all the backups.

That drive/mount point is just gone. I want to say it is because that is what I had to comment out of the fstab file to get the repo VM to boot properly. I highlighted it below.

administrator@veeamrepo-edw:~$ cat /etc/fstab
# /etc/fstab: static file system information.
#
# Use 'blkid' to print the universally unique identifier for a
# device; this may be used with UUID= as a more robust way to name devices
# that works even if disks are added and removed. See fstab(5).
#
# <file system> <mount point>   <type>  <options>       <dump>  <pass>
# / was on /dev/ubuntu-vg/ubuntu-lv during curtin installation
/dev/disk/by-id/dm-uuid-LVM-hKPBa7n51jMbBFC0DLlQlugRuASFVqlSxq544ZzaDDUMVVuvk0XFX086fU0Llq6x / ext4 defaults 0 1
# /boot was on /dev/sda2 during curtin installation
/dev/disk/by-uuid/2dcc3f50-b51a-4fb6-891b-e0da3b50c963 /boot ext4 defaults 0 1
/swap.img       none    swap    sw      0       0
# /dev/veeam-vg/veeam-lg        /home/administrator/veeamrepo   xfs     defaults        0       0
administrator@veeamrepo-edw:~$ blkid
/dev/mapper/ubuntu--vg-ubuntu--lv: UUID="10c9ed7e-a705-4837-9a9b-afb08c4c88a0" BLOCK_SIZE="4096" TYPE="ext4"
/dev/sda2: UUID="2dcc3f50-b51a-4fb6-891b-e0da3b50c963" BLOCK_SIZE="4096" TYPE="ext4" PARTUUID="94fba7e1-ff79-435d-84e6-9e6792f0fc16"
/dev/sda3: UUID="F8esYk-gSCj-P13A-vxIn-M3xY-2A6n-c1uDlv" TYPE="LVM2_member" PARTUUID="95235f35-7573-49cd-8ae6-dea94fb0b363"
administrator@veeamrepo-edw:~$

I do have a support case open with Veeam, but the agents I have been working with don’t seem to have much better ideas than I do.

Well, first off, you shouldn’t use a direct path in your fstab file. The Volume’s UUID is what is preferred. Can you re-add your Volume mount point, save your fstab file, then run mount -a and see if your Repo is retained? Rescan your Repo in Veeam then see if your Job works.

You should be able to retrieve the Repo Volume UUID by running: ls -lha /dev/disk/by-uuid 

Sorry, again my lack of Linux experience is showing.

Are you saying to uncomment this in fstab:

# /dev/veeam-vg/veeam-lg        /home/administrator/veeamrepo   xfs     defaults        0       0

Then save fstab.

Then run mount -a.

Check if repo is retained (I don’t know how I would do this.)

Rescan repo in VB&R.

Our replies crossed there.

Here’s what I get from that command.

administrator@veeamrepo-edw:~$ ls -lha /dev/disk/by-uuid
total 0
drwxr-xr-x 2 root root 100 Nov 12 12:06 .
drwxr-xr-x 7 root root 140 Nov 12 12:05 ..
lrwxrwxrwx 1 root root  10 Nov 12 12:05 10c9ed7e-a705-4837-9a9b-afb08c4c88a0 -> ../../dm-0
lrwxrwxrwx 1 root root  10 Nov 12 12:05 2dcc3f50-b51a-4fb6-891b-e0da3b50c963 -> ../../sda2
lrwxrwxrwx 1 root root  10 Nov 12 12:06 e2697b5d-d070-4b4d-96ab-76798cf730e0 -> ../../dm-2
 

Well, you are a hero for that.

I did just that, and after a rescan of the repo it jumped backup to having 20.5 TB free.

I restarted one of our backup jobs, and it throws a bad username/password error. But we’re very much closer now.

Thanks!

Well...yes, but I was recommending using UUID instead of the mount point path. But it works too. Glad to help you get a bit farther.

The username/pwd is the account with access to the Veeam Backup mount point directory.

Hi @gmattbrown , how is you machine going, is everything OK now?

Oh, heavens no.

I thought that a restart of the Windows VM that is running the VB&R console might break something loose. Jobs were failing with that username/password error.

Now after the restart, the console doesn’t even recognize that there was a vCenter server connected to it.

Rescanning throws an error about “failed to retrieve object hierarchy.”

I am at an utter loss now. Waiting to see if Veeam support has any input on what could be causing this.

Did you try a rescan of the vCenter? Re-enter the VC credentials? 

You didn't clarify the user/pwd error so assumed it came from the Lnx Repo. 

On a complete lark, I updated the vCenter credentials for our “veeam user” account on both the vCenter and in VB&R.

Backup jobs are now running successfully, at least for the moment. I have 3% progress on one, and 10% on another. They never got that far previously.

Thanks to everyone here for helping out! I’ll check back in tomorrow to give an update on how all the jobs went haha.

The scenario is weird, it seems to me that there are problems in several components.
I would suggest that you start validating your infrastructure, check the
a) NTP of the servers and storage, etc.
b) check if the reverse name resolution is OK in all the components involved. (vCenter, BKP SRV, repos, proxies, your notebook, etc.)
After that, try to connect again

Ok...glad to hear. Keep us posted. 

Ok...glad to hear too . Keep us posted.

Nice to hear fixing the VC credentials helped.  Keep us posted to ensure the jobs complete.

Morning update -

The backup and replication jobs each ran successfully. Huge success!

Only bad news is that the backup copy job we have that copies the backups to a secondary repo failed.

I’m wondering if it isn’t because when it ran there were actually no backups to copy, but that’s just me guessing.

Error message itself was:

11/13/2024 12:58:57 AM :: Error: boost::filesystem::status: Permission denied: "/home/administrator/veeamrepo/backups/Test Backup Job/OSTICKET.vm-1103D2024-11-06T190702_0943.vib"
Failed to open storage for read access. Storage: oOSTICKET.vm-1103D2024-11-06T190702_0943.vib].
Failed to restore file from local backup. VFS link: summary.xml]. Target file: MemFs://RestoreText_{99b95828-511a-4d1b-bbee-de1a3f59f80b}]. CHMOD mask: 0].
Agent failed to process method {DataTransfer.RestoreText}.
  
 

The backup copy job do you have it in Immediate Mode or Periodic (Pruning)?  It is trying to open the location of the files to copy and there is a permissions issue.  This could also be another job running locking the files as well so I would start there.

Glad to hear ​@gmattbrown !

Agree with Chris here on the Copy Job issue.

The copy jobs are in pruning mode.

I did a re-rescan of both the vCenter where the secondary repo is, along with the repo itself.

Jobs are running now, just waiting for them to fully complete.

I need a drink. Cheers to all of you, I’d buy you one if I could.

Great to hear you got things going again.  Keep us update to know they complete.  Cheers

Also be sure to go through all posts and whichever one helped the most answer your question mark that as best answer even if you feel it was your own reply.