Solved

Upgraded 12.1 with previously installed ISO for Veeam Hardened Repository?


Userlevel 3

I built a Hardened Repository using the ISO provided here many months ago.  Very happy with it, until I decided to upgrade Veeam to 12.1.  Now it wants to update components on the Hardened Repository end or it will no longer be usable, which it can’t. 

 

Is there a workflow to deal with this or do I have to start all over again on the Hardened Repository side?

icon

Best answer by HannesK 12 December 2023, 10:31

View original

45 comments

Userlevel 5
Badge +2

Hello,
not sure which advice I should give… if you want to do it now, then you must enable SSH. If you have time to wait for CP1, then you don’t need SSH.

Best regards,
Hannes

Userlevel 7
Badge +17

FYI - Whatever user you’re logged in as, to see if you’re in the sudo group, you can simply type group and a list of groups you’re in will be displayed.

Userlevel 3

 

@kmcdermott -- You need to go in to Single-User mode for the VHR and turn on SSH.  Once you do that you can update the VHR via the console and then reboot to go back in to hardened mode.  I had to do this recently for an upgrade in my homelab - have one more VHR to do but understand the process now.  Here is an article for Single-User mode - Ubuntu Linux Defense: Secure Boot & Single User Mode (veeam.com)

Just follow this which I posted, and it will get the VHR updated.  I have done it already and it is fairly straightforward.

 

The issue when using the hardened ISO isn’t that the user doesnt have sudo rights, it’s that sudo is hevaily restricted as to which executables are allowed to be run. It’s limited to reboot and shutdown.

Exactly and why you need to use single user mode to update it until they come out with CP1.

The guide you linked makes no mention removing the sudoers restrictions, and is instead instructions on adding the account to sudoers, which does not solve the issue.

 

Additionally, it looks like the prebuilt hardening ISO has a script or service that, on startup, overwrites sudoers. Any attempt to change the file to remove the restrictions is lost on boot.

 

Userlevel 7
Badge +10

I had been recommending to people to use the Scale-out Backup Repository; there a construct called Seal Mode. Or Maintenance Mode that can help manage these “hard repos” → Watch episode 146 of the recap where I talk faster than I can write this advice.

Userlevel 3

Are you using the prebuilt hardened ISO that was provided from VeeamON? Your configuration sounds significantly different. By default there is no networking in single user mode and systemctl wont work without systemd which isn’t available since init hasn’t happened.

Yes, I used the ISO provided by Hannes to deploy my VHRs.  It works fine to update as I have noted using the Single-User mode.

 

Could you list the exact steps you take? I am not seeing anything like this at all, even after manually bringing an interface up and starting SSHD I still am having to manually edit sudoers so that the VBR console can connect.

 

It works, but it’s incredibly time consuming.

 

Quick edit:

 

I’m still very interested in the steps you take, ultimately this is what worked best for me without a lot of manual reconfiguration, bringing services up, etc.

 

  1. Boot into single user mode as mentioned in the previous link
  2. Edit /etc/sudoers to remove the restrictions:
    # Allow members of group sudo to execute any command
    %sudo   ALL=(ALL:ALL) ALL
    #%sudo   ALL=(root) NOEXEC: /usr/sbin/reboot, /usr/sbin/shutdown
  3. Set sudoers as immutable so that the script/service on startup cannot revert the changes:
    chattr +i /etc/sudoers
  4. Continue booting to get out of single user mode:
    exec /sbin/init
  5. Upgrade through VBR console
  6. Remove immutability flag:
    chattr -i /etc/sudoers
  7. Reboot so that you’re back in hardened mode
Userlevel 3

Confirmed with other repos that edits to sudoers do not survive the boot process. Additionally, while backups are successful to updated repositories, I receive warnings that files cannot set as immutable as the transport service is unavailable. Wondering if this is because of the new port requirement for 12.1 and the software firewall.

 

Edit: Okay, it looks like only the changes to the sudo group permissions inside of sudoers is reverted (the line starting with `%sudo`). If I instead add a line for the repository user specifically, it works as you describe.

Userlevel 7
Badge +17

I believe you need to temporarily re-enable SSH on the VHR @kmcdermott . Have you tried that yet?

Userlevel 3

It won’t let me; it says I’m not allowed to.  I’m assuming because its harderned.

 

Userlevel 7
Badge +17

Or, re-add the VHR user to sudo. After upgrade, you can remove it. I had to remember what was needed. We had discussed this in a previous thread several mos ago. You can review the post & thread comments here.

Userlevel 3

I only have the one account that I created when I made the repo.

If I run “sudo systemctl start ssh”, I get back the error that the user is “not allow to execute as root”.

If I try to run “usermod -a -G sudo user” I get “permission denied”. “cannot lock /etc/passwd”

 

 

Userlevel 3

Just for giggles, I ran “systemctl start ssh without the sudo and that worked.  Interesting.

New problem, I need to figure out how to create a single-use credential. 

Userlevel 7
Badge +17

Hmm...ok. Well, if your acct can make changes, you can create a new account by:
useradd -m <name-of-user> ; then add it to sudo: adduser <new-username> sudo 

The thing is though, since your VHR is already “working” (minus the upgrade part), you should already have a single-cred user you’re using. Search the passwd file to see if there is a “veeam user” in it (cat /etc/passwd).

Userlevel 3

Chris.Childerhose - I don’t see GRUB when I boot up.  It immediately goes to a blue screen that says “Hardened Repository”

Userlevel 7
Badge +8

If you don’t want to play with ssh, wait CP1 for V12.1 as described on r&d forum:

Upgrade of VBR to 12.1 with Hardened Repo - R&D Forums (veeam.com)

Maybe @HannesK could give some advices if you want to update now.

Userlevel 7
Badge +6

Or, re-add the VHR user to sudo. After upgrade, you can remove it. I had to remember what was needed. We had discussed this in a previous thread several mos ago. You can review the post & thread comments here.

I was just going to reference my thread but you beat me to it.  Shane and company got me through the same process when upgrading from 11 to 12.  I think that thread may lay it out pretty well but feel free to check back with questions.

Userlevel 3

 

@kmcdermott -- You need to go in to Single-User mode for the VHR and turn on SSH.  Once you do that you can update the VHR via the console and then reboot to go back in to hardened mode.  I had to do this recently for an upgrade in my homelab - have one more VHR to do but understand the process now.  Here is an article for Single-User mode - Ubuntu Linux Defense: Secure Boot & Single User Mode (veeam.com)

Just follow this which I posted, and it will get the VHR updated.  I have done it already and it is fairly straightforward.

 

The issue when using the hardened ISO isn’t that the user doesnt have sudo rights, it’s that sudo is hevaily restricted as to which executables are allowed to be run. It’s limited to reboot and shutdown.

Userlevel 7
Badge +21

 

@kmcdermott -- You need to go in to Single-User mode for the VHR and turn on SSH.  Once you do that you can update the VHR via the console and then reboot to go back in to hardened mode.  I had to do this recently for an upgrade in my homelab - have one more VHR to do but understand the process now.  Here is an article for Single-User mode - Ubuntu Linux Defense: Secure Boot & Single User Mode (veeam.com)

Just follow this which I posted, and it will get the VHR updated.  I have done it already and it is fairly straightforward.

 

The issue when using the hardened ISO isn’t that the user doesnt have sudo rights, it’s that sudo is hevaily restricted as to which executables are allowed to be run. It’s limited to reboot and shutdown.

Exactly and why you need to use single user mode to update it until they come out with CP1.

Userlevel 3

 

@kmcdermott -- You need to go in to Single-User mode for the VHR and turn on SSH.  Once you do that you can update the VHR via the console and then reboot to go back in to hardened mode.  I had to do this recently for an upgrade in my homelab - have one more VHR to do but understand the process now.  Here is an article for Single-User mode - Ubuntu Linux Defense: Secure Boot & Single User Mode (veeam.com)

Just follow this which I posted, and it will get the VHR updated.  I have done it already and it is fairly straightforward.

 

The issue when using the hardened ISO isn’t that the user doesnt have sudo rights, it’s that sudo is hevaily restricted as to which executables are allowed to be run. It’s limited to reboot and shutdown.

Exactly and why you need to use single user mode to update it until they come out with CP1.

The guide you linked makes no mention removing the sudoers restrictions, and is instead instructions on adding the account to sudoers, which does not solve the issue.

 

 

It does solve the issue to update the VHR as I have done this.  You remove the user from the sudo group once done and get it updated.  Then you reboot the VHR once again to revert it back to being hardened and not in Single-User mode.  It works and does fix the update issue.  I have now done it on about 12 machines we have running here without any problems.

So you’re pushing the update from the VBR console while still in single user mode?

Userlevel 3

I’m going to stop quoting because it’s making the replies gigantic, but there’s no systemd in single user mode. How are you initializing networking and starting SSH?

Userlevel 3

I got it worked out.  Thank you everyone for the help!

Userlevel 7
Badge +21

Steps are below -

  • Single user Mode Boot
  • visudo to edit the sudoers File
  • Added my service user veeam with "ALL"
  • Booted
  • Enabled SSH
  • Update Veeam Transport Role
  • Disabled SSH
  • Used visudo to comment out # my service user again
  • Reboot to get back to Hardened mode
Userlevel 3

So it seems like we’re basically following the same steps, but for some reason my modified sudoers does not survive the boot process. This is extremely weird. I’m going to try a few more immutable repos of mine and see if they exhibit similar behavior.

Userlevel 7
Badge +17

I don’t think think it’s the SSH service that’s the issue, but the user you’re using for VHR needs sudo rights re-added temporarily. If the acct you’re using to login with doesn’t have the ability to make “administrative” changes on the Linux server, you need to login with one that does.

Userlevel 7
Badge +17

To make administrative changes, you need to add “sudo” at the beginning of the command you’re trying to run. If the account your logged in with is not in the sudo group on the server, then it won’t work. You need to either login as root or login with an account with sudo permissions.

Userlevel 7
Badge +17

@kmcdermott the account you’re logged in with wasn’t given sudo rights then. Do you know the root account login credentials?

Comment