Solved

Upgraded 12.1 with previously installed ISO for Veeam Hardened Repository?


Userlevel 3

I built a Hardened Repository using the ISO provided here many months ago.  Very happy with it, until I decided to upgrade Veeam to 12.1.  Now it wants to update components on the Hardened Repository end or it will no longer be usable, which it can’t. 

 

Is there a workflow to deal with this or do I have to start all over again on the Hardened Repository side?

icon

Best answer by HannesK 12 December 2023, 10:31

View original

45 comments

Userlevel 5
Badge +2

Hello,
not sure which advice I should give… if you want to do it now, then you must enable SSH. If you have time to wait for CP1, then you don’t need SSH.

Best regards,
Hannes

Userlevel 7
Badge +20

@kmcdermott -- You need to go in to Single-User mode for the VHR and turn on SSH.  Once you do that you can update the VHR via the console and then reboot to go back in to hardened mode.  I had to do this recently for an upgrade in my homelab - have one more VHR to do but understand the process now.  Here is an article for Single-User mode - Ubuntu Linux Defense: Secure Boot & Single User Mode (veeam.com)

Just follow this which I posted, and it will get the VHR updated.  I have done it already and it is fairly straightforward.

Userlevel 7
Badge +6

Or, re-add the VHR user to sudo. After upgrade, you can remove it. I had to remember what was needed. We had discussed this in a previous thread several mos ago. You can review the post & thread comments here.

I was just going to reference my thread but you beat me to it.  Shane and company got me through the same process when upgrading from 11 to 12.  I think that thread may lay it out pretty well but feel free to check back with questions.

Userlevel 3

 

@kmcdermott -- You need to go in to Single-User mode for the VHR and turn on SSH.  Once you do that you can update the VHR via the console and then reboot to go back in to hardened mode.  I had to do this recently for an upgrade in my homelab - have one more VHR to do but understand the process now.  Here is an article for Single-User mode - Ubuntu Linux Defense: Secure Boot & Single User Mode (veeam.com)

Just follow this which I posted, and it will get the VHR updated.  I have done it already and it is fairly straightforward.

 

The issue when using the hardened ISO isn’t that the user doesnt have sudo rights, it’s that sudo is hevaily restricted as to which executables are allowed to be run. It’s limited to reboot and shutdown.

Userlevel 7
Badge +20

 

@kmcdermott -- You need to go in to Single-User mode for the VHR and turn on SSH.  Once you do that you can update the VHR via the console and then reboot to go back in to hardened mode.  I had to do this recently for an upgrade in my homelab - have one more VHR to do but understand the process now.  Here is an article for Single-User mode - Ubuntu Linux Defense: Secure Boot & Single User Mode (veeam.com)

Just follow this which I posted, and it will get the VHR updated.  I have done it already and it is fairly straightforward.

 

The issue when using the hardened ISO isn’t that the user doesnt have sudo rights, it’s that sudo is hevaily restricted as to which executables are allowed to be run. It’s limited to reboot and shutdown.

Exactly and why you need to use single user mode to update it until they come out with CP1.

Userlevel 3

 

@kmcdermott -- You need to go in to Single-User mode for the VHR and turn on SSH.  Once you do that you can update the VHR via the console and then reboot to go back in to hardened mode.  I had to do this recently for an upgrade in my homelab - have one more VHR to do but understand the process now.  Here is an article for Single-User mode - Ubuntu Linux Defense: Secure Boot & Single User Mode (veeam.com)

Just follow this which I posted, and it will get the VHR updated.  I have done it already and it is fairly straightforward.

 

The issue when using the hardened ISO isn’t that the user doesnt have sudo rights, it’s that sudo is hevaily restricted as to which executables are allowed to be run. It’s limited to reboot and shutdown.

Exactly and why you need to use single user mode to update it until they come out with CP1.

The guide you linked makes no mention removing the sudoers restrictions, and is instead instructions on adding the account to sudoers, which does not solve the issue.

 

Additionally, it looks like the prebuilt hardening ISO has a script or service that, on startup, overwrites sudoers. Any attempt to change the file to remove the restrictions is lost on boot.

 

Userlevel 7
Badge +20

 

@kmcdermott -- You need to go in to Single-User mode for the VHR and turn on SSH.  Once you do that you can update the VHR via the console and then reboot to go back in to hardened mode.  I had to do this recently for an upgrade in my homelab - have one more VHR to do but understand the process now.  Here is an article for Single-User mode - Ubuntu Linux Defense: Secure Boot & Single User Mode (veeam.com)

Just follow this which I posted, and it will get the VHR updated.  I have done it already and it is fairly straightforward.

 

The issue when using the hardened ISO isn’t that the user doesnt have sudo rights, it’s that sudo is hevaily restricted as to which executables are allowed to be run. It’s limited to reboot and shutdown.

Exactly and why you need to use single user mode to update it until they come out with CP1.

The guide you linked makes no mention removing the sudoers restrictions, and is instead instructions on adding the account to sudoers, which does not solve the issue.

 

 

It does solve the issue to update the VHR as I have done this.  You remove the user from the sudo group once done and get it updated.  Then you reboot the VHR once again to revert it back to being hardened and not in Single-User mode.  It works and does fix the update issue.  I have now done it on about 12 machines we have running here without any problems.

Userlevel 3

 

@kmcdermott -- You need to go in to Single-User mode for the VHR and turn on SSH.  Once you do that you can update the VHR via the console and then reboot to go back in to hardened mode.  I had to do this recently for an upgrade in my homelab - have one more VHR to do but understand the process now.  Here is an article for Single-User mode - Ubuntu Linux Defense: Secure Boot & Single User Mode (veeam.com)

Just follow this which I posted, and it will get the VHR updated.  I have done it already and it is fairly straightforward.

 

The issue when using the hardened ISO isn’t that the user doesnt have sudo rights, it’s that sudo is hevaily restricted as to which executables are allowed to be run. It’s limited to reboot and shutdown.

Exactly and why you need to use single user mode to update it until they come out with CP1.

The guide you linked makes no mention removing the sudoers restrictions, and is instead instructions on adding the account to sudoers, which does not solve the issue.

 

 

It does solve the issue to update the VHR as I have done this.  You remove the user from the sudo group once done and get it updated.  Then you reboot the VHR once again to revert it back to being hardened and not in Single-User mode.  It works and does fix the update issue.  I have now done it on about 12 machines we have running here without any problems.

So you’re pushing the update from the VBR console while still in single user mode?

Userlevel 7
Badge +20

 

@kmcdermott -- You need to go in to Single-User mode for the VHR and turn on SSH.  Once you do that you can update the VHR via the console and then reboot to go back in to hardened mode.  I had to do this recently for an upgrade in my homelab - have one more VHR to do but understand the process now.  Here is an article for Single-User mode - Ubuntu Linux Defense: Secure Boot & Single User Mode (veeam.com)

Just follow this which I posted, and it will get the VHR updated.  I have done it already and it is fairly straightforward.

 

The issue when using the hardened ISO isn’t that the user doesnt have sudo rights, it’s that sudo is hevaily restricted as to which executables are allowed to be run. It’s limited to reboot and shutdown.

Exactly and why you need to use single user mode to update it until they come out with CP1.

The guide you linked makes no mention removing the sudoers restrictions, and is instead instructions on adding the account to sudoers, which does not solve the issue.

 

 

It does solve the issue to update the VHR as I have done this.  You remove the user from the sudo group once done and get it updated.  Then you reboot the VHR once again to revert it back to being hardened and not in Single-User mode.  It works and does fix the update issue.  I have now done it on about 12 machines we have running here without any problems.

So you’re pushing the update from the VBR console while still in single user mode?

Yes.  Then when done remove from sudo and reboot to go back to VHR mode from single-user.

Userlevel 3

I’m going to stop quoting because it’s making the replies gigantic, but there’s no systemd in single user mode. How are you initializing networking and starting SSH?

Userlevel 7
Badge +20

I’m going to stop quoting because it’s making the replies gigantic, but there’s no systemd in single user mode. How are you initializing networking and starting SSH?

I have not had to do anything with networking in this mode as you should have it for VBR communication.  I then just run the command that starts the SSH service and it starts.  Update from Veeam console, remove from sudo group and reboot.  I use - systemctl for SSH service.

Userlevel 3

Are you using the prebuilt hardened ISO that was provided from VeeamON? Your configuration sounds significantly different. By default there is no networking in single user mode and systemctl wont work without systemd which isn’t available since init hasn’t happened.

Userlevel 7
Badge +10

I had been recommending to people to use the Scale-out Backup Repository; there a construct called Seal Mode. Or Maintenance Mode that can help manage these “hard repos” → Watch episode 146 of the recap where I talk faster than I can write this advice.

Userlevel 7
Badge +20

Are you using the prebuilt hardened ISO that was provided from VeeamON? Your configuration sounds significantly different. By default there is no networking in single user mode and systemctl wont work without systemd which isn’t available since init hasn’t happened.

Yes, I used the ISO provided by Hannes to deploy my VHRs.  It works fine to update as I have noted using the Single-User mode.

Userlevel 3

Are you using the prebuilt hardened ISO that was provided from VeeamON? Your configuration sounds significantly different. By default there is no networking in single user mode and systemctl wont work without systemd which isn’t available since init hasn’t happened.

Yes, I used the ISO provided by Hannes to deploy my VHRs.  It works fine to update as I have noted using the Single-User mode.

 

Could you list the exact steps you take? I am not seeing anything like this at all, even after manually bringing an interface up and starting SSHD I still am having to manually edit sudoers so that the VBR console can connect.

 

It works, but it’s incredibly time consuming.

 

Quick edit:

 

I’m still very interested in the steps you take, ultimately this is what worked best for me without a lot of manual reconfiguration, bringing services up, etc.

 

  1. Boot into single user mode as mentioned in the previous link
  2. Edit /etc/sudoers to remove the restrictions:
    # Allow members of group sudo to execute any command
    %sudo   ALL=(ALL:ALL) ALL
    #%sudo   ALL=(root) NOEXEC: /usr/sbin/reboot, /usr/sbin/shutdown
  3. Set sudoers as immutable so that the script/service on startup cannot revert the changes:
    chattr +i /etc/sudoers
  4. Continue booting to get out of single user mode:
    exec /sbin/init
  5. Upgrade through VBR console
  6. Remove immutability flag:
    chattr -i /etc/sudoers
  7. Reboot so that you’re back in hardened mode
Userlevel 3

I got it worked out.  Thank you everyone for the help!

Userlevel 7
Badge +20

Steps are below -

  • Single user Mode Boot
  • visudo to edit the sudoers File
  • Added my service user veeam with "ALL"
  • Booted
  • Enabled SSH
  • Update Veeam Transport Role
  • Disabled SSH
  • Used visudo to comment out # my service user again
  • Reboot to get back to Hardened mode
Userlevel 3

So it seems like we’re basically following the same steps, but for some reason my modified sudoers does not survive the boot process. This is extremely weird. I’m going to try a few more immutable repos of mine and see if they exhibit similar behavior.

Userlevel 3

Confirmed with other repos that edits to sudoers do not survive the boot process. Additionally, while backups are successful to updated repositories, I receive warnings that files cannot set as immutable as the transport service is unavailable. Wondering if this is because of the new port requirement for 12.1 and the software firewall.

 

Edit: Okay, it looks like only the changes to the sudo group permissions inside of sudoers is reverted (the line starting with `%sudo`). If I instead add a line for the repository user specifically, it works as you describe.

Userlevel 7
Badge +11

I believe you need to temporarily re-enable SSH on the VHR @kmcdermott . Have you tried that yet?

Yeah! Its possible to start SSH. You need to inactive ufw temporarly too.

Comment