• EN

All-Demo Session & Installable ISO for Veeam Hardened Repository

All-Demo Session & Installable ISO for Veeam Hardened Repository
Rick Vanover
Userlevel 7
Badge +10

 

22-February 2024 Update:  GOOD NEWS! We are scheming a big update and overhaul of this page at or around VeeamON time. I can’t say more!

 

During the lead-up to VeeamON, it was amazing to see what content was leading the pre-registration information. We have the most data here than ever before. Constantly back and forth with the “Architecting Veeam Backup for Microsoft 365 at Scale” session, the session that @hannesk, Christoph Meyer and I are delivering was racing to the finish line as the leader. The “All Demo Session for Veeam Hardened Repository” was in the end the most popular pre-registered session, and as promised in the delivery in Miami, here is your everything you need to know guide.

Post Publish Updates:

  • 1-June 2023:  The installation requirements for the .ISO have been added.
  • 1-June 2023:  Video of installation of the .ISO has been added.
  • 2-June 2023:  Clarification on support, formatting of document.
  • 5-June 2023:  Updated .ISO with support banner and logon screen update, link updated. No changes to Ubuntu configuration or hardening script.
  • 29-June 2023: Updated .ISO with one enhancement, offline installation works.

Recording Resources

You can watch one of my practice runs of the session from Miami here:

At the VeeamON virtual event experience, we will have the replay from Miami session up very soon. 

Markdown File & Write Up

I have made a markdown file with a lot of the key information ready, you can download it here: rickvanover/VeeamHardenedRepoHub (github.com) Additionally - you can download the Markdown file at the QR code below:
 

This link goes to the VHR Markdown file that I’ve written up.

If you are not doing Markdown or GitHub you can download this as a PDF at the bottom of the page.

Overall advice

My overall advice on the Veeam Hardened Repository is to plan your deployment, this will greatly improve your experience with using the VHR. For example, I’m notorious for not documenting the options I put during the Ubuntu installation at this important stage:

 

Installable .ISO & Hardening Script

Key Update 18-December 2023: We have taken down the link for the .ISO, there will be a  new one coming “Soon” :)! 

One of the key parts of the Miami experience is the update we had from Product Management on new functionality for the Veeam hardened repository, which includes the Hardening Script and the installable .ISO. I recommend watching the replay to hear from Hannes and Christop directly before using the script and .ISO. We recommend at least 100 GB for storage. You can get both below:

  • Hardening Script: https://www.veeam.com/sys507
  • Installable .ISO: )18-December 2023 - We’ve removed this download and a new one is coming “soon”!)

About this installable .ISO…. The principal benefit of the installable .ISO is that it sets up much of the installation pre-defined *and* applies the above linked hardening script. 

Installable .ISO Requirements

For the installable .ISO, we recommend you follow these prerequisites (mainly relevant if you try to install it in a VM for lab purposes):

  • This is now optional from the 29-June update:  Internet connection (HTTP to the Ubuntu update servers for automatic security updates)
  • 2 CPUs, 6 GB RAM
  • UEFI boot is required (BIOS does not work) UEFI secure boot strongly recommended
  • 1 disk with at least 100GB for the operating system (plus cache files)
  • 100GB is a hard requirement. If that’s too high, please give feedback below.
  • Then there should be at least one second disk that is larger 100GB for backup data that would be the Veeam repository

During the installation, follow the wizard and reboot after the installation finished. Then install the hardened repository role. The VHR final layout will have all disks are mounted to /mnt/backup<n>

Important: Reboot again (this second reboot locks down the whole system and you can only reboot / shutdown with sudo. there is no root / sudo anymore at this stage) - this is the result of the hardening.  Think of this result as taking a massive green pill that tells this system its role in life forever on is to be a VHR.

After reading all of this, you are now ready to watch my video of installing the VHR with the .ISO:



Installable .ISO General Information & Support Statement

  • It's a community project. please don't call Veeam support for use of the .ISO. Once a VHR is deployed and configured in Veeam Backup & Replication, it would be subject to support. 
  • Automatic updates are enabled.
  • The user does not have permissions to update manually.
  • We plan to add a note to the banner message where to find more information about the ISO installer.

Get Started with the Veeam Hardened Repository 

Getting started with the VHR doesn’t have to be overwhelming. Give it a try - and let us know how it goes in the comments below.

1 Attachment
Twitter @RickVanover | Email: rick.vanover@veeam.com
This topic has been closed for comments

199 comments

T
Badge

If you enter the ‘normal’ command, the system reboots again. Right after pressing enter after typing the normal command press ESC multiple times until you see the grub menu.

see here:

https://askubuntu.com/questions/381613/how-to-return-from-grub-prompt-to-the-grub-menu

Got it, thanks!

E
Userlevel 1

@efd121Is the boot drive utilizing software RAID? I know a similar case and during the setup of a regular Ubuntu there where many additional steps necessary. Not sure if this can be automated.

No its not utilizing software RAID.  The OS drive is mirrored in the iDRAC as a virtual disk

E
Userlevel 1

I’ve been testing with some Dell Rx730’s and having issues with bonding.  It appears its setting the same MAC address each time.  Is there anything I can do before the 2nd reboot to prevent it or do I need to stay away from bonding?

 

Dave

I

HI, I am new to VHR and Linux. but I was able to set up the VHR using installable ISO. Now I need to add more disk to Linux VM and set them up as VHR. Can you tell me the steps?

Thanks. 

H
Userlevel 5
Badge +2

I’ve been testing with some Dell Rx730’s and having issues with bonding.  It appears its setting the same MAC address each time.  Is there anything I can do before the 2nd reboot to prevent it or do I need to stay away from bonding?

 

Dave

it’s a clean Ubuntu 20.04 and everything that works with Ubuntu 20.04 can be done manually. My guess is, that the bonding technology is wrong. https://www.veeam.com/blog/installing-ubuntu-linux-veeam-hardened-repository.html has a short summary. I guess active-backup should solve the problem

H
Userlevel 5
Badge +2

HI, I am new to VHR and Linux. but I was able to set up the VHR using installable ISO. Now I need to add more disk to Linux VM and set them up as VHR. Can you tell me the steps?

Thanks. 

VMs are against the concept of Hardened Repository, because an attacker could simply delete the whole VM. Manually adding disks is possible because it’s a plain Ubuntu. Without Linux experience, I would not do that. The idea of the ISO is, that all disks exist before installation (like with a physical appliance). Then we will auto-detect the disks and format everything correctly. So my suggestion would be to just re-install the system (not repair).

regnor
Userlevel 7
Badge +14

I did try to enable root access like described in Hannes blog post and it did work. Only issue I had was that with the ‘splash quiet’ option the system stopped and only showed the blue background; switching to terminal via STRG+ALT+FX didn’t work. Removing the option boots to the root shell.

Max M. | Systems Engineer @Veeam | VMCE/VMCA | former Legend & Vanguard
E
Userlevel 1

I’ve been testing with some Dell Rx730’s and having issues with bonding.  It appears its setting the same MAC address each time.  Is there anything I can do before the 2nd reboot to prevent it or do I need to stay away from bonding?

 

Dave

it’s a clean Ubuntu 20.04 and everything that works with Ubuntu 20.04 can be done manually. My guess is, that the bonding technology is wrong. https://www.veeam.com/blog/installing-ubuntu-linux-veeam-hardened-repository.html has a short summary. I guess active-backup should solve the problem

Installing from the non-hardened ISO works as expected, my test servers have unique MACs for the bond (active-backup).  When I install from the hardened ISO they both have the same MAC for the bond using active-backup.  Is is possible to have the system generate a unique MAC before the 2nd reboot when its totally locked down?

Dave

E
Userlevel 1

I’ve been testing with some Dell Rx730’s and having issues with bonding.  It appears its setting the same MAC address each time.  Is there anything I can do before the 2nd reboot to prevent it or do I need to stay away from bonding?

 

Dave

it’s a clean Ubuntu 20.04 and everything that works with Ubuntu 20.04 can be done manually. My guess is, that the bonding technology is wrong. https://www.veeam.com/blog/installing-ubuntu-linux-veeam-hardened-repository.html has a short summary. I guess active-backup should solve the problem

Installing from the non-hardened ISO works as expected, my test servers have unique MACs for the bond (active-backup).  When I install from the hardened ISO they both have the same MAC for the bond using active-backup.  Is is possible to have the system generate a unique MAC before the 2nd reboot when its totally locked down?

Dave

I tried deleting /etc/machine-id but that led to the system becoming unresponsive.

I edited the file, changing the first 2 and last 2 characters, this worked, both servers now have unique MACs for the bond (active-passive).  I assume it would work with other bonding options.

I think for my production systems I will update the first 8 characters with the MAC from one of the physical NICs to be extra certain I don’t end up with dupes.

Does anyone know if there are any other risks using this method?

Dave

Chris.Childerhose
Userlevel 7
Badge +20

For those that have tested on physical servers have you tried it with multiple networks - not bonding?  I need to possibly test this with two different networks possibly where the repo is segregated from our typical VLANs.

Chris Childerhose - VMCA2022 | VMCE2023 | Veeam Vanguard 7* | Veeam Legend 3* | vExpert 5* | VCAP-DCV/VCP-DCV | Object First Ace | Cisco Champion | Twitter: @cchilderhose | Blog Site – https://just-virtualization.tech
Rick Vanover
Userlevel 7
Badge +10

For those that have tested on physical servers have you tried it with multiple networks - not bonding?  I need to possibly test this with two different networks possibly where the repo is segregated from our typical VLANs.

I have not @Chris.Childerhose  → But curious how this goes as it is the standard Ubuntu network configuration at this point, too risky to make assumptions there with the installer.

Twitter @RickVanover | Email: rick.vanover@veeam.com
Chris.Childerhose
Userlevel 7
Badge +20

For those that have tested on physical servers have you tried it with multiple networks - not bonding?  I need to possibly test this with two different networks possibly where the repo is segregated from our typical VLANs.

I have not @Chris.Childerhose  → But curious how this goes as it is the standard Ubuntu network configuration at this point, too risky to make assumptions there with the installer.

Agreed. I am going to test with a VM since I don't have a physical box just yet 

Chris Childerhose - VMCA2022 | VMCE2023 | Veeam Vanguard 7* | Veeam Legend 3* | vExpert 5* | VCAP-DCV/VCP-DCV | Object First Ace | Cisco Champion | Twitter: @cchilderhose | Blog Site – https://just-virtualization.tech
Rick Vanover
Userlevel 7
Badge +10

For those that have tested on physical servers have you tried it with multiple networks - not bonding?  I need to possibly test this with two different networks possibly where the repo is segregated from our typical VLANs.

I have not @Chris.Childerhose  → But curious how this goes as it is the standard Ubuntu network configuration at this point, too risky to make assumptions there with the installer.

Agreed. I am going to test with a VM since I don't have a physical box just yet 

Suggestion when you do that - add different types of VMnic adapter types, VMXNET, E1000 or heck whatever is in there nowadays.

Twitter @RickVanover | Email: rick.vanover@veeam.com
Chris.Childerhose
Userlevel 7
Badge +20

For those that have tested on physical servers have you tried it with multiple networks - not bonding?  I need to possibly test this with two different networks possibly where the repo is segregated from our typical VLANs.

I have not @Chris.Childerhose  → But curious how this goes as it is the standard Ubuntu network configuration at this point, too risky to make assumptions there with the installer.

Agreed. I am going to test with a VM since I don't have a physical box just yet 

Suggestion when you do that - add different types of VMnic adapter types, VMXNET, E1000 or heck whatever is in there nowadays.

Yep can do that for sure. Keep you posted.

Chris Childerhose - VMCA2022 | VMCE2023 | Veeam Vanguard 7* | Veeam Legend 3* | vExpert 5* | VCAP-DCV/VCP-DCV | Object First Ace | Cisco Champion | Twitter: @cchilderhose | Blog Site – https://just-virtualization.tech
H
Userlevel 5
Badge +2

For those that have tested on physical servers have you tried it with multiple networks - not bonding?  I need to possibly test this with two different networks possibly where the repo is segregated from our typical VLANs.

what do you mean with “multiple networks”? Multiple network cards in different networks? Meaning you would create a way around firewalls? We won’t support that for sure (bad design)

Chris.Childerhose
Userlevel 7
Badge +20

For those that have tested on physical servers have you tried it with multiple networks - not bonding?  I need to possibly test this with two different networks possibly where the repo is segregated from our typical VLANs.

what do you mean with “multiple networks”? Multiple network cards in different networks? Meaning you would create a way around firewalls? We won’t support that for sure (bad design)

No one for management and then the other on a secure VLAN for data transfer would be the design.  We want to have the VHR on its own VLAN separate from what we use for Veeam.

Chris Childerhose - VMCA2022 | VMCE2023 | Veeam Vanguard 7* | Veeam Legend 3* | vExpert 5* | VCAP-DCV/VCP-DCV | Object First Ace | Cisco Champion | Twitter: @cchilderhose | Blog Site – https://just-virtualization.tech
H
Userlevel 5
Badge +2

that’s what I mean… the Hardened Repository in your case has access to two networks without any firewall in between. Why would you want to separate a few kbit/s management traffic if you have 10gbit/s+ anyway for data traffic? 

SSH is disabled, so that cannot be the reason.

Chris.Childerhose
Userlevel 7
Badge +20

that’s what I mean… the Hardened Repository in your case has access to two networks without any firewall in between. Why would you want to separate a few kbit/s management traffic if you have 10gbit/s+ anyway for data traffic? 

SSH is disabled, so that cannot be the reason.

Ah I got you. That makes sense. So no need to test just ensure it is on the separate VLAN we are going to use and ensure comms between that and VBR.  Thanks Hannes.

Chris Childerhose - VMCA2022 | VMCE2023 | Veeam Vanguard 7* | Veeam Legend 3* | vExpert 5* | VCAP-DCV/VCP-DCV | Object First Ace | Cisco Champion | Twitter: @cchilderhose | Blog Site – https://just-virtualization.tech
T

What would cause the ISO to fail before it even gets to the network configuration screen? The Network config screen pops up then disappears immediately and says there was an error. while behind that popup there is this line “subiquity/Drivers/_list_driver/wait_apt” with a spinning bar after it. 

This is on a Dell R730XD with UEFI on, Secureboot Enabled, a Raid1 OS array, and a Raid 6 backup array both configured in the Bios before attempting install. I have attempted install without any arrays configured and i get the failure at the same point. 

I have attempted once with an IP configured before the install to see if not seeing internet initially is the cause and that did not change anything.

Chris.Childerhose
Userlevel 7
Badge +20

What would cause the ISO to fail before it even gets to the network configuration screen? The Network config screen pops up then disappears immediately and says there was an error. while behind that popup there is this line “subiquity/Drivers/_list_driver/wait_apt” with a spinning bar after it. 

This is on a Dell R730XD with UEFI on, Secureboot Enabled, a Raid1 OS array, and a Raid 6 backup array both configured in the Bios before attempting install. I have attempted install without any arrays configured and i get the failure at the same point. 

I have attempted once with an IP configured before the install to see if not seeing internet initially is the cause and that did not change anything.

I am interested in the outcome of this as I am going to be testing a similar setup but on HPE DL380 G9 server but similar RAID configs, etc.

Chris Childerhose - VMCA2022 | VMCE2023 | Veeam Vanguard 7* | Veeam Legend 3* | vExpert 5* | VCAP-DCV/VCP-DCV | Object First Ace | Cisco Champion | Twitter: @cchilderhose | Blog Site – https://just-virtualization.tech
Rick Vanover
Userlevel 7
Badge +10

What would cause the ISO to fail before it even gets to the network configuration screen? The Network config screen pops up then disappears immediately and says there was an error. while behind that popup there is this line “subiquity/Drivers/_list_driver/wait_apt” with a spinning bar after it. 

This is on a Dell R730XD with UEFI on, Secureboot Enabled, a Raid1 OS array, and a Raid 6 backup array both configured in the Bios before attempting install. I have attempted install without any arrays configured and i get the failure at the same point. 

I have attempted once with an IP configured before the install to see if not seeing internet initially is the cause and that did not change anything.

Hey @Tyicus → Does ‘vanilla’ Ubuntu 20.04 behave the same?

Twitter @RickVanover | Email: rick.vanover@veeam.com
T

What would cause the ISO to fail before it even gets to the network configuration screen? The Network config screen pops up then disappears immediately and says there was an error. while behind that popup there is this line “subiquity/Drivers/_list_driver/wait_apt” with a spinning bar after it. 

This is on a Dell R730XD with UEFI on, Secureboot Enabled, a Raid1 OS array, and a Raid 6 backup array both configured in the Bios before attempting install. I have attempted install without any arrays configured and i get the failure at the same point. 

I have attempted once with an IP configured before the install to see if not seeing internet initially is the cause and that did not change anything.

Hey @Tyicus → Does ‘vanilla’ Ubuntu 20.04 behave the same?

Negative. Ubuntu 20.04.6 LTS gets past that point and proceeds with installation. Im able to set network configs and continue.

H
Userlevel 5
Badge +2

@Tyicus : hmm, that’s strange. Because efd121  reported above, that the R730xd installs fine. The problem he sees is that MAC addresses get duplicated with network bonding. And the reason for that is, that /etc/machine-id is identical on multiple machines.

T

@Tyicus : hmm, that’s strange. Because efd121  reported above, that the R730xd installs fine. The problem he sees is that MAC addresses get duplicated with network bonding. And the reason for that is, that /etc/machine-id is identical on multiple machines.

When i use the VHR iso, it doesnt even get to the Network configuration screen, so there isnt a network bond present for the MAC address to get duplicated, Unless im misunderstanding how this issue is happening. In the standalone Ubuntu 20.04 LTS install, i have a network bond configured and did not encounter the MAC duplication Issue.

Chris.Childerhose
Userlevel 7
Badge +20

Has anyone testing the ISO run the upgrade to the OS and did it break anything?  I know the ISO is based on 20.04 but when the first load of the VHR and log in you can see that there is an update to 22.04 OS which is newer.  I was wondering if the update is a breaking change.

I am testing the ISO on physical boxes now and I know my security team and host management team may want the device patched.  😁

Chris Childerhose - VMCA2022 | VMCE2023 | Veeam Vanguard 7* | Veeam Legend 3* | vExpert 5* | VCAP-DCV/VCP-DCV | Object First Ace | Cisco Champion | Twitter: @cchilderhose | Blog Site – https://just-virtualization.tech
Terms & Conditions