All-Demo Session & Installable ISO for Veeam Hardened Repository

All-Demo Session & Installable ISO for Veeam Hardened Repository
Userlevel 7
Badge +10

 

22-February 2024 Update:  GOOD NEWS! We are scheming a big update and overhaul of this page at or around VeeamON time. I can’t say more!

 

During the lead-up to VeeamON, it was amazing to see what content was leading the pre-registration information. We have the most data here than ever before. Constantly back and forth with the “Architecting Veeam Backup for Microsoft 365 at Scale” session, the session that @hannesk, Christoph Meyer and I are delivering was racing to the finish line as the leader. The “All Demo Session for Veeam Hardened Repository” was in the end the most popular pre-registered session, and as promised in the delivery in Miami, here is your everything you need to know guide.

Post Publish Updates:

  • 1-June 2023:  The installation requirements for the .ISO have been added.
  • 1-June 2023:  Video of installation of the .ISO has been added.
  • 2-June 2023:  Clarification on support, formatting of document.
  • 5-June 2023:  Updated .ISO with support banner and logon screen update, link updated. No changes to Ubuntu configuration or hardening script.
  • 29-June 2023: Updated .ISO with one enhancement, offline installation works.

Recording Resources

You can watch one of my practice runs of the session from Miami here:

At the VeeamON virtual event experience, we will have the replay from Miami session up very soon. 

Markdown File & Write Up

I have made a markdown file with a lot of the key information ready, you can download it here: rickvanover/VeeamHardenedRepoHub (github.com) Additionally - you can download the Markdown file at the QR code below:
 

This link goes to the VHR Markdown file that I’ve written up.

If you are not doing Markdown or GitHub you can download this as a PDF at the bottom of the page.

Overall advice

My overall advice on the Veeam Hardened Repository is to plan your deployment, this will greatly improve your experience with using the VHR. For example, I’m notorious for not documenting the options I put during the Ubuntu installation at this important stage:

 

Installable .ISO & Hardening Script

Key Update 18-December 2023: We have taken down the link for the .ISO, there will be a  new one coming “Soon” :)! 

One of the key parts of the Miami experience is the update we had from Product Management on new functionality for the Veeam hardened repository, which includes the Hardening Script and the installable .ISO. I recommend watching the replay to hear from Hannes and Christop directly before using the script and .ISO. We recommend at least 100 GB for storage. You can get both below:

  • Hardening Script: https://www.veeam.com/sys507
  • Installable .ISO: )18-December 2023 - We’ve removed this download and a new one is coming “soon”!)

About this installable .ISO…. The principal benefit of the installable .ISO is that it sets up much of the installation pre-defined *and* applies the above linked hardening script. 

Installable .ISO Requirements

For the installable .ISO, we recommend you follow these prerequisites (mainly relevant if you try to install it in a VM for lab purposes):

  • This is now optional from the 29-June update:  Internet connection (HTTP to the Ubuntu update servers for automatic security updates)
  • 2 CPUs, 6 GB RAM
  • UEFI boot is required (BIOS does not work) UEFI secure boot strongly recommended
  • 1 disk with at least 100GB for the operating system (plus cache files)
  • 100GB is a hard requirement. If that’s too high, please give feedback below.
  • Then there should be at least one second disk that is larger 100GB for backup data that would be the Veeam repository

During the installation, follow the wizard and reboot after the installation finished. Then install the hardened repository role. The VHR final layout will have all disks are mounted to /mnt/backup<n>

Important: Reboot again (this second reboot locks down the whole system and you can only reboot / shutdown with sudo. there is no root / sudo anymore at this stage) - this is the result of the hardening.  Think of this result as taking a massive green pill that tells this system its role in life forever on is to be a VHR.

After reading all of this, you are now ready to watch my video of installing the VHR with the .ISO:



Installable .ISO General Information & Support Statement

  • It's a community project. please don't call Veeam support for use of the .ISO. Once a VHR is deployed and configured in Veeam Backup & Replication, it would be subject to support. 
  • Automatic updates are enabled.
  • The user does not have permissions to update manually.
  • We plan to add a note to the banner message where to find more information about the ISO installer.

Get Started with the Veeam Hardened Repository 

Getting started with the VHR doesn’t have to be overwhelming. Give it a try - and let us know how it goes in the comments below.


This topic has been closed for comments

199 comments

Userlevel 7
Badge +10

Hi @Chris.Palmisano.26 

RE: Automatic Updates: Then they will fail/run without the ability to check.

 

RE: update manually:  This isn’t panned at this time as far as I know.

 

Will share with everyone a new .ISO is coming soon with some additional enhancements.

Userlevel 7
Badge +20

Hi @Chris.Palmisano.26 

RE: Automatic Updates: Then they will fail/run without the ability to check.

 

RE: update manually:  This isn’t panned at this time as far as I know.

 

Will share with everyone a new .ISO is coming soon with some additional enhancements.

Ooooh - new enhancements.  🤔

Hi @Chris.Palmisano.26 

RE: Automatic Updates: Then they will fail/run without the ability to check.

​​​​​RE: update manually:  This isn’t panned at this time as far as I know.


​​So just to be clear then without internet connectivity we won’t be able to patch these systems?


Will share with everyone a new .ISO is coming soon with some additional enhancements.

​​​​​Look forward to the new enhancements

@Rick Vanover  thanks for the information and quick response!

 

Userlevel 5
Badge +2

HTTP proxy is the only option today

Do you have a Ubuntu mirror on-prem where you would have all security updates? If yes, then we could think about allowing to select APT sources and you can point to your on-prem repository.

Userlevel 7
Badge +20

@Rick Vanover @HannesK --- I am trying to run the ISO file in one of my DC environments to create 3 VHRs.  It crashes and never seems to complete with either of the ISO files.  See the message below but is it due to CPU?  I have the VM set up with 16 vCPU and 32GB of RAM.  Maybe I just need 8x16?

 

Userlevel 7
Badge +10

It seems like plenty of resources @Chris.Childerhose  → What does the plain Ubuntul 20.04 install media do on the same equipment?

Userlevel 7
Badge +20

It seems like plenty of resources @Chris.Childerhose  → What does the plain Ubuntul 20.04 install media do on the same equipment?

The plain one works fine.  I am wondering if it has anything to do with having to use a VLAN?  When I do this, it creates another entry - so does the IP then go on the main NIC or the new VLAN one that is associated to the primary?

Userlevel 7
Badge +20

Here is a cleaner screenshot of my last attempt after scaling down to 8 vCPU leaving 32GB of RAM -

 

Userlevel 7
Badge +10

It seems like plenty of resources @Chris.Childerhose  → What does the plain Ubuntul 20.04 install media do on the same equipment?

The plain one works fine.  I am wondering if it has anything to do with having to use a VLAN?  When I do this, it creates another entry - so does the IP then go on the main NIC or the new VLAN one that is associated to the primary?

It seems it is a VM, but you are passing the VLAN assignment thru to the install?

Userlevel 7
Badge +20

It seems like plenty of resources @Chris.Childerhose  → What does the plain Ubuntul 20.04 install media do on the same equipment?

The plain one works fine.  I am wondering if it has anything to do with having to use a VLAN?  When I do this, it creates another entry - so does the IP then go on the main NIC or the new VLAN one that is associated to the primary?

It seems it is a VM, but you are passing the VLAN assignment thru to the install?

Yes - at the network screen I select the NIC and create the VLAN which creates another entry on the screen.  Then I just add the IP required to the first entry but not the second VLAN entry.  If you want a screenshot I can do that or jump on a call?

Userlevel 7
Badge +20

@Rick Vanover This is the screen after the VLAN creation.  You can see the IP is blurred out but in the first entry so does it go there or the VLAN created entry?

 

So do you disable the first entry and enable the VLAN entry to put the IP?

Userlevel 7
Badge +20

@Rick Vanover -- so I just ran the plain Ubuntu ISO just to confirm and that is working without crashing so far.  Not sure what it is with the custom ISO for the VHR.

Install completed with standard ISO for Ubuntu. 😪

Userlevel 7
Badge +20

@Rick Vanover - chalk it up to Friday, fried brain, or whatever but my always preaching the “RTFM” I did not take my own advice.  When I created my VMs to do this with I did not change the BIOS settings to EFI (Secure).  So once that change was made, reboot from ISO and work through the install it worked this time.

Happy Friday and always RTFM!  🤣😋

Userlevel 7
Badge +10

@Chris.Childerhose → Makes me think of an easier way to test it as a Virtual Machine, here is a .VBK of a VM that I use in the lab. It has 2 drives thin provisioned, OS drive is 200 GB, Data Drive is 24TB. This VM would need network to be re-assigned from “Lab VMs” to your network, but this has the VM settings correct for testing purposes. you can download it from the same folder as the .ISO, in this folder:

#VBKsOverOVAs

Userlevel 7
Badge +20

@Chris.Childerhose → Makes me think of an easier way to test it as a Virtual Machine, here is a .VBK of a VM that I use in the lab. It has 2 drives thin provisioned, OS drive is 200 GB, Data Drive is 24TB. This VM would need network to be re-assigned from “Lab VMs” to your network, but this has the VM settings correct for testing purposes. you can download it from the same folder as the .ISO, in this folder:

#VBKsOverOVAs

Thanks for that Rick but I figured out the issue with EFI boot and Secure option being checked in the VM.  I left BIOS boot on - duh.

This will be a nice way to test things though so will download the VBK file.  👍🏼

Userlevel 1
Badge

I have two VHR images, both deployed in ESXi (An 8.0 and 7.0). After the initial reboot, everything works, but I can’t SSH in anymore AND I can’t input anything via the ESXi console even trying VMRC. Both installs have this issue. Basically, if I need to access the VHR, I’m boned. Is this the intended behavior?

 

Userlevel 7
Badge +20

I have two VHR images, both deployed in ESXi (An 8.0 and 7.0). After the initial reboot, everything works, but I can’t SSH in anymore AND I can’t input anything via the ESXi console even trying VMRC. Both installs have this issue. Basically, if I need to access the VHR, I’m boned. Is this the intended behavior?

 

That is the intended behaviour for this ISO.  You need to restart it in single user mode or something like that to get SSH working in the console.  It was meant to be that way.

Userlevel 1
Badge

Chris. So I would just need to reboot the VM, then hold down shift until I get into the GRUB menu?

Thanks for your quick response, by the way.

Userlevel 7
Badge +20

Chris. So I would just need to reboot the VM, then hold down shift until I get into the GRUB menu?

Thanks for your quick response, by the way.

I believe so but not an expert on these just yet.  LOL

I have deployed them for use and connected to Veeam but that is all.  Check the first page for the docs and video to see as I think it is covered.

Userlevel 1
Badge

Unfortunately, I can’t even enter any inputs when I get into rescue mode, either. I guess it really is locked down tight, I can’t even get access to it.

Userlevel 7
Badge +20

Unfortunately, I can’t even enter any inputs when I get into rescue mode, either. I guess it really is locked down tight, I can’t even get access to it.

Yeah, it is really meant as a “Set it & Forget it” type thing.  🤣

Userlevel 7
Badge +14

@ThatsNASt I haven’t done it myself but @HannesK describes the process in his blog post: https://www.veeam.com/blog/ubuntu-linux-defense-secure-boot-single-user.html

Another possibility would be to do a repair installation, which gives you roo/sudo access till the second reboot.

Any reason why you need root/ssh access?

Userlevel 1
Badge

I had to re-build a vcenter and all new active-fulls were created. I’ll need to delete the old active-full in order to have enough space for backups. I can’t get a break this week =).

Userlevel 7
Badge +14

Then I would check the blog post from above. Or an easier solution, boot a live CD and clean up the files from there 😉

Running the VHR inside VMs isn't recommended as an attacker could just delete the VM including it's storage.

Userlevel 7
Badge +10

I had to re-build a vcenter and all new active-fulls were created. I’ll need to delete the old active-full in order to have enough space for backups. I can’t get a break this week =).

I am glad you going thru this drill @ThatsNASt → The installable .ISO is meant to super-harden the system. If you don’t want that - use the normal way with off the shelf Ubuntu. 

If you are stuck and need these backups put somewhere - you can put them into a SOBR and seal the extent or use Veeamover after it is expired immutability term.