• EN

All-Demo Session & Installable ISO for Veeam Hardened Repository

All-Demo Session & Installable ISO for Veeam Hardened Repository
Rick Vanover
Userlevel 7
Badge +10

 

22-February 2024 Update:  GOOD NEWS! We are scheming a big update and overhaul of this page at or around VeeamON time. I can’t say more!

 

During the lead-up to VeeamON, it was amazing to see what content was leading the pre-registration information. We have the most data here than ever before. Constantly back and forth with the “Architecting Veeam Backup for Microsoft 365 at Scale” session, the session that @hannesk, Christoph Meyer and I are delivering was racing to the finish line as the leader. The “All Demo Session for Veeam Hardened Repository” was in the end the most popular pre-registered session, and as promised in the delivery in Miami, here is your everything you need to know guide.

Post Publish Updates:

  • 1-June 2023:  The installation requirements for the .ISO have been added.
  • 1-June 2023:  Video of installation of the .ISO has been added.
  • 2-June 2023:  Clarification on support, formatting of document.
  • 5-June 2023:  Updated .ISO with support banner and logon screen update, link updated. No changes to Ubuntu configuration or hardening script.
  • 29-June 2023: Updated .ISO with one enhancement, offline installation works.

Recording Resources

You can watch one of my practice runs of the session from Miami here:

At the VeeamON virtual event experience, we will have the replay from Miami session up very soon. 

Markdown File & Write Up

I have made a markdown file with a lot of the key information ready, you can download it here: rickvanover/VeeamHardenedRepoHub (github.com) Additionally - you can download the Markdown file at the QR code below:
 

This link goes to the VHR Markdown file that I’ve written up.

If you are not doing Markdown or GitHub you can download this as a PDF at the bottom of the page.

Overall advice

My overall advice on the Veeam Hardened Repository is to plan your deployment, this will greatly improve your experience with using the VHR. For example, I’m notorious for not documenting the options I put during the Ubuntu installation at this important stage:

 

Installable .ISO & Hardening Script

Key Update 18-December 2023: We have taken down the link for the .ISO, there will be a  new one coming “Soon” :)! 

One of the key parts of the Miami experience is the update we had from Product Management on new functionality for the Veeam hardened repository, which includes the Hardening Script and the installable .ISO. I recommend watching the replay to hear from Hannes and Christop directly before using the script and .ISO. We recommend at least 100 GB for storage. You can get both below:

  • Hardening Script: https://www.veeam.com/sys507
  • Installable .ISO: )18-December 2023 - We’ve removed this download and a new one is coming “soon”!)

About this installable .ISO…. The principal benefit of the installable .ISO is that it sets up much of the installation pre-defined *and* applies the above linked hardening script. 

Installable .ISO Requirements

For the installable .ISO, we recommend you follow these prerequisites (mainly relevant if you try to install it in a VM for lab purposes):

  • This is now optional from the 29-June update:  Internet connection (HTTP to the Ubuntu update servers for automatic security updates)
  • 2 CPUs, 6 GB RAM
  • UEFI boot is required (BIOS does not work) UEFI secure boot strongly recommended
  • 1 disk with at least 100GB for the operating system (plus cache files)
  • 100GB is a hard requirement. If that’s too high, please give feedback below.
  • Then there should be at least one second disk that is larger 100GB for backup data that would be the Veeam repository

During the installation, follow the wizard and reboot after the installation finished. Then install the hardened repository role. The VHR final layout will have all disks are mounted to /mnt/backup<n>

Important: Reboot again (this second reboot locks down the whole system and you can only reboot / shutdown with sudo. there is no root / sudo anymore at this stage) - this is the result of the hardening.  Think of this result as taking a massive green pill that tells this system its role in life forever on is to be a VHR.

After reading all of this, you are now ready to watch my video of installing the VHR with the .ISO:



Installable .ISO General Information & Support Statement

  • It's a community project. please don't call Veeam support for use of the .ISO. Once a VHR is deployed and configured in Veeam Backup & Replication, it would be subject to support. 
  • Automatic updates are enabled.
  • The user does not have permissions to update manually.
  • We plan to add a note to the banner message where to find more information about the ISO installer.

Get Started with the Veeam Hardened Repository 

Getting started with the VHR doesn’t have to be overwhelming. Give it a try - and let us know how it goes in the comments below.

1 Attachment
Twitter @RickVanover | Email: rick.vanover@veeam.com
This topic has been closed for comments

199 comments

Chris.Childerhose
Userlevel 7
Badge +20

Hello,

I’ve just deployed a test repo, and now I’m starting to play with it. Once rebooted, I’ve realized that it’s impossible to run sudo to change disk layout, for example I wan to modify a LVM instance, or extend a filesystem and I can’t see how to do this…..

Any idea?

 

I believe you need to log in as root or enable the login again after the deployment since the hardening of the server removes this.  Once you have the system running why would you want to change it though?  The OS drive is set up with the ISO and the larger drive becomes the backup.  Just curious on the use case to make changes.

I believe the other way to make changes is during the deployment of the ISO as you can manually set up the disk layout.

***UPDATE - Hannes beat me to it LOL ***

What if your backup drive needs to be extended?…..sometimes data grows, or even grows a lot….. We run our production backup in a physical self deployed repo server with hardening tips from Gostev, and others, and I’ve had to extend the repo filesystem because of this… I was just wondering if this ISO was a better option for us

Ok.  That makes sense then as I was just curious.  Then you will need to follow what Hannes said about single-user mode and the URL he posted to do this.

Chris Childerhose - VMCA2022 | VMCE2023 | Veeam Vanguard 7* | Veeam Legend 3* | vExpert 5* | VCAP-DCV/VCP-DCV | Object First Ace | Cisco Champion | Twitter: @cchilderhose | Blog Site – https://just-virtualization.tech
H
Userlevel 5
Badge +2

we also have ideas to support LVM in the future. The plan is to make one large LVM of all volumes except the operating system volume.

extending volumes will probably not be on the roadmap for longer time, because it sounds like a corner case to me with internal disks (customers who are able to manage SAN storage & Linux don’t need the ISO from my point of view)

J
Userlevel 1

we also have ideas to support LVM in the future. The plan is to make one large LVM of all volumes except the operating system volume.

extending volumes will probably not be on the roadmap for longer time, because it sounds like a corner case to me with internal disks (customers who are able to manage SAN storage & Linux don’t need the ISO from my point of view)

I agree, but you should think on customers that had servers with highly expandable internal storage (we’ve just bought a server that can hold up to 50 internal disks…..)

H
Userlevel 5
Badge +2

yes, these high density servers are a primary target.

just to be sure: you bought them with less than 50 disks and you plan to add disks somewhen later?

J
Userlevel 1

yes, these high density servers are a primary target.

just to be sure: you bought them with less than 50 disks and you plan to add disks somewhen later?

not in this case because we run our systems on SAN storage. This server only has a RAID-1 pair of HDD for boot. But it’s an example of a use case for a hardened repo whose storage can be expanded without a SAN environment….

regnor
Userlevel 7
Badge +14

Expanding or changing the storage layout could be a task which happens once or twice during the lifecycle of a repository server. I do see this with customers who start small and expand later on. In addition changing the network settings could be a task which will have to be done regularly. Probably I would just go the way Hannes describes in his blog post.

Or one could do a repair installation which gives you root/sudo access before the second reboot. I once had to reconfigure NIC Teaming via CLI and was already thinking about reinstalling Ubuntu. For this case I would certainly use the repair installation.

 

Max M. | Systems Engineer @Veeam | VMCE/VMCA | former Legend & Vanguard
H
Userlevel 5
Badge +2

@JavierLarrea : could you maybe tell us, what type of server / model / configuration you used? I’m trying to create a list of “configurations that work”.

yes, we are discussing a configuration tool that allows to configure network and some other settings. 

 

 

J
Userlevel 1

@JavierLarrea : could you maybe tell us, what type of server / model / configuration you used? I’m trying to create a list of “configurations that work”.

yes, we are discussing a configuration tool that allows to configure network and some other settings. 

 

 

Hi, We are running our hardened repo in an old IBM xSeries model 3550M3 with a 4 cores Xeon E5630 processor and 8GB RAM. The network connection is a IBM Intel X540-T2 Dual Port 10GBaseT Adapter LP configured as a LACP bond, and the storage HBA is a Qlogic QLE2562 8 GB Dual port card. It is a decomisioned IBM 2805-MC5 System Storage Productivity Center.

You may see the specs in this link https://www.ibm.com/common/ssi/ShowDoc.wss?docURL=/common/ssi/rep_sm/4/877/ENUS2805-_h04/index.html&lang=en&request_locale=en

The only change is the dual port 10G NIC

Rick Vanover
Userlevel 7
Badge +10

FYI Christoph and Hannes have issued a new version of the .ISO. Updated in the main post.

Twitter @RickVanover | Email: rick.vanover@veeam.com
Chris.Childerhose
Userlevel 7
Badge +20

FYI Christoph and Hannes have issued a new version of the .ISO. Updated in the main post.

Sweet!  Is there a changelog for it @Rick Vanover ?

Chris Childerhose - VMCA2022 | VMCE2023 | Veeam Vanguard 7* | Veeam Legend 3* | vExpert 5* | VCAP-DCV/VCP-DCV | Object First Ace | Cisco Champion | Twitter: @cchilderhose | Blog Site – https://just-virtualization.tech
H
Userlevel 5
Badge +2

the new feature is: it works offline. No other changes :-)

Chris.Childerhose
Userlevel 7
Badge +20

the new feature is: it works offline. No other changes :-)

Very cool. Thanks Hannes. Will be testing this one now for my documentation as we plan to use this moving forward to make deploying XFS Hardened Repos for all our backups.  😎

Chris Childerhose - VMCA2022 | VMCE2023 | Veeam Vanguard 7* | Veeam Legend 3* | vExpert 5* | VCAP-DCV/VCP-DCV | Object First Ace | Cisco Champion | Twitter: @cchilderhose | Blog Site – https://just-virtualization.tech
regnor
Userlevel 7
Badge +14

the new feature is: it works offline. No other changes :-)

Just deployed it without network/internet connection and it just works 😅

Max M. | Systems Engineer @Veeam | VMCE/VMCA | former Legend & Vanguard
Chris.Childerhose
Userlevel 7
Badge +20

Question - is this ISO eventually going to be posted on a public URL?  I ask as I am doing documentation and will update the URL at that point.  I prefer not to put the one from this post with a password since that changes.  😁

Chris Childerhose - VMCA2022 | VMCE2023 | Veeam Vanguard 7* | Veeam Legend 3* | vExpert 5* | VCAP-DCV/VCP-DCV | Object First Ace | Cisco Champion | Twitter: @cchilderhose | Blog Site – https://just-virtualization.tech
Rick Vanover
Userlevel 7
Badge +10

Question - is this ISO eventually going to be posted on a public URL?  I ask as I am doing documentation and will update the URL at that point.  I prefer not to put the one from this post with a password since that changes.  😁

We will reach a milestone at some point to then move it to a Veeam.com download. That takes a bit more time - for example, it may take a week to load and stage. We’re not quite sure when that will happen, but I would use http://vee.am/vhrhub for now.  I actually plan on a major reformat of this post at some point.

Twitter @RickVanover | Email: rick.vanover@veeam.com
Chris.Childerhose
Userlevel 7
Badge +20

Question - is this ISO eventually going to be posted on a public URL?  I ask as I am doing documentation and will update the URL at that point.  I prefer not to put the one from this post with a password since that changes.  😁

We will reach a milestone at some point to then move it to a Veeam.com download. That takes a bit more time - for example, it may take a week to load and stage. We’re not quite sure when that will happen, but I would use http://vee.am/vhrhub for now.  I actually plan on a major reformat of this post at some point.

Thanks for the update, Rick.  Appreciate it.

Chris Childerhose - VMCA2022 | VMCE2023 | Veeam Vanguard 7* | Veeam Legend 3* | vExpert 5* | VCAP-DCV/VCP-DCV | Object First Ace | Cisco Champion | Twitter: @cchilderhose | Blog Site – https://just-virtualization.tech
keiranshelden
Userlevel 4
Badge +1

This is an awesome addition. Thanks for all the team effort you guys put in to make it so much easier! 

vExpert ‘14-‘23 | NSX ‘18 | Drummer | @Techfieldday delegate | @tech4breakfast Australian Sr. Field Correspondent
G

The installer errors out as soon as I reach the network config screen early in the setup process.

Trying to install it on a Dell PowerEdge R720 we had lying around.  Downloaded the latest Ubuntu Server 22 LTS and it works fine, so not sure what is wrong with this install image for this particular hardware?

Rick Vanover
Userlevel 7
Badge +10

The installer errors out as soon as I reach the network config screen early in the setup process.

Trying to install it on a Dell PowerEdge R720 we had lying around.  Downloaded the latest Ubuntu Server 22 LTS and it works fine, so not sure what is wrong with this install image for this particular hardware?

Hey @Graham → can you PM me the Service Tag # since it is a Dell and we’ll look up the inventory and check on our side?

Twitter @RickVanover | Email: rick.vanover@veeam.com
H
Userlevel 5
Badge +2

The installer errors out as soon as I reach the network config screen early in the setup process.

Trying to install it on a Dell PowerEdge R720 we had lying around.  Downloaded the latest Ubuntu Server 22 LTS and it works fine, so not sure what is wrong with this install image for this particular hardware?

@Graham : can you maybe test a Ubuntu 20.04 installer whether it detects the network card? I remember the R720 from one of my earlier jobs and they normally work fine with Ubuntu. 

Upgrading the ISO to Ubuntu 22.04 depends on Canonical / DISA providing STIGs. In best case, I guess that’s end of the year.

N

I want to install Ubuntu Hardened repository on open stack cloud based on KVM.

With normal ISO downloaded from ubuntu, i dont see an option to boot with H/W Kernel during boot phase from ISO

can we use this custom hardened ISO for Cloud based on KVM ? please confirm

H
Userlevel 5
Badge +2

the HWE kernel requires UEFI. I guess that’s the reason (I don’t have OpenStack / KVM in my lab). The ISO requires UEFI and might even require “UEFI secure boot” in future. If OpenStack can do that, then it should work.

Please remember: The ISO is designed for hardware to avoid an attacker simply deleting the VM. Virtual machines are only for test / lab and not for production.

N

Thanks for Swift response.

However, if i download ISO from UBUNTU Website and use it to boot on KVM Based cloud, we dont see that option available. Just want to confirm, if we can boot with normal kernel and than install HWE and reboot with new kernel works ? 

we are trying to build Ubuntu Hardened repository in either ways possible, but we dont see a way

H
Userlevel 5
Badge +2

it’s out of scope of the ISO: I assume there is no reason for the HWE kernel on OpenStack at all. I mean, HWE is “Hardware Enablement Stack”… a VM is not hardware.

 

If you really want to do it, then you can apply the hardening script manually

https://www.veeam.com/blog/backup-repository-security-disa-stig-ubuntu-step-by-step-guide.html

https://github.com/VeeamHub/veeam-hardened-repository

C

 

@Rick Vanover 

  • Automatic updates are enabled.
    Is this for systems with access to the internet? what about systems without internet access?
  • The user does not have permissions to update manually.
    Will this be an option in the future for example for systems in a secure network without internet access?
Terms & Conditions