• EN

All-Demo Session & Installable ISO for Veeam Hardened Repository

All-Demo Session & Installable ISO for Veeam Hardened Repository
Show first post
This topic has been closed for comments

199 comments

H
Userlevel 5
Badge +2

HTTP proxy is the only option today

Do you have a Ubuntu mirror on-prem where you would have all security updates? If yes, then we could think about allowing to select APT sources and you can point to your on-prem repository.

I
Userlevel 1

It is very good tools, thank you very much.

I tried to install on my ESxi and I cant make a connection between the Veeam backup and VHR as you can see below, what I missed her? Thank you

 

 

Hi George, did you reboot it after the .ISO the second time? Then the hardening is applied and SSH is gone.

Let me ask a question related to this, If installed properly and the connection is initially established before the reboot to harden...what would happen if the connected Veeam backup machine later is ransomwared and completely wiped out at a later date? Would one be able to start fresh with a new OS install and then reconnect to the VHR to utilize the immutable backups?

regnor
Userlevel 7
Badge +14

Would it be possible to dual boot this AND Windows Server 2019?

 

My thought process was that most of the time my 12TB server would be the VHR but in the event that I needed it as a backup server to run the company I could load windows on boot instead.

While you would be able to dual boot Windows, you won’t be able to access any backups stored on the VHR volumes; Windows can’t read the filesystem.

Max M. | Systems Engineer @Veeam | VMCE/VMCA | former Legend & Vanguard
I
Userlevel 1

Would it be possible to dual boot this AND Windows Server 2019?

 

My thought process was that most of the time my 12TB server would be the VHR but in the event that I needed it as a backup server to run the company I could load windows on boot instead.

While you would be able to dual boot Windows, you won’t be able to access any backups stored on the VHR volumes; Windows can’t read the filesystem.

You’re sure that this iso file would allow me to install a VHR (Machine 1) along side of windows (Also Machine 1), and then, after hardening the VHR, I could kill the power to (Machine 1) and then before the VHR starts I could have the option to boot windows instead, though?

 

I would be using a 2nd separate machine (Machine 2) with Veeam installed, while the VHR on (Machine 1) was loaded, if I needed the backups that are stored on the VHR (Machine 1). I would not try to access the files on (Machine 1 VHR) using (Machine 1 Windows).

The reason I would want this is I really only need 2 database files to run our company for the day, so, in case of a hardware failure on my primary server (Machine 3) I would retrieve those 2 files from the VHR (Machine 1) using (Machine 2) which would only take an hour or so of downtime. However, I still want the ability to completely backup the primary server (Machine 3) so that I could do a complete restore on the weekend when there was no business happening. Currently, as long as I have the database files I need, (Machine 1) can seamlessly operate the business, I’d like to leave it that way in case of an emergency.

 

I just need the potential dual-boot VHR(Machine 1)/Windows (Machine 1) to be versatile enough to perform one action at a time depending on my choice at the moment. VHR OR Windows server, but with the ability to switch between the two on the fly.

I hope that’s not overly complicated I tried to make it as clear as possible.

Rick Vanover
Userlevel 7
Badge +10

It seems like plenty of resources @Chris.Childerhose  → What does the plain Ubuntul 20.04 install media do on the same equipment?

Twitter @RickVanover | Email: rick.vanover@veeam.com
I
Userlevel 1

Oh sorry. I was talking about a manual Linux installation in parallel to the Windows server. The VHR ISO itself won't work in dual boot as it completely wipes the server including the existing windows installation.

Ok, so I can’t make it work with this iso. Thanks for the help.

 

Do you know if I could create my own VHR manually that would allow for what I’m describing?

Rick Vanover
Userlevel 7
Badge +10

@Chris.Childerhose → Makes me think of an easier way to test it as a Virtual Machine, here is a .VBK of a VM that I use in the lab. It has 2 drives thin provisioned, OS drive is 200 GB, Data Drive is 24TB. This VM would need network to be re-assigned from “Lab VMs” to your network, but this has the VM settings correct for testing purposes. you can download it from the same folder as the .ISO, in this folder:

#VBKsOverOVAs

Twitter @RickVanover | Email: rick.vanover@veeam.com
Chris.Childerhose
Userlevel 7
Badge +20

@Chris.Childerhose → Makes me think of an easier way to test it as a Virtual Machine, here is a .VBK of a VM that I use in the lab. It has 2 drives thin provisioned, OS drive is 200 GB, Data Drive is 24TB. This VM would need network to be re-assigned from “Lab VMs” to your network, but this has the VM settings correct for testing purposes. you can download it from the same folder as the .ISO, in this folder:

#VBKsOverOVAs

Thanks for that Rick but I figured out the issue with EFI boot and Secure option being checked in the VM.  I left BIOS boot on - duh.

This will be a nice way to test things though so will download the VBK file.  👍🏼

Chris Childerhose - VMCA2022 | VMCE2023 | Veeam Vanguard 7* | Veeam Legend 3* | vExpert 5* | VCAP-DCV/VCP-DCV | Object First Ace | Cisco Champion | Twitter: @cchilderhose | Blog Site – https://just-virtualization.tech
MarcoLuvisi
Userlevel 7
Badge +5

Hello @bob.lenz 

I have the same issue, I have tried with my vCenter Lab with ESXi 6.7 (HPE ProLiant DL380 G10) and my PC (HP CPU i7, 16 GRAM, 500 G SSD) with Workstation 16 but I receive same error when try to install this .ISO

 

 

If I try to install the same version Ubuntu LTS not customized I don’t have any issues.

Marco Luvisi | VCP-VCAP | MCP | NCDA-NCSE | VMTSP | SCA | VUG Italy Leader
MarcoLuvisi
Userlevel 7
Badge +5

Hello @bob.lenz !

I have solved this issue 🤓 
I have changed the dimension of second disk installation finish without issues.

First disk need 100 GB.

For example this my configuration:

 

 

Good luck !

Marco Luvisi | VCP-VCAP | MCP | NCDA-NCSE | VMTSP | SCA | VUG Italy Leader
T
Userlevel 1
Badge

I have two VHR images, both deployed in ESXi (An 8.0 and 7.0). After the initial reboot, everything works, but I can’t SSH in anymore AND I can’t input anything via the ESXi console even trying VMRC. Both installs have this issue. Basically, if I need to access the VHR, I’m boned. Is this the intended behavior?

 

T
Userlevel 1
Badge

Chris. So I would just need to reboot the VM, then hold down shift until I get into the GRUB menu?

Thanks for your quick response, by the way.

H
Userlevel 5
Badge +2

Hello,
yes, the script does not disable SSH. That’s manual steps as described in the blog posts mentioned earlier in this thread (yes, this thread is a mix of many topics and having one thread instead of a dozen is sub-optimal… we know)

Errors sounds wrong, but I can say that we will continue work on the ISO / script after 12.1 is out. 

For now it would be great to hear if specific hardware works / does not work (we know that it works with VMs if system requirements are met :-))

Best regards,
Hannes

T
Userlevel 1
Badge

Unfortunately, I can’t even enter any inputs when I get into rescue mode, either. I guess it really is locked down tight, I can’t even get access to it.

regnor
Userlevel 7
Badge +14

@ThatsNASt I haven’t done it myself but @HannesK describes the process in his blog post: https://www.veeam.com/blog/ubuntu-linux-defense-secure-boot-single-user.html

Another possibility would be to do a repair installation, which gives you roo/sudo access till the second reboot.

Any reason why you need root/ssh access?

Max M. | Systems Engineer @Veeam | VMCE/VMCA | former Legend & Vanguard
T
Userlevel 1
Badge

I had to re-build a vcenter and all new active-fulls were created. I’ll need to delete the old active-full in order to have enough space for backups. I can’t get a break this week =).

David Tate BIG
Userlevel 1
Badge

Hey @David Tate BIG  → This is really good.

Can you try the bond with the ‘plain’ Ubuntu HWE 20.04 installation media, does it behave the same?

I can certainly give that a try.

David Tate BIG
Userlevel 1
Badge

I just had one of those AHA! moments.  I tried the ‘plain’ Ubuntu install and still had issues, but then I remembered something about ESXi, which is essentially a very special Linux distribution.  It is recommended that when you Team NICs in ESXi and connect to a Meraki switch you leave the switchports as standard access or trunk ports as appropriate.  You don’t put the ports in an Aggregate.

I did that and the ‘plain’ Ubuntu install worked like a charm.  I ran ifconfig and saw basically equal traffic on the two NICs in the bond.

So, I deployed the Hardened Linux Repository from the ISO downloaded here and everything looks great so far.

The bond mode I used was balance-rr.

David Tate BIG
Userlevel 1
Badge

Another AHA! moment.  I took a look at the file structure (lsblk) and found that the deployment ended up on the internal SD Card (ISDM).  It really was out of space.  I had to go in to the BIOS and disable the ISDM.  Reinstalled and now I’m able to add the Repository.

wolff.mateus
Userlevel 7
Badge +11

This session rocks!

I deploy VHR on my homelab right now.

Just a question:

What are the steps that hardening script does on the system?

I didn't find this on the github.

IT Lover | Veeam Vanguard | VUG Leader | Blogger
Rick Vanover
Userlevel 7
Badge +10

Can now also report that the hardened ISO Installer also crashes on a brand new Dell R250 PowerEdge, in addition to the old R720 we had lying around that I tested earlier.  It does progress further than it did on the R720 though, which crashed on the network setup screen.  Now it crashes near the end of the configuration.  So something about it does not seem to jive with Dell hardware - maybe it’s the hardware RAID card, or iDRAC controller?

I’ve gone ahead and installed a vanilla copy of LTS 22 without issue.  My question is can I run the hardening script on it, despite it being designed for LTS 20?  I’m guessing not, and that I should just wait until work has been completed on 22 compatibility.

I’ve had similar issues testing with Dell R730 and R740’s.  I initialized the mirrored boot drive, setup two 10+2 R6 data drives and all (9 so far) have acted the same.  The install crashes just after the networking.  I’ve had success by installing vanilla 20.04 customizing the disks, selecting the mirrored drive as the boot drive and creating a GPT partition for the remainder of the space.  I then format both the data drives as xfs and set the mount point to /mnt/backup1 and /mnt/backup2. 

Immediately after that I boot from the hardened ISO and it installs as expected.  I’m not sure exactly what the hardened ISO doesnt like about the system on the initial install but this is working for me

Dave

Hey @efd121  → Thanks for this. 

Everyone else:
I confirmed with QA we will be taking a look at the next version here after a separate milestone is made, but please keep these PoV’s coming.

Twitter @RickVanover | Email: rick.vanover@veeam.com
H
Userlevel 5
Badge +2

Can now also report that the hardened ISO Installer also crashes on a brand new Dell R250 PowerEdge, in addition to the old R720 we had lying around that I tested earlier.  It does progress further than it did on the R720 though, which crashed on the network setup screen.  Now it crashes near the end of the configuration.  So something about it does not seem to jive with Dell hardware - maybe it’s the hardware RAID card, or iDRAC controller?

I’ve gone ahead and installed a vanilla copy of LTS 22 without issue.  My question is can I run the hardening script on it, despite it being designed for LTS 20?  I’m guessing not, and that I should just wait until work has been completed on 22 compatibility.

Thanks for reporting the crashes. Yes, incompatible hardware is very likely the reason, because we did not change Ubuntu itself.

For 22.04: probably some settings would fail. We can only start on 22.04 once DISA STGs are released for 22.04 (Canonical is working on it, but it will take time...)

coolsport00
Userlevel 7
Badge +17

I missed this session in MIA, but am about to re-watch. I deployed the LHR server; just need to add the server to VBR to deploy the role on the server. Can’t wait to watch the actual recorded session. Thanks for your efforts Rick!

@wolff.mateus - the script probably contains similar cmds as Paolo uses from his VHR post here...at least, that’s my guess. 😊

Shane Williford - Veeam VMCA/VMCE | Veeam Legend | VUG Leader | VCP-DCV | Twitter: @coolsport00
Mildur
Userlevel 7
Badge +12

@Chris.Childerhose 

We don‘t test against specific hardware. But you may check the compatibility list from Cisco to see if your model is compatible with Ubuntu 20.04:

https://ucshcltool.cloudapps.cisco.com/public/

 

Best,

Fabian

Product Management Analyst @ Veeam Software
A

HI, I tried the iso of the hardened linux install and run into an error on physical and virtual test machines. See snip. Did anyone run into the same error and if yes, how did you get around it?

 

Thanks,

 

Terms & Conditions