Solved

Hardened repo access after upgrading to v12


Userlevel 7
Badge +6

I just got all of our systems upgraded to v12.  With that said, I have two clients using hardened repo’s and two different people set them up, neither of them were me, and the system I’m working on now - the engineer that set that up left the company and moved on a couple months ago.  I planned on dipping into these a while ago but never had the time and am now thrust into it.  And my linux skills are still fairly basic, maybe intermediate at best.

So here’s what I have:

The configuration, not that I think it matters, but is a Ubuntu VM running on a Synology NAS with a virtual disk mounted to the VM.  I upgraded from v11 to v12 but the repository became unavailable during this process.  Single-use credentials are in place and I think I’ve mostly got around the learning curve on this, but can’t quite make connections on things.

There are two accounts, I’ll call them “repouser” and “admin”.  Repouser is the account that should have permissions to the repo at /mnt/veeamrepo/backups.  Admin is a member of the sudoers.  I don’t know the root password - guessing I can reset it if needed.

I’m performing the steps found here as that seems to be along the lines of what I need to do.  I’ve logged in with the admin account added the permissions specified to the mount using the below commands.

sudo chown -R repouser:repouser /mnt/veeamrepo/backups

sudo chown repouser:repouser /mnt/veeamrepo/backups

(Pretty sure I had to sudo on those)

Of course I can’t modify the existing backup files that have immutability flags set on them, but if I don’t use -R, I can grant the permission to the folders I believe.  I also added the permissions to /mnt/veeamrepo but I suspect that wasn’t needed.

After doing so, when I rescan the linux server with the one-time credentials using the admin account, it was able to upgrade the services and scan the disks without error (it was erroring before doing so).  I was not able to scan with repouser, which, I’m not sure if I should be able to or not.

That said, now when I try to add the existing mount/folder back as a new hardened repo, I get the below error after it checks for XFS Fastclone requirements.

 

I had also found an article that mentioned chmod’ing the folder to 700 which I did apply but didn’t note anything changing.  I applied that to both /mnt/veeamrepo and /mnt/veeamrepo/backups.

Any idea’s what I’m missing here?  Or am I using the wrong accounts in places?  My apologies as it’s all new to me on how this works, and it sounds like some things may have changed between v11 and v12, so crash course in learning this, so I really appreciate the help.

icon

Best answer by regnor 9 March 2023, 21:14

View original

35 comments

I had the exact same problem.

 

I can confirm that resaving the one time ssh credentials resolved the problem.


Under “ backup infrastructure” →  “managed servers” - > Linux → Properties then add the one time ssh credentials again and let it complete. The repository is back online!

 

Userlevel 7
Badge +6

Glad I could help!  I imagine I may even have to come back and reference it for other clients, but perhaps not since this should be easier once on v12 to newer versions!

Userlevel 7
Badge +17

I re-referenced myself when I upgraded 😊

Hey guys,

Just wanted to say thank you as I just upgraded to V12 and had the same issues.

this thread saved me a lot of time and heartache.

Cheers

Userlevel 7
Badge +17

Nice! Appreciate the share @PDXdoug !

Userlevel 3

Ok, this issue has been resolved.

Here is what I did. Run all commands as the root user.

1. Enable and start ssh on the linux box 

sudo systemctl enable ssh
sudo systemctl start ssh

2. Open firewall ports for ssh

sudo ufw allow ssh

3. Add the user with repository access to the Sudoers group.

sudo usermod -aG sudo "username" or you could use sudo adduser "username" sudo

4. In the B&R Console go to Backup Infrastructure and find the linux box under Managed Servers
Right click the server and go to Properties > SSH Connection > add the linux user with repository permissions to the Credentials drop down list as a Single Use Credentials. 

5. Once the ssh connection has passed testing and the linux server updated, remove the linux user from the Sudoers group - sudo deluser "username" sudo, stop the ssh service - sudo systemctl stop ssh, disable the ssh service - sudo systemctl disable ssh, and finally block ssh in the firewall - sudo ufw deny ssh.

That should do the trick. Happy Backups!

- Doug
 

Userlevel 7
Badge +17

Ah ok. Gotcha. I was actually thinking about that yesterday (your question). Thus why I was curious what support would say. How does one ‘reauthenticate’ against the VHR after it’s already been upgraded? 🤷🏻‍♂️

Userlevel 3

Oh I totally understand that one can add different account types there @coolsport00, and I did add the correct linux account and tested the ssh single use authentication and it worked fine.

Guess what I’m trying to say is how does one use this account to authenticate to the repository via either the backup job or the repository settings as the backups keep failing due to authentication.

6/23/2023 8:33:08 AM :: Error: Permission denied
Failed to get file system object info: '/media/local48/VeeamRepository/VeeamBackups/Exchange Linux Immutable/Exchange Linux Immutable.vbm'
Agent failed to process method {FileSystem.Exists}.
  
 

Userlevel 7
Badge +17

@PDXdoug no sir..that isn’t correct. You can add all kinds of accounts there:

 

Userlevel 3

Hey @regnor 

 

You are most likely correct, but I do love the keywords “should” and “maybe” :-)

There are three options I can think of.

  1. Delete the Linux Repo, then recreate it using the correct ssh single use authentication for the storage directories. Just not sure what this would do to the backup chains.
  2. Change permissions on the storage directories to match the single use ssh credentials I used to upgrade the host.
  3. Somehow edit the SQL database where the credentials are stored to match the current storage directory permissions.

Support suggested going into Credentials & Passwords and creating the correct account. Great idea but there is no way to add those account credentials for repository access. Access Permissions in the repository only allows for AD accounts. 

Unless I’m missing something here.

 

  • Doug
Userlevel 7
Badge +14

@PDXdoug V12 should prevent the usage of the root account, but maybe this only counts for new repositories. You can check which serviceuser is used in the config file on your hardened repository: /opt/veeam/transport/VeeamTransportConfig

Userlevel 7
Badge +17

Hmm..odd. I’d be curious to hear what Support says. Keep us posted @PDXdoug 

Userlevel 3

This is the error I receive btw:

 

6/22/2023 10:43:20 AM :: Error: Permission denied
Failed to get file system object info: '/media/local48/VeeamRepository/VeeamBackups/Santiam Linux Immutable/Santiam Linux Immutable.vbm'
Agent failed to process method {FileSystem.Exists}.
  
 

Userlevel 3

I’ve opened  support ticket on this. Looking forward to seeing what they say.

Userlevel 3

No, the permissions are correct

drwx------  12 veeamrepo veeamrepo 4096 Dec 13  2022 VeeamBackups

And the admin/root account doesn’t nor ever did have permissions on those folders.

Userlevel 7
Badge +17

The admin/root account shouldn’t even have had permissions on the folder/files...yes, even for being root.

Userlevel 7
Badge +17

Did the permissions on the folder & files change? Check, using ls -l on the folder & files.

Userlevel 3

So I just upgraded to V12 and of course my Linux repo required updating.

My linux repo has two users, the admin (root) account, and veeamrepo for repository access.

Instead of adding the veeamrepo user with repository permissions to the sudo users group and then upgrading the Linux repo to V12, I stupidly used the other admin (root) account to perform the SSL  upgrade.

The upgrade worked fine, but now it seems B&R is using the other admin (root) account to access the linux repositories instead of the veeamrepo account with the correct CHMOD & CHOWN permissions on the directories.

Is there any way to change the repository account in B&R back to the veeamrepo one, or do I need to change the directory permissions on the linux repo to match the admin (root) account?

Doing so would require me to enable sudo on the veeamrepo account and disable it on the admin(root) account I believe.

Thoughts?

  • Doug
Userlevel 7
Badge +17

A very welcomed change 😊

Userlevel 7
Badge +6

Yes, was just going to say that one of my takeaways from VeeamON was that with v12 and beyond, SSH no longer needs to be re-enabled for updates.  Way to go Veeam!  

Userlevel 7
Badge +17

Glad to also hear future updates won’t require the couple extra steps. They aren’t difficult, but convenient for the hardened repo to upgrade/update as ‘normal’ repos do. Thanks!

Userlevel 7
Badge +17

Hi @Mildur - thanks for the updated info. Though I reviewed info about the ugprade, I pretty much had it all, but there was 1 item I forgot to do before I started the repo upgrade → enable SSH. Oops! 😊 Once I enabled it, I reperformed the upgrade attempt & it went without a hitch. I did re-disable SSH as well as removed my user from sudo after I completed it. Appreciate the detailed response. Hope you’ve had a good weekend...and thanks again.

Userlevel 7
Badge +12

Hi Shane


Please replace reposuser with the username you use for the deployment. 
 

1) Enable SSH
sudo systemctl start ssh

2) Add Repo User to Sudo
sudo usermod -aG sudo repouser

3) Update Backup Repo in VBR Console with Single Use Credentials

4) Remove Repo User from Sudo
sudo deluser repouser sudo

5) Disable SSH
sudo systemctl stop ssh

 

Future updates (v12 to a later version) will not require that process and SSH anymore.

 

Best,

Fabian

Userlevel 7
Badge +17

Hey @dloseke ...I’m following up on this as I’m now beginning my VBR upgrade. I have 2 servers - main/prod one for backups and a DR one or replication. I upgraded my repl VBR server last wk without hitch. I’m just getting ready to start the upgrade of my main VBR server today, which has a hardened repo attached to it. So, all which was needed was to re-add the local linux user to sudo to be able to finalize the upgrade of the repo component piece, correct?

Thanks!

Userlevel 7
Badge +17

There is indeed. I used Paolo’s posts myself. Thought his was detailed, start to finish, & easy to follow.

Comment