Question

Pre-freeze and post-that scripts throwing error


Userlevel 1

Hi all,

I have a VM running on Hyper-V 2016, its a Ubuntu 20.04.6 with MariaDB installed.

So I defined a (cold) pre-freeze.sh and and post-thaw.sh and tested both in /tmp directory, it will work as expected when done with sudo, under the specified user.

 

However when the backup job does that, it throws an error. This is what you can see in the Console windows under the details of that VM:

 

08.04.2023 16:47:10 :: VM is now in the required state for processing  
08.04.2023 16:47:11 :: Preparing to create snapshot  
08.04.2023 16:47:13 :: Pre-freeze script finished execution with exit code -1  
08.04.2023 16:47:14 :: Creating VM recovery checkpoint (mode: Crash consistent)  
08.04.2023 16:47:25 :: Post-thaw script finished execution with exit code -1

 

Other than that the VM is properly backed up.

This is the content oif the Task.db01.something.log on the VBR server:

 

[08.04.2023 16:47:13.652] <14> Info         [ScriptInvoker] No scripts are set to invoke, creating empty invoker
[08.04.2023 16:47:13.652] <28> Info         [ScriptInvoker] Starting pre-freeze script execution
[08.04.2023 16:47:13.669] <28> Info         [VssCredFactory] Getting guest credentials. OsType: ubuntu64Guest. OsKind: Linux. WinCreds: False. LinCreds: True
[08.04.2023 16:47:13.683] <28> Info         [ScriptInvoker] Running Linux script 'C:\Program Files\Veeam\Backup and Replication\Backup\Scripts\MariaDb\pre-freeze.sh'
[08.04.2023 16:47:13.683] <28> Info         Can't find physical host information for virtual host: HostId:375e3938-20c4-48bd-80fc-6a34a8809637, objectId:d56ec8ee-e9b1-4597-949f-fca5d428a0ff.
[08.04.2023 16:47:13.683] <28> Info         [Ssh] Creating new cached connection 256a5038-7eb5-4646-8e5a-a89ea08d8b72 [host: 'db01', port: 22, elevation to root: 'yes', autoSudo: no, use su if sudo fails: no, host name: db01, IPs: [fe80::215:5dff:fe00:104, 10.0.0.8, 10.1.0.1], AuthenticationData: [UserName: youname, AuthTypes: [KeyboardInteractive, Password]]].
[08.04.2023 16:47:13.683] <28> Info         [Ssh] Creating SSH connection 256a5038-7eb5-4646-8e5a-a89ea08d8b72 to server db01
[08.04.2023 16:47:13.683] <28> Info         [Ssh] Creating Rebex SSH connection '256a5038-7eb5-4646-8e5a-a89ea08d8b72' (unknown protocol)
[08.04.2023 16:47:13.683] <28> Info         [Ssh] Establishing SSH Rebex connection '256a5038-7eb5-4646-8e5a-a89ea08d8b72' to host db01
[08.04.2023 16:47:13.715] <28> Info         [Ssh] Rebex connection '256a5038-7eb5-4646-8e5a-a89ea08d8b72' has been established
[08.04.2023 16:47:13.715] <28> Info         [SshFingerprint] Connection accepted for host [(db01),fe80::215:5dff:fe00:104,10.0.0.8,10.1.0.1] (fingerprint: ssh-rsa e7:06:76:7e:35:2e:62:a3:8f:67:02:b8:71:2e:55:26): Trust all rule.
[08.04.2023 16:47:13.730] <28> Info         [Ssh] SSH connection 256a5038-7eb5-4646-8e5a-a89ea08d8b72 to server db01 created successfully
[08.04.2023 16:47:14.251] <28> Info         [Ssh] Connection '256a5038-7eb5-4646-8e5a-a89ea08d8b72': Elevate method: "sudo" in TTY
[08.04.2023 16:47:14.251] <28> Info         Ssh CommandsExecutor has been created.
[08.04.2023 16:47:14.251] <28> Info         [ScriptInvoker] CommandsExecutor has been created
[08.04.2023 16:47:14.704] <28> Info         [ScriptInvoker] Exception thrown during script execution.
[08.04.2023 16:47:14.704] <28> Info         [LinuxFile] Deleting file /tmp/33b3ae77-45c6-4c6c-b32b-c8eeac2ce2da_pre-freeze.sh
[08.04.2023 16:47:14.769] <28> Info         [Ssh] Connection 256a5038-7eb5-4646-8e5a-a89ea08d8b72 - [host: 'db01', port: 22, elevation to root: 'yes', autoSudo: no, use su if sudo fails: no, host name: db01, IPs: [fe80::215:5dff:fe00:104, 10.0.0.8, 10.1.0.1], AuthenticationData: [UserName: youname, AuthTypes: [KeyboardInteractive, Password]]] is disposing.
[08.04.2023 16:47:14.769] <28> Error        [ScriptInvoker] Failed to execute script in lin guest machine. Script path: C:\Program Files\Veeam\Backup and Replication\Backup\Scripts\MariaDb\pre-freeze.sh.
[08.04.2023 16:47:14.769] <28> Error        Error: sudo: unable to execute /tmp/33b3ae77-45c6-4c6c-b32b-c8eeac2ce2da_pre-freeze.sh: No such file or directory; Exit code: 1 (Veeam.Backup.RemoteInteractionLib.CommandsExecutor.CCommandsExecutorException)
[08.04.2023 16:47:14.769] <28> Error           at Veeam.Backup.RemoteInteractionLib.CommandsExecutor.CCommandResult.CheckResult()
[08.04.2023 16:47:14.769] <28> Error           at Veeam.Backup.Core.CScriptInvoker.RunScript(CScriptFile scriptFile, TimeSpan timeout, Boolean collectLogs, String stdOutFilePath, String stdErrFilePath, Boolean checkStdErr)
[08.04.2023 16:47:14.769] <28> Error           at Veeam.Backup.Core.CScriptInvoker.ExecScriptInner(String localPath, TimeSpan timeout, Boolean collectLogs, String stdOutFilePath, String stdErrFilePath, Boolean checkStdErr)
[08.04.2023 16:47:14.769] <28> Info         [ScriptInvoker] Script finished with exit code = '-1'

 

 

And this is the auth.log which shows the actual commands which were issued:
 

Apr  8 16:47:13 db01 sshd[14944]: Accepted password for youname from 10.0.0.17 port 59299 ssh2
Apr  8 16:47:13 db01 sshd[14944]: pam_unix(sshd:session): session opened for user youname by (uid=0)
Apr  8 16:47:13 db01 systemd-logind[511]: New session 233 of user youname.
Apr  8 16:47:14 db01 sudo:   youname : TTY=pts/1 ; PWD=/home/youname ; USER=root ; COMMAND=/usr/bin/whoami
Apr  8 16:47:14 db01 sudo: pam_unix(sudo:session): session opened for user root by youname(uid=0)
Apr  8 16:47:14 db01 sudo: pam_unix(sudo:session): session closed for user root
Apr  8 16:47:14 db01 sudo:   youname : TTY=pts/1 ; PWD=/home/youname ; USER=root ; COMMAND=/usr/bin/touch /tmp/33b3ae77-45c6-4c6c-b32b-c8eeac2ce2da_pre-freeze.sh
Apr  8 16:47:14 db01 sudo: pam_unix(sudo:session): session opened for user root by youname(uid=0)
Apr  8 16:47:14 db01 sudo: pam_unix(sudo:session): session closed for user root
Apr  8 16:47:14 db01 sudo:   youname : TTY=pts/1 ; PWD=/home/youname ; USER=root ; COMMAND=/bin/chmod 0700 /tmp/33b3ae77-45c6-4c6c-b32b-c8eeac2ce2da_pre-freeze.sh
Apr  8 16:47:14 db01 sudo: pam_unix(sudo:session): session opened for user root by youname(uid=0)
Apr  8 16:47:14 db01 sudo: pam_unix(sudo:session): session closed for user root
Apr  8 16:47:14 db01 sudo:   youname : TTY=pts/1 ; PWD=/home/youname ; USER=root ; COMMAND=/bin/cp -f /home/youname/3fb75bd9-5ed8-4e6d-b77f-b1c58f6587ca /tmp/33b3ae77-45c6-4c6c-b32b-c8eeac2ce2da_pre-freeze.sh
Apr  8 16:47:14 db01 sudo: pam_unix(sudo:session): session opened for user root by youname(uid=0)
Apr  8 16:47:14 db01 sudo: pam_unix(sudo:session): session closed for user root
Apr  8 16:47:14 db01 sudo:   youname : TTY=pts/1 ; PWD=/home/youname ; USER=root ; COMMAND=/bin/chmod 0700 /tmp/33b3ae77-45c6-4c6c-b32b-c8eeac2ce2da_pre-freeze.sh
Apr  8 16:47:14 db01 sudo: pam_unix(sudo:session): session opened for user root by youname(uid=0)
Apr  8 16:47:14 db01 sudo: pam_unix(sudo:session): session closed for user root
Apr  8 16:47:14 db01 sudo:   youname : TTY=pts/1 ; PWD=/home/youname ; USER=root ; COMMAND=/tmp/33b3ae77-45c6-4c6c-b32b-c8eeac2ce2da_pre-freeze.sh
Apr  8 16:47:14 db01 sudo: pam_unix(sudo:session): session opened for user root by youname(uid=0)
Apr  8 16:47:14 db01 sudo: pam_unix(sudo:session): session closed for user root
Apr  8 16:47:14 db01 sudo:   youname : TTY=pts/1 ; PWD=/home/youname ; USER=root ; COMMAND=/bin/rm -f /tmp/33b3ae77-45c6-4c6c-b32b-c8eeac2ce2da_pre-freeze.sh
Apr  8 16:47:14 db01 sudo: pam_unix(sudo:session): session opened for user root by youname(uid=0)
Apr  8 16:47:14 db01 sudo: pam_unix(sudo:session): session closed for user root

 

 

I have no idea why that happens. What else could I check?

I am running on the latest VBR Release 12. The VM its running on is Windows 2022

The really strange thing is, there is also a Postgres DB on that server, and if I enable the AAB, it works. In auth.log I can see the commands issued.


8 comments

Userlevel 7
Badge +20

Guessing you haven’t mounted /tmp as noexec but I’d be interested if you’re doing anything with permissions on the location. I’d also connect to the machine to see if the file exists when Veeam is trying to execute it.

Userlevel 7
Badge +20

Remember that scripts run as the service account in Veeam and if that same account does not have permission you run in to issues like this.

Userlevel 7
Badge +20

Remember that scripts run as the service account in Veeam and if that same account does not have permission you run in to issues like this.

Sorry to disagree but this isn’t true, it’ll run as the account specified within guest processing:

 

When the job starts, Veeam Backup & Replication uploads scripts to the VM guest OS and executes them under the account specified in the Guest OS credentials section of the job settings.

 

Source: https://helpcenter.veeam.com/docs/backup/vsphere/pre_post_scripts.html?ver=120

Userlevel 7
Badge +20

Remember that scripts run as the service account in Veeam and if that same account does not have permission you run in to issues like this.

Sorry to disagree but this isn’t true, it’ll run as the account specified within guest processing:

 

When the job starts, Veeam Backup & Replication uploads scripts to the VM guest OS and executes them under the account specified in the Guest OS credentials section of the job settings.

 

Source: https://helpcenter.veeam.com/docs/backup/vsphere/pre_post_scripts.html?ver=120

Don't be sorry I must have been thinking of something else or maybe a previous version.  Thanks for clarifying.

Userlevel 7
Badge +20

Remember that scripts run as the service account in Veeam and if that same account does not have permission you run in to issues like this.

Sorry to disagree but this isn’t true, it’ll run as the account specified within guest processing:

 

When the job starts, Veeam Backup & Replication uploads scripts to the VM guest OS and executes them under the account specified in the Guest OS credentials section of the job settings.

 

Source: https://helpcenter.veeam.com/docs/backup/vsphere/pre_post_scripts.html?ver=120

Don't be sorry I must have been thinking of something else or maybe a previous version.  Thanks for clarifying.

Could be job level scripts as those run under the service account?

Userlevel 7
Badge +20

Remember that scripts run as the service account in Veeam and if that same account does not have permission you run in to issues like this.

Sorry to disagree but this isn’t true, it’ll run as the account specified within guest processing:

 

When the job starts, Veeam Backup & Replication uploads scripts to the VM guest OS and executes them under the account specified in the Guest OS credentials section of the job settings.

 

Source: https://helpcenter.veeam.com/docs/backup/vsphere/pre_post_scripts.html?ver=120

Don't be sorry I must have been thinking of something else or maybe a previous version.  Thanks for clarifying.

Could be job level scripts as those run under the service account?

Probably what I was thinking. 😜

Userlevel 1

Remember that scripts run as the service account in Veeam and if that same account does not have permission you run in to issues like this.

This is not a Windows VM - its a Debian VM and it uses the Linux user I assigned to it.

Userlevel 1

Guessing you haven’t mounted /tmp as noexec but I’d be interested if you’re doing anything with permissions on the location. I’d also connect to the machine to see if the file exists when Veeam is trying to execute it.

/tmp is not mounted as noexec. As I have written, executing the script with the same user in /tmp is working if I do it myself via SSH.
I managed to see the uploaded script in the /tmp directory, however that is difficult as its visible only for a very short time, it gets deleted after a fraction of a second.

Also: the Postgres application aware script  (which is included in VBR) is running also from /tmp, and that is working without an issue.

Comment