Background

With the recent release of Veeam Backup for Microsoft 365 (VB365) v8 there were several awesome additions to how VB365 proxies work. Scalable performance and high availability are now built in with the introduction of Proxy Pools and a the NATs queue that manages the processing of proxy tasks. We can now add or remove proxies from a proxy pool during backup processing and the queue will redistribute any remaining tasks among the active proxies. This opens the door for some dynamic proxy automation to either conserve resources while backups are not being processed or to save money by deallocating the unnecessary resources (Public Cloud).

This lead to a discussion between Brad Barker and I about how we could potentially pave the way, or at least point in the right direction, for our customers to take advantage of this opportunity. We decided to start with a simple idea of automating proxy power state to have proxies come online when needed and take them offline when they are no longer needed. The ultimate goal for us is the cost savings that could be found by deallocating these proxies when VB365 is deployed in the Public Cloud (Azure or AWS).

To kick things off, I started by outlining and building the underlying logic for the script using my lab built on VMware. This gave me the ability to build and test without occurring cloud costs until I was ready for some focused testing. There may not be such obvious value with automating VMware proxies, but there is some potential that could be expanded on. Since everything was built to work with VMware first, it makes sense for it to be the first walkthrough for our project.

How the Proxy Automation works

Since there are no pre or post job scripting options in VB365 like there is in VBR, the script is designed to run at intervals and check for running job sessions. If a running job session is identified, the proxy automations will launch and the script will then monitor the running session waiting for its completion. The proxy automation tasks are launched as background PowerShell jobs so that we can process multiple proxies in parallel. When the running job session is completed the script will run a check again for a timeout period to make sure that we do not stop the proxies if another job is running or starts within the timeout period. Then we will revert the proxy changes by applying maintenance mode and powering off any proxies that we designate to automate.

Since we are using VMware proxies, the script uses VMware PowerCLI to check and modify the VM power states.

Script logic flow

Start

1. Initialize Parameters
   • Define vCenter Server credentials, check time, buffer time, output file path, proxy pool name, and proxy VM information.

1. Check for Running VBO Job Sessions
   • Use GetVBOSession to check for running sessions within the specified CheckTime.
2. If Running Job Found:
   • Start VBO proxies if needed (using StartVBOProxies).
   • Monitor the VBO job session until completion (using MonitorVBOSession).
3. If No Running Job Found:
   • Check VBO proxy states (using CheckVBOProxies).
     • This will check proxy maintenance mode and power states and change as needed. 
4. Stop Additional Proxies (if needed):
   • If any "Auto" proxies were started, stop them (using StopVBOProxies)

End

To run the script at intervals, we use a Windows scheduled task on the VB365 server to manage the script execution and to relaunch it daily in case of any errors or reboots, etc.

Configuration

Requirements

• VB365 v8 or higher
• VMware PowerCLI needs to be installed on the VB365 server for the script to interact with VMware proxies
  • This requires vCenter credentials with the permissions to read the VM parameters and modify the power state.
  • https://developer.broadcom.com/powercli/installation-guide

Script Parameters

• You will need to either download or copy the script and place it on the VB365 server.
  • You can find the VMware Proxy Automation script here: https://github.com/VeeamHub/powershell/tree/master/VBO-ProxyAutomation/VMware
• The script has several parameters that will need to be modified to work in your environment
  • The required variables that will need to be configured are the vCenter server name and credentials, the output (log) file location, the proxy pool name that we are running the automation against, and the proxy information found in the $VBOAutomatedProxies array.

** You will find these variables at the top of the script under the "** Parameters **" comment heading. Anything found below the "** Functions **" heading should not need to be modified.

** The script is commented with explanations of each variable and property that needs to be added.

#   ** Parameters **

# VMware PowerCLI variables
$vCenterServer = "<String:vCenterServer DNSName or IP>"
$vCenterServerCredsUsername = "<String: username>"
$vCenterServerCredsPassword = "<String: password>"

# Time, in minutes, to run GetVBOSession (will check for running jobs every 30 seconds for the duration specified)
$CheckTime = 4

# Time, in minutes, to check for additional sessions before ending
$BufferTime = 1

# Filepath for logging
$OutputFile = "<String: Path\to\log\file>"

# Name of proxy pool to be automated
$ProxyPoolName = "<String: Proxy pool name>"

<#
List of Proxy VM attributes used to manage proxy power and maintenance mode operations 
    ProxyVMName: 
        Name of the VM in Azure
    ProxyVMIPv4:
        The local IPv4 address of the proxy (potentially used to add proxy to VBO)
    ProxyMode:
        Set which proxy is to stay online ('Primary') and which are to be stopped after jobs complete ('Auto')
    ProxyBootTime:
        Sets the wait time (in seconds) after starting the VM for the guest OS to be up before sending proxy commands
    PoxyUserName:
        Windows- Start a local account username with "`.\" The period must be escaped with a back tick character. Domain account can use "domain\username" format.
        Linux - no domain needed
    ProxyUserPass:
        Password string
    ProxyRootPass:
        Root password string which may be required for Linux proxies.        
#>
$VBOAutomatedProxies = @(
    =PSCustomObject]@{
        ProxyVMName = "<String>";
        ProxyVMIPv4 = "<String>";
        ProxyMode = "Primay";
        ProxyOS = "<Windows/Linux>";
        ProxyBootTime = 120;
        ProxyUserName = '<String>';
        ProxyUserPass = '<String>';
        ProxyRootPass = '<String>'
    }
     PSCustomObject]@{
        ProxyVMName = "<String>";
        ProxyVMIPv4 = "<String>";
        ProxyMode = "Auto";
        ProxyOS =  "<Windows/Linux>";
        ProxyBootTime = 120;
        ProxyUserName = '<String>';
        ProxyUserPass = '<String>';
        ProxyRootPass = '<String>'
    }
)

• The $CheckTime and BufferTime variables can either be left with the defaults or modified as desired.

Scheduled Task

• Launch Task Scheduler, right-click on the "Task Scheduler Library" and select "Create Task"
• Name your task. The naming can be whatever you choose. For my testing I named it VBOJobMonitor.

• On the "Triggers" tab, create a new trigger to launch the script once daily and at intervals that fit with your $CheckTime value. Example - $CheckTime of 4 minutes would fit nicely with a repeat interval of 5 minutes

• On the "Actions" tab, create a new action with the following settings:
  • Program/script:
    • Powershell.exe
  • Add arguments (optional):
    • -ExecutionPolicy Bypass -File "C:\Path\To\File.ps1"
      • Replace the file path with the file path of the ps1 script on the VB365 server
      • You can remove the "-ExecutionPolicy Bypass" if that is not needed.

• The rest of the settings can stay with the defaults and save the task.

With all of this completed, you should be able to watch the proxy automations in action! The last item to cover is how to monitor the automation progress.

Monitor

The script has been designed to log throughout it's execution. It will write these logs to the log file that you designated in the script parameters. Here are some excerpts from the script execution as an example:

Running job session found and automations launched:

Rechecking for additional jobs:

Once all jobs are completed automate proxies again:

Troubleshooting

There is some "Self-healing" built into the script to try and correct potential manual changes to the proxies or if a failure occurs during a previous run. So some errors or warnings encountered in the logging can be resolved without intervention. However, if some troubleshooting is needed the logs should show you where in the execution any failures occurred.