Skip to main content

Background

 

With the recent release of Veeam Backup for Microsoft 365 (VB365) v8 there were several awesome additions to how VB365 proxies work. Scalable performance and high availability are now built in with the introduction of Proxy Pools and a the NATs queue that manages the processing of proxy tasks. We can now add or remove proxies from a proxy pool during backup processing and the queue will redistribute any remaining tasks among the active proxies. This opens the door for some dynamic proxy automation to either conserve resources while backups are not being processed or to save money by deallocating the unnecessary resources (Public Cloud).

 

This lead to a discussion between Brad Barker and I about how we could potentially pave the way, or at least point in the right direction, for our customers to take advantage of this opportunity. We decided to start with a simple idea of automating proxy power state to have proxies come online when needed and take them offline when they are no longer needed. The ultimate goal for us is the cost savings that could be found by deallocating these proxies when VB365 is deployed in the Public Cloud (Azure or AWS).

 

To kick things off, I started by outlining and building the underlying logic for the script using my lab built on VMware. This gave me the ability to build and test without occurring cloud costs until I was ready for some focused testing. There may not be such obvious value with automating VMware proxies, but there is some potential that could be expanded on. Since everything was built to work with VMware first, it makes sense for it to be the first walkthrough for our project.

 

How the Proxy Automation works

 

Since there are no pre or post job scripting options in VB365 like there is in VBR, the script is designed to run at intervals and check for running job sessions. If a running job session is identified, the proxy automations will launch and the script will then monitor the running session waiting for its completion. The proxy automation tasks are launched as background PowerShell jobs so that we can process multiple proxies in parallel. When the running job session is completed the script will run a check again for a timeout period to make sure that we do not stop the proxies if another job is running or starts within the timeout period. Then we will revert the proxy changes by applying maintenance mode and powering off any proxies that we designate to automate.

 

Since we are using VMware proxies, the script uses VMware PowerCLI to check and modify the VM power states.

 

Script logic flow

Start

  1. Initialize Parameters
    • Define vCenter Server credentials, check time, buffer time, output file path, proxy pool name, and proxy VM information.
  1. Check for Running VBO Job Sessions
    • Use GetVBOSession to check for running sessions within the specified CheckTime.
  2. If Running Job Found:
    • Start VBO proxies if needed (using StartVBOProxies).
    • Monitor the VBO job session until completion (using MonitorVBOSession).
  3. If No Running Job Found:
    • Check VBO proxy states (using CheckVBOProxies).
      • This will check proxy maintenance mode and power states and change as needed. 
  4. Stop Additional Proxies (if needed):
    • If any "Auto" proxies were started, stop them (using StopVBOProxies)

End

 

To run the script at intervals, we use a Windows scheduled task on the VB365 server to manage the script execution and to relaunch it daily in case of any errors or reboots, etc.

 

Configuration

 

Requirements

  • VB365 v8 or higher
  • VMware PowerCLI needs to be installed on the VB365 server for the script to interact with VMware proxies

 

Script Parameters

  • You will need to either download or copy the script and place it on the VB365 server.
  • The script has several parameters that will need to be modified to work in your environment
    • The required variables that will need to be configured are the vCenter server name and credentials, the output (log) file location, the proxy pool name that we are running the automation against, and the proxy information found in the $VBOAutomatedProxies array.

 

** You will find these variables at the top of the script under the "** Parameters **" comment heading. Anything found below the "** Functions **" heading should not need to be modified.

** The script is commented with explanations of each variable and property that needs to be added.

#   ** Parameters **

# VMware PowerCLI variables
$vCenterServer = "<String:vCenterServer DNSName or IP>"
$vCenterServerCredsUsername = "<String: username>"
$vCenterServerCredsPassword = "<String: password>"

# Time, in minutes, to run GetVBOSession (will check for running jobs every 30 seconds for the duration specified)
$CheckTime = 4

# Time, in minutes, to check for additional sessions before ending
$BufferTime = 1

# Filepath for logging
$OutputFile = "<String: Path\to\log\file>"

# Name of proxy pool to be automated
$ProxyPoolName = "<String: Proxy pool name>"

<#
List of Proxy VM attributes used to manage proxy power and maintenance mode operations
ProxyVMName:
Name of the VM in Azure
ProxyVMIPv4:
The local IPv4 address of the proxy (potentially used to add proxy to VBO)
ProxyMode:
Set which proxy is to stay online ('Primary') and which are to be stopped after jobs complete ('Auto')
ProxyBootTime:
Sets the wait time (in seconds) after starting the VM for the guest OS to be up before sending proxy commands
PoxyUserName:
Windows- Start a local account username with "`.\" The period must be escaped with a back tick character. Domain account can use "domain\username" format.
Linux - no domain needed
ProxyUserPass:
Password string
ProxyRootPass:
Root password string which may be required for Linux proxies.
#>
$VBOAutomatedProxies = @(
=PSCustomObject]@{
ProxyVMName = "<String>";
ProxyVMIPv4 = "<String>";
ProxyMode = "Primay";
ProxyOS = "<Windows/Linux>";
ProxyBootTime = 120;
ProxyUserName = '<String>';
ProxyUserPass = '<String>';
ProxyRootPass = '<String>'
}
PSCustomObject]@{
ProxyVMName = "<String>";
ProxyVMIPv4 = "<String>";
ProxyMode = "Auto";
ProxyOS = "<Windows/Linux>";
ProxyBootTime = 120;
ProxyUserName = '<String>';
ProxyUserPass = '<String>';
ProxyRootPass = '<String>'
}
)

 

  • The $CheckTime and BufferTime variables can either be left with the defaults or modified as desired.

 

Scheduled Task

  • Launch Task Scheduler, right-click on the "Task Scheduler Library" and select "Create Task"
  • Name your task. The naming can be whatever you choose. For my testing I named it VBOJobMonitor.
  • On the "Triggers" tab, create a new trigger to launch the script once daily and at intervals that fit with your $CheckTime value. Example - $CheckTime of 4 minutes would fit nicely with a repeat interval of 5 minutes
  • On the "Actions" tab, create a new action with the following settings:
    • Program/script:
      • Powershell.exe
    • Add arguments (optional):
      • -ExecutionPolicy Bypass -File "C:\Path\To\File.ps1"
        • Replace the file path with the file path of the ps1 script on the VB365 server
        • You can remove the "-ExecutionPolicy Bypass" if that is not needed.
  • The rest of the settings can stay with the defaults and save the task.

 

With all of this completed, you should be able to watch the proxy automations in action! The last item to cover is how to monitor the automation progress.

 

Monitor

 

The script has been designed to log throughout it's execution. It will write these logs to the log file that you designated in the script parameters. Here are some excerpts from the script execution as an example:

 

Running job session found and automations launched:

 

Rechecking for additional jobs:

 

Once all jobs are completed automate proxies again:

Troubleshooting

There is some "Self-healing" built into the script to try and correct potential manual changes to the proxies or if a failure occurs during a previous run. So some errors or warnings encountered in the logging can be resolved without intervention. However, if some troubleshooting is needed the logs should show you where in the execution any failures occurred.

I will be publishing Part 2 (Azure) soon!


Good job so far, I will try to test it as soon as I will have my next deployment.

Really looking forward for the Azure part!


Great post, @tfewins! Can’t wait for Part 2, @Brad!


Quick question: Is there a plan to introduce pre or post job options in VB365 in the next release?


Part 2 (Azure) is live! You can find it at:

 


Quick question: Is there a plan to introduce pre or post job options in VB365 in the next release?

Haven’t heard if that is road map, but that would be an awesome way to run these kinds of automations without having to have the scheduled task approach. I would upvote that feature  😁


Going to test out this script and see how it works with our VB365 v8 deployments.  Will provide feedback here.


Such an knowledgeable post thank you for posting.


Hi ​@tfewins 

 

I have a question it would work with GCP ? Is there any limitations ? How can we see this with GCP?

 

Thank you,

 

 


Hi ​@tfewins 

 

I have a question it would work with GCP ? Is there any limitations ? How can we see this with GCP?

 

Thank you,

 

 

The script is written specifically for the platform in question. In this case it’s looking at vSphere. So the code interacts with esxcli. There are also scripts targeted at Azure and AWS. However we have not made a script for GCP.

 

Are you looking to do this with GCVE or native GCP?


@tfewins -- can this script be used to just deploy new Proxies versus the monitoring/shutting down of them?  I would assume so by setting all of them to “Primary” vs “Auto”?


@tfewins -- can this script be used to just deploy new Proxies versus the monitoring/shutting down of them?  I would assume so by setting all of them to “Primary” vs “Auto”?

This version of the script is meant to automate bringing up the “Auto” proxies by powering them on and toggling maintenance mode, and then taking them down when jobs are detected as running within the vSphere environment.

 

You could definitely modify it to deploy proxies from an OVA, template, or something else within vSphere, but the listener aspect is really there so that we can tell when jobs are running and then do the bits of automation to spin up the machines in the proxy pool.

 

Is that more in line with what you’re looking for? Deployment of the proxy components itself and then full tear down and removal of them from the environment?


@tfewins -- can this script be used to just deploy new Proxies versus the monitoring/shutting down of them?  I would assume so by setting all of them to “Primary” vs “Auto”?

This version of the script is meant to automate bringing up the “Auto” proxies by powering them on and toggling maintenance mode, and then taking them down when jobs are detected as running within the vSphere environment.

 

You could definitely modify it to deploy proxies from an OVA, template, or something else within vSphere, but the listener aspect is really there so that we can tell when jobs are running and then do the bits of automation to spin up the machines in the proxy pool.

 

Is that more in line with what you’re looking for? Deployment of the proxy components itself and then full tear down and removal of them from the environment?

Hey Brad - I am looking at ways to deploy new proxies for an environment and keep them there for scaling up.  I am looking in to Ansible and some other tools we use here as well but was wondering about this script, but will forego that since it is meant as you explained.


@tfewins -- can this script be used to just deploy new Proxies versus the monitoring/shutting down of them?  I would assume so by setting all of them to “Primary” vs “Auto”?

This version of the script is meant to automate bringing up the “Auto” proxies by powering them on and toggling maintenance mode, and then taking them down when jobs are detected as running within the vSphere environment.

 

You could definitely modify it to deploy proxies from an OVA, template, or something else within vSphere, but the listener aspect is really there so that we can tell when jobs are running and then do the bits of automation to spin up the machines in the proxy pool.

 

Is that more in line with what you’re looking for? Deployment of the proxy components itself and then full tear down and removal of them from the environment?

Hey Brad - I am looking at ways to deploy new proxies for an environment and keep them there for scaling up.  I am looking in to Ansible and some other tools we use here as well but was wondering about this script, but will forego that since it is meant as you explained.

Tyson and I have discussed a v2 of this that builds the VMs and tears them down afterwards in vsphere, as well as using auto scaling within AWS/Azure. We have not really had the cycles to delve into it heavily though.


@tfewins -- can this script be used to just deploy new Proxies versus the monitoring/shutting down of them?  I would assume so by setting all of them to “Primary” vs “Auto”?

This version of the script is meant to automate bringing up the “Auto” proxies by powering them on and toggling maintenance mode, and then taking them down when jobs are detected as running within the vSphere environment.

 

You could definitely modify it to deploy proxies from an OVA, template, or something else within vSphere, but the listener aspect is really there so that we can tell when jobs are running and then do the bits of automation to spin up the machines in the proxy pool.

 

Is that more in line with what you’re looking for? Deployment of the proxy components itself and then full tear down and removal of them from the environment?

Hey Brad - I am looking at ways to deploy new proxies for an environment and keep them there for scaling up.  I am looking in to Ansible and some other tools we use here as well but was wondering about this script, but will forego that since it is meant as you explained.

Tyson and I have discussed a v2 of this that builds the VMs and tears them down afterwards in vsphere, as well as using auto scaling within AWS/Azure. We have not really had the cycles to delve into it heavily though.

Sounds good.  I will circle back once you get to v2. 😁


Comment