VRO: Restore from Backup, fundamentals and deployment considerations

The overall intent of this article is to consolidate major considerations when it comes to using Veeam Recovery Orchestrator to Recover from Backups. 

Note: We will cover Backup Copies in another article, which is generally for recovery of VMs to another site during a critical site failure. 

Key:

• VBR: Veeam Backup & Replication
• VRO: Veeam Recovery Orchestrator

Orchestrating recovery from Backups - Use-Case Overview:

• Below you will see a is common customer design example.
  • Both sites are Active/Active with production virtual workloads being protected with Backup Jobs.
  • (1) VBR server at Site-Z which is the DR location.
  • Backup Proxy at each location.
  • Backup Repository at each location.
  • (1) vCenter at Site-Z and vSphere clusters at each location.
  • Agent deployed protecting a physical workload.
  • Our Veeam Transport (Data Mover) service is represented by a (M) icon here for these machines.
  • Our Agent services are represented by a (A), although the Agent Backup and the one on VBR server is an Orchestrator Agent, they are different services.
• Backup Proxies:
  • Virtual or Physical Proxy server transporting/source data from the Production vSphere clusters at each location.
• Backup Repositories:
  • These are deployed at each location, generally a Windows/Linux server with direct attached storage, NFS/SMB or Storage appliance specific.
  • RPO-1-A is the backup repository for Site-A VMs and Agent Backups.
  • RPO-1-Z is the backup repository for Site-Z VMs.
  • This is also often the Mount server, which is used to mount data for instant recovery with our vNFS protocol.
    • You will also see Gateway mentioned here, this is specific for a certain backup repository which are unable to have the Data Mover installed locally. (IE: NFS/SMB)
    • Note: This representation is acting a Direct Attached Storage on a Virtual Machine which has the Data Mover installed locally.
• Enterprise Manager and Veeam Recovery Orchestrator:
  • For this example, we have these management overlay servers in the cloud, it's best when they are not on-prem if a site fails.
  • Even better would be to have VBR (VBR-Z) also in the cloud, but for the sake of common configurations we will leave it at Site-Z.
    • There are also considerations regarding Virtual Labs and testing that will come up in later articles I plan to develop.

Additional Site Recovery Considerations:

• Backups are for recovering data from it’s own location and recovering it back to the same location. (Site-A → Site-A)

• If Site-Z fails, VBR-Z is sitting on location, Veeam Recovery Orchestrator requires VBR to be online for recovery, as we need access to the policies and repositories.

  • There are ways around this, as previously mentioned having your VBR server in the cloud or witness location.

  • Replicating the VBR server with Veeam and doing a manual recovery of the VM at the other location.

  • Taking a configuration backup of Veeam, having a cold server in-place at Site-A, cloud or witness site to recover the VBR server during an orchestration event.

    • Note: That this failure will cause an increased RTO for a Site-Z failure, Site-A will not be impacted during a critical site failure as VBR will be online.

How do Restores from Backup work: (Referencing the above image)

Sourcing Data:

• VBR-Z has backup policies configured to protect both Site-A and Site-Z.
• B1-PRX-A and B1-PRX-B source data from Site-A and Site-Z respectively.
  • Agent Backup has its own in-guest OS Agent installed that facilitates source and moving data to the onsite repository.
• The backup chains/data are stored on RPO-1-A and RPO-1-Z respectively. 

Restoring Data:

• We have multiple ways to restore data in Veeam, the one we will focus on is VM Restore.
  • VM Restore rehydrates the entire VM from the backup Repository with our Transport service into a ESXi Host and Datastore you specify during the recovery operation.
    • Reference: https://helpcenter.veeam.com/docs/backup/vsphere/full_recovery.html

Veeam Backup Repository location: (Restore Source)

• One of the most important design aspects is the location of the backup data in which you plan to restore from.
  • The best option here is a local repository for locally protected workloads.
    • Note: Pulling from a Data Domain appliance which needs to rehydrate globally deduplicated data could add restore latency.

Veeam Recovery Orchestrator:

• It's important to note that for Veeam Recovery Orchestrator (VRO) we are utilizing components created and configured from the Veeam Backup & Replication interface.
• Veeam Recovery Orchestrator is an additional product and deployment on top of your existing Veeam Backup & Replication server, it connects to your Enterprise Manager and/or VBR instances and creates orchestration overlays.
  • VRO Placement: If you have presence in the cloud, we would recommend that this be placed in the cloud or at a Witness location.
  • Additional considerations: Veeam University (VU) https://veeam.looop.co/topic/834185 
• VMware Tagging: A popular but very important aspect of our Veeam product when grouping and categorizing workloads to protect and in this case grouping of servers for restoration.
  • With Veeam Recovery Orchestrator mapping and grouping your workloads for orchestration, it's best practice to-do so with VMware Tags.

Veeam Recovery Orchestrator Core Components:

• Recovery Locations: We create and utilize these in VRO to pinpoint the resources we are going to utilizing during a recovery event. Some of the selections include source data and placement of converted workloads. (Compute, Storage, Networking and Re-IP Rules.)
• Recovery Plans: A recovery plan is where we select the workloads we will orchestrate and the Recovery Location in which those workloads will be recovered to.

How to Configure VRO for Orchestration of a Test VM in Site-A:

• Below I am going to walk you through how to configure an example recovery of a Site-A VM being recovered back to the same site.
• We will configure VMware Tags, Recovery Locations, and Recovery Plans.

  • Note: Agent Recovery, you can't use VMware Tags but you can select Agent Backup policies protecting that VM instead.

Creating VMware Tags:

• We need to create at a minimum (3) VMware tags for each Site on our vCenters or single vCenter if it manages both sites.
• These Tags will be used to define Compute and Storage locations for the entire VM restore of a Virtual or Agent workloads.
• Create Tags:
  • In vCenter you will want to go to your menu in the upper left-hand corner and then select (Tags & Custom Attributes) from the list.
  • Create a new Category and name it VRO or something similar.
  • Create (3) New Tags for each location and associate them to the VRO Category you created. (IE: VRO_COMPUTE_SITE-A, VRO_STORAGE_SITE-A, and VRO_TESTVM_SITE-A)
    • Reference: I would link a VMware article but that might change, so google "What are VMware Tags and How to use them"
• Associate Tags:
  • Go back to Inventory on the upper left-hand menu in vCenter and find your Compute and Storage to Tag.
  • Associate the compute Tag you created to an ESXi host or Cluster.
  • Associate the Storage Tag you created to a Datastore or Cluster.
  • Associate the TESTVM Tag to a Test server that you have protected with a Backup Job on the VBR server.
• Updating VRO Inventory:
  • Use the Tags created above, if you don't see them in the VRO inventory, login to the Veeam One Web client (https://VROVM:1239/) and run a rescan from the configuration page.

Creating Recovery Locations: 

• Now that we've identified the Compute and Storage locations, we can now use these VMware Tags in our Recovery Location in Veeam Recovery Orchestrator.
  • Login to your VRO server: (https://VROVM:9898), click the gear icon in the upper right-hand corner to enter Config/Administration mode.
  • On the left-hand side in the Navigation pane click (Recovery Locations), then click (Add).
  • Once in the New Recovery Location Wizard fill out the following:
    • Select (VMware) as the Location Type.
    • Give a Location Name: (IE: VRO_RECOVERY_LOCATION_SITE-A)
    • Recovery Options: Select (Recovery Agent Backups) and/or (Recover VM Backups) as needed.
    • Compute Resources: Select the (VRO_COMPUTE_SITE-A) Tag we created earlier.
    • Storage Resource: Select the (VRO_STORAGE_SITE-A) Tag we created earlier.
    • Storage Options: Don't select backup copies, this is for cross site recovery. (%80 of Capacity is fine)
    • Agent / VM Networks: This will map source VMware Network/Port Groups, back to the Source or different VMware Network/Port Groups as needed.
      • If you are restoring back to the same site and same network through VRO in a use-case such as ransomware.
      • Note: We capture the Port Group the VM is on during the backup which is stored in the .VMX file.
    • Re-IP Rules: This is only required if you are restoring to a different destination Port Group/network.
    • Data Sovereignty: This is defined in VBR and can be enforced here in VRO If needed.
      • Reference: https://helpcenter.veeam.com/docs/vro/userguide/adding_restore_locations.html

Creating Recovery Plans:

• Now that we've created a Recovery Location, we will create a Recovery Plan that will associate to our newly created Recovery Location.
• Exit Administrator by clicking (Exit Administration) in the upper lefthand corner.
  • On the left-hand side in the Navigation pane click (Recovery Plans), select (Manage), and select (New) 
  • Once in the New Restore Plan Wizard fill out the following:
  • Give a Recovery Plan Name: (IE: VRO_RECOVERY_PLAN_SITE-A)
  • Plan Type: Select (Restore) plan type.
  • Recovery Location: Select the (IE: VRO_RECOVERY_LOCATION_SITE-A) Recovery Location that we created.
  • Inventory Groups: This is where we can select the (VRO_TESTVM_SITE-A) Tag we created.
    • Or Agent Backup Policy for Agent testing.
    • Note: You can also select the Backup Job you have created as well, tags are best for scaled out deployments or single VM testing.
  • VM Recovery Options: Choose defaults here.
  • VM Steps: Add Ping Test
  • Protect Inventory Groups: Only for reprotection of the Virtual Machine after a commit failover.
  • RTO&RPO: This simply validates whether the protected workloads are achieving these marks for the associated Backup Plan.
  • Report Template: Default
  • Report Scheduling: Default
    • Reference: https://helpcenter.veeam.com/docs/vro/userguide/creating_restore_plans.html

Closing:

• Once you have this all created you are ready to run a live recovery of a Virtual Machine you backed up with Veeam Backup & Replication.
  • Note: This is a live recovery and should be taken at your own risk, Veeam is not responsible for any production outages or loss of data for your recovery configurations and actions.

Recovery_Workbook: https://docs.google.com/spreadsheets/d/1qyZHWLFMg64q9CfQ3Ohnv1gRWNzT9LSzpW7L8JKkCEQ/edit?usp=sharing