On of the large risks in IT isn’t just Ransomware, hardware faults, or even disaster recovery events.
What happens when the person who knows how everything works quits?
In many organizations, especially smaller ones, the Veeam Admin is not just responsible for dealing with backups and restores all day.
Some of the tasks us Administrators do are:
- Windows Server Administrators
- Virtualization Administrators
- Storage Administrators
- Security Administrators
- Network Administrators
- Domain Administrators
- Cloud Administrators
- Disaster Recovery
You get the point, we wear many hats, and have a lot of access and knowledge.
Here are a few things you need to do in the case someone who is responsible for these items do.
1. Change Every password they knew.
Veeam Service accounts, Repositories, Hypervisors, OS’s, Storage, Tape libraries, Storage arrays, Switches, Firewalls, Server management, Cloud accounts, Enterprise applications, Encryption passwords…. The list goes on and on, but you don’t know what was backed up to a drive in a text file at some point. Hopefully everyone was using a password vault, but to be 100% sure, this needs to be a priority.
2. Review Privileged Access
Review all groups, record, and remove what is necessary.
Domain Admins, Backup Operators, VMware admins, Storage admins, and whatever permissions have been grated over the time should be revoked, and local accounts also checked for.
3 Review software and licenses
Find the software, servers, and licenses they were responsible for, verify things are valid, and well documented, if not, you will have your work cut out for you trying to find every setting and configuration and why they are set that way. Take this as a good time to start the process and do it correctly.
4 Alerting and monitoring
In all alerting and monitoring tools and applications it’s now time to see alerts that were set to a single account rather than a group. This is very common and those alerts should not go unnoticed. It’s also risky if someone has used a personal account and receives these after they leave the company. Use this as a time to verify notifications, and SMTP settings are tested and actually working as expected, to a group for a team to receive.
5 Veeam Specific Tasks
Inventory every job, find out what is being protected and for how long, document everything. This is a great opportunity to use Veeam One to generate reports. Validate and test backups to make sure they are working, make sure you are following 3-2-2-1-0 and SureBackup, copy jobs and tape jobs are are working as expected.
Preform some Test restores as well to make sure the environment is working and you can continue to restore without someone who was maintaining the environment available.
Review your repository capacity. If someone intended to leave, they may have not felt it necessary to request additional storage in their last few weeks of employment. The last thing you want is for all the backups to start failing. Make sure to have alerts and monitor this and other backup processes while a new employee is being hired.
Find any documents about recovery or disaster recovery. If they don’t exist you need to work on this right away. Veeam Admins are often responsible for DR plans. If most of it is in there head, you need to be able to recover as quickly as possible in the event of a disaster.
6 Crosstrain Someone Else
I always make sure at least 2 people can Restore a VM, Files, recover AD, preform security updates on Veeam, Recovery from Ransomware, and preform Disaster Recovery tasks. I also want them to be able to create, modify, and trouble shoot failed jobs. Even if there is an open position for a backup admin, 2 people is the minimum as someone could get sick or injured, or also find other employment.
7 Don’t forget the Little Things
Many years ago I had a coworker retire, and I was finding his credentials everywhere. Even on AD authenticated devices. While he was extremely smart, hardworking, and an excellent employee, convenience is often favored due to the amount of workload us admins have. Thinks like BIOS, iDRAC, iLO credentials, Local accounts on SAN’s, SAN Switches, Applications, local administrator accounts popped up for years. After many years of service keeping his Email PST around was a life saver for some support related events as well. Many users sign up to licenses, or portals with their own accounts. It can be difficult sometimes to switch that over to a new user. Now i make sure multiple people have access to portals, even with limited accounts, or create secondary accounts in a password vault on t he chance I am away and my supervisor can get access.
When one of these administrators leaves, the focus shouldn't be limited to changing a Veeam and domain password or assigning a new job owner. It should trigger a full review of privileged access, documentation, operational procedures, and the overall recoverability of the environment.
You will have your work cut out for you, and you will find some things you didn’t expect, but treat it as a good way to verify documentation and procedures and assisting the next person to step in to the job role.
