What is this about?
There's a saying in IT security that it's never a question of if, but when you'll be attacked by intruders.
We're already doing a lot to harden our VBR and VB365 servers.
However, we must always assume that attackers will be able to bypass our defenses and gain access to our systems. That's why we always have at least one immutable repository (VHR, S3 w/object-lock, tape, ...).
But even then, it could be a matter of time before an attacker can bypass time-based object locks.
So we would definitely want to know as soon as possible when bad guys are strolling through our perimeter.
Have little birds tweet you
The concept of placing so called “honeypots” in an IT environment is well established: Honeypot (computing) - Wikipedia. Designing, placing and maintaining smart honeypots - like e.g. a fake VBR server - is a cumbersome process that needs attention.
We can do something much simpler that will also give us knowledge of intruders roaming our networks.
In former times miners took little canary birds with them. They are way more sensitive to carbon monoxide (CO) and other toxic gases than we are. So, once the canary next to you fainted, you should have been running (Canary used for testing for carbon monoxide 1926 US Bureau of Mines (youtube.com) - no animals had been harmed...).
used for testing carbon monoxide gas in 1928 (picture public domain).
In analogy to the duty of those birds Thinkst Canary has established a free of charge web service that can be used for our matters. The solution is both, modular and open source. The technique to trigger the alarm is either a HTTP-GET request or a DNS resolution to a randomly generated FQDN. The domain being accessed by the tokens can either be the free public web service provided by Thinkst Canary or you might run it in a Docker container in your own environment (GitHub - thinkst/canarytokens-docker: Docker configuration to quickly setup your own Canarytokens).
In the following examples I will show the straightforward way using the public web service.
Canary flavors
Using https://canarytokens.org/ we can generate those tokens for our backup environment in a matter of seconds. There are several types of tokens we will elaborate on in a second. In addition you can specify the action to take. Could e.g. be an email address or a webhook URL. On top you can prepare a reminder text for the token. We would want to understand what just happened, when the alarm is fired.
The following types of tokens can be useful for our mission:
- DNS token - alarm triggered when a hostname is requested
- Sensitive command token - alarm triggered when a suspicious Windows command is run
- Word document token - alarm triggered when a document is opened in Word
- Excel document token - alarm triggered when a document is opened in Excel
- Custom EXE/binary - alarm triggered when the EXE or DLL is execute
There are more tokens than that. But e.g. the PDF token needs the PDF to be opened in Acrobat Reader. If you open it in a regular browser, the token wont trigger. So I consider this to be less useful in our case.
The idea is to tripwire some actions we expect an attacker to take to silently raise an alarm. It should be an action that we rarely perform, but that an attacker is very likely to take advantage of.
It’s a trap! Some examples for tripwiring our VBR server
So let's say we are intruders in our own environment. What would you do if you reached, say, the VBR server?
Attackers usually need to determine who and where they are. Two commands immediately come to mind: whoami.exe and netsh.exe. These two are rarely used in normal operation, but have a high probability of being used during an intrusion.
We could either use the sensitive command token or the custom EXE token to prepare them for the alarm.
I would recommend the sensitive command token as it only requires a small addition to the registry.
The custom EXE token would need you to sign and replace the EXEs and this easily can get lost during the next Windows update. Technically, they are using the certificate revocation list (CRL) with it’s FQDN as a trigger here. The CRL is automatically checked when the signed EXE is executed.
Downside of the sensitive command token in contrast is, that it triggers when the command exits, not when it starts. So to tripwire cmd.exe with the sensitive command token would not be the best idea, as the attacker might just leave the shell open. But for the examples above, it’s just fine.
Now let’s prepare for netsh.exe to be tripwired. Create the token first:
Create the token on Canarytokens
All we have to do is specify the target email address for the notification and the EXE file. We can also add a description of what exactly happened and where.
In the next step we are offered to download our token in the form of a *.reg file.
Note that you have to import the *.reg to both registry branches (32bit and 64bit) to make sure it fires for both architectures.
Let’s have a look into the reg-file. The name actually represents the token ID.
b34pmz70igycoz2diztbiv5jc.reg in our example:
Windows Registry Editor Version 5.00
; Sensitive command token generated by Thinkst Canary
; Run the following commands with admin privs on a Windows machine:
; reg import FILENAME /reg:64
; reg import FILENAME /reg:32
; command that will be watched for
[HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows NT\CurrentVersion\Image File Execution Options\netsh.exe]
"GlobalFlag"=dword:00000200
; magic unique canarytoken that will be fired when this command is executed
[HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows
NT\CurrentVersion\SilentProcessExit\netsh.exe]
"ReportingMode"=dword:00000001
"MonitorProcess"="cmd.exe /c start /min powershell.exe -windowstyle hidden -command \"$($u=$(\\\"u$env:username\\\" -replace('[^a-zA-Z0-9\\-]+', ''))[0..63] -join '';$c=$(\\\"c$env:computername\\\" -replace('[^a-zA-Z0-9\\-]+', ''))[0..63] -join '';Resolve-DnsName -Name \\\"$c.UN.$u.CMD.b34pmz70igycoz2diztbiv5jc.canarytokens.com\\\")\""
Looking closely we can see, that it’s actually triggering a DNS resolution to b34pmz70igycoz2diztbiv5jc.canarytokens.com via an invisible monitoring process.
BTW, as we’re talking about it, check your “Image File Execution Options” (IFEO) hive. If you find anything suspicious here, you might already be pwned. This is a common attack vector as you can repeatedly run things in the background once other binaries are run: Invoke Malware with SilentProcessExit – Cyber Security Architect | Red/Blue Teaming | Exploit/Malware Analysis (rioasmara.com)
So once the netsh.exe is run...
...we will receive an email seconds later...
Using the links in the email you can disable and enable the token any time. Also the trigger history can be shown together with a visualization of the originating location.
Bear in mind, that we see a DNS resolver here. This is not an IP from your perimeter, but the upstream resolver the token was using on the last hop towards canarytokens.org’s own DNS. The token reminder you’ve set is your only way to get info about the originating device. Make sure to put something meaningful here.
What else to booby-trap?
Other ideas to set tripwires to:
- Netsh.exe
- Whoami.exe
- Ipconfig.exe
- Systeminfo.exe
- Net.exe
- Netstat.exe
- Tasklist.exe
- Qprocess.exe
- Ping.exe
- Sc.exe
- At.exe
For all of them: check if you’re using them regularly for good reasons (e.g. in scripts).
You could also add things you don’t have and use, but expect an attacker to bring and execute.
Like e.g:
- Mimikatz.exe
- lc7.exe (l0phtcrack)
- ...
Another smart option would be to set veeam.backup.shell.exe as the target. According to best practices we should be using the console remotely with 2FA and avoid it on VBR itself.
Don’t tripwire cmd.exe or powershell.exe. The Canary Token spawns those shells in hidden mode to carry out the DNS request. That would lead to recursive fun… ;)
Bear in mind though, that if you or the attacker renames the *.exe, your canary will not trigger. An easy way to circumvent the tripwire is to rename them before executing them. But as the attacker doesn’t know – will he consequently do so?
IT-Security is a process - not a single product
Also hopefully this is not our only line of defense. VBR should be hardened by all means of V12.1+ functionality: Security & Compliance Analyzer - User Guide for VMware vSphere (veeam.com)
This was just a brief example of the usage. You can bake this into a full blown SIEM by using webhooks or maybe the commercial version of Thinkst Canary. Or you might run it completely on your premises or within your cloud using the containerized version.
