Veeam S3 Error Log Parsing Script -- Check S3 errors on all your gateways automatically

Not sure how useful this will be for anyone else, but should you need to parse through your logs on all gateway servers for S3 errors returned during Veeam operations with Object Storage, this script is for you.

Script requires ripgrep to be installed on the Backup Server: https://github.com/BurntSushi/ripgrep?tab=readme-ov-file#installation

If you cannot install / use chocolately in your environment, download the ripgrep binary appropriate for your OS, and edit the script as noted in comment [2].

What it does:

1. Collect all Object Storage Repsositories
2. Get Ids for all assigned gateways
3. Collect all gateways with Get-VBRServer cmdlet to create array of gateways
4. Iterate over each gateway and do the following:
   1. Export logs JUST for that gateway server to the VBR server, uncompressed
   2. grep the logs for the relevant S3 error string with ripgrep (rg command)
   3. Write all found errors in full to a .log file
   4. Pares all found errors and write in human friendly format to CSV
   5. Clean up the exported logs, continue iterating over gateways

The work will be stored under C:\temp\$backupServerName, and each gateway will have its own folder.

The script will take some time to run. Like a long time with many gateways. Main purpose of the script is to avoid gigantic log bundle, needing space for gigantic log bundle, and to collect this information completely unattended.

This has been used in rather large Veeam environments (hundreds of buckets + gateways) to help narrow down Object Storage issues, giving us better visibility on whether a specific gateway is being trouble or if there are periods of time when the target S3 server cannot handle request load, etc.

Note that you will need free space on the Backup Server for the log export; by design the script exports logs from each gateway one at a time and cleans up before continuing to the next gateway server. Space required for the log bundle per gateway server will be the same as the space used by the log directory on the gateway server. 

Please see the comments in the script for additional details; I realize this is extremely niche situation, but perhaps others will find it useful, or at least interesting.

Get-VBRLicenseAutoUpdateStatus | Out-Null
$workspacePath = "C:\temp\$($env:COMPUTERNAME)-s3workspace" #Comment[0] 
If(-not(Test-Path $workspacePath)){New-Item -ItemType Directory -Force -Path $workspacePath | Out-Null}
$allObjStgRepos = [Veeam.Backup.Core.CBackupRepository]::GetAll() | Where-Object {$_.Type -like "*S3*"}
$gatewayIDs = $allObjStgRepos.GetAssignedGatewayIDs() | Sort -Unique
$gateways = Get-VBRServer | Where-Object {$_.Id -in $gatewayIDs}
Foreach($g in $gateways){
	If($g.id -eq '6745a759-2205-4cd2-b172-8ec8f7e60ef8'){Continue} #Comment[1]
	$gName = $g.name
	$logPath = New-Item -ItemType Directory -Force -Path "$workspacePath\$($g.Name)"
	$logExport = Export-VBRLogs -Server $g -FolderPath $logPath.FullName -Compress:$false
	$logFiles = Get-ChildItem -Path $logPath.FullName
	rg 'Amazon REST exception with status' "$($logFiles.FullName)" | out-file $logPath\$gName-s3errors.log #Comment[2]
	$content = Get-Content $logPath\$gName-s3errors.log
	$report = @()
	Foreach($c in $content){
                $name = $c.Split(".")[1]
                $timestamp = $c.Split("[")[1].Split("]")[0]
                $date = $timestamp.Split()[0]
                $time = $timestamp.Split()[1]
                $errorCode = $c.Split("=")[1].Split(",")[0]
                $data = [PSCustomObject]@{
                                machine = $name
                                date = $date
                                time = $time
                                s3response = $errorCode
                }
                $report += $data
                Remove-Variable data
	}
	$report | Export-CSV -NoTypeInformation $logPath\$($gName)-errorreport.csv
	Get-ChildItem $logFiles.FullName -File -Recurse | Sort-Object FullName -Descending | Remove-Item -Force -Confirm:$false
	Get-ChildItem $logFiles.FullName -Directory -Recurse | Sort-Object FullName -Descending | Remove-Item -Force -Confirm:$false
	Remove-Item $logFiles.FullName -Force
}


<#

0 - Change this path if desired to a disk with ample free space. Log exports can be quite big, and by default the Export-VBRLogs command only exports for last 1 day

1 - Script is designed for large environments with many gateways, and the assumption is that the Backup Server itself IS NOT A GATEWAY. We hardcode the backup server ID and skip it as the logs for the backup server are expected to be extensive. For environments where Backup server may or is used as a gateway (or environments with tons of space), you can comment out this line and the backup server logs will be parsed also.

2 - If installation of ripgrep via chocolatey is not allowed in the environment, you can download the rg.exe binary from ripgrep github and replace `rg` with full path to the binary, e.g., C:\temp\rg\rg.exe

#>