Upgraded to VBR 13.1? You might want to re-run your plug-in wizard!

Here's a short one from the field that might save you some troubleshooting time.

The setup

We had an unmanaged Oracle RMAN plug-in on SUSE Linux. It ran fine for a long time under VBR 13.0.2 and earlier with the matching plug-in version. We upgraded VBR to 13.1.1.18 and then updated the plug-in. The 13.1 plug-in also needs a new OpenSSL package, so we installed that and checked it as well. Everything looked clean.

The problem

After the upgrade, the RMAN backups failed. The plug-in could no longer talk to the backup server properly.

Nothing in the upgrade documentation suggested any further step was needed, so we opened a support case.

What changed under the hood

We compared the plug-in's veeam_config.xml from before and after the upgrade:

• The stored VBR fingerprint changed from a 40-character value (SHA-1 length) to a 64-character value (SHA-256 length).
• The encrypted passwords (VBR user and Oracle DB credentials) are stored in a different format afterwards.

So the ciphers and hashing behind the plug-in configuration changed. My guess is that this is linked to the new OpenSSL dependency, but Veeam hasn't confirmed that yet. What is clear: a configuration written by the old plug-in isn't fully valid for the new one.

The fix

Run the configuration wizard again:

OracleRMANConfigTool --wizard

Go through all the steps. The wizard then writes the fingerprint and the credentials in the new format, and the backups work again.

Two pitfalls:

• At the password prompt, press Enter without typing anything to keep the existing password. If you've typed something by accident, you can't see it because input is hidden. Cancel with Ctrl+C and start over.
• If the wizard stops with Invalid AES-GCM payload size detected / Failed to decrypt buffer using AES algorithm while it reads the existing DB credentials, enter the credentials again instead of reusing the stored ones.
•  

Not just Oracle, and not just unmanaged

The same thing should affect the other application plug-ins, because the configuration mechanism works the same way.

We also saw backup failures with managed plug-ins right after the upgrade. We think it's the same issue. There, though, it went away on its own after a while, without us doing anything. Our explanation is that the Protection Group reapplied the configuration and wrote it in the new format. With unmanaged plug-ins nothing does that for you, so you have to run the wizard yourself.

Takeaway

If you run unmanaged application plug-ins, plan a wizard run on every plug-in server after upgrading to 13.1 together with the update of the plugins themselves. With managed plug-ins, give the Protection Group some time or start a rescan manually before you start troubleshooting. I've asked Veeam support to add this step to the upgrade documentation.

Hope this helps someone. Let me know in the comments if you've seen the same.

Cheers, Michael