Houston, we have a Connection Problem – When Veeam Enterprise Manager no longer communicates with VBR and/or no longer synchronizes the Catalog Service

Recently, a customer came to me with what seemed to be a simple question:

“Our Veeam Enterprise Manager can suddenly no longer properly connect to the Veeam Backup & Replication server, and the Catalog Service no longer synchronizes. But nothing was changed. What could be the reason?”

Ah, the famous IT words: Nothing was changed.

Of course, I basically understand what is meant by that. But it does not necessarily mean that a person changed something. Maybe a process, an automation, an update, a migration, etc. changed something. And of course, it is also possible that another person changed something without informing anyone, or that you changed something yourself but do not remember it anymore — or you do not see the impact because, at first glance, it does not seem to have anything to do with the issue.

Well then, time to roll up our sleeves and go through the checklist:

• Was there a Veeam update?
  No
• Was there a configuration change in Veeam Backup Enterprise Manager or Veeam Backup & Replication?
  No
• Was there a Microsoft update?
  No, or not known
• Was there a reboot?
  Yes

Well then, it could be that something changed on VEM or VBR, because during a reboot certain things are sometimes only executed or applied afterwards. In addition, you should also check the Windows Update logs.
 

So let’s take a look at the logs… what do we find there?
 

Unable to connect to VBR [net.tcp://fqdn-vbr:9405/certauth]...The certificate chain was issued by an untrusted certification authority (System.ComponentModel.Win32Exception)Failed to connect to backup server "<fqdn>" ...

This made one thing clear already:

The problem is not simply the Catalog Service itself, but rather the communication between Veeam Backup Enterprise Manager and the Veeam Backup & Replication Server in general.

For the rest of this article, I will abbreviate Veeam Backup Enterprise Manager as EM and Veeam Backup & Replication as VBR.

Fortunately, there are some good troubleshooting points and approaches for this type of issue.

Spoiler alert: In our customer case, the classic checks did not help. Nevertheless, they are important, because all of these prerequisites must be fulfilled before looking deeper into certificates, which was necessary in our case.

 

Communication ports between Enterprise Manager and VBR
 

Since both Veeam Backup & Replication v12 and v13 are currently still in use in the field, it is worth taking a closer look at the required ports.
 

You can find the official port documentation here:

(v13) https://helpcenter.veeam.com/docs/vbr/em/used_ports.html?ver=13 . (v12) https://helpcenter.veeam.com/archive/backup/120/em/used_ports.html

v12 (Windows Based):

v13 (Windows Based)

v13 (VSA)

How do I test the ports?
 

The easiest way is to use PowerShell with Test-NetConnection.
 

Important:
If possible, test with the same FQDN that is also configured for the Backup Server in Enterprise Manager. A successful IP test proves routing and firewall reachability, but not necessarily that name resolution, certificate validation, or FQDN-based communication are working correctly.

Which ports you need to test depends, of course, on your version.

If one of these tests fails, firewall rules should be checked — both on the Windows servers themselves and on external firewalls between EM and VBR.
 

Is the Catalog Service port really in use?
 

On the corresponding server, you can check which process is using a port.

For v12 and TCP 9393, for example:

If port 9393 is not in use, or if it is being used by another process, this explains why Enterprise Manager cannot correctly use the Catalog endpoint.
 

Are all required services running?
 

The Windows service Veeam Guest Catalog Service should be running.

If it is not running: start it.
If it is running: restart it in a controlled way and then check whether synchronization starts again.

Depending on the version and role, additional Veeam services may also be relevant, for example Enterprise Manager services or REST/API components.
 

Antivirus measures
 

If the ports are reachable and the Veeam services are running, the next thing to check is antivirus, EDR, or other security solutions.

Veeam provides its own KB article for this:
https://www.veeam.com/kb1999
I have also written an article about this myself: (Virus Scanning and Performance (English))

The important thing here is not only to check classic file system and path exclusions. Modern security products go much deeper.

Relevant items can include, for example:

• Antivirus file scanning
• Process blocking
• EDR policies
• Application Control
• SSL/TLS inspection
• Certificate replacement by security products
• Hardening policies
• Blocked access to Veeam directories or logs

It gets especially interesting when such policies were already distributed earlier but only become fully effective after a reboot.

 

Check the database engine
 

Another point that is often overlooked:
Enterprise Manager and all added Backup Servers must use the same database engine.

This means:

• EM on Microsoft SQL Server and VBR on Microsoft SQL Server: okay
• EM on PostgreSQL and VBR on PostgreSQL: okay
• EM on Microsoft SQL Server and VBR on PostgreSQL: not okay
• EM on PostgreSQL and VBR on Microsoft SQL Server: not okay
   

This is especially relevant after migrations, partial upgrades, or when “just quickly” migrating a single VBR from Microsoft SQL Server to PostgreSQL.

The official Veeam documentation states that the configuration databases of the Enterprise Manager server and the added Backup Servers must use the same database engine:

https://helpcenter.veeam.com/docs/vbr/em/setup_database.html?ver=13

Reboot and reconfigure – always a good idea, right?
 

If ports, services, security software, and the database engine all look clean, a controlled restart of Enterprise Manager and VBR can make sense.

If the problem still persists, you can check or reconfigure the connection in Enterprise Manager.

However, I would not immediately remove the Backup Server from Enterprise Manager at this point. It is better to start with the following:

1. Re-enter the stored credentials in Enterprise Manager.
2. Save the connection again.
3. Trigger the certificate/fingerprint update this way.
4. Then test synchronization again.

Only if this does not help should you consider removing and re-adding the Backup Server — coordinated with the customer and depending on the environment.

Documentation for adding and removing a Backup Server in Enterprise Manager:

Add Backup Server:
v12: https://helpcenter.veeam.com/archive/backup/120/em/adding_backup_server.html
v13: https://helpcenter.veeam.com/docs/vbr/em/adding_backup_server.html?ver=13

Remove Backup Server:
v12: https://helpcenter.veeam.com/archive/backup/120/em/removing_backup_server.html
v13: https://helpcenter.veeam.com/docs/vbr/em/removing_backup_server.html?ver=13
 

If none of this helps: check certificates
 

Now things get interesting.

The log message was clear:
 

The certificate chain was issued by an untrusted certification authority

The question is: was there a change here? Most of the time, people will say no — because how often do you change or delete certificates?

But what is often forgotten is that software, for example EDR, a Windows update, or Group Policy, can also interfere with this process. And Veeam requires external certificates for encrypted communication both on the Veeam Enterprise Manager server and on the Veeam Backup Server. And of course, a certificate itself can also expire.

In addition, there are, of course, your own internal CA certificates or the self-signed certificates from Veeam. Usually, these are valid for 10 years and should therefore not cause a problem, but checking them certainly does not hurt.

In Enterprise Manager, the certificate can be updated as follows:
v12: https://helpcenter.veeam.com/archive/backup/120/em/updating_security_certificate.html
v13: https://helpcenter.veeam.com/docs/vbr/em/updating_security_certificate.html?ver=13

Information about internal CA certificates or self-signed certificates:
v12: https://helpcenter.veeam.com/archive/backup/120/vsphere/backup_server_certificate.html
v13: https://helpcenter.veeam.com/docs/vbr/userguide/backup_server_certificate.html?ver=13

Also re-enter the credentials for the connection to VBR in Enterprise Manager.
Important: do not remove the Backup Server for this. Only re-enter the credentials. This updates the fingerprint of the EM certificate on the VBR side.
v12: https://helpcenter.veeam.com/docs/backup/em/editing_backup_server.html?ver=120
v13: https://helpcenter.veeam.com/docs/vbr/em/editing_backup_server.html?ver=13

After that, restart the Enterprise Manager REST API service.
After the restart, the service reads the current certificate again.
 

The actual root cause in our case
 

If all of this does not help, then it may be exactly what we had in our customer case.

Several root and intermediate certificates are actually required for Veeam Backup & Replication to work properly. Removing these certificates from the Backup Server or from an infrastructure component such as Enterprise Manager can limit the functionality of Veeam Backup & Replication or even cause failures.
 

Side note: Linux-based components of the backup infrastructure that were configured with the Veeam Software Appliance or the Veeam Infrastructure Appliance have all required certificates installed by default.
 

In most cases, the required certificates are already installed on Microsoft Windows systems. However, some installations do not contain the required certificate authorities as trusted certificates or have outdated certificates. This can happen especially on servers with heavily restricted security settings, on servers without internet access, or if the latest updates are not installed.
 

Okay, so which certificates are required?
 

Root certificates (In German Windows installations, these are located under: „Vertrauenswürdige Stammzertifizierungsstellen“)

• https://www.digicert.com/CACerts/DigiCertAssuredIDRootCA.crt (DigiCert Assured ID Root CA)
• https://www.digicert.com/CACerts/DigiCertHighAssuranceEVRootCA.crt (DigiCert High Assurance EV Root CA)
• https://support.globalsign.com/ca-certificates/globalsign-root-certificates (install R1 and R3 certificates)
  • R1 Zertifikat Direkt URL: https://secure.globalsign.net/cacert/Root-R1.crt
  • R3 Zertifikat Direkt URL: https://secure.globalsign.net/cacert/Root-R3.crt
• https://files.entrust.com/root-certificates/entrust_g2_ca.cer (Entrust Root Certification Authority — G2)
• https://files.entrust.com/root-certificates/CSBR1.cer (Entrust Code Signing Root Certification Authority — CSBR1)

Intermediate certificates (In German Windows installations, these are located under: „Zwischenzertifizierungsstellen“)

• Intermediate certificates:
• https://www.digicert.com/CACerts/DigiCertEVCodeSigningCA-SHA2.crt (DigiCert EV Code Signing CA — SHA2)
• https://files.entrust.com/subca-certificates/EVCS2-CSBR1-crosscert.cer  (Entrust Extended Validation Code Signing CA — EVCS2)

See the official Veeam documentation:
https://helpcenter.veeam.com/docs/vbr/userguide/trusted_root_certificates.html?zoom_highlight=r3+certificates&ver=13

Check whether all of these certificates are installed or not. Download any missing certificates and install them into the appropriate certificate store.

Attention: Not all certificates appear under the expected names. For example, the R1 and R3 certificates from GlobalSign are shown as “GlobalSign” and “GlobalSign Root CA”. You only see whether it is R1 or R3 when looking at the certificate properties.

 

To be on the safe side, the servers should be restarted after installing the certificates. If necessary, the VBR should also be added to EM again. After that, the problem should be resolved.