• EN
Question

Veeam B&R 12 "Veeam GCP Service" and "Veeam Kubernetes Service" don't start.

H
Userlevel 3
  • Han
  • Comes here often
  • 18 comments

Fresh default install of B&R 12 on Windows Server 2022.
"Veeam GCP Service" and "Veeam Kubernetes Service" refuse to start. Produce the following error in de System Log:

“The Veeam Kubernetes Service service terminated with the following error: 
An exception occurred in the service when handling the control request.”

24 comments

JMeixner
Userlevel 7
Badge +17

Sorry, but standard questions first:

  • Did you reboot?
  • What is in the system event logs?
  • Are there some logfiles you can post?
Jochen (Joe) Meixner | Veeam Vanguard 2024 | Veeam Legend 2021 - 2023 | Veeam Certified Architect (VMCA) 2022 | Twitter: @JoMeix | BlueSky: @jmeixner.bsky.social
Chris.Childerhose
Userlevel 7
Badge +20

Also when you installed v12 did you use Local System for the services or a specific service account?  This along with Joe's questions will help us.

Chris Childerhose - VMCA2022 | VMCE2023 | Veeam Vanguard 7* | Veeam Legend 3* | vExpert 5* | VCAP-DCV/VCP-DCV | Object First Ace | Cisco Champion | Twitter: @cchilderhose | Blog Site – https://just-virtualization.tech
Iams3le
Userlevel 7
Badge +9

Hello @Han, I would advise you to take a look at the Windows Event log as this can help you pin-point the root cause of this issue. See the “System Event Log” as mentioned above or the “Application Event Log” as well. Did you change your servername recently? 

Microsoft MVP | Veeam Legend 5* | VMware vExpert 4* | Cisco Champion 3*
Iams3le
Userlevel 7
Badge +9

Hi @Han, here is a similar error (same error message) I resolved. You may want to follow the troubleshooting steps to have yours resolved as well: https://techdirectarchive.com/2022/02/24/no-connection-can-be-made-because-the-active-machine-rejected-it-an-exception-occurred-in-the-service-when-handling-the-control-request-error-1064/

Microsoft MVP | Veeam Legend 5* | VMware vExpert 4* | Cisco Champion 3*
T
Userlevel 2
Badge +1

I think this is caused by a known issue that keeps some plugin services from starting when the locale does not use YYYY for date format (some locales use YY instead) due to a failure to parse the certificate date correctly.  An update is being created to address this but, in the interim, a workaround is to temporarily set the date format to use YYYY.

@Han am I correct that you are using such a locale?

H
Userlevel 3

Sorry, but standard questions first:

  • Did you reboot?
  • What is in the system event logs?
  • Are there some logfiles you can post?
  • Yes of course I the has been restarted.
  • See my original post.
  • Which log files would you like me to post?

 

H
Userlevel 3

Also when you installed v12 did you use Local System for the services or a specific service account?  This along with Joe's questions will help us.

All Veeam B&R services run under the Local System account.

H
Userlevel 3

Hello @Han, I would advise you to take a look at the Windows Event log as this can help you pin-point the root cause of this issue. See the “System Event Log” as mentioned above or the “Application Event Log” as well. Did you change your servername recently? 

System Log, please see my original post.

Nothing in the application Log.

Chris.Childerhose
Userlevel 7
Badge +20

Also when you installed v12 did you use Local System for the services or a specific service account?  This along with Joe's questions will help us.

All Veeam B&R services run under the Local System account.

Have you tried to run a repair of the installation?  If that does not work maybe a clean uninstall/reinstall - you can point to the same DB as that won’t be affected so all data remains.

Chris Childerhose - VMCA2022 | VMCE2023 | Veeam Vanguard 7* | Veeam Legend 3* | vExpert 5* | VCAP-DCV/VCP-DCV | Object First Ace | Cisco Champion | Twitter: @cchilderhose | Blog Site – https://just-virtualization.tech
H
Userlevel 3

Sorry, but standard questions first:

  • Did you reboot?
  • What is in the system event logs?
  • Are there some logfiles you can post?

Ok, found some interesting log entries for the GCP plugin, one of the services that don’t start.

2023-03-08 21:00:48.4643 Error  <___1>:    Veeam.GCP.PlatformService.CertificateManager.WindowsCertificateStorage
2023-03-08 21:00:48.4643 Error  <___1>:    Certificate was not found. Thumbprint: '7E96792192CCE1649D6D6EB141DF1A9BEE9592B7'
2023-03-08 21:00:48.7350 Info   <___1>:    Veeam.GCP.PlatformService.CertificateManager.WindowsCertificateStorage
2023-03-08 21:00:48.7350 Info   <___1>:    Saved the certificate '7B6053BD0A88BB39AAC5BE9D20E84380FE23A12E' to the store.
2023-03-08 21:00:48.7379 Error  <___1>:    Veeam.GCP.PlatformService.CertificateManager.WindowsCertificateStorage
2023-03-08 21:00:48.7379 Error  <___1>:    Certificate was not found. Thumbprint: '7B6053BD0A88BB39AAC5BE9D20E84380FE23A12E'
2023-03-08 21:00:48.7542 Error  <___1>:    Veeam.GCP.PlatformService.WebService.Program+<Main>d__0
2023-03-08 21:00:48.7542 Error  <___1>:    Error during start service
2023-03-08 21:00:48.7542 Error  <___1>:    Veeam.GCP.PlatformService.CertificateManager.CertificateNotFoundException: Cannot find backup appliance certificate

 

Also I have found a lot of GCP and Kasten plugin self-signed certificates in the local machine store.

H
Userlevel 3

Also when you installed v12 did you use Local System for the services or a specific service account?  This along with Joe's questions will help us.

All Veeam B&R services run under the Local System account.

Have you tried to run a repair of the installation?  If that does not work maybe a clean uninstall/reinstall - you can point to the same DB as that won’t be affected so all data remains.

This is a clean next next finish install on a fresh VM.

Chris.Childerhose
Userlevel 7
Badge +20

Sorry, but standard questions first:

  • Did you reboot?
  • What is in the system event logs?
  • Are there some logfiles you can post?

Ok, found some interesting log entries for the GCP plugin, one of the services that don’t start.

2023-03-08 21:00:48.4643 Error  <___1>:    Veeam.GCP.PlatformService.CertificateManager.WindowsCertificateStorage
2023-03-08 21:00:48.4643 Error  <___1>:    Certificate was not found. Thumbprint: '7E96792192CCE1649D6D6EB141DF1A9BEE9592B7'
2023-03-08 21:00:48.7350 Info   <___1>:    Veeam.GCP.PlatformService.CertificateManager.WindowsCertificateStorage
2023-03-08 21:00:48.7350 Info   <___1>:    Saved the certificate '7B6053BD0A88BB39AAC5BE9D20E84380FE23A12E' to the store.
2023-03-08 21:00:48.7379 Error  <___1>:    Veeam.GCP.PlatformService.CertificateManager.WindowsCertificateStorage
2023-03-08 21:00:48.7379 Error  <___1>:    Certificate was not found. Thumbprint: '7B6053BD0A88BB39AAC5BE9D20E84380FE23A12E'
2023-03-08 21:00:48.7542 Error  <___1>:    Veeam.GCP.PlatformService.WebService.Program+<Main>d__0
2023-03-08 21:00:48.7542 Error  <___1>:    Error during start service
2023-03-08 21:00:48.7542 Error  <___1>:    Veeam.GCP.PlatformService.CertificateManager.CertificateNotFoundException: Cannot find backup appliance certificate

 

Also I have found a lot of GCP and Kasten plugin self-signed certificates in the local machine store.

So based on this line here -

2023-03-08 21:00:48.7379 Error  <___1>:    Certificate was not found. Thumbprint: '7B6053BD0A88BB39AAC5BE9D20E84380FE23A12E'

It is looking for the SSL with this thumbprint on the server to start the service by the sounds of it even though it saved it to the store from what it says.  You might need to regenerate the self-signed certs again and try to see what happens.

Chris Childerhose - VMCA2022 | VMCE2023 | Veeam Vanguard 7* | Veeam Legend 3* | vExpert 5* | VCAP-DCV/VCP-DCV | Object First Ace | Cisco Champion | Twitter: @cchilderhose | Blog Site – https://just-virtualization.tech
H
Userlevel 3

Sorry, but standard questions first:

  • Did you reboot?
  • What is in the system event logs?
  • Are there some logfiles you can post?

Ok, found some interesting log entries for the GCP plugin, one of the services that don’t start.

2023-03-08 21:00:48.4643 Error  <___1>:    Veeam.GCP.PlatformService.CertificateManager.WindowsCertificateStorage
2023-03-08 21:00:48.4643 Error  <___1>:    Certificate was not found. Thumbprint: '7E96792192CCE1649D6D6EB141DF1A9BEE9592B7'
2023-03-08 21:00:48.7350 Info   <___1>:    Veeam.GCP.PlatformService.CertificateManager.WindowsCertificateStorage
2023-03-08 21:00:48.7350 Info   <___1>:    Saved the certificate '7B6053BD0A88BB39AAC5BE9D20E84380FE23A12E' to the store.
2023-03-08 21:00:48.7379 Error  <___1>:    Veeam.GCP.PlatformService.CertificateManager.WindowsCertificateStorage
2023-03-08 21:00:48.7379 Error  <___1>:    Certificate was not found. Thumbprint: '7B6053BD0A88BB39AAC5BE9D20E84380FE23A12E'
2023-03-08 21:00:48.7542 Error  <___1>:    Veeam.GCP.PlatformService.WebService.Program+<Main>d__0
2023-03-08 21:00:48.7542 Error  <___1>:    Error during start service
2023-03-08 21:00:48.7542 Error  <___1>:    Veeam.GCP.PlatformService.CertificateManager.CertificateNotFoundException: Cannot find backup appliance certificate

 

Also I have found a lot of GCP and Kasten plugin self-signed certificates in the local machine store.

Every time I try to start the GCP or Kubernetes service a new certificate is generated.

What the heck is going on?

H
Userlevel 3

Sorry, but standard questions first:

  • Did you reboot?
  • What is in the system event logs?
  • Are there some logfiles you can post?

Ok, found some interesting log entries for the GCP plugin, one of the services that don’t start.

2023-03-08 21:00:48.4643 Error  <___1>:    Veeam.GCP.PlatformService.CertificateManager.WindowsCertificateStorage
2023-03-08 21:00:48.4643 Error  <___1>:    Certificate was not found. Thumbprint: '7E96792192CCE1649D6D6EB141DF1A9BEE9592B7'
2023-03-08 21:00:48.7350 Info   <___1>:    Veeam.GCP.PlatformService.CertificateManager.WindowsCertificateStorage
2023-03-08 21:00:48.7350 Info   <___1>:    Saved the certificate '7B6053BD0A88BB39AAC5BE9D20E84380FE23A12E' to the store.
2023-03-08 21:00:48.7379 Error  <___1>:    Veeam.GCP.PlatformService.CertificateManager.WindowsCertificateStorage
2023-03-08 21:00:48.7379 Error  <___1>:    Certificate was not found. Thumbprint: '7B6053BD0A88BB39AAC5BE9D20E84380FE23A12E'
2023-03-08 21:00:48.7542 Error  <___1>:    Veeam.GCP.PlatformService.WebService.Program+<Main>d__0
2023-03-08 21:00:48.7542 Error  <___1>:    Error during start service
2023-03-08 21:00:48.7542 Error  <___1>:    Veeam.GCP.PlatformService.CertificateManager.CertificateNotFoundException: Cannot find backup appliance certificate

 

Also I have found a lot of GCP and Kasten plugin self-signed certificates in the local machine store.

So based on this line here -

2023-03-08 21:00:48.7379 Error  <___1>:    Certificate was not found. Thumbprint: '7B6053BD0A88BB39AAC5BE9D20E84380FE23A12E'

It is looking for the SSL with this thumbprint on the server to start the service by the sounds of it even though it saved it to the store from what it says.  You might need to regenerate the self-signed certs again and try to see what happens.

Every time I try to start the GCP or Kubernetes service a new certificate is being generated.

M
Userlevel 1

Same issue here.

I’ve just ran the update from V11 to V12 and reaching the last update steps, I got the message : 

“Failed to start service ‘VeeamKastenSvc'. “

I retried once, but got the same error. 
I canceled, then the upgrade finished.

I rebooted the server to get the same error log :

“The Veeam Kubernetes Service service terminated with the following error: 
An exception occurred in the service when handling the control request.”

& 

“The Veeam GCP Service service failed to start due to the following error: 
The service did not respond to the start or control request in a timely fashion.”

The Veeam Console opens and nothing seems to fail so far, but that is kinda weird.

I setup the installation with a service account, but these services run as Local System.

No other services fails.

Does anyone else had this issue or found something ?

M
Userlevel 1

Same errors here :

For Kasten : 

“[09.03.2023 12:45:48] <32b6> Info     VeeamAuth executable path: 'C:\Program Files\Veeam\Plugins\Kasten K10\Service\VeeamAuth.exe'.
[09.03.2023 12:45:48] <32b6> Info     [AsyncTaskService] Watch thread started
[09.03.2023 12:45:48] <32b6> Info     Configuring WEB services.
[09.03.2023 12:45:48] <32b6> Info     Creating automapper namespaces: 
[09.03.2023 12:45:48] <32b6> Info     0. Veeam.Backup.PlatformServices.Kasten.v1_0_rev0.Models
[09.03.2023 12:45:48] <32b6> Info     1. Veeam.Backup.PlatformServices.Kasten.v_latest.Models
[09.03.2023 12:45:49] <32b6> Info     Web services are configured.
[09.03.2023 12:45:49] <32b6> Info     Starting application as windows service.
[09.03.2023 12:45:49] <7cbc> Info     Starting service.
[09.03.2023 12:45:49] <7cbc> Info     Loading certificate thumbprint
[09.03.2023 12:45:49] <7cbc> Info     Certificate thumbprint loaded
[09.03.2023 12:45:49] <7cbc> Info     Searching certificate by thumbprint in certificate store My-LocalMachine
[09.03.2023 12:45:49] <7cbc> Error    Certificate was not found. Thumbprint: '2942DD4C38BAE9A263776D6A12640D5A58350458' (Veeam.Backup.PlatformServices.Kasten.CertificateNotFoundException)
[09.03.2023 12:45:49] <7cbc> Error       at Veeam.Backup.PlatformServices.Kasten.WindowsCertificateStorage.GetCertificateByThumbprint(String rawThumbprint)
[09.03.2023 12:45:49] <7cbc> Error       at Veeam.Backup.PlatformServices.Kasten.CCertificateProvider.LoadCertificate()
[09.03.2023 12:45:49] <7cbc> Info     Saved the certificate '84ECDC34751CFCA8192572D19E559C2CAE2232E7' to the store.
[09.03.2023 12:45:49] <7cbc> Info     Saving certificate thumbprint
[09.03.2023 12:45:49] <7cbc> Info     Certificate thumbprint saved
[09.03.2023 12:45:49] <7cbc> Info     Searching certificate by thumbprint in certificate store My-LocalMachine
[09.03.2023 12:45:49] <32b6> Error    Failed to run Kasten Platform Service.
[09.03.2023 12:45:49] <32b6> Error    Certificate was not found. Thumbprint: '84ECDC34751CFCA8192572D19E559C2CAE2232E7' (Veeam.Backup.PlatformServices.Kasten.CertificateNotFoundException)”

For GCP : 

“2023-03-09 12:32:38.3377 Error  <___1>:    Veeam.GCP.PlatformService.CertificateManager.WindowsCertificateStorage
2023-03-09 12:32:38.3377 Error  <___1>:    Certificate was not found. Thumbprint: 'D99B72D66DDAEA340D98670CF2920CBA4EA31C62'
2023-03-09 12:32:38.5706 Info   <___1>:    Veeam.GCP.PlatformService.CertificateManager.WindowsCertificateStorage
2023-03-09 12:32:38.5706 Info   <___1>:    Saved the certificate '8397FD3449CE35C37D54E8EBA68FFB737844FD96' to the store.
2023-03-09 12:32:38.5732 Error  <___1>:    Veeam.GCP.PlatformService.CertificateManager.WindowsCertificateStorage
2023-03-09 12:32:38.5732 Error  <___1>:    Certificate was not found. Thumbprint: '8397FD3449CE35C37D54E8EBA68FFB737844FD96'
2023-03-09 12:32:38.6661 Error  <___1>:    Veeam.GCP.PlatformService.WebService.Program+<Main>d__0
2023-03-09 12:32:38.6661 Error  <___1>:    Error during start service
2023-03-09 12:32:38.6661 Error  <___1>:    Veeam.GCP.PlatformService.CertificateManager.CertificateNotFoundException: Cannot find backup appliance certificate”

 

I might just create a support case.

We’ll see what they say.

H

Error message: “failed to start service veeamkastensvc” while upgrading to version 12.

Had the same problem (for the “Veeam Kubernet Service” and “Veeam GCP Service”).

I changed the “logon as” in services.msc for both services to a local account/service account, rather than the local system account. This solved it for me. 

M
Userlevel 1

Error message: “failed to start service veeamkastensvc” while upgrading to version 12.

Had the same problem (for the “Veeam Kubernet Service” and “Veeam GCP Service”).

I changed the “logon as” in services.msc for both services to a local account/service account, rather than the local system account. This solved it for me. 

I just tried to run these services as the same service user that runs the other ones. 

Unfortunately, still no luck :(, same error : 

2023-03-09 14:11:04.9978 Error  <___1>:    Veeam.GCP.PlatformService.CertificateManager.WindowsCertificateStorage
2023-03-09 14:11:04.9978 Error  <___1>:    Certificate was not found. Thumbprint: 'D99B72D66DDAEA340D98670CF2920CBA4EA31C62'
2023-03-09 14:11:05.2934 Info   <___1>:    Veeam.GCP.PlatformService.CertificateManager.WindowsCertificateStorage
2023-03-09 14:11:05.2934 Info   <___1>:    Saved the certificate '211AC0F0C0346E4964A9F0EFFC995C228B8962E9' to the store.
2023-03-09 14:11:05.2967 Error  <___1>:    Veeam.GCP.PlatformService.CertificateManager.WindowsCertificateStorage
2023-03-09 14:11:05.2967 Error  <___1>:    Certificate was not found. Thumbprint: '211AC0F0C0346E4964A9F0EFFC995C228B8962E9'
2023-03-09 14:11:05.3937 Error  <___1>:    Veeam.GCP.PlatformService.WebService.Program+<Main>d__0
2023-03-09 14:11:05.3937 Error  <___1>:    Error during start service
2023-03-09 14:11:05.3937 Error  <___1>:    Veeam.GCP.PlatformService.CertificateManager.CertificateNotFoundException: Cannot find backup appliance certificate

 

Chris.Childerhose
Userlevel 7
Badge +20

Would suggest support case at this point.

Chris Childerhose - VMCA2022 | VMCE2023 | Veeam Vanguard 7* | Veeam Legend 3* | vExpert 5* | VCAP-DCV/VCP-DCV | Object First Ace | Cisco Champion | Twitter: @cchilderhose | Blog Site – https://just-virtualization.tech
M
Userlevel 1

Would suggest support case at this point.

Thanks, I’ve created a support case right after posting here.
I’ll keep you updated on the case :)

L

I think this is caused by a known issue that keeps some plugin services from starting when the locale does not use YYYY for date format (some locales use YY instead) due to a failure to parse the certificate date correctly.  An update is being created to address this but, in the interim, a workaround is to temporarily set the date format to use YYYY.

@Han am I correct that you are using such a locale?

Hello everyone,

I just experienced exact same error in my environment.

This was the solution for me.

Windows Settings → Time & Language → Region → Regional format data

Short date had YY format → updating this to YYYY format resolved the issue.

Thank you @tsightler for the solution!

M
Userlevel 1

Tried that as well, no luck :(
Support suggest that it is because these services are not configured.

It’s true that I do not use Kubernetes or Google Cloud Platform.

The annoying thing is that now, I get warning on my Configuration Backup that these services cannot be accessed.

I’ll keep posting.

Chris.Childerhose
Userlevel 7
Badge +20

Tried that as well, no luck :(
Support suggest that it is because these services are not configured.

It’s true that I do not use Kubernetes or Google Cloud Platform.

The annoying thing is that now, I get warning on my Configuration Backup that these services cannot be accessed.

I’ll keep posting.

I don't have the services configured and they still work. So not sure that is an answer to the issue.  What if you don't use those services?  🤔

Chris Childerhose - VMCA2022 | VMCE2023 | Veeam Vanguard 7* | Veeam Legend 3* | vExpert 5* | VCAP-DCV/VCP-DCV | Object First Ace | Cisco Champion | Twitter: @cchilderhose | Blog Site – https://just-virtualization.tech
I

Hi
Solution still not found? What did tech support say?

Comment


Terms & Conditions