Skip to main content
  • EN

Currently performing POC for a client with 2 x K10 instance setup, deployed on AKS, each running on different region. Backup was successful. However, when comes to recovery via import job, PV and pods seems to ok. A couple of pods are stuck at Status: ContainerCreating 0/1. Check the pods, and discover some issues here related to secretproviderclass which is not restoring.  Any idea what is going on, and how is the next step here.

Events: Type Reason Age From Message ---- ------ ---- ---- ------- Normal Scheduled 47m default-scheduler Successfully assigned sanofi-net/peer0-0 to aks-agentpool-11584630-vmss00000e Warning FailedMount 46m (x3 over 47m) kubelet MountVolume.SetUp failed for volume "tlscacerts" : rpc error: code = Unknown desc = failed to get secretproviderclass sanofi-net/peer0-tlscacerts, error: SecretProviderClass.secrets-store.csi.x-k8s.io "peer0-tlscacerts" not found Warning FailedMount 46m (x3 over 47m) kubelet MountVolume.SetUp failed for volume "cacerts" : rpc error: code = Unknown desc = failed to get secretproviderclass sanofi-net/peer0-cacerts, error: SecretProviderClass.secrets-store.csi.x-k8s.io "peer0-cacerts" not found Warning FailedMount 46m (x3 over 47m) kubelet MountVolume.SetUp failed for volume "user-cred" : rpc error: code = Unknown desc = failed to get secretproviderclass sanofi-net/peer0-user-cred, error: SecretProviderClass.secrets-store.csi.x-k8s.io "peer0-user-cred" not found Warning FailedMount 46m (x4 over 47m) kubelet MountVolume.SetUp failed for volume "tls" : rpc error: code = Unknown desc = failed to get secretproviderclass sanofi-net/peer0-tls, error: SecretProviderClass.secrets-store.csi.x-k8s.io "peer0-tls" not found Warning FailedMount 46m (x4 over 47m) kubelet MountVolume.SetUp failed for volume "signcerts" : rpc error: code = Unknown desc = failed to get secretproviderclass sanofi-net/peer0-signcerts, error: SecretProviderClass.secrets-store.csi.x-k8s.io "peer0-signcerts" not found Warning FailedMount 46m (x4 over 47m) kubelet MountVolume.SetUp failed for volume "keystore" : rpc error: code = Unknown desc = failed to get secretproviderclass sanofi-net/peer0-keystore, error: SecretProviderClass.secrets-store.csi.x-k8s.io "peer0-keystore" not found Warning FailedMount 20m (x20 over 47m) kubelet MountVolume.SetUp failed for volume "admincerts" : rpc error: code = Unknown desc = failed to get secretproviderclass sanofi-net/peer0-admincerts, error: SecretProviderClass.secrets-store.csi.x-k8s.io "peer0-admincerts" not found Warning FailedMount 2m2s (x59 over 40m) kubelet (combined from similar events): Unable to attach or mount volumes: unmounted volumes=mkeystore user-cred cacerts signcerts admincerts tlscacerts tls], unattached volumes=mdatadir peer0-msp-config-volume external-builder-release builders-config keystore external-builder-detect external-builder-build user-cred kube-api-access-nmn8g cacerts certificates datadir-couchdb dockersocket signcerts admincerts tlscacerts tls]: timed out waiting for the condition


 

Events:
  Type     Reason       Age                  From               Message
  ----     ------       ----                 ----               -------
  Normal   Scheduled    47m                  default-scheduler  Successfully assigned sanofi-net/peer0-0 to aks-agentpool-11584630-vmss00000e
  Warning  FailedMount  46m (x3 over 47m)    kubelet            MountVolume.SetUp failed for volume "tlscacerts" : rpc error: code = Unknown desc = failed to get secretproviderclass sanofi-net/peer0-tlscacerts, error: SecretProviderClass.secrets-store.csi.x-k8s.io "peer0-tlscacerts" not found
  Warning  FailedMount  46m (x3 over 47m)    kubelet            MountVolume.SetUp failed for volume "cacerts" : rpc error: code = Unknown desc = failed to get secretproviderclass sanofi-net/peer0-cacerts, error: SecretProviderClass.secrets-store.csi.x-k8s.io "peer0-cacerts" not found
  Warning  FailedMount  46m (x3 over 47m)    kubelet            MountVolume.SetUp failed for volume "user-cred" : rpc error: code = Unknown desc = failed to get secretproviderclass sanofi-net/peer0-user-cred, error: SecretProviderClass.secrets-store.csi.x-k8s.io "peer0-user-cred" not found
  Warning  FailedMount  46m (x4 over 47m)    kubelet            MountVolume.SetUp failed for volume "tls" : rpc error: code = Unknown desc = failed to get secretproviderclass sanofi-net/peer0-tls, error: SecretProviderClass.secrets-store.csi.x-k8s.io "peer0-tls" not found
  Warning  FailedMount  46m (x4 over 47m)    kubelet            MountVolume.SetUp failed for volume "signcerts" : rpc error: code = Unknown desc = failed to get secretproviderclass sanofi-net/peer0-signcerts, error: SecretProviderClass.secrets-store.csi.x-k8s.io "peer0-signcerts" not found
  Warning  FailedMount  46m (x4 over 47m)    kubelet            MountVolume.SetUp failed for volume "keystore" : rpc error: code = Unknown desc = failed to get secretproviderclass sanofi-net/peer0-keystore, error: SecretProviderClass.secrets-store.csi.x-k8s.io "peer0-keystore" not found
  Warning  FailedMount  20m (x20 over 47m)   kubelet            MountVolume.SetUp failed for volume "admincerts" : rpc error: code = Unknown desc = failed to get secretproviderclass sanofi-net/peer0-admincerts, error: SecretProviderClass.secrets-store.csi.x-k8s.io "peer0-admincerts" not found
  Warning  FailedMount  2m2s (x59 over 40m)  kubelet            (combined from similar events): Unable to attach or mount volumes: unmounted volumes=[keystore user-cred cacerts signcerts admincerts tlscacerts tls], unattached volumes=[datadir peer0-msp-config-volume external-builder-release builders-config keystore external-builder-detect external-builder-build user-cred kube-api-access-nmn8g cacerts certificates datadir-couchdb dockersocket signcerts admincerts tlscacerts tls]: timed out waiting for the condition

 

@nantheless Thanks for posting your question.

It seems that the secretproviderclass is missing from the cluster.

I am not very familiar with the secret store CSI driver. However, do you already have the CRD for secretProviderClass in the destination cluster ?
 

It seems that you need to export the cluster-scoped-resources and use that to import and restore cluster-scoped resources and then attempt application restore.

@nantheless Thanks for posting your question.

It seems that the secretproviderclass is missing from the cluster.

I am not very familiar with the secret store CSI driver. However, do you already have the CRD for secretProviderClass in the destination cluster ?
 

It seems that you need to export the cluster-scoped-resources and use that to import and restore cluster-scoped resources and then attempt application restore.

You are correct. However, Microsoft cloud team had also advised customer to enable Azure Key Vault Provider which they have duly enabled. But, nothing works. Microsoft had advise if there is any issue with the recovery process. 

https://docs.microsoft.com/en-us/azure/aks/csi-secrets-store-driver#create-an-aks-cluster-with-azure-key-vault-provider-for-secrets-store-csi-driver-support

 

Comment


Terms & Conditions