• EN
Solved

Kasten Stateful backup failing on Azure AKS cluster

A

Hello noticing failing backup on our AKS cluster: 

```

cause:
  cause:
    cause:
      cause:
        message: 'Failed to get VolumeSnapshotClass (true) specified in Storageclass
          (default) annotations: volumesnapshotclasses.snapshot.storage.k8s.io
          "true" not found'
      file: kasten.io/k10/kio/exec/phases/backup/snapshot_precheck.go:105
      function: kasten.io/k10/kio/exec/phases/backup.runCSIPrechecks
      linenumber: 105
      message: Failed to get K10 VolumeSnapshotClass
    fields:
      - name: PVC
        value: nginx-pvc
    file: kasten.io/k10/kio/exec/phases/backup/snapshot_precheck.go:87
    function: kasten.io/k10/kio/exec/phases/backup.isCSIPVC
    linenumber: 87
    message: Failed to run CSI prechecks for PVC
  file: kasten.io/k10/kio/exec/phases/backup/snapshot_data_phase.go:268
  function: kasten.io/k10/kio/exec/phases/backup.processArtifacts.func1
  linenumber: 268
  message: Failed snapshot precheck for namespace
message: Job failed to be executed
fields: []

```

 

Kasten version 5.0.1

AKS version 1.23.5

 

We have another cluster (which also has same version 1.23.5) and same K10 version and where everything is running without any issues. 

 

 

 

Any advices?

 

Thank in advance. 

icon

Best answer by Iurii 22 July 2022, 14:49

View original

6 comments

jaiganeshjk
Userlevel 6
Badge +2

@andriktr Thanks for the question.

It seems that you are annotating the storageclass with the annotation 

kubectl annotate storageclass ${SC_NAME} \    k10.kasten.io/volume-snapshot-class=”true”

 

This annotation is for specifying the volumesnapshotclass name to use for a specific storageClass.

This means that K10 looks for the volumesnapshotclass named “true” and it fails.

 

You will have to either remove it and annotate the volumesnapshotclass with k10.kasten.io/is-snapshot-class=true or change the annotation to point to the the correct volumesnapshotclass name instead of true

kubectl annotate storageclass ${SC_NAME} \ k10.kasten.io/volume-snapshot-class=<volumesnapshotclassName>

Jaiganesh
A

Hi @jaiganeshjk 

Thanks for response.

I’m not adding any annotations myself to any storage class by default (But I have tried to do this during troubleshooting). In my example a default storage class was used for PV which looks like this: 

```

allowVolumeExpansion: true
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  annotations:
    storageclass.kubernetes.io/is-default-class: "true"
  creationTimestamp: "2022-07-03T00:12:27Z"
  labels:
    addonmanager.kubernetes.io/mode: EnsureExists
    kubernetes.io/cluster-service: "true"
  name: default
  resourceVersion: "1632529"
  uid: 7d2e2478-19b4-44e8-bcf0-11fefcde780e
parameters:
  skuname: StandardSSD_LRS
provisioner: disk.csi.azure.com
reclaimPolicy: Delete
volumeBindingMode: WaitForFirstConsumer

```

As you can see mention annotation is absent. 

And here is the error from the last backup job

```

cause:
  cause:
    cause:
      cause:
        message: "Failed to find any VolumeSnapshotClass in the cluster: <nil>"
      file: kasten.io/k10/kio/exec/phases/backup/snapshot_precheck.go:105
      function: kasten.io/k10/kio/exec/phases/backup.runCSIPrechecks
      linenumber: 105
      message: Failed to get K10 VolumeSnapshotClass
    fields:
      - name: PVC
        value: nginx-pvc
    file: kasten.io/k10/kio/exec/phases/backup/snapshot_precheck.go:87
    function: kasten.io/k10/kio/exec/phases/backup.isCSIPVC
    linenumber: 87
    message: Failed to run CSI prechecks for PVC
  file: kasten.io/k10/kio/exec/phases/backup/snapshot_data_phase.go:268
  function: kasten.io/k10/kio/exec/phases/backup.processArtifacts.func1
  linenumber: 268
  message: Failed snapshot precheck for namespace
message: Job failed to be executed
fields: []
 

```

 

Thanks

Hagag
Userlevel 5
Badge +2

@andriktr Please share the output of the below script 

curl https://docs.kasten.io/tools/k10_primer.sh | bash

A

Hi @Hagag, 

Here is the output: 

Kubernetes Version Check:
  Valid kubernetes version (v1.23.5)  -  OK

RBAC Check:
  Kubernetes RBAC is enabled  -  OK

Aggregated Layer Check:
  The Kubernetes Aggregated Layer is enabled  -  OK

         Found multiple snapshot API group versions, using preferred.
CSI Capabilities Check:
  Using CSI GroupVersion snapshot.storage.k8s.io/v1  -  OK

         Found multiple snapshot API group versions, using preferred.
I0711 12:11:52.533396       9 request.go:601] Waited for 1.047216909s due to client-side throttling, not priority and fairness, request: GET:https://192.168.0.1:443/apis/networking.k8s.io/v1
         Found multiple snapshot API group versions, using preferred.
Validating Provisioners: 
disk.csi.azure.com:
  Is a CSI Provisioner  -  OK
  Missing/Failed to Fetch CSIDriver Object
  CSI Provisioner doesn't have VolumeSnapshotClass  -  Error
  Storage Classes:
    default
      Valid Storage Class  -  OK
    managed
      Valid Storage Class  -  OK
    managed-csi
      Valid Storage Class  -  OK
    managed-csi-premium
      Valid Storage Class  -  OK
    managed-premium
      Valid Storage Class  -  OK

file.csi.azure.com:
  Is a CSI Provisioner  -  OK
  Missing/Failed to Fetch CSIDriver Object
  CSI Provisioner doesn't have VolumeSnapshotClass  -  Error
  Storage Classes:
    azurefile
      Valid Storage Class  -  OK
    azurefile-csi
      Valid Storage Class  -  OK
    azurefile-csi-premium
      Valid Storage Class  -  OK
    azurefile-premium
      Valid Storage Class  -  OK

Validate Generic Volume Snapshot:
  Pod created successfully  -  OK
  GVS Backup command executed successfully  -  OK
  Pod deleted successfully  -  OK

 

I
Badge

Hi @andriktr, 

 

Based on the output of the preflight checks it looks like you don’t have a valid VolumeSnapshotClass instantiated for your Azure Disk CSI. That VSC needs to be added to your cluster. You can find the associated info for Azure disk CSI’s VSC: https://github.com/kubernetes-sigs/azuredisk-csi-driver/blob/master/deploy/example/snapshot/storageclass-azuredisk-snapshot.yaml/

 

Once you’ve applied the above you will then need to follow the steps to configure Kasten to use this VSC they can be found here: https://docs.kasten.io/latest/install/storage.html#csi-snapshot-configuration

 

Specifically, you’ll need to run the following command to annotate your storage class with the Kasten VSC annotation: 

kubectl annotate storageclass ${SC_NAME} \ k10.kasten.io/volume-snapshot-class=<volumesnapshotclassName>`

Additionally, if you need to use the Azure files CSI for any of your workloads, you would need to enable the generic backup feature for these as currently, VolumeSnapshotClass is unsupported. See more on enabling Generic backup in our docs here: https://docs.kasten.io/latest/install/generic.html#generic-storage-backup-and-restore

 

Hope this helps! 

 

A

Thank you @Iurii your provided solution fixed the issue. 

Comment


Terms & Conditions