Skip to main content
Solved

Kasten Stateful backup failing on Azure AKS cluster

A

Hello noticing failing backup on our AKS cluster: 

```

cause:
  cause:
    cause:
      cause:
        message: 'Failed to get VolumeSnapshotClass (true) specified in Storageclass
          (default) annotations: volumesnapshotclasses.snapshot.storage.k8s.io
          "true" not found'
      file: kasten.io/k10/kio/exec/phases/backup/snapshot_precheck.go:105
      function: kasten.io/k10/kio/exec/phases/backup.runCSIPrechecks
      linenumber: 105
      message: Failed to get K10 VolumeSnapshotClass
    fields:
      - name: PVC
        value: nginx-pvc
    file: kasten.io/k10/kio/exec/phases/backup/snapshot_precheck.go:87
    function: kasten.io/k10/kio/exec/phases/backup.isCSIPVC
    linenumber: 87
    message: Failed to run CSI prechecks for PVC
  file: kasten.io/k10/kio/exec/phases/backup/snapshot_data_phase.go:268
  function: kasten.io/k10/kio/exec/phases/backup.processArtifacts.func1
  linenumber: 268
  message: Failed snapshot precheck for namespace
message: Job failed to be executed
fields: []

```

 

Kasten version 5.0.1

AKS version 1.23.5

 

We have another cluster (which also has same version 1.23.5) and same K10 version and where everything is running without any issues. 

 

 

 

Any advices?

 

Thank in advance. 

Best answer by Iurii

Hi @andriktr, 

 

Based on the output of the preflight checks it looks like you don’t have a valid VolumeSnapshotClass instantiated for your Azure Disk CSI. That VSC needs to be added to your cluster. You can find the associated info for Azure disk CSI’s VSC: https://github.com/kubernetes-sigs/azuredisk-csi-driver/blob/master/deploy/example/snapshot/storageclass-azuredisk-snapshot.yaml/

 

Once you’ve applied the above you will then need to follow the steps to configure Kasten to use this VSC they can be found here: https://docs.kasten.io/latest/install/storage.html#csi-snapshot-configuration

 

Specifically, you’ll need to run the following command to annotate your storage class with the Kasten VSC annotation: 

kubectl annotate storageclass ${SC_NAME} \ k10.kasten.io/volume-snapshot-class=<volumesnapshotclassName>`

Additionally, if you need to use the Azure files CSI for any of your workloads, you would need to enable the generic backup feature for these as currently, VolumeSnapshotClass is unsupported. See more on enabling Generic backup in our docs here: https://docs.kasten.io/latest/install/generic.html#generic-storage-backup-and-restore

 

Hope this helps! 

 

View original
    Chris.Childerhose
    April M
    2 people like this
    • Chris.Childerhose
      Chris.Childerhose
    • April M
      April M
    Did this topic help you find an answer to your question?

    6 comments

    jaiganeshjk
    Forum|alt.badge.img+2
    • jaiganeshjk
    • Experienced User
    • 274 comments
    • July 8, 2022

    @andriktr Thanks for the question.

    It seems that you are annotating the storageclass with the annotation 

    kubectl annotate storageclass ${SC_NAME} \    k10.kasten.io/volume-snapshot-class=”true”

     

    This annotation is for specifying the volumesnapshotclass name to use for a specific storageClass.

    This means that K10 looks for the volumesnapshotclass named “true” and it fails.

     

    You will have to either remove it and annotate the volumesnapshotclass with k10.kasten.io/is-snapshot-class=true or change the annotation to point to the the correct volumesnapshotclass name instead of true

    kubectl annotate storageclass ${SC_NAME} \ k10.kasten.io/volume-snapshot-class=<volumesnapshotclassName>

    Jaiganesh
    A
    • andriktr
    • Author
    • Not a newbie anymore
    • 3 comments
    • July 8, 2022

    Hi @jaiganeshjk 

    Thanks for response.

    I’m not adding any annotations myself to any storage class by default (But I have tried to do this during troubleshooting). In my example a default storage class was used for PV which looks like this: 

    ```

    allowVolumeExpansion: true
    apiVersion: storage.k8s.io/v1
    kind: StorageClass
    metadata:
      annotations:
        storageclass.kubernetes.io/is-default-class: "true"
      creationTimestamp: "2022-07-03T00:12:27Z"
      labels:
        addonmanager.kubernetes.io/mode: EnsureExists
        kubernetes.io/cluster-service: "true"
      name: default
      resourceVersion: "1632529"
      uid: 7d2e2478-19b4-44e8-bcf0-11fefcde780e
    parameters:
      skuname: StandardSSD_LRS
    provisioner: disk.csi.azure.com
    reclaimPolicy: Delete
    volumeBindingMode: WaitForFirstConsumer

    ```

    As you can see mention annotation is absent. 

    And here is the error from the last backup job

    ```

    cause:
      cause:
        cause:
          cause:
            message: "Failed to find any VolumeSnapshotClass in the cluster: <nil>"
          file: kasten.io/k10/kio/exec/phases/backup/snapshot_precheck.go:105
          function: kasten.io/k10/kio/exec/phases/backup.runCSIPrechecks
          linenumber: 105
          message: Failed to get K10 VolumeSnapshotClass
        fields:
          - name: PVC
            value: nginx-pvc
        file: kasten.io/k10/kio/exec/phases/backup/snapshot_precheck.go:87
        function: kasten.io/k10/kio/exec/phases/backup.isCSIPVC
        linenumber: 87
        message: Failed to run CSI prechecks for PVC
      file: kasten.io/k10/kio/exec/phases/backup/snapshot_data_phase.go:268
      function: kasten.io/k10/kio/exec/phases/backup.processArtifacts.func1
      linenumber: 268
      message: Failed snapshot precheck for namespace
    message: Job failed to be executed
    fields: []
     

    ```

     

    Thanks

    Hagag
    Forum|alt.badge.img+2
    • Hagag
    • Experienced User
    • 154 comments
    • July 11, 2022

    @andriktr Please share the output of the below script 

    curl https://docs.kasten.io/tools/k10_primer.sh | bash

    jaiganeshjk
    1 person likes this
    • jaiganeshjk
      jaiganeshjk
    A
    • andriktr
    • Author
    • Not a newbie anymore
    • 3 comments
    • July 11, 2022

    Hi @Hagag, 

    Here is the output: 

    Kubernetes Version Check:
  Valid kubernetes version (v1.23.5)  -  OK

RBAC Check:
  Kubernetes RBAC is enabled  -  OK

Aggregated Layer Check:
  The Kubernetes Aggregated Layer is enabled  -  OK

         Found multiple snapshot API group versions, using preferred.
CSI Capabilities Check:
  Using CSI GroupVersion snapshot.storage.k8s.io/v1  -  OK

         Found multiple snapshot API group versions, using preferred.
I0711 12:11:52.533396       9 request.go:601] Waited for 1.047216909s due to client-side throttling, not priority and fairness, request: GET:https://192.168.0.1:443/apis/networking.k8s.io/v1
         Found multiple snapshot API group versions, using preferred.
Validating Provisioners: 
disk.csi.azure.com:
  Is a CSI Provisioner  -  OK
  Missing/Failed to Fetch CSIDriver Object
  CSI Provisioner doesn't have VolumeSnapshotClass  -  Error
  Storage Classes:
    default
      Valid Storage Class  -  OK
    managed
      Valid Storage Class  -  OK
    managed-csi
      Valid Storage Class  -  OK
    managed-csi-premium
      Valid Storage Class  -  OK
    managed-premium
      Valid Storage Class  -  OK

file.csi.azure.com:
  Is a CSI Provisioner  -  OK
  Missing/Failed to Fetch CSIDriver Object
  CSI Provisioner doesn't have VolumeSnapshotClass  -  Error
  Storage Classes:
    azurefile
      Valid Storage Class  -  OK
    azurefile-csi
      Valid Storage Class  -  OK
    azurefile-csi-premium
      Valid Storage Class  -  OK
    azurefile-premium
      Valid Storage Class  -  OK

Validate Generic Volume Snapshot:
  Pod created successfully  -  OK
  GVS Backup command executed successfully  -  OK
  Pod deleted successfully  -  OK

     

    jaiganeshjk
    1 person likes this
    • jaiganeshjk
      jaiganeshjk
    I
    Forum|alt.badge.img
    • Iurii
    • Not a newbie anymore
    • 1 comment
    • Answer
    • July 22, 2022

    Hi @andriktr, 

     

    Based on the output of the preflight checks it looks like you don’t have a valid VolumeSnapshotClass instantiated for your Azure Disk CSI. That VSC needs to be added to your cluster. You can find the associated info for Azure disk CSI’s VSC: https://github.com/kubernetes-sigs/azuredisk-csi-driver/blob/master/deploy/example/snapshot/storageclass-azuredisk-snapshot.yaml/

     

    Once you’ve applied the above you will then need to follow the steps to configure Kasten to use this VSC they can be found here: https://docs.kasten.io/latest/install/storage.html#csi-snapshot-configuration

     

    Specifically, you’ll need to run the following command to annotate your storage class with the Kasten VSC annotation: 

    kubectl annotate storageclass ${SC_NAME} \ k10.kasten.io/volume-snapshot-class=<volumesnapshotclassName>`

    Additionally, if you need to use the Azure files CSI for any of your workloads, you would need to enable the generic backup feature for these as currently, VolumeSnapshotClass is unsupported. See more on enabling Generic backup in our docs here: https://docs.kasten.io/latest/install/generic.html#generic-storage-backup-and-restore

     

    Hope this helps! 

     

    A
    • andriktr
    • Author
    • Not a newbie anymore
    • 3 comments
    • July 22, 2022

    Thank you @Iurii your provided solution fixed the issue. 

    I
    1 person likes this
    • I
      Iurii

    Comment


    Terms & Conditions