• EN
Solved

kasten k10 Pre-Flight Checks reports longhorn as Not a supported CSI driver

V

Hi Team,

We installed longhorn for our storage. We want to take CSI snapshots using kasten k10, i have already installed snapshot CRDs . But when i run K10 primer check it fails ,

 

Starting CSI Checker. Could take up to 5 minutes
CSI Snapshot Walkthrough:
Not a supported CSI driver (driver.longhorn.io) - Error

 

 

But i can take volumesnapshots and those are created in longhorn and saved to our s3 storage. This works fine.

kubectl get volumesnapshotcontent
NAME READYTOUSE RESTORESIZE DELETIONPOLICY DRIVER VOLUMESNAPSHOTCLASS VOLUMESNAPSHOT AGE
snapcontent-c25957d7-01a5-4bbc-8d7d-bea22f9a28e4 true 2147483648 Delete driver.longhorn.io longhorn test-snapshot-pvc 18m
snapcontent-d9e9d0f4-61fd-487f-8f41-f5b30c3d4dd8 true 2147483648 Delete driver.longhorn.io longhorn test-snapshot-pvc2 9s

kubectl get volumesnapshot
NAME READYTOUSE SOURCEPVC SOURCESNAPSHOTCONTENT RESTORESIZE SNAPSHOTCLASS SNAPSHOTCONTENT CREATIONTIME AGE
test-snapshot-pvc true nginx-readwrite-many 2Gi longhorn snapcontent-c25957d7-01a5-4bbc-8d7d-bea22f9a28e4 19m 18m
test-snapshot-pvc2 true nginx-readwrite-many 2Gi longhorn snapcontent-d9e9d0f4-61fd-487f-8f41-f5b30c3d4dd8 4m20s 14s

But i don't understand why kasten k10 thinks longhorn is not capable when all the CRDs and snapshot-controller are present. I even tried to add the annotation "k10.kasten.io/is-snapshot-class: "true"" to VolumeSnapshotClass but still the same issue.

 

Regards,

Balaji

icon

Best answer by Hagag 27 May 2022, 11:32

View original

10 comments

Hagag
Userlevel 5
Badge +2

Hello @vsbsapbasis 

Please look at the below link to confirm snapshot support and confirm the longhorn version

https://kubernetes-csi.github.io/docs/drivers.html

 

Thanks

Ahmed Hagag

 
V

Hi Ahmed

 

Our longhorn is on v1.2.4 and as per link it is v1.2 , so it should be fine.

 

 

Regards,

Balaji

Hagag
Userlevel 5
Badge +2

@vsbsapbasis  Please share the output of the pre-flight check.

 

Thanks

V

Hi Hagag,

 

url -s https://docs.kasten.io/tools/k10_primer.sh  | bash
Namespace option not provided, using default namespace
Checking for tools
 --> Found kubectl
 --> Found helm
Checking if the Kasten Helm repo is present
 --> The Kasten Helm repo was found
Checking for required Helm version (>= v3.0.0)
 --> No Tiller needed with Helm v3.8.2
K10Primer image
 --> Using Image (gcr.io/kasten-images/k10tools:4.5.14) to run test
Checking access to the Kubernetes context kubernetes-admin@kubernetes
 --> Able to access the default Kubernetes namespace
K10 Kanister tools image
 --> Using Kanister tools image (ghcr.io/kanisterio/kanister-tools:0.78.0) to run test

Running K10Primer Job in cluster with command-
     ./k10tools primer
serviceaccount/k10-primer created
clusterrolebinding.rbac.authorization.k8s.io/k10-primer created
job.batch/k10primer created
Waiting for pod k10primer-r8znj to be ready - ContainerCreating
Waiting for pod k10primer-r8znj to be ready - ContainerCreating
Waiting for pod k10primer-r8znj to be ready - ContainerCreating
Waiting for pod k10primer-r8znj to be ready - ContainerCreating
Waiting for pod k10primer-r8znj to be ready - ContainerCreating
Waiting for pod k10primer-r8znj to be ready - ContainerCreating
Waiting for pod k10primer-r8znj to be ready - ContainerCreating
Waiting for pod k10primer-r8znj to be ready - ContainerCreating
Pod Ready!

Kubernetes Version Check:
  Valid kubernetes version (v1.23.5)  -  OK

RBAC Check:
  Kubernetes RBAC is enabled  -  OK

Aggregated Layer Check:
  The Kubernetes Aggregated Layer is enabled  -  OK

CSI Capabilities Check:
  Cluster isn't CSI capable

Validating Provisioners:
driver.longhorn.io:
  Storage Classes:
    longhorn
      Supported via K10 Generic Volume Backup. See https://docs.kasten.io/latest/install/generic.html.

Validate Generic Volume Snapshot:
  Pod Created successfully  -  OK
  GVS Backup command executed successfully  -  OK
  Pod deleted successfully  -  OK

serviceaccount "k10-primer" deleted
clusterrolebinding.rbac.authorization.k8s.io "k10-primer" deleted
job.batch "k10primer" deleted

 

 

 

 

 

curl -s https://docs.kasten.io/tools/k10_primer.sh  | bash /dev/stdin -s longhorn
Namespace option not provided, using default namespace
Checking for tools
 --> Found kubectl
 --> Found helm
Checking if the Kasten Helm repo is present
 --> The Kasten Helm repo was found
Checking for required Helm version (>= v3.0.0)
 --> No Tiller needed with Helm v3.8.2
K10Primer image
 --> Using Image (gcr.io/kasten-images/k10tools:4.5.14) to run test
Checking access to the Kubernetes context kubernetes-admin@kubernetes
 --> Able to access the default Kubernetes namespace
K10 Kanister tools image
 --> Using Kanister tools image (ghcr.io/kanisterio/kanister-tools:0.78.0) to run test

Running K10Primer Job in cluster with command-
     ./k10tools primer storage csi-checker -s longhorn
serviceaccount/k10-primer created
clusterrolebinding.rbac.authorization.k8s.io/k10-primer created
job.batch/k10primer created
Waiting for pod k10primer-mp99x to be ready - ContainerCreating
Waiting for pod k10primer-mp99x to be ready - ContainerCreating
Pod Ready!

Starting CSI Checker. Could take up to 5 minutes
CSI Snapshot Walkthrough:
  Not a supported CSI driver (driver.longhorn.io)  -  Error

serviceaccount "k10-primer" deleted
clusterrolebinding.rbac.authorization.k8s.io "k10-primer" deleted
job.batch "k10primer" deleted

 

Regards,

Balaji

Hagag
Userlevel 5
Badge +2

Hello @vsbsapbasis 

It seems CSI driver issue.
Try to check the logs for csi-attacher-0 and csi-provisioner-0, as well as containers in longhorn-csi-plugin-xxx

 

Regards,

Ahmed

Geoff Burke
Userlevel 7
Badge +22

HI Folks,

 

I had this issue as well and Adam Berg posted in Rancher slack that there was an issue with Longhorn and a specific version. Let me find that and I will post here.

VMCA2022, VMCE2023, CKA, CKAD, KCNA, LFCS, PCA,TARS
Geoff Burke
Userlevel 7
Badge +22

Someone had a similar issue it seems. Adam posted his exact instructions on how to get things working. The other problem might have been from earlier versions:

 

Here is what he posted in the Rancher Slack:

 

Prepare K3s Nodes for Longhorn ISCSI (CentOS 7)--------------------------------yum install -y iscsi-initiator-utilscat <<EOF > iscsid.temodule iscsid 1.0;require {        type iscsid_t;        class system module_load;}allow iscsid_t self:system module_load;EOFcheckmodule -M -m -o iscsid.mod iscsid.tesemodule_package -m iscsid.mod -o iscsid.ppsemodule -i iscsid.pprm -f iscsid.mod iscsid.te iscsid.pp

(Install Longhorn from Rancher UI)

Install snaphot CRDs------------------------------------git clone https://github.com/kubernetes-csi/external-snapshotter.gitkubectl kustomize external-snapshotter/client/config/crd | kubectl create -f -kubectl -n kube-system kustomize external-snapshotter/deploy/kubernetes/snapshot-controller | kubectl create -f -Enable Longhorn Snapshot Class--------------------------------------cat <<EOF>> longhorn-csi-snapshotclass.yamlapiVersion: snapshot.storage.k8s.io/v1kind: VolumeSnapshotClassmetadata:  name: longhorn  annotations:    k10.kasten.io/is-snapshot-class: "true"driver: driver.longhorn.iodeletionPolicy: DeleteEOFkubectl apply -f longhorn-csi-snapshotclass.yaml --kubeconfig k3sconfigInstall local minio for longhorn--------------------------------------kubectl create -f https://raw.githubusercontent.com/longhorn/longhorn/master/deploy/backupstores/minio-backupstore.yamlGo to the Longhorn UI. In the top navigation bar, click Settings. In the Backup section, set Backup Target tos3://backupbucket@us-east-1/And set Backup Target Credential Secret to:minio-secretTesting CSI---------------------------------------kubestr csicheck -s longhorn -v longhornInstall Kasten K10----------------------------------------helm repo add kasten https://charts.kasten.io/helm repo updatekubectl create namespace kasten-iohelm install k10 kasten/k10 --namespace=kasten-ioPort Forwarding Kasten UI---------------------------------------kubectl --namespace kasten-io port-forward service/gateway --address 172.18.117.110 8080:8000 > /dev/null 2>&1 & Demo Application (Pacman)---------------------------------kubectl create namespace pacmanhelm repo add veducate https://saintdle.github.io/helm-charts/helm install pacman veducate/pacman -n pacman
VMCA2022, VMCE2023, CKA, CKAD, KCNA, LFCS, PCA,TARS
V

Hi Geoff,

 

I have already followed those steps, still it says below,

 

Starting CSI Checker. Could take up to 5 minutes
CSI Snapshot Walkthrough:
  Not a supported CSI driver (driver.longhorn.io)  -  Error

 

 

interestingly

 

kubestr csicheck -s longhorn -v longhorn
Creating application
  -> Created pod (kubestr-csi-original-podvk972) and pvc (kubestr-csi-original-pvcvq6gh)
Taking a snapshot
  -> Created snapshot (kubestr-snapshot-20220527152744)
Restoring application
  -> Restored pod (kubestr-csi-cloned-podxkg4c) and pvc (kubestr-csi-cloned-pvckvj7d)
Cleaning up resources
CSI checker test:
  CSI application successfully snapshotted and restored.  -  OK

 

I am even able to create snapshots manually as i posted earlier.

 

 

Regards,

Balaji

V

Hi Geoff,

 

Just to update our kubernetes is on 1.23. Not sure if that is the problem.

 

 

Regards,

Balaji

V

Hi Geoff,

It works on 1.21 , i referred the k10 documentation again, it has support only until 1.22.

But while running 1.21 curl -s https://docs.kasten.io/tools/k10_primer.sh  | bash /dev/stdin -s longhorn

 

i face below issue, have you come across below issue before ?

 

CSI Snapshot Walkthrough:
  Using annotated VolumeSnapshotClass (longhorn)
  Failed to fetch data from pod. Failure may be due to permissions issues. Try again with runAsUser=1000 option.: Failed to exec command in pod: command terminated with exit code 1  -  Error

serviceaccount "k10-primer" deleted
clusterrolebinding.rbac.authorization.k8s.io "k10-primer" deleted
job.batch "k10primer" deleted

 

Regards,

Balaji

Comment


Terms & Conditions