VeeamON 2024 - Use Code "COMMUNITY10" for 10% Off!
one more thing, what is the type of your EKS nodes, is it Fargate?
@joao.neri are statically or dynamically provisioned EFS volumes?if it is dynamically provisioned EFS volumes, please review and follow the below linkhttps://docs.kasten.io/latest/install/shareable-volume.html#shareable-volume-backup-and-restorealso please share the YAML file of the EFS storage class and the volume (PV ) you are trying to backup.ThanksAhmed Hagag
Hi @lgromb It seems there may be an issue originating from the Ceph side, as the responsibility for managing snapshots lies with the Ceph csi-snapshotter. K10 merely requests deletion from the CSI snapshotter. I recommend reaching out to the Ceph or storage team to investigate why errors occur during snapshot deletion.you can also try to manually create and delete the snapshot without involving K10 and check if you have the same issue. Additionally, I came across a link to a bug report detailing a similar issue.https://bugzilla.redhat.com/show_bug.cgi?id=1951399BR,Ahmed Hagag
@rajeshvarun Please share the PVC YAML file and the output of the below command,Please sanitize the output to remove any sensitive information. helm get values k10 --output yaml --namespace=kasten-io ThanksAhmed Hagag
@Raff please share the description command for the pod catalog which is in crashloopbackoff, we might find any errors telling why the pod did not startedalso share the list of PVCs running in kasten-io namespace kubectl describe pod -l component=catalog -n kasten-io kubectl get pvc -n kasten-ioThanksAhmed Hagag
Hi @atinivelli as you stated it could be a network issue, would you be able to reach your NFS host from the cluster? you can kubectl exec any pod and try to ping/curl the NFS endpoint.also, try to mount that PVC in a dummy volume and check if it works.ThanksAhmed Hagag
@george.tor2018 To activate the Generic Storage Backup feature, you must obtain a token by requesting one through my.veeam.com. To do so, open a support case, specifying "Kasten by Veeam K10 Trial" as the product. Once you receive the token, you can proceed to edit the K10 operand YAML file. In the file's specifications, add the provided parameters. genericStorageBackup: token: <TOKEN>injectKanisterSidecar: enabled: “true”You need to follow the same procedure for any additional parameters necessary to enable injectKanisterSidecar. You can find all the required parameters in the link provided below.https://docs.kasten.io/latest/install/generic.html?highlight=injectkanistersidecar#generic-storage-backup-and-restoreThanksAhmed Hagag
@terrencejun321 According to the error message, the token extracted by k10tool for validation does not match the one you've created. Could you please provide details on how you obtained the clientSecret token, the YAML file for the service account k10-dex-sa, and the list of secrets?ThanksAhmed Hagag
Hello @issen007 if you check the warining message when describing the catalog pod, It indicates that youe EKS cluster is having trouble scheduling a pod due to resource limitations on your nodes (worker machines in your cluster).No nodes out of your 3 available nodes are currently suitable to run the pod, Three nodes lack sufficient memory to run the pod. Its resource requirements exceed the available memory on those nodes. Warning FailedScheduling 3m55s (x77 over 6h23m) default-scheduler 0/3 nodes are available: 1 Too many pods, 3 Insufficient memory. preemption: 0/3 nodes are available: 3 No preemption victims found for incoming pod.BR,Ahmed Hagag
Hi @B-Lehcim As per your mention, Kubernetes version 1.29 is currently not supported, which could potentially cause some issues.Nevertheless, we can proceed with investigating the problem further. Could you please provide a screenshot of the error encountered during volume snapshotting?Additionally, if the primer is persistently stuck in the snapshotting process, it might indicate a problem with your CSI provisioner. I suggest checking its logs for any errors.Furthermore, you can attempt manual volume snapshotting using this provisioner without involving K10. This might help us diagnose the issue more effectively.
Hello @vbrizuela To initiate the start of the copy-vol-data pod, a PVC is established within the kasten-io namepace. This volume is derived from a cloned snapshot and must be ready and mounted to the copy-vol-data pod.Based on the error message provided, it appears that the volume creation process might be encountering issues.It could be pending, not yet created, or facing some other problem. It's essential to investigate why the volume isn't being created and bounded within the kasten-io namespace.Error Details:Failed to attach to the node 'node' with attachmentID 'csi-5e91702a070b2db664a6a7e3ba50e5b37c71b2c440628a746b8bc21ec3a02792': Volume 'pvc-27883e4b-f9d4-43dd-8fac-565045bf302d' has already been attached to node 'dell-750-9' with incompatible parameters.ThanksAhmed Hagag
@sriramadasu please provide the Helm command used to install k10. did you use the below helm option?services.dashboardbff.hostNetworkThanksAhmed Hagag
Hello @peterturnip currently, migrating workloads with nodeselector on the source cluster poses a challenge. However, we are actively working to fully support this without requiring manual interventions. As a temporary workaround for restoration, you can manually add the labels to the nodes in the destination cluster.
Hi @hafesi if you are using NFS PVCs, you need to enable the GenericStorageBackup as the below link.https://docs.kasten.io/latest/install/generic.html?highlight=generic%20volume%20snapshot BR,Ahmed Hagag
@donaldleung yea kasten k10 can be installed from the openshift marketplace/operatorhub BR,Ahmed Hagag
@jimenro you need to check CPU Compatibility, you can use the lscpu command on the host node to verify supported features (look for "x86-64-v2" in flags).you might need to adjust Virtualization Settings if applicable, Consult your virtualization platform's documentation for instructions on enabling x86-64-v2 features. BR,Ahmed Hagag
@rbhowmik-sds my colleague has ownership of the case and will contact you soon.I also had a quick view of the logs, and I found the jobs-svc pod is not started, so the executor pods won't be able to pull the job from the queue.jobs-svc-6b999bbdc9-c82bg This pod is not started also, you have many pods restarting and some of them are in crashloopbackupoff status.CC @Haythem Elkhouly
@rbhowmik-sds I see the case is taken by my colleague, he should check it and get back to you but from the logs I see the jobs-svc pod is not started which explains why there is an issue with executor pods unable to retrieve the job from the queue for execution.jobs-svc-6b999bbdc9-c82bg 0/1 Init:0/1 0 18hI see also many K10 pods restarting and in CrashLoopBackOff. services/executor-svc-8d6ff74c4-h9cc8.txt:{"File":"kasten.io/k10/kio/exec/internal/queuepoll/queuepoll.go","Function":"kasten.io/k10/kio/exec/internal/queuepoll.(*QueuePoll).getQueuedJobWithBackoff.func1","Line":58,"cluster_name":"a852c906-4194-4486-b00a-8f0be53f817d","error":{"message":"Error fetching queued job","function":"kasten.io/k10/kio/exec/internal/jobs.(*JobManager).NextQueuedJob","linenumber":145,"file":"kasten.io/k10/kio/exec/internal/jobs/jobmanager.go:145","cause":{"message":"Get \"http://10.111.62.71:8000/v0/queuedjobs\": dial tcp 10.111.62.71:8000: connect: connec
@rbhowmik-sds i need to check the debug logs, Would you mind opening a case with us through `my.veeam.com` and use `Kasten by veeam K10 Trial` in products while opening a case.Please collect the debug logs (https://docs.kasten.io/latest/operating/support.html#gathering-debugging-information) and upload the same to the case. We will get in touch and take a deep look at what’s going on.
Hi @rbhowmik-sds The executor pods seem unable to retrieve the job from the queue for execution, which differs from the initial error you provided.it is telling that you have read only FS and could not fetch queued jobsdid you get the same kopia error in the dashbard?Ahmed
@rbhowmik-sds I would suggest during the backup/export to check logs of the following pods executor-svc and kanister-svc and focus on kopia errors, it would be nice if you shared the log snippets with uskubectl logs -l component=executor -n kasten-io -fkubectl logs -l component=kanister -n kasten-io -fAhmed
@hafesi looks race condition with one another concurrent operation.error that occurs when you try to create a snapshot of a PersistentVolumeClaim (PVC - tomcat-empal-email-api-logs ) that has been modified since you last fetched it from the API server.is it happened during the backup or export phase?i would suggest to try to capture what is changing during takeing snapshot from that PVCyou can capture the YAML file of the PVC before starting the snapshot, then start the backup/export job and try to capture the PVC YAML file again to check if there were any changes!If you continue to receive the error message, we may need to investigate further to determine what is causing the PVC to be modified.Ahmed Hagag
@rbhowmik-sds I see the Kubelet is unable to attach or mount volumes for a pod. The exact cause of the error may be because the storage is already utilised as it is mentioned in the volume description.yea, you might need to check if you have sufficient space in the nodes.BR,Ahmed Hagag
@WWK-Cloud-Plattform if could guess whic RBAC is missing if you share the role/clusterrole assinged to that user.BR,Ahmed Hagag
@WWK-Cloud-Plattform what is your k10 version?This issue is fixed in k10 v 6.5.0, please upgrade to that version.BR,Ahmed Hagag
Already have an account? Login
Enter your username or e-mail address. We'll send you an e-mail with instructions to reset your password.
Sorry, we're still checking this file's contents to make sure it's safe to download. Please try again in a few minutes.
Sorry, our virus scanner detected that this file isn't safe to download.
