VeeamON 2024 - Use Code "COMMUNITY10" for 10% Off!
@Madhuri Thanks for posting this question.Yes, K10 gets this metrics from Prometheus. If you have disabled prometheus, you won’t see any data populated in the usage metrics.K10 integrates deeply with prometheus so that a lot of operations/health checks need prometheus to be running.We recommend you to enable prometheus or if you already have a prometheus running in the cluster, configure K10 to use that(https://docs.kasten.io/latest/operating/monitoring.html#integrating-external-prometheus-with-k10).
@emre.ozer Thank you for posting this question.Can you describe and verify if your SA has a token secret created for it ?From K8s 1.24, Automatic creation of token secrets for SA is stopped by default and you need to request token from the kubeapiserver using TokenRequest API.So if you are using k8s 1.24, there’s a possibility that you don’t have that secret.You can however generate one token as per your convenience using the below command.kubectl -n kasten-io create token <serviceAccountName> --duration=1h - Jaiganesh
Glad that you were able to find the issue. We are trying to enhance this and are tracking this internally.However, for now, kanister-sidecar uses the same security-context as the application container. You might have to manually remove the read-only root filesystem SecurityContext from the kanister-sidecar or patch the kanister-sidecar to add an emptyDir volume for the `/tmp` path. This way, K10 can write to the `/tmp` directory.
Hi @schnapsidee , Thank you for posting this question.I see the error message Failed to exec command in pod: command terminated with exit code 1 in the first failure details.You would be able to find more details on why this exec failed in the kanister-svc pod logs.Would you be able to spot the error message in the kanister logs or post the logs so that I can have a look into it ?Second error message usually comes when you don’t have a location profile specified under Location Profile for Kanister Action. I couldn’t find any reason why setting this in the U, will still miss it in the YAML under backupParameters. I will try to see if I can reproduce the same in my test env.
@Smugness2774 Thanks for posting the question.The snapshot count and snapshot data in the report corresponds to the local snapshot (CSI based snapshot created through Volumesnapshot K8s resource or provider snapshots created using K10s direct integration) created by a backup action.Assuming, you are just running K10 DR policy/Cluster-scoped resource/GVS backup, the data is copied to your object store/NFS and not stored locally as snapshots during backup action. This is not counted as a local snapshot in the reports.You will be able to find the utilisation of the same under the object store utilisation.Hope this answers your question.
@andriktr Thanks for the question.It seems that you are annotating the storageclass with the annotation kubectl annotate storageclass ${SC_NAME} \ k10.kasten.io/volume-snapshot-class=”true” This annotation is for specifying the volumesnapshotclass name to use for a specific storageClass.This means that K10 looks for the volumesnapshotclass named “true” and it fails. You will have to either remove it and annotate the volumesnapshotclass with k10.kasten.io/is-snapshot-class=true or change the annotation to point to the the correct volumesnapshotclass name instead of truekubectl annotate storageclass ${SC_NAME} \ k10.kasten.io/volume-snapshot-class=<volumesnapshotclassName>
@cheslz Would you be able to confirm from where did you copy the export string (config data for import)?Is it from the policy ?I could see that the export you have done is a manual one from the above outputs.The manual exports are stored in a different location and if you were using the export string from the policy, it might probably not find the manifests in the correct directory.Would you be able to validate the same ?
One question, maybe you guys know this: Where are volumesnapshots stored (the storage needs should only be relevant for the delta, right?) inside the cluster? It depends on the backend storage that you use and in most of the cases the snapshots are only delta.I cant create a case in this particular outage, because the problem is simply gone. But it has a bad taste.I understand. The reason I wanted to work with you over a case is because there are few variables(NFS PVC claimname, repo path(this is derived from NS UID) that are used for generating the repository password. If any of those above changed as a part of the restore(if you deleted the namespace and recreated it with the restore) that you did, it might be a reason for `invalid repository password`Basically, K10 doesn’t store the keys. It is derived onDemand based on the above variables whenever needed.
@lemassacre The below error means that the kanister action timed out (default is 45 mins)This can be adjusted if your data is more and if you anticipate it to take more time.The helm value kanister.backupTimeout can be used to upgrade K10 to increase the timeout.This document will provide you all the available helm values that you can use to configure K10. function: kasten.io/k10/kio/poll.waitWithBackoffWithRetriesHelper linenumber: 95 message: Context done while polling fields: - name: duration value: 44m59.963318481s
@Smugness2774 Thank you for posting this question.While looking at the event messages, the attach-detach controller is failing to attach the volume to the node where `k10-grafana` pod is scheduled.Do you still see the older instance of K10-grafana pod in terminating state ?
@Yvan Thank you for posting this question.Would you be able to list the configmaps in K10 namespace and share the output here.kubectl get cm -n kasten-ioWe create a lock whenever a backup starts to run(to ensure no other jobs run against the particular policy). This lock will be removed once the policy-run completes. However, for some reason if the job did not finish/complete gracefully, these locks will be left over.This can be one case for the job to be in waiting state until timeout.
@SMT-Almato Would you be able to re-run the backup(since this is old and probably we might not have relevant logs anymore) and generate a debug bundle ?This needs further troubleshooting. Would you be able to create a case with us by selecting Kasten By Veeam K10 Trial under the select products drop-down including the debug logs ?
This needs further troubleshooting. Would you be able to create a case with us by selecting Kasten By Veeam K10 Trial under the select products dropdown
Hi @SMT-Almato Thank you for sharing the logs.Invalid repository password happens usually when you reinstall K10. This way, the MasterKey K10 uses changes causing the mismatch while connecting to the kopia repository in your target NFS location.However, I don’t see any re-installation recently from the logs you have shared. Also I see that all the K10 secrets are intact. This shouldn’t be happening unless there is something wrong.Would you be able to elaborate more on the network issue that you mentioned in the post ?
Hi @KastenJohn Thank you for posting this question.Automatic sidecar injection won’t work in older OCP clusters.I believe your cluster doesn’t have V1 API version for the mutatingwebhookconfiguration.You might have to install K10 without enabling the sidecar injection and use K10tools to inject sidecars manually.
It’s just a few MB that are copied on that snapshot, only took a few seconds before the upgrade. Are there any logs that i can investiagte? This shouldn’t take that much time for few MB. You can look at the kanister-svc logs to see if there are any clues.If not, You can raise a case with us by signing up to veeam portal https://my.veeam.com/ with the debug logs.
Thanks for confirming that your issue with the gateway service is resolved.kanister.backupTimeoutI would suggest you to mark the comment with the KB article as the answer for this question. Unfortunatelly it doesn’t solve the problems that i have with volume snapshots since the upgrade:cause: cause: cause: cause: cause: message: context deadline exceeded file: kasten.io/k10/kio/poll/poll.go:95 function: kasten.io/k10/kio/poll.waitWithBackoffWithRetriesHelper linenumber: 95 message: Context done while polling fields: - name: duration value: 44m5kanister.backupTimeout9.963318481s file: kasten.io/k10/kio/poll/poll.go:65 function: kasten.io/k10/kio/poll.waitWithBackoffWithRetries linenumber: 65 message: Timeout while polling fields: - name: actionSet value: k10-backuptoserver-k10-deployment-generic-volume-2.0.17-piznnbl file: kasten.io/k10/kio/kanister/operation.go:284 functio
@lemassacre Thank you for posting this question.K10 has started using Emissary . There is a change in the image registry for emissary and it is now docker.io/emissaryingress/emissary:2.2.2You might probably be using older values.yaml file.Please have a look at this published KB article and let us know if it helps
@Pablo_Madriaga I understand that this is not mentioned in our documentation. I will see if this can be added to the Vsphere integration page.
@Arun Prasad It seems that you are trying to import the backups into the same cluster that you have exported from.Is that the case ? This will not work in K10,Import is actually designed to import the restorePoints into a new cluster and restore into them.It can be directly restored from the the exported restorePoint in the same cluster. You don’t need to import it.
@Pablo_Madriaga I believe you are using Tanzu Kubernetes Grid Service(TKGs)/Vsphere with Tanzu.In that case, K10 infrastructure profile doesn’t support TKGs yet.It is on the roadmap and will be supported in future.
@marcinbojko The container name is mandatory field in the override and the container name should match the actual pods container name.In case of K10, all the container in pods created by kanister has a default name `container`.Can you try the below ConfigMap ? It should work. apiVersion: v1 data: override: | kind: Pod spec: containers: - name: container resources: requests: cpu: "100m" memory: "512Mi" limits: cpu: "200m" memory: "1Gi" kind: ConfigMap metadata: name: pod-spec-override namespace: kasten-io
@marcinbojkoWhy do you think you will lose it after the upgrades ?Would you be able to let me know how you are configuring the podOverrides ? IMO, You won’t lose it unless the configMap is deleted manually.
Yea I assume this kanister-job-* pod is from a blueprint based backup. right ?You might have to use pod Overrides for this use case I believe.Because if you have an admission controller that forbids pods without resource limits, Kanister pod Override is the way to go for dynamically created pods (by kanister).
@marcinbojko , You can use the helm value genericVolumeSnapshot.resources.[requests|limits].[cpu|memory] to set the default resources requests/limits for the kanister-sidecar containers.For example, if you want to set the CPU and memory requests/limits to 1 core and 1Gi respectively,use the below upgrade command to do the same.helm get values k10 --output yaml --namespace=kasten-io > k10_val.yaml && \helm upgrade k10 kasten/k10 --namespace=kasten-io -f k10_val.yaml \ --set genericVolumeSnapshot.resources.requests.cpu=1 \ --set genericVolumeSnapshot.resources.requests.memory=1 \ --set genericVolumeSnapshot.resources.limits.cpu=1 \ --set genericVolumeSnapshot.resources.limits.memory=1However, this won’t have any effect on already injected `kanister-sidecar` containers. You might have to uninject them to get them updated.We have an in-house tool k10tools that can be used to easily un-inject all the sidecars.
Already have an account? Login
Enter your E-mail address. We'll send you an e-mail with instructions to reset your password.
Sorry, we're still checking this file's contents to make sure it's safe to download. Please try again in a few minutes.
Sorry, our virus scanner detected that this file isn't safe to download.
