VeeamON 2024 - Use Code "COMMUNITY10" for 10% Off!
@kelvin koh Not sure.Probably admins can.@Madi.Cristil / @Debarshi_K10 , Would you be able to change the marked answer or remove the answer in this post, so that Kelvin can mark the right one ?
@tinto.1970 I just saw your post.From what I see, You are trying to get alerts for the failed backups in your email.I think it is just matter of tweaking the alert rules in grafana .You probably have only the example alert rule configured from the docs.I can help you with the more promql queries which can be used to configured alert rules.LMK if you are still on it.
In this topic I’d like to ask the community if there is a way to tell K10 “send me one email every morning, containing the list of iussues occurred yesterday with the protection of namespaces/project.” Also, we don’t have a way to send daily reports over the email for time being.
@kelvin koh Can you mark the right one now?
@Matteo.Gazzadi As Emmanuel mentioned earlier, K10 has a master encryption Key that is used to generate keys per application export.This key changes if the K10 is reinstalled. However, it will be recovered using K10 DR recovery.K10-restore chart restores catalog database as well as the resources that were in Kasten-io namespace(which includes profiles, policies, secrets) unless these are skipped during restore.There is something going wrong in your installation/recovery which we would like to dig in further. Would you be able to open a support request by creating an account on https://my.veeam.com/ ?If you are using free version, select the Kasten K10 by Veeam Trial under the product to open a case with us including the debug logs as well.
@voarsh Thanks for posting the question. You will have to get in a toolbox pod (https://rook.io/docs/rook/v1.10/Troubleshooting/ceph-toolbox/) and run rbd cli commands.Please refer this documentation https://docs.ceph.com/en/mimic/rbd/rados-rbd-cmds/ for RBD commands. I don’t think you would need to remove the rbd images.You will have to run through a for loop for listing all the snapshots for each rbd image and remove those snapshots.Below is the list of commands that might come handy for you.#list all the images for a poolrbd ls {poolname}#list all the snapshots for an imagerbd snap ls {pool-name}/{image-name}#remove a snapshot with the image and pool name as inputrbd snap rm {pool-name}/{image-name}@{snap-name}#To delete all snapshots for an image with rbd, specify the snap purge option and the image name.rbd snap purge {pool-name}/{image-name}
Yes @voarsh That should purge all the snapshots.
@schnapsidee Thanks for posting your question.This might need some efforts & digging up and recreating the error locally in the test environments to find out if there is a limitation here.Would you be able to open a case with us through https://my.veeam.com/ If you don’t have an active enterprise subscription, You could select `Kasten by Veeam K10 Trial` under the products category.Also, please upload the blueprint(after obfuscating sensitive information) that you are using and debug logs to the case.Once we find the root cause of this issue through the case, We could come up with a summary and add it as an answer here.
@flavio_bitstan , Thanks for posting the question here.From the error I could see that the kanister-job pod did not transition to a running state.If it errored out for some reason, You could find it in the kanister controller(kanister-svc pod in kasten-io namespace). Are you using the same blueprint as specified in the example or do you have it customized based on the deployment ?How did you deploy your mysql ? Is it using bitnami charts ?
Thanks @flavio_bitstan . Let me take a look at this
@tinto.1970 Thank you for posting this question.We recently added a KB article to understand how this works and how it can be used along with Openshift/Tanzu.Please take a look at this KB article and let me know if you have further questions.
@Harish KM Though the NFS is a part of the CSI drivers list, It should be have CSI based snapshot support. However, I don’t think generic NFS CSI driver supports volumesnapshots.https://github.com/kubernetes-csi/csi-driver-nfs/tree/master/deploy/example K10 relies on CSI snapshotting capability which this driver doesn’t seem to have
Hi Flavio,Sorry I got stuck in something else and this slipped my mind.I will take a look at this today and definitely get back to you.
Thank you for your patience Flavio,I could see that the dump command fails with below error from the logs that you have shared.mysqldump: Got error: 2005: Unknown MySQL server host 'mysql' (-2) when trying to connectIt seems that the name resolution of the mysql service is problematic in your cluster.The ActionSet gets the name of the service from the helm release name as per the blueprint. Logs says that it is mysql. So assuming your helm releaseName is mysql,Can you confirm if there is a service with the name `mysql` ?
hi @tinto.1970 , Thanks for posting your question here.I understand that there is a problem with the policies. This usually happens when the secret that has the migration token is deleted. By re-validating, K10 policy controller recreates/finds that secret and add a reference in the profile. This could also happen if you manually edit the policy to remove the migrationToken/receiveString fields. Do you know if there is something that deletes this secrets or edit the policies ? Do you use ArgoCD or any gitOps tools to sync your policy ?
@stefanodemartini Thanks for posting the question here.From the above error messages, I understand that there is a problem with the permissions of the PVCs.All the pods which use PVCs are failing to start/initialise and the error message that I see in the logs are permission denied(except Grafana, which uses initContainer to change permissions of the directory).There is something wrong with the directory permissions and ownership for those PVCs. Let me take a look at the CephFS if it has any limitations wrt fsGroup settings in the workload.
Yes the example is designed for standalone mysql and not mysql replicated cluster.That is just an example for the end users to understand the functionality and you can customise it based on your requirements.In this case, It can be fixed easily.All you need is to change the hostname in the dump and restore commands that has the hostname configured.Currently it is pointing to this label value {{ index .Object.metadata.labels "app.kubernetes.io/instance" }} from the workload to get the service name.Value of this label is `mysql` and hence the template is rendered and the dump command uses mysql as hostname.You can actually edit the blueprint to add some other template which might render into `mysql-primary` or hardcode the hostname in the blueprint.You will have to change templates after `-h` in the below places.https://github.com/kanisterio/kanister/blob/master/examples/mysql/blueprint-v2/mysql-blueprint.yaml#L34https://github.com/kanisterio/kanister/blob/master/examples/mysql/blueprint
Even this is a weird behaviour for ArgoCD as we had earlier tested with the same and we did not come across this issue.We did not support policies being created with gitops until 5.5 major release of k10. So we started supporting it very recently. Can you confirm if you are using any specific sync-options with ArgoCD ?I can test it for you from my side as well.
I found it is the RWX volume is causing the issue. So, I believe Kasten K10 does not support volume backed by vSphere CNS which mounted as NFS from VSAN File Service. Is that right? @brian96 Thank you for posting your question.I suppose you are right with the support for VSAN file service. We haven’t tested this.Let me confirm if that is the case and update you.
@ntiwari38 Thank you for posting the question.From the events, I could see that the kanister-sidecar container from the catalog-svc is failing as it couldn’t pull the image from ghcr.io.Can you confirm if your environment allows pulling images from github container registry (ghcrio) ? If yes, it could be transient problem, You can try recreating the pod.You can also try to confirm if the image is accessible from you environment by running another pod with that image.kubectl run kanister-tools --rm -it --image ghcr.io/kanisterio/kanister-tools:0.85.0 -n kasten-io -- bash BTW, I can confirm that this image is available and accessible without any problem❯ docker pull ghcr.io/kanisterio/kanister-tools:0.85.00.85.0: Pulling from kanisterio/kanister-tools1d6f30850896: Pull complete 97d7b37e7232: Pull complete 196b571ab112: Pull complete 74d0dc5eeb47: Pull complete 187f5a0f24c6: Pull complete 701b65989276: Pull complete Digest: sha256:0c8da74a06fcfba32c7c278861b1e7e29b40e40f2a5e38c195f4339b81
@ntiwari38 Thanks for confirming. Glad that it worked for you.Would you be able to mark the comment as answer ?
@lidw Thanks for posting your question.The redirect URL should be the URL pointing to gateway svc which should have your NodePort in it.Would you please update your redirectURL value to include the nodePort as well. Your command should look like this.helm upgrade k10 k10-5.5.4.tgz --namespace=kasten-io -f k10_val.yaml \ --set auth.oidcAuth.enabled=true \ --set auth.oidcAuth.providerURL="http://192.168.40.10:8080/realms/kasten" \ --set auth.oidcAuth.redirectURL="http://192.168.40.11:32051" \ --set auth.oidcAuth.scopes="groups profile email" \ --set auth.oidcAuth.usernameClaim="email" \ --set auth.oidcAuth.groupClaim="groups" \ --set auth.oidcAuth.prompt="login" \ --set auth.oidcAuth.clientID="kasten" \ --set auth.oidcAuth.clientSecret="RbyRI3FgBrmuqNM2NhEQ7lMNzWp1663L"
@thatman Thank you for posting your question.Does the useraccount you use to login to the UI, have permissions to create profile in kasten-io namespace ?kubectl auth can-i create policies.config.kio.kasten.io -n kasten-io --as=<username>#if you are using serviceAccount token, then use the below commandkubectl auth can-i create policies.config.kio.kasten.io -n mysql --as=system:serviceaccount:<namespaceName:serviceaccountName>
@thatman Thanks for your detailed explanation with the timelines.I think I may know what happened with you. When you install K10 for the first time, you get a pop-up to accept EULA with your email and company name.This is stored as a configMap named k10-eula-info in K10 namespace.Without accepting the EULA, most of the actions in UI will be disabled.Since you mentioned you accepted that earlier, I am assuming this ConfigMap is somehow deleted from your kasten-io namespace, forcing you to accept it again.Since you accepted the second time, it might have started working for you again.
Just an FYI, It could be useful for someone who comes across this post at a later point. If you think some external gitops controller could be responsible for the deletion of this configMap because of the declarative state, You could use the below helm values to set the EULA info declaratively. eula.accept Whether to enable accept EULA before installation false eula.company Company name. Required field if EULA is accepted None eula.email Contact email. Required field if EULA is accepted None
Already have an account? Login
Enter your E-mail address. We'll send you an e-mail with instructions to reset your password.
Sorry, we're still checking this file's contents to make sure it's safe to download. Please try again in a few minutes.
Sorry, our virus scanner detected that this file isn't safe to download.
