VeeamON 2024 - Use Code "COMMUNITY10" for 10% Off!
@KelianSB still, the fix is in progress.
@andriktr Please share the output of the below script curl https://docs.kasten.io/tools/k10_primer.sh | bash
Hello @markbon I think your configuration is ok , For each CSI driver, ensure that a VolumeSnapshotClass has been added with K10 annotation (k10.kasten.io/is-snapshot-class: "true").Why k10-clone-csi-cinder is created? Note that CSI snapshots are not durable. In particular, CSI snapshots have a namespaced VolumeSnapshot object and a non-namespaced VolumeSnapshotContent object. With the default (and recommended) deletionPolicy, if there is a deletion of a volume or the namespace containing the volume, the cleanup of the namespaced VolumeSnapshot object will lead to the cascading delete of the VolumeSnapshotContent object and therefore the underlying storage snapshot.Setting deletionPolicy to Delete isn't sufficient either as some storage systems will force snapshot deletion if the associated volume is deleted (snapshot lifecycle is not independent of the volume). Similarly, it might be possible to force-delete snapshots through the storage array's native management interface. Enabling b
@markbon It seems you are using kanister blueprints to backup the application,the pod "kanister-job-s5vl8" that created during the backup in your app namespace is failed to be in Running statusmight be a problem with pulling the image "Image:ghcr.io/kanisterio/kanister-kubectl-1.18:0.84.0”Please try to track the pods initiated in your app namespace and do kubectl describe to get more insight about the error kubectl get pods -n <NAMESPACE> -wyou can create a while loop to be able to catch the pod before its termination. while true; do kubectl describe pod -l createdBy=kanister -n <NAMESPACE> ; sleep 2 ; donekubectl describe pod -l createdBy=kanister -n <NAMESPACE>
Hello @gerardotapianqn Can you describe any of these pending pods and share the output ? ThanksAhmed Hagag
@gerardotapianqn do you have Preemption enabled in your cluster, I see the below error:default-scheduler 0/3 nodes are available: 3 pod has unbound immediate PersistentVolumeClaims. preemption: 0/3 nodes are available: 3 Preemption is not helpful for scheduling. It seems the pending pods are in the queue and waiting to be scheduled. pods are waiting to be scheduled and will stay in the scheduling queue, until sufficient resources are free, and they can be scheduled.The scheduler picks a Pod from the queue and tries to schedule it on a Node. If no Node is found that satisfies all the specified requirements of the Pod, preemption logic is triggered for the pending Pod.can you share the output of the below command:kubectl get priorityclass
Hello @Luiz Eduardo Serrano it is mandatory to have your preflight script run successfully.This script runs in a pod in the cluster and does the following: Validates if the Kubernetes settings meet the K10 requirements. Catalogs the available StorageClasses. If a CSI provisioner exists, it will also perform a basic validation of the cluster's CSI capabilities and any relevant objects that may be required. 2nf point is you can not use the kasten GUI/ dashboard without having most of the k10 up and running, for example, I see the catalog pod is in pending status, similar to etcd in Kubernetes, catalog is a backend store, all the information for everything happens in k10 is stored in the catalog.Jobs pod also is not running, you can not run any action without that pod up and running.i recommend fixing the pre-flight check prerequisites in order to have it run and get a summary of what is missing or required in order to have k10 working properly.
@Luiz Eduardo Serrano you can describe the pod to get more insight into why the pods are not running, using the command and trying to fix the issue based on the error message you have in the output.kubectl describe <POD_NAME> -n kasten-io but it is much easier to fix the pre-flight checks script to have an overview of what is missing or required in order to have k10 working properly. for jq package, your OS has to be able to locate the correct executable. PATH specifies program directories and instructs the system where to search for a program to run.try to the below commands to search the path of jq ( i think it will fail )which jqcommand -v jq
@gerardotapianqn I see you have set a PriorityClass with preemption enabled, The priority level of a PriorityClass is used by the Kubernetes scheduler to determine the order in which pods should be scheduled. If not enough resources are available to schedule all pods, the scheduler will prioritize higher-priority pods over lower-priority pods.That is why you have some K10 pods in a pending state, could you make sure that you have enough cluster resources for the Kubernetes scheduler to be able to schedule the rest of the pods?or you can try to disable the preemption & priority class if not needed.
Hello @voarsh Try to have a look into the status of the retire-actions that get created and check if it is failing! kubectl get retireactions.actions.kio.kasten.io -o custom-columns=NAME:.metadata.name,"CREATED AT":.metadata.creationTimestamp,STATUS:.status.stateand what is the deletion policy specified in the volumesnapshot class(that the volumesnapshot uses)? Ahmed Hagag
Hello @widura it seems, the catalog deployment is already scaled down and the restore pod is trying to scale it down too, and eventually, you got the error “unable to scale down...)plz share the output of the below command: kubectl get deployments catalog-svc -n kasten-ioand please clarify if you trying to restore on the same cluster.If you are reinstalling K10 on the same cluster, cleaning up the namespace in which K10 was previously installed before the above passphrase creation is important.and K10 must be reinstalled before recovery. Ahmed Hagag
@Brandonrbt when you run k10multicluster bootstrap …, it uses the kubeconfig on the machine from which you ran the command and attempts to connect to the cluster. But it can't even connect to the cluster using the kubeconfig, so something's wrong there (that's why you're getting the ERROR: API server connection failed) On the same machine that you're running k10multicluster bootstrap …, what does this give you?kubectl --kubeconfig=<your secondary cluster .kube/conf> --context=<secondary-context> --namespace kasten-io get serviceaccounts
@mcoul Enabling Kanister Sidecar Injection adds an additional sidecar container to your deployment, which can cause additional resource usage and may lead to instability in your cluster. This could be the reason why your MinIO instances are restarting frequently when sidecar injection is enabled.To troubleshoot this issue, you can try the following steps: Check the logs of the MinIO pods to see if there are any errors or warnings that might indicate the cause of the instability. You can use the kubectl logs command to view the logs of a specific pod. Check the resource usage of your cluster to see if it is reaching its limits. You can use the kubectl top command to view the resource usage of your pods and nodes. Try increasing the resource limits for your MinIO deployment to see if it improves stability. You can do this by modifying the resources section of the deployment YAML
@jaiganeshjk
Hello @Kittikom.Pa I noticed that you're using the incorrect port number "8000" instead of "8080" in the URL. Additionally, I assume that the IP address you're using is that of your local machine.
@Kittikom.Pa the best option is to follow the recommendation provided by @jaiganeshjk in the pervios comment.
Hi @Daniel Zentai Allow me to verify this in my testing lab, and I will provide you with an update shortly.
@Daniel Zentai Despite the warning about a duplicated license in my secondary cluster, it is still able to lease nodes from the contributed pool instead of the local one, as shown in the screenshot from my secondary cluster. I recommend restarting the deployment of the controllermanager-svc in the primary cluster and rechecking the issue. kubectl rollout restart deployment controllermanager-svc -n kasten-ioSuppose you have three clusters using the same starter license ID. Only one of the clusters should have a valid license ID for license contribution, while the remaining two clusters with duplicated licenses will not be utilized for contribution.P.SIt is not essential for the contribution to come from your primary cluster. It could originate from any of your secondary clusters. Our Engineering team needs to confirm whether this is the expected behavior or not.In order to gain a more comprehensive understanding of your environment, kindly provide the license page for each of yo
@Daniel Zentai According to the information you provided, if you are utilizing the same license ID across all of your clusters, this is the expected behavior. One of the installed clusters will be chosen as the "owner" of the license when (in this case, the starter license) is installed in several clusters.and I can see that one of your clusters is using 4 nodes plus 6 nodes for contributions, which adds up to the 10 nodes that the starter license allows you to use in total.You are renting 5 of the 6 nodes for the other clusters, but your final cluster, which has 4 nodes, is no longer able to rent from the pool due to a dearth of contributed nodes.To solve this issue, I recommend either upgrading your license or disabling Multi-Cluster License Management by editing the Cluster resource for the primary cluster and removing the ingress URL field. If you choose to disable this feature, each of your clusters will be able to use its own starter license while still utilizing the multi-cluste
@Florian Lacrampe please share the version of VDDK you are using.do you have any errors in VDDK logs?
@meijianwei The regular snapshot should function correctly. Could you please provide the K10 values file and the YAML file of your MySQL statefulset?how you disabled the injection of the Kanister sidecar?
@meijianwei The "injectKanisterSidecar.enabled" parameter is utilized when performing a Generic Storage Backup and Restore and is not associated with the logical backup of MySQL. It appears that you have confused two configurations. Earlier, you had mentioned that you were following a GitHub link to back up your database.(https://github.com/kanisterio/kanister/tree/master/examples/mysql)The link employs Kanister, which is a tool for managing data protection workflows.If you want to enable logical MySQL backup, I suggest using the link below. However, before proceeding, it's important to remove any previously created Artifacts and CRDs.https://docs.kasten.io/latest/kanister/mysql/install.html?highlight=logical#logical-mysql-backupTo delete Artifactshttps://github.com/kanisterio/kanister/tree/master/examples/mysql#delete-the-artifactsTo delete CRDshttps://github.com/kanisterio/kanister/tree/master/examples/mysql#delete-crs
@meijianwei I have successfully reproduced the issue. It appears to occur when using replicas in your workload. I will need some time to investigate this behavior and provide you with further information. In the meantime, a possible workaround is to remove the annotation from your workload, such as StatefulSets ( to disable blueprint usage ) , and perform a normal backupkubectl --namespace mysql-test annotate statefulset/mysql-bk-sfs kanister.kasten.io/blueprint=mysql-dep-config-blueprint-P.S The blueprint will work with one replica
@meijianwei P.S. When attempting to connect to your MySQL host in the blueprint, you are connecting through the service cluster IP which acts as a load balancer between your workload endpoints. However, this does not ensure that you are connecting to the host containing the test database. Prior to utilizing the blueprint, it is necessary to manage the database synchronization between the endpoints.
@markbon I think Flux CD also has a similar feature called ignore that allows you to ignore certain differences between the desired and observed state of the policy resource during a sync operation.you can specify a JSONPath expression in the annotations section of the manifest for the resource. using fluxcd.io/ignore:The Jason PathExpression: → .spec.actions[].exportParametersthe 2nd option is to ignore syncing the policy resources by using FluxCD --ignore flag.
Already have an account? Login
Enter your E-mail address. We'll send you an e-mail with instructions to reset your password.
Sorry, we're still checking this file's contents to make sure it's safe to download. Please try again in a few minutes.
Sorry, our virus scanner detected that this file isn't safe to download.
