VeeamON 2024 - Use Code "COMMUNITY10" for 10% Off!
@anjul Good to talk to you to.Thanks for confirming that you used helm to do the upgrade of the values.If it fails with the same error, then for some reason this update in the config is not taken into account. The commands are executed from the executor to the kanister-sidecar container during the K10 DR backup. Can you search for any other error messages in the executor pod logs ?
@anjul Thanks for posting the question.can you confirm if you updated the value using helm value `kanister.backupTimeout` or just edited the value of KanisterBackupTimeout in k10-config configMap. ? If you have manually updated the configmap, You will have to restart the executor-svc and kanister-svc pods to make the changes active(As these values are set with environment variables).You can run the below commands to rollout restart the deploymentskubectl rollout restart deploy executor-svc -n kasten-iokubectl rollout restart deploy kanister-svc -n kasten-io
@giorgionanfa Thanks for the updates.From what I understand, We do scale down of the workloads even during the restore of the exported restorepoints(same as we do for local restorepoints). However, your tests with restoreprehook and restoreposthook eventually shows that it doesn’t. I will have to test this locally, try to reproduce it and file a bug.
@giorgionanfa Thanks for your explanation.Generally during the restore workflow, K10 deletes the PVC and recreates it based on the backups and restore the data into it from the external target(MinIO in your case) .From the log messages that you have shared, It says that k10 is failing to delete the PVC. Since you mentioned that release + its PVCs were deleted, did you validate if the PVC actually got removed from the cluster or was it in terminating state ?If it is stuck in terminating state, k10 would also fail to delete the PVC.However, if the namespace was clean, we wouldn’t attempt to delete the PVC as it wouldn’t be there. If possible can you retry the restore to the same namespace after deleting the entire namespace(and ensuring that the resources are removed) ?
@markbon Thank you for posting your question here.K10 requires profiles to be there in the namespace as the K10 is installed(By default it is kasten-io)If you have installed K10 in a namespace called kasten-io, Profile resources are expected to be in kasten-io namespace. However, Policies are allowed to be created in other namespaces. We call them application-scoped policies. This is to allow non-admin users to create policies for their namespaces, Do you have a specific use-case to create profile out of K10s namespace ?
The logical blueprint that you are using is an example created for mysql Standalone and not HA.I don’t know how the data is synced between the replicas. You will have to keep it in mind how the HA works before backing it up. Snapshot field computes the capacity of the local snapshots created by the CSI driver or the provider.However, your backup was blueprint based and it is pushed to your location target. So it comes under object storage and not Snapshots.
Recent versions probably has improvements in how we handle the phonehome reports
This is usually a problem with the storage backends which exposes client access to snapshot directory.K10 tries to run chown recursively and fails since `.snapshot` directory is read-onlyMay I know what NAS backend you are using in this case ?Is there a way you can disable client access snapshot directory for this volume ?
@JCandela Thanks for posting your question here. Would you be able to share the values file that you have ? (Please redact any sensitive info in it)
@meijianwei Thank you for posting this question.From your screenshot, I could see that the phase `restoreFromStore` is completed. However, the progress tracking from the job is actually failing.I am not sure about the exact reason for this. I see that you are in very old & probably unsupported version of K10 5.0.11.My suggestion is to upgrade k10 to latest or n-1 version and see if you can run the blueprint based backup again.If you still get the same issue, we can take a dig at debugging the issue by looking deeper into the logs.
Yes Madhuri. Port-forwarding procedure should be same for all the applications provided you use the correct ports.In the above scenario, You are forwarding requests to port 3000 to port 80. This will work if your application listens in that port. For example, K10 listens on port 8000. So, We use the below command.kubectl port-forward svc/gateway 8080:8000 -n kasten-io
K10 doesn’t directly interact with Ceph when you are using a CSI driver. K10 creates corresponding kubernetes resources(volumesnapsot/PVC) and CSI driver acts on it.We just create a VolumeSnapshot resource when a backup action runs. Similarly, We create a temp PVC with the source as volumesnapshot during the export(With which ceph internally clones the volume)We delete the temp PVC as soon as the export completes. We also delete the volumesnapshot resource when the restorepoints crosses its retention period.
Hi @jesus.rosales Thank you for posting the questions here.The mentioned kanister PVC are created during an export and are cleaned up when the export is completed.Since you mentioned CephFS, There might be a chance the K10 initiates a PVC creation from volumesnapshot source and for some reason(due to timeout maybe) the operation fails, and K10 cleans up the PVC. But the clone operation happening in the background started by Ceph CSI driver still runs and doesn’t get cleaned up as the CSI driver doesn’t handle the cancellation/removal of the clone operation when the PVC(Created from snapshot source) is removed. K10 has a pod wait timeout set to 15 mins by default when it waits for the temporary cloned volume to be ready for exports. However, I have seen cephFS(as default clone operation in CephFS is a full copy) taking more time to clone volumes.The above is my hunch and it could be entirely something else as well.Can you confirm what is the largest PVC(in terms of utilization) in your
Not all non-admin users have need to use all the profiles. In a multi-tenant environment, Profiles are segregated for each tenant.Since, the idea of this cluster role is to restrict the non-admin users to use only a particular profile or blueprint(specified explicitly by name), It is not added by default during K10 installation.
@banjo Thank you for posting your question here.From your screenshots, I see that you have a rolebinding for your user to the clusterrole `k10-admin` in bookinfo namespace.However, Profiles are part of `kasten-io` namespace and your user should have at-least get/list permissions for profile resources in `kasten-io` namespace. The documentation here will help you with a basic-config clusterrole and rolebinding creation for listing some/all profiles and blueprints for a non-cluster-admin user.
@banjo Thank you for posting your question here.At the moment, we don’t have any architecture diagrams for k10 available publicly.We will check internally if there is something we can do about it.
@banjo In this case, as the screenshot says, Backupaction doesn’t have a snapshot(the namespace you are backing up doesn’t seem to have a PVC)Only the app config is backed up in this case and this is stored in catalog-svc database. Below is the high level details of where the data of specific resource is stored as per the action.BackupAction - App config → K10s catalogBackupAction - PVC → Local provider snapshot if infrastructure profile is used/CSI based Volume snapshotExportAction → - App config → Location profile targetExportAction - PVC → Location profile target
Hi @banjo , Thank you for posting this topic.From the initial looks, The route fails to get created not because of SCC but it could be some admission controller that is stopping the custom hostname. I have not seen this behaviour while changing the route hostname with the K10 APII have earlier heard that the openshift starter plan doesn’t support using custom hostname for route. Not sure if that is the case in your setup.For updating the Openshift auth values in the K10 CR, there could be an easier way to patch the K10 resource with oc/kubectl command rather than manually editing the large k10 CR manifest.we are drafting an article and will share it here once it is published.
@banjo We use Vsphere govmomi library to create FCD snapshots in Vsphere directly. https://vdc-download.vmware.com/vmwb-repository/dcr-public/b83a47dc-134c-4295-a7a0-212b858e2a3c/9e342828-face-41ab-9f23-c539f72468c5/GUID-3FB348EE-46F0-46F6-A99E-BC1388604FC4.html
@Kittikom.Pa For port-forwarding through a jump host, you might have to follow the below steps.Running a port-forward to the service from the jump-box. Another terminal connecting ssh to the jump-host through with forward(SSH-tunnelling). ssh -L 8080:<ip-address>:8080 user@jumpbox-serverAlternatively and more easier & reliable is to use a ingress/loadbalancer service(with metalllb) to access k10.https://docs.kasten.io/latest/access/dashboard.html
K10 is currently set-up to use the infrastructure profile for all the PVCs provisioned using Vsphere CSI driver and doesn’t take the CSI snapshot route.Currently Kasten doesn’t use the volumeSnapshotClass for the Vsphere PVCs.With Infrastructure profile in K10, Exports can be set to use blockmode and CBT using VADP API.I am not sure if this is possible with the CSI snapshots.
@lejav At this point, K10 doesn’t support CSI snapshotting with Vsphere driver.As @Geoff Burke mentioned, You will have to use Infrastructure profile to setup backup/exports for Vsphere CSI driver based volumes.This internally uses vsphere govmomi libraries to connect to your vsphere infra and do snapshot and volume operations.
@stephw Thank you for posting your question.You are right. Policy controller adds the recieveString and migrationToken to the policy spec.We started supporting creating policies using gitops approach sometime back.Even if the argocd syncs the resources and reverts the changes, We ensured that it doesn’t affect the policy runs in k10. Not sure where the issue might be, We will take a look at this.For the time-being, I would request you to ignore syncing the policy resources with argoCD until we have a solution for this.
You could try removing the older apiVersions for snapshot CRDs from your cluster just keeping the V1.K10 supports V1 snapshot CRDs as well.
@rust84 Thank you for your support and patience thus far.We found the root cause of the issue in this case and we are currently working on enhancing the product to avoid such situation in future.We don’t expect/support importing the restorepoints in the same k10 installation where it was initially exported from.Below is the gist of what caused this issue. You seem to have run an import for the media namespace after the catalog is restored using K10 DR.Prior to 5.5.4 we didn't have the concept of imported repositories.We introduced a change after 5.5.4 for repositoryArtifact to track and find the difference between imported and exported repositories(imported repositories are set to read-only)with this particular timing, you happened to DR restore data to a catalog state where the export-side repo artifact didn't yet have API keys, prior to 5.5.4. Then, prior to running another export to that same repo using 5.5.4, you happened to run an import into the same k10 instance first. Due to th
Already have an account? Login
Enter your E-mail address. We'll send you an e-mail with instructions to reset your password.
Sorry, we're still checking this file's contents to make sure it's safe to download. Please try again in a few minutes.
Sorry, our virus scanner detected that this file isn't safe to download.
