VeeamON 2024 - Use Code "COMMUNITY10" for 10% Off!
@Refresh2260 Thank you for posting your question here.Openstack infrastructure profile is supported only for PVCs provisioned with Cinder In-Tree provisioner(kubernetes.io/cinder). Most of the times, we use infrastructure profiles only in cases where the provisoner doesn’t support taking snapshots the CSI way(Except for Vsphere).Since Cinder CSI driver already supports snapshotting through volumeSnapshot resources, I would suggest you to make use of that by creating volumesnapshotclass and annotating it with the above annotation.https://docs.kasten.io/latest/install/storage.html#csi-snapshot-configuration
@lidw Thanks for confirming that it worked.Would you be able to mark the comment as answer ?
@rust84 I am not exactly sure on this error, However from the message, it suggests that there is an artifact in the catalog that has the same key/value as the resource that you are trying to backup. There are few logs and outputs which we would like to look at and it could be better handled and tracked through a case(As this might involve looking through the saved artifacts from your namespace). Would you be able to open a case with us from my.veeam.com and select Veeam Kasten by K10 Trial under the product to raise a case with the debug logs?
@ntiwari38 In addition to what Satish had asked, Can you confirm if the useraccount that you are using to access K10 UI, is a K10-admin/Cluster-admin user ?Or is it just a namespace admin or a basic user with access to few namespace ?The RunOnce action is an admin action and can only be done by a user with cluster-admin privilege or at-least with the privilege to create runaction resource(which is a cluster-scoped resource at the moment)
Hi Neha,For immediate assistance, I would suggest you to open a case with us by logging into my.veeam.com and selecting the product Veeam Kasten by K10 Trial.You can upload the entire log bundle to the case.
Thank you for your interest. We are tracking this internally.However, we don’t have any timelines yet.@pascalzero You are right. As I mentioned earlier, We reuse the accessMode of the PVC from the original PVC that is being exported.We would update you once we have the ability in the product.
Hi @Garland7362 Thank you for posting the question.From the error message, it seems that the kopia binary is not available in the pod that is spinned-up for exports.It could be either create-repo or copy-vol-data pod. Do you see which image it is trying to use ?
I am just checking how k10 runs exec command in the newly created pods. We could be trying to run k10 kopia commands in first container but for some reason it could be running in linkerD proxy container rather than the kanister-tools container
I see that you have a support case with us as well. Let me confirm if that is the cause of the issue and see what could be done if this is the case.
@lidw Thanks for posting your question.From your message and the topic, I assume that you are trying to export your application by injecting Sidecar(Generic Volume snapshot) to an NFS target. Is that right ?Error message says that it is failing to create a repository in the target location. It could be a permission issue if it is an NFS target location profile.You could take a look at the executor pod logs to find out the reason for the failure of the repository create command.
@Garland7362I can confirm that the linkerd proxy container is where the commands are being run.We seem to be running exec commands in first container as we don't expect to have multiple containers with the pod that we dynamically spin-up.We will have to enhance this behaviour for your use case. I have filed an enhancement request to support this configuration. I went through the Linkerd issues to find out the reason for adding the proxy container as the first container.They mention that this change was purposefully added to avoid startup issues in this github issue .It seems that you can use annotation in the workload config.linkerd.io/proxy-await: "disabled" to disable this feature.Thinking about working around this, You could try using custom kanister annotations helm value to add this annotation to the pods spinned up by K10.This would make sure that the kanister-tools container be the first container and should resolve your issue for time-being.https://docs.kasten.io/latest/kanis
I think I know the answer. It seems to be a bug from our side(I will file a bug and get it fixed).The keyReference(from the configMap) that is added to the workloads is kanisterPodCustomAnnotationsWhere as the Key that was added in the configmap is KanisterPodCustomAnnotationsNotice the upper case K in the configMap and lowercase in the deployments.You might have to workaround it for the time-being by editing the configmap and changing the key to kanisterPodCustomAnnotations (with lowercase k)
@Garland7362 Glad that the workaround works for you now.I don’t have any timelines. It has to go through the PM review and it will be worked upon.All I can say that the bug will be fixed before the enhancement. IMO, your workaround to add annotations to the kanister pod will ensure your exports work until we have the enhancement in this workflow, unless something changes in the linkerd wrt to the annotation.
@ntiwari38 Init crashlooping of Catalog SVC mostly occurs when it cannot write data into its Persistent Volume.If you are using an NFS StorageClass which doesn’t support fsGroup fields in the workload securityContext, You might have to manually change the permissions & ownership for the directory to the user which catalog uses(UID/GID is 1000) You can look at the reason for crashlooping with the below commandkubectl logs deploy/catalog-svc --all-containers -n kasten-io
@zimbres Thank you for posting your question here.During restore of an Application, K10 always deletes the PVC of the application and recreates it with the snapshot.If the snapshot of your PVC is tied to the lifecycle of the volume, then you will see problems with restore as the snapshot would eventually be destroyed when the PVCs are deleted for restore.In this case, it would be better to have your application exported to an external Target and use it to do an in-place restore.Please let me know if you have more questions.
@sukhumsae Thank you for posting your question here.Kasten doesn’t support using a proxy configuration for location profiles or backup targets.Using proxy server for backups might actually not be a great idea with respect to performance.There might be interruptions/bottle-neck/network-latencies while writing backups to a target through a proxy server. Our suggestions would be to allow GCS access from your network bypassing the proxy.
@voarsh Do you mean, you still have a lot of stale RBD images that were created by K10 ? K10 integrates with the CSI driver to create/remove snapshots and images when the corresponding volumesnapshot and PVC resources are created.If there are RBD images left over in your cluster, I suppose there is an issue with the cleanup with RBD images when the PVCs are deleted. Also, Verify the deletion policy for your storageClass and snapshotClass and see if it is set to `Delete`.Setting it to retain, may not delete the images when K10 deletes the corresponding PVC/PV pairs.
@ntiwari38 With this Primer checks, K10 spins up a pod and validates GVS backup in your K8s cluster.The error in the output seems to be related to your cluster DNS where this is a warning when you have more than 6 search domains in your nodes /etc/resolv.conf file(Kubernetes adds 3 custom domains default.svc.cluster.local, svc.cluster.local, cluster.local).This means that you can add only 3 more custom domains in your resolv.conf file and if you have more than that few would be ignored.This doesn’t mean that K10 validation failed because of this DNS issue, but the pod that primer tries to spin up did not come to a running state properly.You can look at the events of the namespace to see why the pod did not come up.
@voarsh Thank you for posting your question and being a patron by using K10 for a long time. Do you see K10 databases crashing due to storage I/O issues ? Which databases do you see such issues with ? Is it Catalog/jobs/metering ?We don’t use SQLlite in any of our K10 services except for upstream grafana that we deploy along with K10. K10 uses a golang based key-value store in all of its stateful services.
We are still improving the maintenance process.However, If you need help in running a maintenance for your S3, Please open up a case with us through my.veeam.com
@rust84 Thank you for your support and patience thus far.We found the root cause of the issue in this case and we are currently working on enhancing the product to avoid such situation in future.We don’t expect/support importing the restorepoints in the same k10 installation where it was initially exported from.Below is the gist of what caused this issue. You seem to have run an import for the media namespace after the catalog is restored using K10 DR.Prior to 5.5.4 we didn't have the concept of imported repositories.We introduced a change after 5.5.4 for repositoryArtifact to track and find the difference between imported and exported repositories(imported repositories are set to read-only)with this particular timing, you happened to DR restore data to a catalog state where the export-side repo artifact didn't yet have API keys, prior to 5.5.4. Then, prior to running another export to that same repo using 5.5.4, you happened to run an import into the same k10 instance first. Due to th
You could try removing the older apiVersions for snapshot CRDs from your cluster just keeping the V1.K10 supports V1 snapshot CRDs as well.
@stephw Thank you for posting your question.You are right. Policy controller adds the recieveString and migrationToken to the policy spec.We started supporting creating policies using gitops approach sometime back.Even if the argocd syncs the resources and reverts the changes, We ensured that it doesn’t affect the policy runs in k10. Not sure where the issue might be, We will take a look at this.For the time-being, I would request you to ignore syncing the policy resources with argoCD until we have a solution for this.
@lejav At this point, K10 doesn’t support CSI snapshotting with Vsphere driver.As @Geoff Burke mentioned, You will have to use Infrastructure profile to setup backup/exports for Vsphere CSI driver based volumes.This internally uses vsphere govmomi libraries to connect to your vsphere infra and do snapshot and volume operations.
K10 is currently set-up to use the infrastructure profile for all the PVCs provisioned using Vsphere CSI driver and doesn’t take the CSI snapshot route.Currently Kasten doesn’t use the volumeSnapshotClass for the Vsphere PVCs.With Infrastructure profile in K10, Exports can be set to use blockmode and CBT using VADP API.I am not sure if this is possible with the CSI snapshots.
Already have an account? Login
Enter your E-mail address. We'll send you an e-mail with instructions to reset your password.
Sorry, we're still checking this file's contents to make sure it's safe to download. Please try again in a few minutes.
Sorry, our virus scanner detected that this file isn't safe to download.
