VeeamON 2024 - Use Code "COMMUNITY10" for 10% Off!
Hi @voarsh Thanks for posting this question. Did you have K10 DR policy enabled in your cluster ? If you had a backup of catalog, it can be easily recovered by doing a K10 DR restore(https://docs.kasten.io/latest/operating/dr.html#recovering-k10-from-a-disaster). Once restored, the retire of these snapshots will happen as per the set policy retention. If you don’t have this enabled, All you can do is manually delete the volumesnapshots/volumesnapshotcontents from your kubernetes cluster. Thanks, yup. I painfully deleted them all.
K10 doesn’t support Ceph FS at this time :/
@voarsh Thanks for posting the question. You will have to get in a toolbox pod (https://rook.io/docs/rook/v1.10/Troubleshooting/ceph-toolbox/) and run rbd cli commands. Please refer this documentation https://docs.ceph.com/en/mimic/rbd/rados-rbd-cmds/ for RBD commands. I don’t think you would need to remove the rbd images. You will have to run through a for loop for listing all the snapshots for each rbd image and remove those snapshots. Below is the list of commands that might come handy for you. #list all the images for a poolrbd ls {poolname}#list all the snapshots for an imagerbd snap ls {pool-name}/{image-name}#remove a snapshot with the image and pool name as inputrbd snap rm {pool-name}/{image-name}@{snap-name}#To delete all snapshots for an image with rbd, specify the snap purge option and the image name.rbd snap purge {pool-name}/{image-name} Thanks for your reply. I put together this example:for x in $(rbd list --pool replicapool); dorbd snap purge replicapool/$xdoneI
I had a quick question around purging images.See, I’ve backed up, restored, and backed up, restored, creating images from snapshot after snapshot over the usage - there hasn’t been any image flattening, because that would be up to if Kasten K10 does flattening. I’m wondering if this will mean I still have old images from restored volumes (that came from a different Ceph Pool, for example)
Argh. My reply was over the char limit….An example of a stuck retire job:kubectl get --raw /apis/actions.kio.kasten.io/v1alpha1/retireactions/retire-r6qdh8jdps/details{"kind":"RetireAction","apiVersion":"actions.kio.kasten.io/v1alpha1","metadata":{"name":"retire-r6qdh8jdps","uid":"3a637be2-b801-11ed-874e-36f9cb9e1653","resourceVersion":"687634","creationTimestamp":"2023-03-01T07:18:13Z","labels":{"k10.kasten.io/policyName":"chatwoot-backup","k10.kasten.io/policyNamespace":"kasten-io"}},"status":{"state":"Running","startTime":"2023-03-01T07:18:13Z","endTime":null,"restorePoint":{"name":""},"result":{"name":""},"actionDetails":{"phases":[{"endTime":null,"name":"Retiring RestorePoint","startTime":"2023-03-01T07:18:13Z","state":"waiting","updatedTime":"2023-03-01T11:25:13Z"}]},"progress":14},"spec":{"subject":{"apiVersion":"internals.kio.kasten.io/v1alpha1","kind":"Manifest","name":"59c2ae22-b7f6-11ed-874e-36f9cb9e1653"},"scheduledTime":"2023-03-01T07:00:00Z","retireIndex":614}}I have hund
and what is the deletion policy specified in the volumesnapshot class(that the volumesnapshot uses)?ceph-block (delete), and k10-clone-ceph-block (retain)… though, looking at new snaps in VolumeSnapshots, the class seems to be using ceph-block, can’t speak for the 200+ that I had deleted earlier, assume it was the same. As it currently stands, I have 27 snaps in Ceph trash that I can’t delete… I would need to manually dig into Ceph to see what the parent image is, is it safe to delete all child snaps… because Kasten has definitely messed up I think. kubectl get retireactions.actions.kio.kasten.io -o custom-columns=NAME:.metadata.name,"CREATED AT":.metadata.creationTimestamp,STATUS:.status.stateAnd for sure this shows some failed, not much reasons on why, the output is way too large to post here, and doesn’t offer much detail
Would be great to have some work looked into this
So, I finally ran:for x in $(rbd list --pool replicapool); do rbd snap purge replicapool/$x done And upon checking images in the Ceph Dashboard I still have like 800 images, when I only have something like 80 PVC’s, it’s not deleting all the images? Could this be because the volumes have snaps and aren’t flattened? Keep in mind, for all my Kasten K10 backups, I am keeping 0 snapshots to be kept in Ceph.
@voarsh Thank you for posting your question and being a patron by using K10 for a long time. Do you see K10 databases crashing due to storage I/O issues ? Which databases do you see such issues with ? Is it Catalog/jobs/metering ? We don’t use SQLlite in any of our K10 services except for upstream grafana that we deploy along with K10. K10 uses a golang based key-value store in all of its stateful services. Oh my bad, I assumed it was SQLlite because I had an IO/unclean shutdown a few days ago and a lot of my SQLlite based PVCs (I have a few), including K10 that were corrupted (it’s funny, because they’re Ceph backed, so there can’t really be corruption, but SQLite images need repairing when it’s not a clean shutdown/image close) - not had this problem with real database engines though… I have had to repair Redis AppendAOF files (unclean close on the file), but there’s a tool for that so easy to fix….... it’s usually the Catalog PVC/database (that has all the backup/K10) (e
If I remember correctly from when I ran into this, either K10 or the K8S RBD driver takes a copy-on-write clone first and then snapshots the clone. This makes it pretty difficult to track down because you have to find RBD images with a parent ID of the one you are trying to remove. That COW image will then be a root for one or more snapshots which you’ll have to flatten or purge. I guess I don’t know if there’s a bug in the K8s snapshot driver or K10.It keeps happening after a while when I have unattended hourly snapshot backups. I don’t want my cluster filled with orphaned images that take up unnecessary space. My previous Ceph cluster ended up having several TB’s of images (several hundred images that couldn’t be purged) I couldn’t remove through conventional means. Have you got any tips on how I can proceed to get to the parent image and delete the parent and clone? Bearing in mind that I can’t even query the image (as per my first post here)
Thank you for your interest. We are tracking this internally. However, we don’t have any timelines yet. @pascalzero You are right. As I mentioned earlier, We reuse the accessMode of the PVC from the original PVC that is being exported. We would update you once we have the ability in the product. Was hoping after 4 months I’d hear something else about this issue. Just to give you an idea of how this affects me, I cannot backup CephFS volumes with K10, because I have hundreds and hundreds of GB’s that must be copied every time K10 makes a backup with a snapshot, not only is it crazy IO expensive, but it times out and the jobs fail because of x3 retires (that I can’t increase).For the most part RSYNC works (for my other workloads that use CephFS), but I am deploying Bitbucket and it doesn’t reserve file permissions, so I must find another way to backup (if the file permissions aren’t retained it breaks my backup), so I can’t use RSYNC for backing up this application, and I can’t use
Glad that you’re at least looking into the issue. I will investigate your workaround for now. There was another question I had about CephFS clones as-is, it creates a linked clone within the CephFS filesystem, when it is deleted either by retention policy or manually, does it actually delete the snapshot within CephFS? Trying to clear up old CephFS snapshots for volumes within the Ceph Admin UI is atrociously painful and slow by hand, and the Ceph tooling around managing CephFS snapshots is sorely lacking - that’s another reason I don’t like playing with CephFS snapshots…….Ceph RBD snapshots are a little more transparent about this, and know that Kasten K10 definitely deletes snapshots (although I’ve mentioned numerous times about orphaned images, etc that don’t get deleted and hang around).
Hi all, Just to keep you all informed that much awaited support for shallow read-only volume snapshots for cephFS during export operations is available in k10 from version 6.5.2 Here's the documentation for the same. https://docs.kasten.io/latest/install/storage.html#snapshots-as-shallow-read-only-volumes-cephfs-only Finally - I saw this just today by chance - hadn’t updated K10 since August 2023. The documentation isn’t really clear enough for me on setting up CephFS shallow volume clones.I am using external snapshotter v4 - I am looking to upgrade to v6 shortly, not sure if that’s a prerequisite…. Here’s my VolumeSnapshotClass that K10 created: apiVersion: snapshot.storage.k8s.io/v1deletionPolicy: Retaindriver: rook-ceph.cephfs.csi.ceph.comkind: VolumeSnapshotClassmetadata: annotations: meta.helm.sh/release-name: rook-ceph-cluster meta.helm.sh/release-namespace: rook-ceph labels: kanister-cloned-from: ceph-filesystem name: k10-clone-ceph-filesystemparameters: clu
Basically, You don’t change the existing storageClass that you use(as restores from local volumesnapshot will fail when you do that). Instead, create a new storageClass with backingSnapshot parameter set to true and use the override in the policy. Below note is important as well as you will need to preserve SELinux options while using CephFS shallow volume copy for export. This annotation should be added to the original storageClass. Additionally, in the case of SELinux usage, it may be necessary to preserve SELinuxOptions of the original Pod into the Kanister Pod during the Export phase. $ kubectl annotate storageclass regular-cephfs-csi-storage-class \ k10.kasten.io/sc-preserve-selinux-options="true" Thanks for the quick reply.I’ve annotated the original storageclass:kubectl annotate storageclass ceph-filesystem k10.kasten.io/sc-preserve-selinux-options="true"Cloned it, with new params backingSnapshot: "true"Edited a backup policy to include: exportData:
Already have an account? Login
Enter your username or e-mail address. We'll send you an e-mail with instructions to reset your password.
Sorry, we're still checking this file's contents to make sure it's safe to download. Please try again in a few minutes.
Sorry, our virus scanner detected that this file isn't safe to download.
