Skip to main content
Solved

Backup files not deteled in MinIO Object Store

M

Hello everyone!

I am using kasten k10 v4.5.8 (with airgapped installation) on kubernetes rke v1.21.8 and I am performing some logical backups for some db (Mongo/Postgre/MsSQL).
The snapshot are exported to a S3 Compatible MinIO Object Store (version RELEASE.2022-02-18T01-50-10Z) that is installed on barematel infrastructure.
I defined some policy to backup application three times at hours and as retain I have defined 'ONE HOURLY EXPORTED SNAPSHOT' to have one local and exported restore point.

In the blueprint I have the delete phase that is 

  delete:
    inputArtifactNames:
    # The kopia snapshot info created in backup phase can be used here
    # Use the `--kopia-snapshot` flag in kando to pass in `mongoBackup.KopiaSnapshot`
      - mongoBackup
    phases:
    - func: KubeTask
      name: deleteFromStore
      args:
        namespace: "{{ .Namespace.Name }}"
        image: ghcr.io/kanisterio/mongodb:0.72.0
        command:
          - bash
          - -o
          - errexit
          - -o
          - pipefail
          - -c
          - |
            backup_file_path='rs_backup.gz'
            kopia_snap='{{ .ArtifactsIn.mongoBackup.KopiaSnapshot }}'
            kando location delete --profile '{{ toJson .Profile }}' --path "${backup_file_path}" --kopia-snapshot "${kopia_snap}"


I left kasten doing its job and effectively although kasten executes snapshot/backup/delete actions three times in an hour I have always one local and exported restore point. 


This looks great but if I go on minio server into the defined bucket I can found the object of the previous (and deleted) restore point.
I am now wondering if this is normal or if I have skip some preliminary minio server configuration 

Many thanks for help!

Best answer by shuguet

The maintenance task will clean up files/blocks (and metadata associated) from the catalog, and then that will be reflected on the overall MinIO bucket yes. Although there is no direct relationship between the blobs/objects you see in the MinIO bucket and the actual files being backed up.

So you might see changes in some objects sizes, or new objects created, or objects deleted.

As it’s a catalog and not a list of files, the “important” part is the overall size of the bucket, not the actual behaviour of any individual object in the bucket.

Best regards,

Sylvain Huguet

Kasten by Veeam

View original
Did this topic help you find an answer to your question?

5 comments

shuguet
  • shuguet
  • Comes here often
  • 19 comments
  • April 7, 2022

Hello there @mario996 ,

Thank you for using K10 & Kanister!

Can you clarify what you mean by

I can found the object of the previous (and deleted) restore point.

Are you talking about the 'rs_backup.gz' files from previous runs, or the actual content of restore points, or both?

Based on what you wrote above, you should see backup repositories (K10’s catalog + data) exported. Deleted RestorePointContent are not immediately removed from the remote location on deletion though, instead we “mark them as deleted” but the actual removal is done as part of a maintenance/background task in a future run.

This is in place to give a small (but sometime useful) margin to recover data with the help of our support team (no commitment!) in case this was ever needed.

Hope that helps!

Best regards,

Sylvain Huguet

Kasten by Veeam

Sylvain Huguet
Chris.Childerhose
Geoff Burke
2 people like this
  • Chris.Childerhose
    Chris.Childerhose
  • Geoff Burke
    Geoff Burke
M
  • mario996
  • Author
  • Not a newbie anymore
  • 4 comments
  • April 7, 2022

Hi @shuguet, many thanks for replying!

When I say “I can found the object of the previous (and deleted) restore point.”  I refer to files like “kopia.repository”, “kopia.maintenance” and other compressed and encrypted ones that were pushed to minio bucket during the export phase of backup.
So this files are marked as “deleted” and then there will be a maintenance/background task that will delete them effectively?
If I understand correctly, how many times in one day this task is executed?  Is there some way to customize the schedule of this task?

Many thanks in advance!

Chris.Childerhose
Geoff Burke
2 people like this
  • Chris.Childerhose
    Chris.Childerhose
  • Geoff Burke
    Geoff Burke
shuguet
  • shuguet
  • Comes here often
  • 19 comments
  • April 7, 2022

Ha :)


Those files are expected to be there. K10 actually orchestrate the use of an open source tool called Kopia (https://kopia.io) to ”copy” data to out of your local cluster and into an external location (like your Minio bucket.

Kopia uses a concept of a catalog (a sort of database) to keep track of anything it has backed up, blocks, files, etc. and use those informations to do things like deduplication, compressions, differential backups, etc.

So when you are looking at the bucket, you are actually seeing files that Kopia can read/understand, not a 1:1 copy of the files in your application’s data volumes.

If we “simply” did a 1:1 copy, we would have to re-copy 100% of your data each time, from your local cluster storage, over the wire, and into the object repository.

Having Kopia in the loop with its handy catalog actually reduce that.

The maintenance task I mentioned takes care of “cleaning up” files/blocks that are not necessary from the catalog, without you having to do anything. You cannot customize this behaviour no how/when it is performed, nor should you worry about it. K10 & Kopia takes care of that for you.

Just know that taking a backup+export, then deleting the export from the GUI doesn’t meant you’ll see additional space freed up right away from the Minio bucket, as mentioned previously, there will be a delay (Up to a couple of days in certain scenarios) before Kopia performs the final cleanup & actual deletion of the blocks/files marked for deletion in its catalog.

Best regards,

Sylvain Huguet

Kasten by Veeam.

Sylvain Huguet
Chris.Childerhose
Geoff Burke
2 people like this
  • Chris.Childerhose
    Chris.Childerhose
  • Geoff Burke
    Geoff Burke
M
  • mario996
  • Author
  • Not a newbie anymore
  • 4 comments
  • April 7, 2022

Hi @shuguet,

Thanks a lot! 
So, in summary, the maintenance task will clean up files/block from the catalog and then from MinIO bucket? It is correct ? 

 

Chris.Childerhose
Geoff Burke
jaiganeshjk
3 people like this
  • Chris.Childerhose
    Chris.Childerhose
  • Geoff Burke
    Geoff Burke
  • jaiganeshjk
    jaiganeshjk
shuguet
  • shuguet
  • Comes here often
  • 19 comments
  • Answer
  • April 7, 2022

The maintenance task will clean up files/blocks (and metadata associated) from the catalog, and then that will be reflected on the overall MinIO bucket yes. Although there is no direct relationship between the blobs/objects you see in the MinIO bucket and the actual files being backed up.

So you might see changes in some objects sizes, or new objects created, or objects deleted.

As it’s a catalog and not a list of files, the “important” part is the overall size of the bucket, not the actual behaviour of any individual object in the bucket.

Best regards,

Sylvain Huguet

Kasten by Veeam

Sylvain Huguet
Chris.Childerhose
Geoff Burke
M
3 people like this
  • Chris.Childerhose
    Chris.Childerhose
  • Geoff Burke
    Geoff Burke
  • M
    mario996

Comment


Terms & Conditions