Question

Possible export repository tampering detected


Userlevel 4

Hey! 

I have added an immutable S3 Bucket to be able to export my backups to. Everything works as expected, I can backup and restore, but I get perdiodically this error message and I don’t know why, because the data wasn’t modified by anyone (only kasten has access to it). 

 

Blobs containing immutable backup data were found to be in an unexpected state that may indicate tampering has occurred. The affected repository location is of type S3, at the endpoint https://minio.domain.tld, in the bucket "k10-minio" at the path "k10/b5c2fd3e-6d52-41b0-8675-84dc6f2acaab/migration/repo/52717ccf-8f42-4d31-bec4-c3122e5ccee5/". The profile being used to access this repository is called "k10-minio" in the "kasten-io" namespace. Issue description: one or more data blobs have unexpected version history. See logs for more details.

What can I do to prevent these messages and be sure that my data is reliable?

 

Thanks a lot!


14 comments

Userlevel 7
Badge +7

@Yongkang 😊

Userlevel 6
Badge +2

@SMT-Almato 
Thank you for posting this.

Would you be able to confirm which version of K10 are you using ?
I see that an issue with false positive alerts was fixed in 5.0.5 (https://docs.kasten.io/latest/releasenotes.html#relnotes-5-0-5)

If you are using older version of K10, We suggest you to upgrade to latest version 5.0.7 and observe if you are still seeing these false positive events from K10.

Userlevel 4

@jaiganeshjk 

Thanks a lot! I will definitely try an upgrade from 5.0.0 to 5.0.7 and report back! I searched in the release notes before, but that was in the time 5.0.2 was released!

Userlevel 4

Hey! 

I tried an update to 5.0.7. Sorry the late answer. 

The tampering seems to stay - how can I solve this?

Userlevel 6
Badge +2

@SMT-Almato Can you confirm if the alert was triggered recently(after the upgrade to 5.0.7) or this was an older event ?

Userlevel 4

Yes, I can confirm, that this is a new message. I deleted all messages after the upgrade.

 

 

Userlevel 4

Any news to this? Its really annoying! 

Same kind of notifications with version 5.0.10 and dedicated Minio with immutable backups.

Userlevel 6
Badge +2

@SMT-Almato @Sébastien Prud'homme 
We might need more logs to narrow down the issue. Both of the issue might be related or totally unrelated.

Would you be able to open a case using https://my.veeam.com/ with the debug logs to assist further ?

Userlevel 4

Is this possible without a license / with a free license?

Userlevel 6
Badge +2

Yes it is possible to create a support case selecting a trial product.
You will have to select `Kasten by Veeam K10 Trial` under products while creating a case.

Userlevel 4

Okay, I have opened a case now. Thank you. Hopefully there is a solution to this!

Userlevel 6
Badge +2

Thanks @SMT-Almato @Sébastien Prud'homme 
After looking at the logs, We found a potential improvement in the integrity check mechanism.

From one of the logs, we found double delete markers which were probably due the the retry mechanism of the snapshot retire actions.

We would be fixing this in the upcoming version and will keep you posted on the same.

Userlevel 6
Badge +2

@SMT-Almato @Sébastien Prud'homme ,

We have a fix available as a part of 5.5.1 release(https://docs.kasten.io/5.5.1/releasenotes.html#bug-fixes)

Please upgrade your K10 to latest version and let us know if it resolves your issue.

Comment