• EN

Replies posted by Aaron Oneal

29 Comments

A
Aaron OnealComes here often
asked in Kasten K10 Support

Removing restore point didn't remove snapshots

Velero, though I decided to go with its Kopia snapshotting instead of CSI / filesystem.

9
A
3 months ago
A
Aaron OnealComes here often
asked in Kasten K10 Support

Removing restore point didn't remove snapshots

I had to manually remove the restore points and ultimately move to a different backup solution.

9
A
3 months ago
A
Aaron OnealComes here often
asked in Kasten K10 Support

High apiserver CPU usage due to K10

Anyone else seeing these repeated policy checks?Kasten team, any thoughts?

4
Y
6 months ago
A
Aaron OnealComes here often
asked in Kasten K10 Support

High apiserver CPU usage due to K10

It appears that Kasten is continually polling the API server to see if new profiles are created.I don’t understand why: (1) this is done so frequently (2) polling was used instead of a watch.This is clogging up my API server queue to the point that other services are being denied and affecting the stability of the entire cluster because the request queue is full.

4
Y
6 months ago
V
voarshComes here often
asked in Kasten K10 Support

Ceph RBD - Kasten still creates leftover orphaned RBD images

If I remember correctly from when I ran into this, either K10 or the K8S RBD driver takes a copy-on-write clone first and then snapshots the clone. This makes it pretty difficult to track down because you have to find RBD images with a parent ID of the one you are trying to remove. That COW image will then be a root for one or more snapshots which you’ll have to flatten or purge.

3
V
10 months ago
A
Aaron OnealComes here often
asked in Kasten K10 Support

Removing restore point didn't remove snapshots

Provisioner is Ceph RBD and policy is Delete.

9
A
3 months ago
A
Aaron OnealComes here often
asked in Kasten K10 Support

Removing restore point didn't remove snapshots

I used `kubectl` to remove `restorepointcontents`. I saw the retire actions spawn and complete. However, the `volumesnapshots` remain and I’m having to delete them all manually.

9
A
3 months ago
A
Aaron OnealComes here often
asked in Kasten K10 Support

Remove PVC snapshot from all restore points

Thanks, I considered that but would like to retain the historical backups for the other PVCs in the namespace.

2
A
1 year ago
A
Aaron OnealComes here often
asked in Kasten K10 Support

10 hour timeout backing up 1TB

@Hagag -BTW, this also applies to local backups, not just network. I have a 10TB application getting backed up over USB and it fails at 10 hours as well. Please allow configuration of this limit.

6
A
1 year ago
A
Aaron OnealComes here often
asked in Kasten K10 Support

10 hour timeout backing up 1TB

The case I opened was closed without an opportunity to respond so I will continue discussion here and open a new case if needed.10M link to do backup, even with a VM it would fail... It is better to review your architecture because backup is not the only issue you will have, if you want to resume backup where it stopped or failed, you will rely on a different snapshot , that`s a real consistency issue. It’s hard to parse what support is saying.First of all, my understanding is that Kopia repos just store blocks based on a hash. So from one snapshot to the next, it doesn’t really matter if the preceding snapshot didn’t finish uploading. While true I can’t use a failed snap1 to restore, the blocks are still in the repo from the failed snap1 so snap2 doesn’t have to resend them. Eventually all the data will make it to the repo and snaps will complete.Which leads to the fact that we are talking about 1TB over a 10M link taking a while is only true for the initial backup. Once that repo is

6
A
1 year ago
A
Aaron OnealComes here often
asked in Kasten K10 Support

Policy run fails even though all actions succeed

Attached to support case. What I noted there is creating a new policy with a new name works. Creating a policy with the old name does not. It appears there must be some old state in the catalog related to the original policy.

19
A
1 year ago
A
Aaron OnealComes here often
asked in Kasten K10 Support

Bug: Cluster scoped resources and labels

While that happens to be how the policy is configured in a screenshot, there is nothing in the text that says filter by name is required. Since configuring otherwise breaks the functionality, it would be better to disable the snapshot toggle in the UI when labels are configured because it leads to this non-functional behavior. I am baffled this is considered as designed and not an obvious bug.

9
FRubens
1 year ago
A
Aaron OnealComes here often
asked in Kasten K10 Support

Policy run fails even though all actions succeed

Actually, it still fails when time for export too. Looks like I was able to get a single manual run to work but anything scheduled still fails.

19
A
1 year ago
A
Aaron OnealComes here often
asked in Kasten K10 Support

Policy run fails even though all actions succeed

@FRubens I am already using that version and it still fails after running snapshots if it is not time for export.E.g. hourly snapshot, daily export.

19
A
1 year ago
A
Aaron OnealComes here often
asked in Kasten K10 Support

Policy run fails even though all actions succeed

These were submitted. Backups still failing.

19
A
1 year ago
A
Aaron OnealComes here often
asked in Kasten K10 Support

Policy run fails even though all actions succeed

I still don’t know what database format `model-store.db` is so I can’t correct any records, but I managed to extract the manifest from the catalog with a hex dump. The error indicates that `ArtifactReferenceGroup` is not a valid entry in the `entries` collection. My backup cluster is still failing due to this. Please advise.{ "creationTime": "2022-05-09T10:57:05.341Z", "destructionTime": "2022-05-10T10:59:57.196Z", "id": "c374f458-cf86-11ec-a067-0ef546421892", "meta": { "manifest": { "action": "snapshot", "apiKeys": [ "/actions.kio.kasten.io/runactions/run-m9dp9v4pwg" ], "apiMeta": { "annotations": null, "labels": [ { "key": "k10.kasten.io/policyName", "value": "cloud-daily-backup" }, { "key": "k10.kasten.io/policyNamespace", "value": "kasten-io" } ] }, "endTime": "2022-05-09T11:04:44.306Z", "entries": [ { "artifactReferen

19
A
1 year ago
A
Aaron OnealComes here often
asked in Kasten K10 Support

Bug: Cluster scoped resources and labels

@Hagag, have you reproduced this? I’m unclear why your question above was marked as a solution. 

9
FRubens
1 year ago
A
Aaron OnealComes here often
asked in Kasten K10 Support

Policy run fails even though all actions succeed

It appears these manifests are kept in the `model-store.db` of either the `catalog-pv-claim` or `jobs-pv-claim`. What format is the database?

19
A
1 year ago
A
Aaron OnealComes here often
asked in Kasten K10 Support

Policy run fails even though all actions succeed

@Hagag, deleting and recreating the policy did not fix this.I cannot share the debug logs as there is information in them that would be a privacy breach. However, I am happy to share specifics from them or redact files if you can tell me what you’re looking for. `executor-svc*` appears to be the only file that is relevant. Clearly K10 is attempting to “retire a policy” and manifest "c374f458-cf86-11ec-a067-0ef546421892" has an unexpected entry.Too bad it doesn’t say what the unexpected entry is.How can I find and clean up this manifest and any related data manually? { "File": "kasten.io/k10/kio/exec/phases/phase/retire_policy.go", "Function": "kasten.io/k10/kio/exec/phases/phase.(*retirePolicyPhase).checkRetirePolicyRun", "JobID": "c378bd89-cf86-11ec-ac01-6e30f4d8aeb4", "Line": 316, "ManifestID": "c374f458-cf86-11ec-a067-0ef546421892", "QueuedJobID": "c378bd89-cf86-11ec-ac01-6e30f4d8aeb4", "SubjectRef": "kasten-io:cloud-daily-backup", "cluster_name": "053cf100-3f34-47f3-a4b5-0d

19
A
1 year ago
A
Aaron OnealComes here often
asked in Kasten K10 Support

Policy run fails even though all actions succeed

K10 support, @Hagag , @Satish-- What manifest is K10 complaining about and how can I fix this? My critical backups are currently failing.

19
A
1 year ago
A
Aaron OnealComes here often
asked in Kasten K10 Support

Bug: Cluster scoped resources and labels

@Hagag, and was it a label-based policy? This repros every time. v4.5.14.Create new policy Select Applications by Labels Add a label Check Snapshot Cluster-Scoped Resources Save policy (Create Policy button) Edit the just created policy and witness it unchecked snapshot resources and added an extra label. 

9
FRubens
1 year ago
A
Aaron OnealComes here often
asked in Kasten K10 Support

Policy run fails even though all actions succeed

Thanks. I am storing to S3, however the error indicates an unexpected manifest entry and not an error communicating with the object store or a full bucket.

19
A
1 year ago
A
Aaron OnealComes here often
asked in Kasten K10 Support

Bug: Cluster scoped resources and labels

@Hagag, Point 4 is due to the bug reported here: The other 3 bullets are easy to reproduce. As I mentioned, create a label-based policy and then check the Snapshot Cluster-Scoped Resources toggle. After you save and then edit the policy again you will see the results I mentioned.

9
FRubens
1 year ago
A
Aaron OnealComes here often
asked in Kasten K10 Support

Policy run fails even though all actions succeed

I didn’t see anything on the details page but I just discovered if I view the YAML there is an error listed. cause: '{"fields":[{"name":"entry","value":{"type":"ArtifactReferenceGroup"}}],"file":"kasten.io/k10/kio/exec/phases/phase/retire_policy.go:177","function":"kasten.io/k10/kio/exec/phases/phase.(*retirePolicyPhase).retireMultiActionPolicyEntries","linenumber":177,"message":"Unexpected manifest entry"}' message: Job failed to be executed 

19
A
1 year ago
A
Aaron OnealComes here often
asked in Kasten K10 Support

10 hour timeout backing up 1TB

K10 support—This is clearly a bug for Monitoring Actions to be hard coded to 10 hours when it’s possible (and desirable) to configure backups to run longer than that.Will this be addressed?Ideally Monitoring Actions should inherit the longest timeout configured.

6
A
1 year ago

Badge winners

Show all badges
Terms & Conditions