Skip to main content

Upgraded K10 to 7.0.7.

The catalog service fails with the following error:

“Back-off restarting failed container schema-upgrade-check in pod catalog-svc-75ff7bb8cf-zpnhd_kasten-io(e1cd5a5c-32e1-4549-b4be-c0fd8ba68874)”

Any ideas on what could cause this?

Many thanks,

 

-Mark

The schema-upgrade-check container log shows:

{"File":"kasten.io/k10/kio/modelstore/store.go","Function":"kasten.io/k10/kio/modelstore.NewStore","Line":215,"cluster_name":"ef31068b-5d9a-4942-8c88-803b69c098d2","hostname":"catalog-svc-75ff7bb8cf-2wtfw","level":"info","modelStoreDir":"//mnt/k10state/kasten-io/","modelVersion":"0.0.33","msg":"store version","time":"2024-08-31T22:21:38.653Z","version":"7.0.7"}
{"File":"kasten.io/k10/kio/modelstore/store.go","Function":"kasten.io/k10/kio/modelstore.IsForcedUpgrade","Line":120,"cluster_name":"ef31068b-5d9a-4942-8c88-803b69c098d2","hostname":"catalog-svc-75ff7bb8cf-2wtfw","level":"info","msg":"checking for forced model upgrade","name":"k10-force-model-store-upgrade","namespace":"kasten-io","time":"2024-08-31T22:21:38.654Z","version":"7.0.7"}
{"File":"kasten.io/k10/kio/modelstore/store.go","Function":"kasten.io/k10/kio/modelstore.removeDataStore","Line":702,"cluster_name":"ef31068b-5d9a-4942-8c88-803b69c098d2","description":"old upgrade","hostname":"catalog-svc-75ff7bb8cf-2wtfw","level":"info","msg":"removing datastore","path":"/mnt/k10state/kasten-io/catalog/model-store.db-upgrading","time":"2024-08-31T22:21:38.776Z","version":"7.0.7"}
{"File":"kasten.io/k10/kio/modelstore/store.go","Function":"kasten.io/k10/kio/modelstore.NewStore","Line":215,"cluster_name":"ef31068b-5d9a-4942-8c88-803b69c098d2","hostname":"catalog-svc-75ff7bb8cf-2wtfw","level":"info","modelStoreDir":"//mnt/k10state/kasten-io/","modelVersion":"0.0.33","msg":"store version","time":"2024-08-31T22:21:38.777Z","version":"7.0.7"}
{"File":"kasten.io/k10/kio/oninit/oninit.go","Function":"kasten.io/k10/kio/oninit.Do","Line":27,"cluster_name":"ef31068b-5d9a-4942-8c88-803b69c098d2","error":{"message":"failed to open upgrade datastore","function":"kasten.io/k10/kio/modelstore.(*ModelStore).upgradeStore","linenumber":341,"file":"kasten.io/k10/kio/modelstore/store.go:341","cause":{"message":"Could not open Bolt database","function":"kasten.io/k10/kio/modelstore.(*ModelStore).openDataStore","linenumber":755,"file":"kasten.io/k10/kio/modelstore/store.go:755","cause":{"message":"Failed to open Bolt store","function":"kasten.io/k10/kio/store.OpenBoltDB","linenumber":88,"file":"kasten.io/k10/kio/store/bolt.go:88","cause":{"message":"write /mnt/k10state/kasten-io/catalog/model-store.db-upgrading: no space left on device"}}}},"hostname":"catalog-svc-75ff7bb8cf-2wtfw","level":"error","msg":"Init Container Failed","time":"2024-08-31T22:21:38.777Z","version":"7.0.7"}


Anyone know how I can give it more space?


@virtualhobbit Thank you for creating this topic. 

There is a prerequisite for upgrade is to have 50% free space for catalog-pv-claim before the upgrade.

It seems that there is a schema upgrade that is needed for your K10 instance and it is failing because it doesn’t have free space to do the upgrade.

Everytime when a schema upgrade happens, We copy our database into a temporary new database, do the schema upgrade and then commit it. Then we will delete the old database, This is the reason why catalog needs to have 50% free space. 

 

If your storageClass supports volume expansion, You can do a helm upgrade with the below command to increase the size of the catalog PVC(make sure that the catalog utilization is below 50 % while setting up the size)

helm get values k10 --output yaml --namespace=kasten-io > k10_val.yaml && \
helm upgrade k10 kasten/k10 --namespace=kasten-io -f k10_val.yaml \
--set global.persistence.catalog.size=<Size with units eg 30Gi>

Once you increase the PVCs capacity, The catalog schma upgrade will go through and it should come up and running after the upgrade.


Fantastic, thank you!!


Comment