Upgraded K10 to 7.0.7.
The catalog service fails with the following error:
“Back-off restarting failed container schema-upgrade-check in pod catalog-svc-75ff7bb8cf-zpnhd_kasten-io(e1cd5a5c-32e1-4549-b4be-c0fd8ba68874)”
Any ideas on what could cause this?
Many thanks,
-Mark
Best answer by jaiganeshjk
There is a prerequisite for upgrade is to have 50% free space for catalog-pv-claim before the upgrade.
It seems that there is a schema upgrade that is needed for your K10 instance and it is failing because it doesn’t have free space to do the upgrade.
Everytime when a schema upgrade happens, We copy our database into a temporary new database, do the schema upgrade and then commit it. Then we will delete the old database, This is the reason why catalog needs to have 50% free space.
If your storageClass supports volume expansion, You can do a helm upgrade with the below command to increase the size of the catalog PVC(make sure that the catalog utilization is below 50 % while setting up the size)
helm get values k10 --output yaml --namespace=kasten-io > k10_val.yaml && \
helm upgrade k10 kasten/k10 --namespace=kasten-io -f k10_val.yaml \
--set global.persistence.catalog.size=<Size with units eg 30Gi>Once you increase the PVCs capacity, The catalog schma upgrade will go through and it should come up and running after the upgrade.
The schema-upgrade-check container log shows:
{"File":"kasten.io/k10/kio/modelstore/store.go","Function":"kasten.io/k10/kio/modelstore.NewStore","Line":215,"cluster_name":"ef31068b-5d9a-4942-8c88-803b69c098d2","hostname":"catalog-svc-75ff7bb8cf-2wtfw","level":"info","modelStoreDir":"//mnt/k10state/kasten-io/","modelVersion":"0.0.33","msg":"store version","time":"2024-08-31T22:21:38.653Z","version":"7.0.7"}
{"File":"kasten.io/k10/kio/modelstore/store.go","Function":"kasten.io/k10/kio/modelstore.IsForcedUpgrade","Line":120,"cluster_name":"ef31068b-5d9a-4942-8c88-803b69c098d2","hostname":"catalog-svc-75ff7bb8cf-2wtfw","level":"info","msg":"checking for forced model upgrade","name":"k10-force-model-store-upgrade","namespace":"kasten-io","time":"2024-08-31T22:21:38.654Z","version":"7.0.7"}
{"File":"kasten.io/k10/kio/modelstore/store.go","Function":"kasten.io/k10/kio/modelstore.removeDataStore","Line":702,"cluster_name":"ef31068b-5d9a-4942-8c88-803b69c098d2","description":"old upgrade","hostname":"catalog-svc-75ff7bb8cf-2wtfw","level":"info","msg":"removing datastore","path":"/mnt/k10state/kasten-io/catalog/model-store.db-upgrading","time":"2024-08-31T22:21:38.776Z","version":"7.0.7"}
{"File":"kasten.io/k10/kio/modelstore/store.go","Function":"kasten.io/k10/kio/modelstore.NewStore","Line":215,"cluster_name":"ef31068b-5d9a-4942-8c88-803b69c098d2","hostname":"catalog-svc-75ff7bb8cf-2wtfw","level":"info","modelStoreDir":"//mnt/k10state/kasten-io/","modelVersion":"0.0.33","msg":"store version","time":"2024-08-31T22:21:38.777Z","version":"7.0.7"}
{"File":"kasten.io/k10/kio/oninit/oninit.go","Function":"kasten.io/k10/kio/oninit.Do","Line":27,"cluster_name":"ef31068b-5d9a-4942-8c88-803b69c098d2","error":{"message":"failed to open upgrade datastore","function":"kasten.io/k10/kio/modelstore.(*ModelStore).upgradeStore","linenumber":341,"file":"kasten.io/k10/kio/modelstore/store.go:341","cause":{"message":"Could not open Bolt database","function":"kasten.io/k10/kio/modelstore.(*ModelStore).openDataStore","linenumber":755,"file":"kasten.io/k10/kio/modelstore/store.go:755","cause":{"message":"Failed to open Bolt store","function":"kasten.io/k10/kio/store.OpenBoltDB","linenumber":88,"file":"kasten.io/k10/kio/store/bolt.go:88","cause":{"message":"write /mnt/k10state/kasten-io/catalog/model-store.db-upgrading: no space left on device"}}}},"hostname":"catalog-svc-75ff7bb8cf-2wtfw","level":"error","msg":"Init Container Failed","time":"2024-08-31T22:21:38.777Z","version":"7.0.7"}
Anyone know how I can give it more space?
+2There is a prerequisite for upgrade is to have 50% free space for catalog-pv-claim before the upgrade.
It seems that there is a schema upgrade that is needed for your K10 instance and it is failing because it doesn’t have free space to do the upgrade.
Everytime when a schema upgrade happens, We copy our database into a temporary new database, do the schema upgrade and then commit it. Then we will delete the old database, This is the reason why catalog needs to have 50% free space.
If your storageClass supports volume expansion, You can do a helm upgrade with the below command to increase the size of the catalog PVC(make sure that the catalog utilization is below 50 % while setting up the size)
helm get values k10 --output yaml --namespace=kasten-io > k10_val.yaml && \
helm upgrade k10 kasten/k10 --namespace=kasten-io -f k10_val.yaml \
--set global.persistence.catalog.size=<Size with units eg 30Gi>Once you increase the PVCs capacity, The catalog schma upgrade will go through and it should come up and running after the upgrade.
Fantastic, thank you!!
Enter your E-mail address. We'll send you an e-mail with instructions to reset your password.