Stuck in Terminating state!

Hi Everyone,

I don’t remember if this has come up here before but every once in a while when deleting namespaces your deletion just hangs in a terminating state:

Yes I am removing Kasten but only because this is a test lab for other purposes. Never delete Kasten in production if you want data protection 😉

So what is up?  I had installed via helm so this would normally not happen but I think I was tinkering and there may have been so objects left over. We can check that by running this command:

kubectl api-resources --verbs=list --namespaced -o name | xargs -n 1 kubectl get -n kasten-io

No objects are present which is good since we don’t want to junk lying around in the cluster.

The problem when this happens is that a finalizer is present for whatever reason so you need to manually remove it. We can get the namespace manifest and check it out:

kubectl get ns kasten-io -o json >kasten.json

Let’s view the file:

We need to remove this in order for the deletion to continue.

Edit the file and remove the “kubernetes” line

The file should now look like this:

Now we need to replace this in the cluster:

kubectl replace --raw "/api/v1/namespaces/kasten-io/finalize" -f kasten.json

You will see a lot of output from that command:

Now when we look for the kasten-io namespace we see that it is gone:

You may be asking just what are finalizers? In short they are there to tell system to wait while everything gets cleaned up in the namespace before deleting it. You can find a deeper dive on finalizers over on the Kubernetes official documentation page:

https://kubernetes.io/docs/concepts/overview/working-with-objects/finalizers/