Minikube Prometheus Issue

Hi Folks,

I wrote a few posts about Kasten on Minikube and mentioned that there was an issue with the Prometheus pod not starting up correctly. Being pressed for time i simply checked the values in the Helm chart and saw that you could disable persistence (just add to the helm install command “--set prometheus.server.persistentVolume.enabled=false”)  and everything worked fine after that.

Nevertheless it bothered me that it was not working. Today I decided to dig a bit deeper and remembered that it had something to do with permissions.

In the container logs I found the prometheus container could not open a file in persitstent volume “/data/queries.active” I grabbed the yaml file

k get po -n kasten-io prometheus-server-5f8795dd55-s8c4c -oyaml >prometheus.yaml

and saw right away that the Kasten Helm chart properly runs the pod as a non root user and remembered that I had seen issues with this before with other types of deployments

securityContext:
    fsGroup: 65534
    runAsGroup: 65534
    runAsNonRoot: true
    runAsUser: 65534

The trick then is to run an init container that changes the permissions on this folder so that the main container can access it. I looked at the Helm chart values and did not see any setting like this (bitnami charts have a setting for this) so decided to try home grown and possibly ugly solutions. I ran the Kasten install and then immediately ran a fixer pod with the following declaration:

apiVersion: v1
kind: Pod
metadata:
  creationTimestamp: null
  labels:
    run: fix
  name: fix
  namespace: kasten-io
spec:
  containers:
  - image: debian:stable
    name: fix
    command: ["chown", "-R", "65534:65534", "/data"]
    command: ["/bin/chmod","-R","777","/data"]
    volumeMounts:
      - name: storage-volume
        mountPath: /data
        subPath: ""
  volumes:
  - name: storage-volume
    persistentVolumeClaim:
      claimName: prometheus-server

The result was that the prometheus-server container in the prometheus pod came up and everything is running smoothly.

You can then delete the fix pod since it has already fulfilled its task.  I have only seen this issue when installing Kasten on Minikube which is only for dev/demo purposes so this is not a major issue but was fun to fix nevertheless.