Minikube Prometheus Issue

Hi Folks,

I wrote a few posts about Kasten on Minikube and mentioned that there was an issue with the Prometheus pod not starting up correctly. Being pressed for time i simply checked the values in the Helm chart and saw that you could disable persistence (just add to the helm install command “--set prometheus.server.persistentVolume.enabled=false”)  and everything worked fine after that.

Nevertheless it bothered me that it was not working. Today I decided to dig a bit deeper and remembered that it had something to do with permissions.

In the container logs I found the prometheus container could not open a file in persitstent volume “/data/queries.active” I grabbed the yaml file

1k get po -n kasten-io prometheus-server-5f8795dd55-s8c4c -oyaml >prometheus.yaml

and saw right away that the Kasten Helm chart properly runs the pod as a non root user and remembered that I had seen issues with this before with other types of deployments

1securityContext:
2    fsGroup: 65534
3    runAsGroup: 65534
4    runAsNonRoot: true
5    runAsUser: 65534

The trick then is to run an init container that changes the permissions on this folder so that the main container can access it. I looked at the Helm chart values and did not see any setting like this (bitnami charts have a setting for this) so decided to try home grown and possibly ugly solutions. I ran the Kasten install and then immediately ran a fixer pod with the following declaration:

1apiVersion: v1
2kind: Pod
3metadata:
4  creationTimestamp: null
5  labels:
6    run: fix
7  name: fix
8  namespace: kasten-io
9spec:
10  containers:
11  - image: debian:stable
12    name: fix
13    command: ["chown", "-R", "65534:65534", "/data"]
14    command: ["/bin/chmod","-R","777","/data"]
15    volumeMounts:
16      - name: storage-volume
17        mountPath: /data
18        subPath: ""
19  volumes:
20  - name: storage-volume
21    persistentVolumeClaim:
22      claimName: prometheus-server

The result was that the prometheus-server container in the prometheus pod came up and everything is running smoothly.

You can then delete the fix pod since it has already fulfilled its task.  I have only seen this issue when installing Kasten on Minikube which is only for dev/demo purposes so this is not a major issue but was fun to fix nevertheless.