Skip to main content
Solved

Federating Prometheus behind Ingress Controller

M

Hello everyone,

I am trying to federate the Kasten setup behind an Ingress Controller (see here for more info https://blog.kasten.io/posts/how-to-set-up-alerts-in-kasten-k10-to-immediately-catch-failed-backups) . It seems that everytime my Prometheus server tries to scrape data from the Kasten K10 Prometheus instance, located at the address http://<ingress>/k10/prometheus/federate it requires a login. I managed to do that by passing “authorization: credentials: inside the Prometheus job config.

Are there anywhere any best practices to do it? What kind of service account, with restricted permissions, shall I create in order to just have the exact permissions to get monitoring data?
Thank you!

Mattia

Best answer by michael-courcy

Hi Mattia

 

If you access through http://<ingress>/k10/prometheus/federate  then you’ll go through the auth process implemented by ambassador. 

 

Hence depending of your authentication setting on kasten you’ll have to adapt. I’m not sure for instance that your external prometheus can handle an OIDC authentification flow, but most likely it should be able to handle a token auth flow or a basic auth flow. What did you do in your case ?

 

Anyway you basically depends on the flow we choose to implement in our auth-svc service.

 

In term of best practice, I’ll choose to be independent of the kasten auth mechanism, so that you can switch without impacting your scraper or enforcing your level of security as you need. I would create a dedicated ingress for the prometheus-server service with tls client auth  (https://kubernetes.github.io/ingress-nginx/examples/auth/client-certs/) and ask the scraper to use the client certificate (https://prometheus.io/docs/prometheus/latest/configuration/configuration/#tls_config)  

 

I did not try this but that would be my approach.

 

regards

View original
Did this topic help you find an answer to your question?

2 comments

M
  • michael-courcy
  • Comes here often
  • 11 comments
  • Answer
  • April 19, 2022

Hi Mattia

 

If you access through http://<ingress>/k10/prometheus/federate  then you’ll go through the auth process implemented by ambassador. 

 

Hence depending of your authentication setting on kasten you’ll have to adapt. I’m not sure for instance that your external prometheus can handle an OIDC authentification flow, but most likely it should be able to handle a token auth flow or a basic auth flow. What did you do in your case ?

 

Anyway you basically depends on the flow we choose to implement in our auth-svc service.

 

In term of best practice, I’ll choose to be independent of the kasten auth mechanism, so that you can switch without impacting your scraper or enforcing your level of security as you need. I would create a dedicated ingress for the prometheus-server service with tls client auth  (https://kubernetes.github.io/ingress-nginx/examples/auth/client-certs/) and ask the scraper to use the client certificate (https://prometheus.io/docs/prometheus/latest/configuration/configuration/#tls_config)  

 

I did not try this but that would be my approach.

 

regards

jaiganeshjk
1 person likes this
  • jaiganeshjk
    jaiganeshjk
M

Hi Michael, thanks for your reply.  I really like your suggestion of not relying on Kasten auth mechanism. 
I thought about it too, and I agree with your point.

Problem is with Prometheus… that it’s not easy to put it behind a reverse proxy. You need to pass the external URL, and for Kasten, it’s /k10/prometheus… so it goes through the ingress of Kasten.

I believe I need to do a host-based ingress and not a path-based. What do you suggest? 


Just brainstorming in order to get the best setup (and to share it with the community): actually I am using the token auth with Kasten and federated Prometheus is able to handle it.

 

Mattia

Comment


Terms & Conditions