Skip to main content
Solved

Helm chart Ingress problem with cert-manager

J

I’m experienced with Kubernetes but new to K10. I’m deploying it on my cluster via the K10 Helm chart. I’m running into a problem with the Ingress settings. I’m using cert-manager to provision a TLS cert via Let’s Encrypt, which I do regularly. Something is wrong with the Ingress resource generated by the Helm chart that means the http01 solver of cert-manager does not complete successfully.

This is the relevant part of my values for the Helm chart: 

global:
  ingress:
    create: true
    urlPath: "" #url path for k10 gateway

ingress:
  create: true
  tls:
    enabled: true
    secretName: ingress-tls
 class: public
  host: backup.mydomain.com
  urlPath: "" #url path for k10 gateway
  pathType: ImplementationSpecific
  annotations:
    cert-manager.io/cluster-issuer: letsencrypt-prod

Cert remains unprovisioned:

$ kubectl get cert
NAME          READY   SECRET        AGE
ingress-tls   False   ingress-tls   25m

The generated Ingress resource:

$ kubectl get ing k10-ingress -o yaml
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  annotations:
    cert-manager.io/cluster-issuer: letsencrypt-prod
    meta.helm.sh/release-name: k10
    meta.helm.sh/release-namespace: kasten-io
  creationTimestamp: "2023-09-05T19:08:51Z"
  generation: 9
  labels:
    app: k10
    app.kubernetes.io/instance: k10
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/name: k10
    helm.sh/chart: k10-6.0.6
    heritage: Helm
    release: k10
  name: k10-ingress
  namespace: kasten-io
  resourceVersion: "89575587"
  uid: 0af872d8-85e3-4634-b580-004b5da7c821
spec:
  ingressClassName: public
  rules:
  - host: backup.mydomain.com
    http:
      paths:
      - backend:
          service:
            name: gateway
            port:
              number: 8000
        path: /k10/
        pathType: ImplementationSpecific
  tls:
  - hosts:
    - backup.mydomain.com
    secretName: ingress-tls
status:
  loadBalancer:
    ingress:
    - ip: 127.0.0.1

I can’t personally see anything wrong with that Ingress record but all my other Ingresses on the cluster work and can successfully provision TLS certs

Best answer by jgazeley

I solved this issue. cert-manager was unable to provision a certificate using the http01 solver due to strict networkPolicies that were applied to the kasten-io namespace.

I realised that the k10 helm chart has a default values entry:

networkPolicy:
  create: true

I changed this to False and it works now. Usually networkPolicy should be set to a default of False to avoid problems like this, and the admin can opt in if they require it.

Ideally the networkPolicy should also include a rule to allow the http01 solver, if the admin wishes to use it.

View original
Geoff Burke
Madi.Cristil
April M
3 people like this
  • Geoff Burke
    Geoff Burke
  • Madi.Cristil
    Madi.Cristil
  • April M
    April M
Did this topic help you find an answer to your question?

5 comments

Madi.Cristil
Forum|alt.badge.img+8
  • Madi.Cristil
  • Community Manager
  • 617 comments
  • September 12, 2023

@jaiganeshjk 

Madi Cristil
jaiganeshjk
Forum|alt.badge.img+2
  • jaiganeshjk
  • Experienced User
  • 274 comments
  • September 13, 2023

@jgazeley Thanks for posting your question here.
I understand that you are facing issues in cert-manager certificate provisioning while creating ingress with K10.

Do you see any issue/events in the certificate CR resource describe output provisioned by the cert-manager for this ingress ?

kubectl describe certificates ingress-tls -n kasten-io

 

Jaiganesh
Madi.Cristil
J
2 people like this
  • Madi.Cristil
    Madi.Cristil
  • J
    jgazeley
J
  • jgazeley
  • Author
  • Not a newbie anymore
  • 2 comments
  • September 15, 2023

Hey @jaiganeshjk. There isn’t much in the output of that command:

 

$ kubectl describe certificates ingress-tls -n kasten-io
Name:         ingress-tls
Namespace:    kasten-io
Labels:       app=k10
              app.kubernetes.io/instance=k10
              app.kubernetes.io/managed-by=Helm
              app.kubernetes.io/name=k10
              helm.sh/chart=k10-6.0.7
              heritage=Helm
              release=k10
Annotations:  <none>
API Version:  cert-manager.io/v1
Kind:         Certificate
Metadata:
  Creation Timestamp:  2023-09-15T19:43:24Z
  Generation:          1
  Owner References:
    API Version:           networking.k8s.io/v1
    Block Owner Deletion:  true
    Controller:            true
    Kind:                  Ingress
    Name:                  k10-ingress
    UID:                   0af872d8-85e3-4634-b580-004b5da7c821
  Resource Version:        91780513
  UID:                     5f7be6b6-8305-4d9f-ba35-7c6c9e426c86
Spec:
  Dns Names:
    backup.gazeley.uk
  Issuer Ref:
    Group:      cert-manager.io
    Kind:       ClusterIssuer
    Name:       letsencrypt-prod
  Secret Name:  ingress-tls
  Usages:
    digital signature
    key encipherment
Status:
  Conditions:
    Last Transition Time:        2023-09-15T19:43:24Z
    Message:                     Issuing certificate as Secret does not exist
    Observed Generation:         1
    Reason:                      DoesNotExist
    Status:                      False
    Type:                        Ready
    Last Transition Time:        2023-09-15T19:43:24Z
    Message:                     Issuing certificate as Secret does not exist
    Observed Generation:         1
    Reason:                      DoesNotExist
    Status:                      True
    Type:                        Issuing
  Next Private Key Secret Name:  ingress-tls-zlzpd
Events:
  Type    Reason     Age    From                                       Message
  ----    ------     ----   ----                                       -------
  Normal  Issuing    2m54s  cert-manager-certificates-trigger          Issuing certificate as Secret does not exist
  Normal  Generated  2m54s  cert-manager-certificates-key-manager      Stored new private key in temporary Secret resource "ingress-tls-zlzpd"
  Normal  Requested  2m54s  cert-manager-certificates-request-manager  Created new CertificateRequest resource "ingress-tls-1"

 

J
  • jgazeley
  • Author
  • Not a newbie anymore
  • 2 comments
  • Answer
  • October 24, 2023

I solved this issue. cert-manager was unable to provision a certificate using the http01 solver due to strict networkPolicies that were applied to the kasten-io namespace.

I realised that the k10 helm chart has a default values entry:

networkPolicy:
  create: true

I changed this to False and it works now. Usually networkPolicy should be set to a default of False to avoid problems like this, and the admin can opt in if they require it.

Ideally the networkPolicy should also include a rule to allow the http01 solver, if the admin wishes to use it.

Geoff Burke
Forum|alt.badge.img+22
  • Geoff Burke
  • Veeam Legend, Veeam Vanguard
  • 1317 comments
  • October 17, 2024

Hey Folks,

Just wondering if some other option might become available. I setup a smallstep ca in podman and then cert-manager in K3S, have it leveraging ACME for auto certificates but had the same issue with Kasten. 

It works if I remove the network polices but otherwise the request for certificate remains pending. 

On a side note if you want to leverage your internal lab DNS server inside K3S you need to add a custom configmap for coredns:

apiVersion: v1
kind: ConfigMap
metadata:
  name: coredns-custom
  namespace: kube-system
data:
  lab1.local: |
    lab5.local {
      forward . 192.168.0.199
    }

 

VMCA2024, VMCE2023, VMCE2024-SP,CKA, LFCS, PCA,TARS
jaiganeshjk
1 person likes this
  • jaiganeshjk
    jaiganeshjk

Comment


Terms & Conditions