fresh install of k10 in kubernetes 1.25 / no dashboard available and problem with fluentbit

Question

Hello togetheryesterday we installed on a dev kubernetes vanilla system v.1.25 the kasten k10 backup application with helm and like described in the doumentation.i installed kasten k10 with the following helm options: helm install k10 kasten/k10 --namespace=kasten-io --namespace=kasten-io --set auth.basicAuth.enabled=true \--set auth.basicAuth.htpasswd='admin:$apr1$qrAVXu.v$Q8YVc50vtiS8KPmiyrkld0' --set externalGateway.externalGateway.fqdn.type=external-dns --set externalGateway.fqdn.name=kasten.some.domain.net --set ingress.create=true --set ingress.class=haproxy --set rbac.create=false --set serviceAccount.create=false --set serviceAccount.name=k10-sa if y try to access to the external url (dashboard) i get the error:{ "message": "the gateway.kasten-io.1 AuthService is misconfigured; see the logs for more information", "request_id": "a42e6308-5eff-461d-a34e-899b99691781", "status_code": 500}the logs of the auth container:root@botprodkubm01 ~ # kubectl logs auth-svc-d45bd5f4c-5pr2l{"File":"kasten.io/k10/kio/tracing/tracing.go","Function":"kasten.io/k10/kio/tracing.StartProfileBuffers","Line":109,"cluster_name":"db7940b1-219c-42d4-9de9-c0d395839ad0","hostname":"auth-svc-d45bd5f4c-5pr2l","level":"info","msg":"no profile buffers configured","time":"2023-11-07T11:58:27.124Z","version":"6.0.12"}{"File":"kasten.io/k10/kio/kube/openshift.go","Function":"kasten.io/k10/kio/kube.IsOSRouteGroupAvailable","Line":70,"available":false,"cacheValid":true,"cluster_name":"db7940b1-219c-42d4-9de9-c0d395839ad0","hostname":"auth-svc-d45bd5f4c-5pr2l","level":"info","msg":"Result cached","time":"2023-11-07T11:58:27.145Z","version":"6.0.12"}{"File":"kasten.io/k10/kio/auth/auth.go","Function":"kasten.io/k10/kio/auth.Start","Line":116,"auth_type":"basic","cluster_name":"db7940b1-219c-42d4-9de9-c0d395839ad0","hostname":"auth-svc-d45bd5f4c-5pr2l","level":"info","msg":"Authentication type enabled","time":"2023-11-07T11:58:27.150Z","version":"6.0.12"}{"File":"kasten.io/k10/rest/srv/authserver/kio_auth_handler.go","Function":"kasten.io/k10/rest/srv/authserver.authHTTPHandler","Line":325,"cluster_name":"db7940b1-219c-42d4-9de9-c0d395839ad0","hostname":"auth-svc-d45bd5f4c-5pr2l","level":"info","msg":"Setup auth handler","time":"2023-11-07T11:58:27.151Z","version":"6.0.12"}{"File":"kasten.io/k10/kio/utils/swagger_utils.go","Function":"kasten.io/k10/kio/utils.ServerLogger","Line":11,"cluster_name":"db7940b1-219c-42d4-9de9-c0d395839ad0","hostname":"auth-svc-d45bd5f4c-5pr2l","level":"info","msg":"Serving auth at http://[::]:8000","time":"2023-11-07T11:58:27.152Z","version":"6.0.12"}Log message dropped (buffer): {"File":"kasten.io/k10/kio/tracing/tracing.go","Function":"kasten.io/k10/kio/tracing.StartProfileBuffers","Level":"info","Line":109,"Message":"no profile buffers configured","Time":"2023-11-07T11:58:27.124964068Z","cluster_name":"db7940b1-219c-42d4-9de9-c0d395839ad0","hostname":"auth-svc-d45bd5f4c-5pr2l","version":"6.0.12"}{"File":"kasten.io/k10/kio/kube/openshift.go","Function":"kasten.io/k10/kio/kube.IsOSRouteGroupAvailable","Level":"info","Line":70,"Message":"Result cached","Time":"2023-11-07T11:58:27.145911736Z","available":false,"cacheValid":true,"cluster_name":"db7940b1-219c-42d4-9de9-c0d395839ad0","hostname":"auth-svc-d45bd5f4c-5pr2l","version":"6.0.12"}{"File":"kasten.io/k10/kio/auth/auth.go","Function":"kasten.io/k10/kio/auth.Start","Level":"info","Line":116,"Message":"Authentication type enabled","Time":"2023-11-07T11:58:27.150168203Z","auth_type":"basic","cluster_name":"db7940b1-219c-42d4-9de9-c0d395839ad0","hostname":"auth-svc-d45bd5f4c-5pr2l","version":"6.0.12"}{"File":"kasten.io/k10/rest/srv/authserver/kio_auth_handler.go","Function":"kasten.io/k10/rest/srv/authserver.authHTTPHandler","Level":"info","Line":325,"Message":"Setup auth handler","Time":"2023-11-07T11:58:27.151801703Z","cluster_name":"db7940b1-219c-42d4-9de9-c0d395839ad0","hostname":"auth-svc-d45bd5f4c-5pr2l","version":"6.0.12"}{"File":"kasten.io/k10/kio/utils/swagger_utils.go","Function":"kasten.io/k10/kio/utils.ServerLogger","Level":"info","Line":11,"Message":"Serving auth at http://[::]:8000","Time":"2023-11-07T11:58:27.152199321Z","cluster_name":"db7940b1-219c-42d4-9de9-c0d395839ad0","hostname":"auth-svc-d45bd5f4c-5pr2l","version":"6.0.12"} Error: {"message":"Fluentbit connection error","function":"kasten.io/k10/kio/log/hooks/fluentbit.(*Hook).handle","linenumber":97,"file":"kasten.io/k10/kio/log/hooks/fluentbit/fluentbit.go:97","cause":{"message":"Fluentbit connection problem","function":"kasten.io/k10/kio/log/hooks/fluentbit.(*Hook).dial","linenumber":88,"file":"kasten.io/k10/kio/log/hooks/fluentbit/fluentbit.go:88","cause":{"message":"dial tcp 10.85.92.26:24224: i/o timeout"}}}the error in the end of the log is the logging-svc container/service:"=>{"message"=>"Eula not accepted", "function"=>"kasten.io/k10/kio/scheduler.checkLicense", "linenumber"=>186, "file"=>"kasten.io/k10/kio/scheduler/scheduler.go:186"}, "hostname"=>"controllermanager-svc-598c74797-5wqh2", "version"=>"6.0.12"}][0] tcp.0: [1699361591.207178165, {"File"=>"kasten.io/k10/kio/scheduler/scheduler.go", "Function"=>"kasten.io/k10/kio/scheduler.(*schedulerRunner).Run", "Level"=>"error", "Line"=>92, "Message"=>"License check failed. Skipping scheduler run.", "Time"=>"2023-11-07T12:53:11.210184004Z", "cluster_name"=>"db7940b1-219c-42d4-9de9-c0d395839ad0", "error"=>{"message"=>"Eula not accepted", "function"=>"kasten.io/k10/kio/scheduler.checkLicense", "linenumber"=>186, "file"=>"kasten.io/k10/kio/scheduler/scheduler.go:186"}, "hostname"=>"controllermanager-svc-598c74797-5wqh2", "version"=>"6.0.12"}][0] tcp.0: [1699361606.234792549, {"File"=>"kasten.io/k10/kio/scheduler/scheduler.go", "Function"=>"kasten.io/k10/kio/scheduler.(*schedulerRunner).Run", "Level"=>"error", "Line"=>92, "Message"=>"License check failed. Skipping scheduler run.", "Time"=>"2023-11-07T12:53:26.208866492Z", "cluster_name"=>"db7940b1-219c-42d4-9de9-c0d395839ad0", "error"=>{"message"=>"Eula not accepted", "function"=>"kasten.io/k10/kio/scheduler.checkLicense", "linenumber"=>186, "file"=>"kasten.io/k10/kio/scheduler/scheduler.go:186"}, "hostname"=>"controllermanager-svc-598c74797-5wqh2", "version"=>"6.0.12"}]Checking the log size...reading config file /mnt/conf/logrotate.confReading state from file: //mnt/k10state/kasten-io//logrotate.statusAllocating hash table for state file, size 64 entriesCreating new stateHandling 1 logsrotating pattern: /mnt/k10state/k10.log 1073741824 bytes (6 rotations)empty log files are rotated, old logs are removedconsidering log /mnt/k10state/k10.log Now: 2023-11-07 12:53 Last rotated at 2023-11-07 12:00 log does not need rotating (log size is below the 'size' thresholdthe k10.primer.sh pre shows goodroot@botprodkubm01: curl https://docs.kasten.io/tools/k10_primer.sh | bash % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0100 10660 100 10660 0 0 14683 0 --:--:-- --:--:-- --:--:-- 14662Namespace option not provided, using default namespace[1;34mChecking for tools[0m[0;32m --> Found kubectl[0m[0;32m --> Found helm[0m[0;32m --> Found jq[0m[0;32m --> Found cat[0m[0;32m --> Found base64[0m[0;32m --> Found tr[0m[1;34mChecking if the Kasten Helm repo is present[0m[0;32m --> The Kasten Helm repo was found[0m[1;34mChecking for required Helm version (>= v3.9.0)[0m[0;32m --> No Tiller needed with Helm v3.12.3[0m[1;34mK10Primer image[0m[0;32m --> Using Image (gcr.io/kasten-images/k10tools:6.0.12) to run test[0m[1;34mChecking access to the Kubernetes context kubernetes-admin@kubernetes[0m[0;32m --> Able to access the default Kubernetes namespace[0m[1;34mK10 Kanister tools image[0m[0;32m --> Using Kanister tools image (gcr.io/kasten-images/k10tools:6.0.12) to run test[0m[1;34mRunning K10Primer Job in cluster with command[0m[0;32m ./k10tools primer [0mserviceaccount/k10-primer createdclusterrolebinding.rbac.authorization.k8s.io/k10-primer createdjob.batch/k10primer createdPod k10primer-knt55 is in Pending phasePod Ready!================================================================Kubernetes Version Check: Valid kubernetes version (v1.25.13) - [1;32mOK[0mRBAC Check: Kubernetes RBAC is enabled - [1;32mOK[0mAggregated Layer Check: The Kubernetes Aggregated Layer is enabled - [1;32mOK[0mCSI Capabilities Check: VolumeSnapshot CRD-based APIs are not installed - [1;31mError[0mI1107 12:43:12.786701 10 request.go:690] Waited for 1.048745081s due to client-side throttling, not priority and fairness, request: GET:https://10.85.0.1:443/apis/batch/v1Validating Provisioners: rook-ceph.rbd.csi.ceph.com: Is a CSI Provisioner - [1;32mOK[0m VolumeSnapshot CRD-based APIs are not installed - [1;31mError[0m Storage Classes: rook-ceph-block Valid Storage Class - [1;32mOK[0mrook-ceph.cephfs.csi.ceph.com: Is a CSI Provisioner - [1;32mOK[0m VolumeSnapshot CRD-based APIs are not installed - [1;31mError[0m Storage Classes: rook-cephfs Valid Storage Class - [1;32mOK[0mValidate Generic Volume Snapshot: Pod created successfully - [1;32mOK[0m GVS Backup command executed successfully - [1;32mOK[0m Pod deleted successfully - [1;32mOK[0m================================================================serviceaccount "k10-primer" deletedclusterrolebinding.rbac.authorization.k8s.io "k10-primer" deletedjob.batch "k10primer" deleted All Pods are running fineroot@botprodkubm01 /home/jf-admin # kubectl get podNAME READY STATUS RESTARTS AGEaggregatedapis-svc-b5cfc77f6-xprzb 1/1 Running 0 6h46mauth-svc-d45bd5f4c-5pr2l 1/1 Running 0 6h46mcatalog-svc-6fd9dbc46d-d2tm9 2/2 Running 0 6h46mcontrollermanager-svc-598c74797-5wqh2 1/1 Running 0 6h46mcrypto-svc-559cb78585-s5dpg 4/4 Running 0 6h46mdashboardbff-svc-6cdd989968-485kp 2/2 Running 0 6h46mexecutor-svc-74bb49cbc-b4f25 2/2 Running 0 6h46mexecutor-svc-74bb49cbc-pvfvl 2/2 Running 0 6h46mexecutor-svc-74bb49cbc-qlpfd 2/2 Running 0 6h46mfrontend-svc-6db75444fb-p6br4 1/1 Running 0 6h46mgateway-8486476798-nqhlj 1/1 Running 0 6h46mjobs-svc-5b78c55d98-5br9r 1/1 Running 0 6h46mk10-grafana-5fd9cbddbb-s7nnt 1/1 Running 0 6h46mkanister-svc-577846d9d6-t2vfc 1/1 Running 0 6h46mlogging-svc-9bb77dc76-lvz6x 1/1 Running 0 6h46mmetering-svc-67fcb9648d-sjdd8 1/1 Running 0 6h46mprometheus-server-84fff47bc-d65tl 2/2 Running 0 6h46mstate-svc-76d88bdf88-4l9rh 3/3 Running 0 6h46mthe services of all containersroot@botprodkubm01 /home/jf-admin # kubectl get svcNAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGEaggregatedapis-svc ClusterIP 10.85.145.42 443/TCP 6h47mauth-svc ClusterIP 10.85.40.207 8000/TCP 6h47mcatalog-svc ClusterIP 10.85.136.250 8000/TCP 6h47mcontrollermanager-svc ClusterIP 10.85.198.246 8000/TCP,18000/TCP 6h47mcrypto-svc ClusterIP 10.85.69.5 8000/TCP,8003/TCP,8001/TCP,8002/TCP 6h47mdashboardbff-svc ClusterIP 10.85.248.149 8000/TCP,8001/TCP 6h47mexecutor-svc ClusterIP 10.85.100.240 8000/TCP 6h47mfrontend-svc ClusterIP 10.85.54.99 8000/TCP 6h47mgateway ClusterIP 10.85.171.7 8000/TCP 6h47mgateway-admin ClusterIP 10.85.106.147 8877/TCP 6h47mgateway-loadbalancer LoadBalancer 10.85.35.248 10.46.0.204 8000:31584/TCP 42djobs-svc ClusterIP 10.85.79.97 8000/TCP 6h47mk10-grafana ClusterIP 10.85.233.207 80/TCP 6h47mkanister-svc ClusterIP 10.85.53.180 8000/TCP 6h47mlogging-svc ClusterIP 10.85.92.26 8000/TCP,24224/TCP,24225/TCP 6h47mmetering-svc ClusterIP 10.85.154.4 8000/TCP 6h47mprometheus-server ClusterIP 10.85.165.194 80/TCP 6h47mprometheus-server-exp ClusterIP 10.85.1.179 80/TCP 6h47mstate-svc ClusterIP 10.85.26.222 8000/TCP,8001/TCP,8002/TCP 6h47mThe IP of the auth and logging container are true root@botprodkubm01 /home/jf-admin # kubectl describe svc logging-svcName: logging-svcNamespace: kasten-ioLabels: app=k10 app.kubernetes.io/instance=k10 app.kubernetes.io/managed-by=Helm app.kubernetes.io/name=k10 component=logging helm.sh/chart=k10-6.0.12 heritage=Helm release=k10 run=logging-svcAnnotations: meta.helm.sh/release-name: k10 meta.helm.sh/release-namespace: kasten-ioSelector: run=logging-svcType: ClusterIPIP Family Policy: SingleStackIP Families: IPv4IP: 10.85.92.26IPs: 10.85.92.26Port: http 8000/TCPTargetPort: 8000/TCPEndpoints: 10.88.9.34:8000Port: logging 24224/TCPTargetPort: 24224/TCPEndpoints: 10.88.9.34:24224Port: logging-metrics 24225/TCPTargetPort: 24225/TCPEndpoints: 10.88.9.34:24225Session Affinity: NoneEvents: root@botprodkubm01 /home/jf-admin # kubectl describe svc auth-svcName: auth-svcNamespace: kasten-ioLabels: app=k10 app.kubernetes.io/instance=k10 app.kubernetes.io/managed-by=Helm app.kubernetes.io/name=k10 component=auth helm.sh/chart=k10-6.0.12 heritage=Helm release=k10 run=auth-svcAnnotations: getambassador.io/config: --- apiVersion: getambassador.io/v3alpha1 kind: Mapping name: auth-mapping prefix: /k10/auth-svc/ rewrite: / service: auth-svc.kasten-io:8000 timeout_ms: 30000 hostname: "*" ambassador_id: [ "kasten.io/k10" ] meta.helm.sh/release-name: k10 meta.helm.sh/release-namespace: kasten-ioSelector: run=auth-svcType: ClusterIPIP Family Policy: SingleStackIP Families: IPv4IP: 10.85.40.207IPs: 10.85.40.207Port: http 8000/TCPTargetPort: 8000/TCPEndpoints: 10.88.4.195:8000Session Affinity: NoneEvents: root@botprodkubm01 /home/jf-admin # so for me looks like the auth-svc can not connect to the logging-svc with fluentbit.After the helm install i would like to go to the kasten k10 dashboard and accept the license etc. like described in the documentationsomebody got the same problem or somebody can help me to investigate or to sove the problem?thx in advance for helpcheers,jose

safiya · Answer

@jaiganeshjk

fresh install of k10 in kubernetes 1.25 / no dashboard available and problem with fluentbit

1 comment

Comment

Comment

Sign up

Login to the community

Scanning file for viruses.

This file cannot be downloaded