Make sure you have a loadbalancer like Metallb running on your cluster. This will assign IP address to your Loadbalancers service or add fixed ip address to the gateway-ext service
regarding service account :
#first create service account:
kubectl create serviceaccount my-kast-sa -n kasten-io
# add k10-admin cluster role to serviceaccount
kubectl create clusterrolebinding k10-admin-sa --clusterrole=k10-admin --serviceaccount=kasten-io:my-kasten-sa
# add cluster-admin role (bit rough) to service account
kubectl create clusterrolebinding k10-admin-cluster --clusterrole=cluster-admin --serviceaccount=kasten-io:kasten-sa
# rerun the token procedure
sorry for the late answer
I’ve installed metallb, try a loadbalancer port with a nginx deployment with no problem but with kasten i see the gateways pods crashing and I have theses errors on one gateway pod:
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Scheduled 22m default-scheduler Successfully assigned kasten-io/gateway-68fbc949dc-zjxb9 to k8s-worker02
Normal Pulling 22m kubelet Pulling image "gcr.io/kasten-images/emissary:6.5.8"
Normal Pulled 22m kubelet Successfully pulled image "gcr.io/kasten-images/emissary:6.5.8" in 34.133s (34.688s including waiting)
Normal Created 22m kubelet Created container ambassador
Normal Started 22m kubelet Started container ambassador
Warning FailedCreatePodSandBox 19m kubelet Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "4b954eddb0b2d442577fe75ca74d6b035f6f441a9b22d683e5adbad585e13003": plugin type="flannel" failed (add): loadFlannelSubnetEnv failed: open /run/flannel/subnet.env: no such file or directory
Warning FailedCreatePodSandBox 19m kubelet Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "a3602588ba6dec7d3ff3c6391daaf1837c4de3a501e4294acb9374fe1cabce1c": plugin type="flannel" failed (add): loadFlannelSubnetEnv failed: open /run/flannel/subnet.env: no such file or directory
Warning FailedCreatePodSandBox 19m kubelet Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "0207cd0a7e68ddbc359c8d786a7c8a358fd7f044615f81ee14f62bd9faa9027f": plugin type="flannel" failed (add): loadFlannelSubnetEnv failed: open /run/flannel/subnet.env: no such file or directory
Normal SandboxChanged 18m (x4 over 19m) kubelet Pod sandbox changed, it will be killed and re-created.
Normal Pulled 18m kubelet Container image "gcr.io/kasten-images/emissary:6.5.8" already present on machine
Normal Created 18m kubelet Created container ambassador
Normal Started 18m kubelet Started container ambassador
Warning Unhealthy 17m (x3 over 17m) kubelet Liveness probe failed: HTTP probe failed with statuscode: 503
Normal Killing 17m kubelet Container ambassador failed liveness probe, will be restarted
Warning BackOff 9m12s (x29 over 15m) kubelet Back-off restarting failed container ambassador in pod gateway-68fbc949dc-zjxb9_kasten-io(e37a91e0-2268-40d4-9377-a8552a02d72a)
Warning Unhealthy 3m56s (x40 over 18m) kubelet Readiness probe failed: HTTP probe failed with statuscode: 503
but now my loadbalancer port in kasten-io namespace is OK :
k get -n kasten-io svc gateway-ext -o wide
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE SELECTOR
gateway-ext LoadBalancer 10.101.224.229 192.168.1.241 80:31856/TCP 8m29s service=gateway
and all other kasten pods are ok:
kubectl get pods --namespace kasten-io
NAME READY STATUS RESTARTS AGE
aggregatedapis-svc-89d76759c-65gkw 1/1 Running 0 13m
auth-svc-6dcf6685cc-84w9f 1/1 Running 0 13m
catalog-svc-85478b567b-jdnkd 2/2 Running 0 12m
controllermanager-svc-99f87f7f6-9n7fv 1/1 Running 0 13m
crypto-svc-6bfbfc4c54-b9b8t 4/4 Running 0 13m
dashboardbff-svc-7b54b8f649-k4f9l 2/2 Running 0 13m
executor-svc-5978df4789-4vmtv 1/1 Running 0 13m
executor-svc-5978df4789-cnbgm 1/1 Running 0 13m
executor-svc-5978df4789-ctj6f 1/1 Running 0 13m
frontend-svc-64c49d55bb-cnrdj 1/1 Running 0 13m
gateway-5775954c65-zmkgb 0/1 CrashLoopBackOff 7 (4m32s ago) 13m
gateway-68fbc949dc-zjxb9 0/1 CrashLoopBackOff 10 (4m31s ago) 29m
jobs-svc-f75878f54-x6h4w 1/1 Running 0 13m
k10-grafana-7687dfc495-rq7ch 1/1 Running 0 13m
kanister-svc-598d7688b5-glgd7 1/1 Running 0 13m
logging-svc-7668586c9d-ddrst 1/1 Running 0 13m
metering-svc-7cb6fc9bc7-pjkm7 1/1 Running 0 13m
prometheus-server-85d896ddb7-pj94d 2/2 Running 0 29m
state-svc-6475f95b88-qrvsv 3/3 Running 0 13m
What I don’t understand is why my flannel network plugin seems in error when all other deployments with a loadbalancer port working correctly
OK, forget about my last reply
I have removed my kasten deployment
reinstall it with helm and now have only one gateway pod
here’s the log from the gateway pod:
https://pastebin.com/M3bSLq2w
And here’s what I see in the events:
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Scheduled 23m default-scheduler Successfully assigned kasten-io/gateway-5775954c65-6sz8f to k8s-worker01
Normal Created 22m (x2 over 23m) kubelet Created container ambassador
Normal Started 22m (x2 over 23m) kubelet Started container ambassador
Normal Pulled 21m (x3 over 23m) kubelet Container image "gcr.io/kasten-images/emissary:6.5.8" already present on machine
Warning Unhealthy 21m (x6 over 22m) kubelet Liveness probe failed: HTTP probe failed with statuscode: 503
Normal Killing 21m (x2 over 22m) kubelet Container ambassador failed liveness probe, will be restarted
Warning Unhealthy 18m (x31 over 22m) kubelet Readiness probe failed: HTTP probe failed with statuscode: 503
Warning FailedCreatePodSandBox 13m kubelet Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "73515b7a039ead65b02ecb4c62ef61ed4aae456fcb553ddfe82413d5a0e7931a": plugin type="flannel" failed (add): loadFlannelSubnetEnv failed: open /run/flannel/subnet.env: no such file or directory
Warning FailedCreatePodSandBox 13m kubelet Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "3af05071bf11cc72e65de617c8ce8606a2c9b73537e748a205929e7748fef397": plugin type="flannel" failed (add): loadFlannelSubnetEnv failed: open /run/flannel/subnet.env: no such file or directory
Warning FailedCreatePodSandBox 13m kubelet Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "33714bc5a5c2ce3c005402fc0896e7477d6f66c76680b01da5ff31e1b42c32d8": plugin type="flannel" failed (add): loadFlannelSubnetEnv failed: open /run/flannel/subnet.env: no such file or directory
Normal SandboxChanged 13m (x4 over 13m) kubelet Pod sandbox changed, it will be killed and re-created.
Normal Pulled 12m (x2 over 13m) kubelet Container image "gcr.io/kasten-images/emissary:6.5.8" already present on machine
Normal Created 12m (x2 over 13m) kubelet Created container ambassador
Normal Started 12m (x2 over 13m) kubelet Started container ambassador
Normal Killing 12m kubelet Container ambassador failed liveness probe, will be restarted
Warning Unhealthy 12m (x4 over 12m) kubelet Liveness probe failed: HTTP probe failed with statuscode: 503
Warning Unhealthy 12m (x7 over 13m) kubelet Readiness probe failed: HTTP probe failed with statuscode: 503
Warning BackOff 3m41s (x31 over 11m) kubelet Back-off restarting failed container ambassador in pod gateway-5775954c65-6sz8f_kasten-io(523dc826-915a-4551-88ba-00561a827676)
Hello @eztech
So, I am looking over this issue and I am noticing that it looks to be you are having problems in your environment with your CNI and the Gateway-svc pods.
failed to setup network for sandbox "33714bc5a5c2ce3c005402fc0896e7477d6f66c76680b01da5ff31e1b42c32d8": plugin type="flannel" failed (add): loadFlannelSubnetEnv failed: open /run/flannel/subnet.env: no such file or directory
I would recommend taking a look over flannel logs during the time the Gateway Pod attempts to start.
Thanks
Emmanuel
