Kasten Fresh installation on K8S cluster issues

Hi All,

I installed the kasten by helm chart on my k8s cluster but im facing the below error:

$ kubectl get pods --namespace kasten-io

NAME                                 READY   STATUS             RESTARTS        AGE
aggregatedapis-svc-698fc8cc7-xzjkc   0/1     CrashLoopBackOff   5 (2m44s ago)   21m
auth-svc-7c6d5dfb99-m2wh8            0/1     CrashLoopBackOff   6 (114s ago)    21m
catalog-svc-cdd4b4875-277lj          2/2     Running            0               21m
config-svc-7c7587c989-brj2p          0/1     CrashLoopBackOff   5 (27s ago)     21m
crypto-svc-fc5fb685d-ggk6d           3/3     Running            0               21m
dashboardbff-svc-64b677dbdf-xtlfd    0/1     Running            3               21m
executor-svc-8645d48645-8c8xn        2/2     Running            0               21m
executor-svc-8645d48645-97xtm        2/2     Running            0               21m
executor-svc-8645d48645-pkcr7        2/2     Running            0               21m
frontend-svc-5bc88d576f-dd66f        1/1     Running            0               21m
gateway-74d6f8cc4f-jh9w9             0/1     CrashLoopBackOff   9 (30s ago)     21m
jobs-svc-6854494bc-7kdkz             1/1     Running            0               21m
k10-grafana-6dbc8d7b65-4wfnq         1/1     Running            0               21m
kanister-svc-6d54d97f75-78v5c        0/1     CrashLoopBackOff   6 (2m10s ago)   21m
logging-svc-bffd6dd6-qtq5n           1/1     Running            0               21m
metering-svc-75ff77488-4xfkq         0/1     CrashLoopBackOff   5 (2m6s ago)    21m
prometheus-server-5f8795dd55-f6xms   2/2     Running            0               21m
state-svc-8849656c6-dfr8j            2/2     Running            0               21m

----------------------------------------------------------------------------------------------------------------------

$ kubectl -n kasten-io logs kanister-svc-6d54d97f75-78v5c
{"File":"pkg/kancontroller/kancontroller.go","Function":"github.com/kanisterio/kanister/pkg/kancontroller.Execute","Line":45,"hostname":"kanister-svc-6d54d97f75-78v5c","level":"info","msg":"Getting kubernetes context","time":"2022-03-10T07:45:45.620685745Z","version":"4.5.10"}
Log message dropped (buffer): {"File":"pkg/kancontroller/kancontroller.go","Function":"github.com/kanisterio/kanister/pkg/kancontroller.Execute","Level":"info","Line":45,"Message":"Getting kubernetes context","Time":"2022-03-10T07:45:45.620685745Z","hostname":"kanister-svc-6d54d97f75-78v5c","version":"4.5.10"}
Error: Fluentbit connection error: Fluentbit connection problem: dial tcp 10.106.74.54:24224: i/o timeout
{"File":"controller/main.go","Function":"main.main","Line":8,"error":"Error getting server version: Get \"https://10.96.0.1:443/version\": dial tcp 10.96.0.1:443: i/o timeout","hostname":"kanister-svc-6d54d97f75-78v5c","level":"info","msg":"Failed to create CustomResources.","time":"2022-03-10T07:46:15.624518644Z","version":"4.5.10"}
{"File":"runtime/asm_amd64.s","Function":"runtime.goexit","Line":1581,"error":"http: Server closed","hostname":"kanister-svc-6d54d97f75-78v5c","level":"info","msg":"Failed to start health check server","time":"2022-03-10T07:46:15.625117855Z","version":"4.5.10"}

------------------------------------------------------------------------------------------------------------------

$ kubectl -n kasten-io logs gateway-74d6f8cc4f-jh9w9
2022-03-10 07:46:50 kubewatch "10 TMainThread] 1.14.1-dev.13087+g1be09b89d DEBUG: looking up ID for namespace kasten-io

------------------------------------------------------------------------------------------------------------------

$ kubectl -n kasten-io logs auth-svc-7c6d5dfb99-m2wh8
{"File":"kasten.io/k10/kio/tracing/tracing.go","Function":"kasten.io/k10/kio/tracing.StartProfileBuffers","Line":109,"hostname":"auth-svc-7c6d5dfb99-m2wh8","level":"info","msg":"no profile buffers configured","time":"20220310-07:46:01.565Z","version":"4.5.10"}
Log message dropped (buffer): {"File":"kasten.io/k10/kio/tracing/tracing.go","Function":"kasten.io/k10/kio/tracing.StartProfileBuffers","Level":"info","Line":109,"Message":"no profile buffers configured","Time":"2022-03-10T07:46:01.56570144Z","hostname":"auth-svc-7c6d5dfb99-m2wh8","version":"4.5.10"}
Error: {"message":"Fluentbit connection error","function":"kasten.io/k10/kio/log/hooks/fluentbit.(*Hook).handle","linenumber":98,"file":"kasten.io/k10/kio/log/hooks/fluentbit/fluentbit.go:98","cause":{"message":"Fluentbit connection problem","function":"kasten.io/k10/kio/log/hooks/fluentbit.(*Hook).dial","linenumber":89,"file":"kasten.io/k10/kio/log/hooks/fluentbit/fluentbit.go:89","cause":{"Op":"dial","Net":"tcp","Source":null,"Addr":{"IP":"10.106.74.54","Port":24224,"Zone":""},"Err":{}}}}
{"File":"kasten.io/k10/kio/assert/assert.go","Function":"kasten.io/k10/kio/assert.NoError","Line":16,"error":"Get \"https://10.96.0.1:443/api?timeout=32s\": dial tcp 10.96.0.1:443: i/o timeout","hostname":"auth-svc-7c6d5dfb99-m2wh8","level":"panic","msg":"Failed to initialize auth handler","time":"20220310-07:46:31.636Z","version":"4.5.10"}
panic: (*logrus.Entry) 0xc00023e310

goroutine 1 krunning]:
github.com/sirupsen/logrus.(*Entry).log(0xc001756350, 0x0, {0xc001754060, 0x21})
        /root/go/pkg/mod/github.com/sirupsen/logrus@v1.8.1/entry.go:259 +0x24f
github.com/sirupsen/logrus.(*Entry).Log(0xc00023e2a0, 0x0, {0xc001756350, 0x22, 0x10})
        /root/go/pkg/mod/github.com/sirupsen/logrus@v1.8.1/entry.go:293 +0x4f
github.com/sirupsen/logrus.(*Entry).Panic(0x27682f0, {0xc001756350, 0xc00060dd00, 0xc00060dd10})
        /root/go/pkg/mod/github.com/sirupsen/logrus@v1.8.1/entry.go:331 +0x29
kasten.io/k10/kio/log.loggerAdapter.Panic({{0x27682f0, 0xc00023e230}}, {0xc001756350, 0x1, 0x1})
        /codefresh/volume/k10/go/src/kasten.io/k10/kio/log/log.go:304 +0x4d
kasten.io/k10/kio/assert.NoError({0x26ca240, 0xc00146a120}, {0xc001756350, 0x1, 0x1})
        /codefresh/volume/k10/go/src/kasten.io/k10/kio/assert/assert.go:16 +0x57
kasten.io/k10/rest/srv/authserver.authHTTPHandler({0x26cbea0, 0xc001935640})
        /codefresh/volume/k10/go/src/kasten.io/k10/rest/srv/authserver/kio_auth_handler.go:269 +0xe6
kasten.io/k10/rest/srv/authserver.setupGlobalMiddleware({0x26cbea0, 0xc001935640})
        /codefresh/volume/k10/go/src/kasten.io/k10/rest/srv/authserver/configure_auth.go:75 +0x25
kasten.io/k10/rest/srv/authserver.configureAPI(0xc0013e8580)
        /codefresh/volume/k10/go/src/kasten.io/k10/rest/srv/authserver/configure_auth.go:51 +0x20a
kasten.io/k10/rest/srv/authserver.(*Server).ConfigureAPI(...)
        /codefresh/volume/k10/go/src/kasten.io/k10/rest/srv/authserver/server.go:58
main.main()
        /codefresh/volume/k10/go/src/kasten.io/k10/cmd/rest/auth-server/main.go:50 +0x2b6

$ kubectl -n kasten-io logs aggregatedapis-svc-698fc8cc7-xzjkc
I0310 07:49:04.744919 1 serving.go:342] Generated self-signed cert (/tmp/apiserver.local.config/certificates/apiserver.crt, /tmp/apiserver.local.config/certificates/apiserver.key)

BR,

Page 1 / 1

Hello @alaaeldin

can you run pre-flight checks by deploying the primer tool. and share the output !

curl https://docs.kasten.io/tools/k10_primer.sh | bash

and please send the output of kubectl describe on the CrashLoopBackOff pods

Thanks

Ahmed Hagag

Kubernetes environment:

v1.22.6
6 nodes (3 master/control panel)

network: Weave

metallb: load balancing

NFS set up as a location for Kasten.

Dear Ahmed,

Thanks a lot for your help.

Please find the attached required outputs.

BR,

Alaa Mohammed

Hello @alaaeldin

i can see from the pre flight check script results , there is error with your kubernetes aggregation api layer.

Aggregated Layer Check:  unable to retrieve the complete list of server APIs: actions.kio.kasten.io/v1alpha1: the server is currently unable to handle the request, apps.kio.kasten.io/v1alpha1: the server is currently unable to handle the request, metrics.k8s.io/v1beta1: an error on the server ("Internal Server Error: \"/apis/metrics.k8s.io/v1beta1\": the server could not find the requested resource") has prevented the request from succeeding, vault.kio.kasten.io/v1alpha1: the server is currently unable to handle the request  -  Error

i believe you need to configure the aggregation layer to allow the Kubernetes apiserver to be extended with additional APIs, which are not part of the core Kubernetes APIs.

{"File":"controller/main.go","Function":"main.main","Line":8,"error":"Error getting server version: Get \"https://10.96.0.1:443/version\": dial tcp 10.96.0.1:443: i/o timeout","hostname":"kanister-svc-6d54d97f75-78v5c","level":"info","msg":"Failed to create CustomResources.","time":"2022-03-

Thanks

Ahmed Hagag

looks like they changed that link try here https://kubernetes.io/docs/tasks/extend-kubernetes/configure-aggregation-layer/

cheers

Hi,

Please find the output of the above command after deleting the Kasten-io namespace and I found that the Kubernetes Aggregated Layer is enabled >> OK.

Is there any required options we need to configure/enable them in the aggregation layer on the Kubernetes during setup the Kasten by helm chart?

$ curl https://docs.kasten.io/tools/k10_primer.sh | bash
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 7045 100 7045 0 0 5360 0 0:00:01 0:00:01 --:--:-- 5365
Namespace option not provided, using default namespace
Checking for tools
--> Found kubectl
--> Found helm
Checking if the Kasten Helm repo is present
--> The Kasten Helm repo was found
Checking for required Helm version (>= v3.0.0)
--> No Tiller needed with Helm v3.2.4
K10Primer image
--> Using Image (gcr.io/kasten-images/k10tools:4.5.10) to run test
Checking access to the Kubernetes context kubernetes-admin@kubernetes
--> Able to access the default Kubernetes namespace
K10 Kanister tools image
--> Using Kanister tools image (ghcr.io/kanisterio/kanister-tools:0.74.0) to run test

Running K10Primer Job in cluster with command-
./k10tools primer
serviceaccount/k10-primer created
clusterrolebinding.rbac.authorization.k8s.io/k10-primer created
job.batch/k10primer created
Waiting for pod k10primer--1-5zxtd to be ready - ContainerCreating
Waiting for pod k10primer--1-5zxtd to be ready - ContainerCreating
Pod Ready!

Kubernetes Version Check:
Valid kubernetes version (v1.22.6) - OK

RBAC Check:
Kubernetes RBAC is enabled - OK

Aggregated Layer Check:
The Kubernetes Aggregated Layer is enabled - OK

CSI Capabilities Check:
Cluster isn't CSI capable

Validating Provisioners:
kubernetes.io/gce-pd:
Storage Classes:
administration-db-storage
Valid Storage Class - OK
business-process-db-storage
Valid Storage Class - OK
commercial-agencies-db-storage
Valid Storage Class - OK
lookups-db-storage
Valid Storage Class - OK
notification-db-storage
Valid Storage Class - OK
security-db-storage
Valid Storage Class - OK

cluster.local/mci-nfs-server-nfs-client-provisioner:
Storage Classes:
nfs-client
Supported via K10 Generic Volume Backup. See https://docs.kasten.io/latest/install/generic.html.

Validate Generic Volume Snapshot:
Pod Created successfully - OK
GVS Backup command executed successfully - OK
Pod deleted successfully - OK

serviceaccount "k10-primer" deleted
clusterrolebinding.rbac.authorization.k8s.io "k10-primer" deleted
job.batch "k10primer" deleted

BR,

Hello @Hagag

Is there any required options we need to configure/enable them in the aggregation layer on the Kubernetes during setup the Kasten by helm chart because the Kubernetes Aggregated Layer is enabled >> OK as in the output in the above comment.

BR,

I have not had to configure or enable the aggregation layer myself and have done quite a few installs on different distributions so I don’t think that would be necessary.

Just for folks that might be new here is a good article on the aggregation layer:

https://medium.com/@vanSadhu/kubernetes-api-aggregation-setup-nuts-bolts-733fef22a504

I don’t know if this will help in this particular case but they do have a section in there on setting up and how things work.

Comment

Sign up

Login to the community

Scanning file for viruses.

This file cannot be downloaded