• EN
Question

Dashboard sometimes not showing Actions and throws code 504

M

Following Error is thrown:

 

I configured following beside from default helm configuration:

  • services.aggregatedapis.hostNetwork: true
  • service.aggregatedApiPort: 10251

Dashboard-Log with the upper configurations:

{"File":"kasten.io/k10/rest/srv/dashboardbffserver/kio_action_handler.go","Function":"kasten.io/k10/rest/srv/dashboardbffserver.(*actionHandler).listActions","Line":454,"cluster_name":"5d6ac71e-7d7d-4209-9399-8b85148be5cd","error":{"message":"Failed to list actions","function":"kasten.io/k10/kio/bff.(*ActionDashboard).ListActions","linenumber":154,"file":"kasten.io/k10/kio/bff/actions.go:154","cause":{"message":"Failed to list run actions","function":"kasten.io/k10/kio/bff.listRootAndSystemActions","linenumber":299,"file":"kasten.io/k10/kio/bff/actions.go:299","cause":{"message":"Failed to list admin run actions","function":"kasten.io/k10/kio/bff.fetchRunActionsElideForbidden","linenumber":1084,"file":"kasten.io/k10/kio/bff/actions.go:1084","cause":{"message":"Failed to list run actions","function":"kasten.io/k10/kio/bff.fetchRunActions","linenumber":1064,"file":"kasten.io/k10/kio/bff/actions.go:1064","cause":{"message":"Get \"https://10.3.0.1:443/apis/actions.kio.kasten.io/v1alpha1/runactions?limit=300\": context canceled"}}}}},"hostname":"dashboardbff-svc-8449cf86b7-m5cnt","level":"error","msg":"Failed to list actions","time":"2023-12-06T09:26:36.878Z","version":"6.5.0"}

I need those configurations, since when dont have it, I cannot see any applications and following Error is thrown:

Dashboard-Log without the upper configurations:

{"File":"kasten.io/k10/rest/srv/dashboardbffserver/kio_action_handler.go","Function":"kasten.io/k10/rest/srv/dashboardbffserver.(*actionHandler).listActions","Line":454,"cluster_name":"5d6ac71e-7d7d-4209-9399-8b85148be5cd","error":{"message":"Failed to list actions","function":"kasten.io/k10/kio/bff.(*ActionDashboard).ListActions","linenumber":154,"file":"kasten.io/k10/kio/bff/actions.go:154","cause":{"message":"Failed to list actions","function":"kasten.io/k10/kio/bff.listActions","linenumber":373,"file":"kasten.io/k10/kio/bff/actions.go:373","fields":[{"name":"actionType","value":"restoreaction"}],"cause":{"message":"Failed to list cluster restore actions","function":"kasten.io/k10/kio/bff.fetchClusterRestoreActionsElideForbidden","linenumber":831,"file":"kasten.io/k10/kio/bff/actions.go:831","cause":{"message":"Failed to list restore actions","function":"kasten.io/k10/kio/bff.fetchRestoreActions","linenumber":810,"file":"kasten.io/k10/kio/bff/actions.go:810","fields":[{"name":"namespace","value":""}],"cause":{"message":"the server is currently unable to handle the request (get restoreactions.actions.kio.kasten.io)"}}}}},"hostname":"dashboardbff-svc-8449cf86b7-m5cnt","level":"error","msg":"Failed to list actions","time":"2023-12-06T09:57:49.485Z","version":"6.5.0"}

 

Iam logged in and I do have unrestricted permissions:

 

Also the actions most of the time doesnt load correctly and if they load successfully, it takes loads of time. 

 

Also inside Dashboard-Logs is a lot of following OIDC Errors/Warning:

{"File":"kasten.io/k10/kio/auth/config.go","Function":"kasten.io/k10/kio/auth.loadOpt","Line":121,"cluster_name":"5d6ac71e-7d7d-4209-9399-8b85148be5cd","error":"open /var/run/secrets/kasten.io/k10-oidc-auth/usernamePrefix: no such file or directory","hostname":"dashboardbff-svc-8449cf86b7-m5cnt","level":"info","msg":"Failed to get OIDC usernamePrefix from config. Will use default value.","time":"2023-12-06T09:18:42.162Z","version":"6.5.0"}

 

I use K10 in Version 6.5.0, logged in via OIDC Keycloak.

13 comments

Madi.Cristil
Userlevel 7
Badge +7

@jaiganeshjk 

Madi Cristil
E
Userlevel 3
Badge +1

Hello @mario.t 

 

Would you be able to tell me what Service correlates with 10.3.0.1. Also would it be possible to gather the results of the command below. 

 

kubectl get apiservices | grep kasten

 

Thanks

Emmanuel

M

@EBrockman 

of course!

10.3.0.1, seems to be kubernetes api:

m@t-nb MINGW64 ~
$ kubectl get svc -A | grep "10.3.0.1"
default                kubernetes                                           ClusterIP      10.3.0.1       <none>                           443/TCP                                                     229d
kube-system            kube-dns                                             ClusterIP      10.3.0.10      <none>                           53/UDP,53/TCP,9153/TCP                                      229d

 

m@t-nb MINGW64 ~
$ kubectl get apiservices | grep kasten
v1alpha1.actions.kio.kasten.io         kasten-io/aggregatedapis-svc   True        180d
v1alpha1.apps.kio.kasten.io            kasten-io/aggregatedapis-svc   True        180d
v1alpha1.auth.kio.kasten.io            Local                          True        187d
v1alpha1.config.kio.kasten.io          Local                          True        187d
v1alpha1.dist.kio.kasten.io            Local                          True        187d
v1alpha1.reporting.kio.kasten.io       Local                          True        187d
v1alpha1.repositories.kio.kasten.io    kasten-io/aggregatedapis-svc   True        180d
v1alpha1.vault.kio.kasten.io           kasten-io/aggregatedapis-svc   True        180d

 

I just upgraded to v6.5.11, still the same issues.

E
Userlevel 3
Badge +1

Hello @mario.t 

 

Would it be possible to gather logs from Aggregated API?

 

Thanks

Emmanuel

M

@EBrockman of course: 

{"File":"kasten.io/k10/kio/aggapi/store/catalog.go","Function":"kasten.io/k10/kio/aggapi/store.CatalogStorage.List","Line":64,"cluster_name":"5d6ac71e-7d7d-4209-9399-8b85148be5cd","hostname":"b2-30-node-ba218f","key":"/actions.kio.kasten.io/backupactions","level":"debug","msg":"API List","time":"2024-04-16T06:43:35.593Z","version":"6.5.11"}
{"File":"kasten.io/k10/kio/aggapi/util/errors.go","Function":"kasten.io/k10/kio/aggapi/util.WrapError","Line":25,"cluster_name":"5d6ac71e-7d7d-4209-9399-8b85148be5cd","error":{"message":"Could not retrieve artifacts for prefix search","function":"kasten.io/k10/kio/rest/clients.FetchArtifactsForSearchPrefix","linenumber":65,"file":"kasten.io/k10/kio/rest/clients/catalogclient.go:65","fields":[{"name":"config","value":{}},{"name":"retries","value":1}],"cause":{"message":"Get \"http://10.3.107.173:8000/v0/artifacts/prefixSearch?key=api-list-prefix-key-exportactions.actions.kio.kasten.io\u0026limit=300\u0026reverseOrder=true\u0026value=%40\": context canceled"}},"hostname":"b2-30-node-ba218f","level":"error","msg":"Internal Error","time":"2024-04-16T06:43:35.636Z","version":"6.5.11"}
E0416 06:43:35.636335       1 writers.go:122] apiserver was unable to write a JSON response: http: Handler timeout
E0416 06:43:35.636358       1 status.go:71] apiserver received an error that is not an metav1.Status: &errors.errorString{s:"http: Handler timeout"}: http: Handler timeout
E0416 06:43:35.637514       1 writers.go:135] apiserver was unable to write a fallback JSON response: http: Handler timeout
I0416 06:43:35.637548       1 trace.go:236] Trace[426382339]: "List" accept:application/json, */*,audit-id:e1126eb2-a4db-4c88-9ecc-c77618f5ef3a,client:10.10.100.1,api-group:actions.kio.kasten.io,api-version:v1alpha1,name:,subresource:,namespace:,protocol:HTTP/2.0,resource:exportactions,scope:cluster,url:/apis/actions.kio.kasten.io/v1alpha1/exportactions,user-agent:dashboardbff-server/v0.0.0 (linux/amd64) kubernetes/$Format,verb:LIST (16-Apr-2024 06:43:05.769) (total time: 29868ms):
Trace[426382339]: [29.868000808s] [29.868000808s] END
E0416 06:43:35.637617       1 timeout.go:142] post-timeout activity - time-elapsed: 1.221545ms, GET "/apis/actions.kio.kasten.io/v1alpha1/exportactions" result: <nil>
{"File":"kasten.io/k10/kio/aggapi/apps/cr_cache_updater.go","Function":"kasten.io/k10/kio/aggapi/apps.(*crCacheUpdater).handleCREvent","Line":200,"cluster_name":"5d6ac71e-7d7d-4209-9399-8b85148be5cd","eventType":"MODIFIED","group":"argoproj.io","hostname":"b2-30-node-ba218f","level":"debug","msg":"Skipping CR modified event","resource":"applications","time":"2024-04-16T06:43:36.046Z","version":"v1alpha1"}
{"File":"kasten.io/k10/kio/aggapi/apps/cr_cache_updater.go","Function":"kasten.io/k10/kio/aggapi/apps.(*crCacheUpdater).handleCREvent","Line":200,"cluster_name":"5d6ac71e-7d7d-4209-9399-8b85148be5cd","eventType":"MODIFIED","group":"argoproj.io","hostname":"b2-30-node-ba218f","level":"debug","msg":"Skipping CR modified event","resource":"applications","time":"2024-04-16T06:43:36.345Z","version":"v1alpha1"}
{"File":"kasten.io/k10/kio/aggapi/store/catalog.go","Function":"kasten.io/k10/kio/aggapi/store.CatalogStorage.List","Line":64,"cluster_name":"5d6ac71e-7d7d-4209-9399-8b85148be5cd","hostname":"b2-30-node-ba218f","key":"/actions.kio.kasten.io/upgradeactions","level":"debug","msg":"API List","time":"2024-04-16T06:43:37.268Z","version":"6.5.11"}
{"File":"kasten.io/k10/kio/aggapi/util/errors.go","Function":"kasten.io/k10/kio/aggapi/util.WrapError","Line":25,"cluster_name":"5d6ac71e-7d7d-4209-9399-8b85148be5cd","error":{"message":"Could not retrieve artifacts for prefix search","function":"kasten.io/k10/kio/rest/clients.FetchArtifactsForSearchPrefix","linenumber":65,"file":"kasten.io/k10/kio/rest/clients/catalogclient.go:65","fields":[{"name":"config","value":{}},{"name":"retries","value":1}],"cause":{"message":"Get \"http://10.3.107.173:8000/v0/artifacts/prefixSearch?key=api-list-prefix-key-exportactions.actions.kio.kasten.io\u0026limit=300\u0026reverseOrder=true\u0026value=%40\": context canceled"}},"hostname":"b2-30-node-ba218f","level":"error","msg":"Internal Error","time":"2024-04-16T06:43:38.647Z","version":"6.5.11"}
E0416 06:43:38.648059       1 writers.go:122] apiserver was unable to write a JSON response: http: Handler timeout
E0416 06:43:38.648075       1 status.go:71] apiserver received an error that is not an metav1.Status: &errors.errorString{s:"http: Handler timeout"}: http: Handler timeout
E0416 06:43:38.649252       1 writers.go:135] apiserver was unable to write a fallback JSON response: http: Handler timeout
I0416 06:43:38.649289       1 trace.go:236] Trace[962619652]: "List" accept:application/json, */*,audit-id:f849cb28-5a3e-4aae-9d2e-921da779efa3,client:10.10.100.1,api-group:actions.kio.kasten.io,api-version:v1alpha1,name:,subresource:,namespace:,protocol:HTTP/2.0,resource:exportactions,scope:cluster,url:/apis/actions.kio.kasten.io/v1alpha1/exportactions,user-agent:dashboardbff-server/v0.0.0 (linux/amd64) kubernetes/$Format,verb:LIST (16-Apr-2024 06:43:08.771) (total time: 29877ms):
Trace[962619652]: [29.877315962s] [29.877315962s] END
E0416 06:43:38.649357       1 timeout.go:142] post-timeout activity - time-elapsed: 1.527736ms, GET "/apis/actions.kio.kasten.io/v1alpha1/exportactions" result: <nil>
{"File":"kasten.io/k10/kio/aggapi/store/catalog.go","Function":"kasten.io/k10/kio/aggapi/store.CatalogStorage.List","Line":64,"cluster_name":"5d6ac71e-7d7d-4209-9399-8b85148be5cd","hostname":"b2-30-node-ba218f","key":"/actions.kio.kasten.io/exportactions","level":"debug","msg":"API List","time":"2024-04-16T06:43:40.225Z","version":"6.5.11"}
{"File":"kasten.io/k10/kio/aggapi/store/catalog.go","Function":"kasten.io/k10/kio/aggapi/store.CatalogStorage.List","Line":64,"cluster_name":"5d6ac71e-7d7d-4209-9399-8b85148be5cd","hostname":"b2-30-node-ba218f","key":"/actions.kio.kasten.io/restoreactions","level":"debug","msg":"API List","time":"2024-04-16T06:43:43.234Z","version":"6.5.11"}
{"File":"kasten.io/k10/kio/aggapi/store/catalog.go","Function":"kasten.io/k10/kio/aggapi/store.CatalogStorage.List","Line":64,"cluster_name":"5d6ac71e-7d7d-4209-9399-8b85148be5cd","hostname":"b2-30-node-ba218f","key":"/actions.kio.kasten.io/migratefcdactions","level":"debug","msg":"API List","time":"2024-04-16T06:43:46.250Z","version":"6.5.11"}
{"File":"kasten.io/k10/kio/aggapi/apps/cr_cache_updater.go","Function":"kasten.io/k10/kio/aggapi/apps.(*crCacheUpdater).handleCREvent","Line":200,"cluster_name":"5d6ac71e-7d7d-4209-9399-8b85148be5cd","eventType":"MODIFIED","group":"argoproj.io","hostname":"b2-30-node-ba218f","level":"debug","msg":"Skipping CR modified event","resource":"applications","time":"2024-04-16T06:43:47.490Z","version":"v1alpha1"}
{"File":"kasten.io/k10/kio/aggapi/util/errors.go","Function":"kasten.io/k10/kio/aggapi/util.WrapError","Line":25,"cluster_name":"5d6ac71e-7d7d-4209-9399-8b85148be5cd","error":{"message":"Could not retrieve artifacts for prefix search","function":"kasten.io/k10/kio/rest/clients.FetchArtifactsForSearchPrefix","linenumber":65,"file":"kasten.io/k10/kio/rest/clients/catalogclient.go:65","fields":[{"name":"config","value":{}},{"name":"retries","value":1}],"cause":{"message":"Get \"http://10.3.107.173:8000/v0/artifacts/prefixSearch?key=api-list-prefix-key-migratefcdactions.actions.kio.kasten.io\u0026limit=300\u0026reverseOrder=true\u0026value=%40\": context canceled"}},"hostname":"b2-30-node-ba218f","level":"error","msg":"Internal Error","time":"2024-04-16T06:43:47.574Z","version":"6.5.11"}
E0416 06:43:47.575044       1 writers.go:122] apiserver was unable to write a JSON response: http: Handler timeout
E0416 06:43:47.575062       1 status.go:71] apiserver received an error that is not an metav1.Status: &errors.errorString{s:"http: Handler timeout"}: http: Handler timeout
E0416 06:43:47.576369       1 writers.go:135] apiserver was unable to write a fallback JSON response: http: Handler timeout
I0416 06:43:47.576481       1 trace.go:236] Trace[1123356241]: "List" accept:application/json, */*,audit-id:0ff0ecfd-3511-4122-a980-bef3cd4e5f1a,client:10.10.100.1,api-group:actions.kio.kasten.io,api-version:v1alpha1,name:,subresource:,namespace:,protocol:HTTP/2.0,resource:migratefcdactions,scope:cluster,url:/apis/actions.kio.kasten.io/v1alpha1/migratefcdactions,user-agent:dashboardbff-server/v0.0.0 (linux/amd64) kubernetes/$Format,verb:LIST (16-Apr-2024 06:43:17.685) (total time: 29891ms):
Trace[1123356241]: [29.891320516s] [29.891320516s] END
E0416 06:43:47.576602       1 timeout.go:142] post-timeout activity - time-elapsed: 1.70787ms, GET "/apis/actions.kio.kasten.io/v1alpha1/migratefcdactions" result: <nil>
{"File":"kasten.io/k10/kio/aggapi/store/catalog.go","Function":"kasten.io/k10/kio/aggapi/store.CatalogStorage.List","Line":64,"cluster_name":"5d6ac71e-7d7d-4209-9399-8b85148be5cd","hostname":"b2-30-node-ba218f","key":"/actions.kio.kasten.io/exportactions","level":"debug","msg":"API List","time":"2024-04-16T06:43:49.233Z","version":"6.5.11"}
{"File":"kasten.io/k10/kio/aggapi/apps/cr_cache_updater.go","Function":"kasten.io/k10/kio/aggapi/apps.(*crCacheUpdater).handleCRDEvent","Line":147,"cluster_name":"5d6ac71e-7d7d-4209-9399-8b85148be5cd","eventType":"MODIFIED","hostname":"b2-30-node-ba218f","level":"debug","msg":"Skipping crd modified event","time":"2024-04-16T06:43:51.408Z","version":"6.5.11"}
{"File":"kasten.io/k10/kio/aggapi/apps/cr_cache_updater.go","Function":"kasten.io/k10/kio/aggapi/apps.(*crCacheUpdater).handleCRDEvent","Line":147,"cluster_name":"5d6ac71e-7d7d-4209-9399-8b85148be5cd","eventType":"MODIFIED","hostname":"b2-30-node-ba218f","level":"debug","msg":"Skipping crd modified event","time":"2024-04-16T06:43:51.418Z","version":"6.5.11"}
{"File":"kasten.io/k10/kio/aggapi/store/catalog.go","Function":"kasten.io/k10/kio/aggapi/store.CatalogStorage.List","Line":64,"cluster_name":"5d6ac71e-7d7d-4209-9399-8b85148be5cd","hostname":"b2-30-node-ba218f","key":"/actions.kio.kasten.io/importactions","level":"debug","msg":"API List","time":"2024-04-16T06:43:52.243Z","version":"6.5.11"}
{"File":"kasten.io/k10/kio/aggapi/store/catalog.go","Function":"kasten.io/k10/kio/aggapi/store.CatalogStorage.List","Line":64,"cluster_name":"5d6ac71e-7d7d-4209-9399-8b85148be5cd","hostname":"b2-30-node-ba218f","key":"/actions.kio.kasten.io/exportactions","level":"debug","msg":"API List","time":"2024-04-16T06:43:55.261Z","version":"6.5.11"}

 

M

@EBrockman of course:

{"File":"kasten.io/k10/kio/aggapi/store/catalog.go","Function":"kasten.io/k10/kio/aggapi/store.CatalogStorage.List","Line":64,"cluster_name":"5d6ac71e-7d7d-4209-9399-8b85148be5cd","hostname":"b2-30-node-ba218f","key":"/actions.kio.kasten.io/upgradeactions","level":"debug","msg":"API List","time":"2024-04-16T06:43:37.268Z","version":"6.5.11"}
{"File":"kasten.io/k10/kio/aggapi/util/errors.go","Function":"kasten.io/k10/kio/aggapi/util.WrapError","Line":25,"cluster_name":"5d6ac71e-7d7d-4209-9399-8b85148be5cd","error":{"message":"Could not retrieve artifacts for prefix search","function":"kasten.io/k10/kio/rest/clients.FetchArtifactsForSearchPrefix","linenumber":65,"file":"kasten.io/k10/kio/rest/clients/catalogclient.go:65","fields":[{"name":"config","value":{}},{"name":"retries","value":1}],"cause":{"message":"Get \"http://10.3.107.173:8000/v0/artifacts/prefixSearch?key=api-list-prefix-key-exportactions.actions.kio.kasten.io\u0026limit=300\u0026reverseOrder=true\u0026value=%40\": context canceled"}},"hostname":"b2-30-node-ba218f","level":"error","msg":"Internal Error","time":"2024-04-16T06:43:38.647Z","version":"6.5.11"}
E0416 06:43:38.648059       1 writers.go:122] apiserver was unable to write a JSON response: http: Handler timeout
E0416 06:43:38.648075       1 status.go:71] apiserver received an error that is not an metav1.Status: &errors.errorString{s:"http: Handler timeout"}: http: Handler timeout
E0416 06:43:38.649252       1 writers.go:135] apiserver was unable to write a fallback JSON response: http: Handler timeout
I0416 06:43:38.649289       1 trace.go:236] Trace[962619652]: "List" accept:application/json, */*,audit-id:f849cb28-5a3e-4aae-9d2e-921da779efa3,client:10.10.100.1,api-group:actions.kio.kasten.io,api-version:v1alpha1,name:,subresource:,namespace:,protocol:HTTP/2.0,resource:exportactions,scope:cluster,url:/apis/actions.kio.kasten.io/v1alpha1/exportactions,user-agent:dashboardbff-server/v0.0.0 (linux/amd64) kubernetes/$Format,verb:LIST (16-Apr-2024 06:43:08.771) (total time: 29877ms):
Trace[962619652]: [29.877315962s] [29.877315962s] END
E0416 06:43:38.649357       1 timeout.go:142] post-timeout activity - time-elapsed: 1.527736ms, GET "/apis/actions.kio.kasten.io/v1alpha1/exportactions" result: <nil>
{"File":"kasten.io/k10/kio/aggapi/store/catalog.go","Function":"kasten.io/k10/kio/aggapi/store.CatalogStorage.List","Line":64,"cluster_name":"5d6ac71e-7d7d-4209-9399-8b85148be5cd","hostname":"b2-30-node-ba218f","key":"/actions.kio.kasten.io/exportactions","level":"debug","msg":"API List","time":"2024-04-16T06:43:40.225Z","version":"6.5.11"}
{"File":"kasten.io/k10/kio/aggapi/store/catalog.go","Function":"kasten.io/k10/kio/aggapi/store.CatalogStorage.List","Line":64,"cluster_name":"5d6ac71e-7d7d-4209-9399-8b85148be5cd","hostname":"b2-30-node-ba218f","key":"/actions.kio.kasten.io/restoreactions","level":"debug","msg":"API List","time":"2024-04-16T06:43:43.234Z","version":"6.5.11"}
{"File":"kasten.io/k10/kio/aggapi/store/catalog.go","Function":"kasten.io/k10/kio/aggapi/store.CatalogStorage.List","Line":64,"cluster_name":"5d6ac71e-7d7d-4209-9399-8b85148be5cd","hostname":"b2-30-node-ba218f","key":"/actions.kio.kasten.io/migratefcdactions","level":"debug","msg":"API List","time":"2024-04-16T06:43:46.250Z","version":"6.5.11"}
{"File":"kasten.io/k10/kio/aggapi/apps/cr_cache_updater.go","Function":"kasten.io/k10/kio/aggapi/apps.(*crCacheUpdater).handleCREvent","Line":200,"cluster_name":"5d6ac71e-7d7d-4209-9399-8b85148be5cd","eventType":"MODIFIED","group":"argoproj.io","hostname":"b2-30-node-ba218f","level":"debug","msg":"Skipping CR modified event","resource":"applications","time":"2024-04-16T06:43:47.490Z","version":"v1alpha1"}
{"File":"kasten.io/k10/kio/aggapi/util/errors.go","Function":"kasten.io/k10/kio/aggapi/util.WrapError","Line":25,"cluster_name":"5d6ac71e-7d7d-4209-9399-8b85148be5cd","error":{"message":"Could not retrieve artifacts for prefix search","function":"kasten.io/k10/kio/rest/clients.FetchArtifactsForSearchPrefix","linenumber":65,"file":"kasten.io/k10/kio/rest/clients/catalogclient.go:65","fields":[{"name":"config","value":{}},{"name":"retries","value":1}],"cause":{"message":"Get \"http://10.3.107.173:8000/v0/artifacts/prefixSearch?key=api-list-prefix-key-migratefcdactions.actions.kio.kasten.io\u0026limit=300\u0026reverseOrder=true\u0026value=%40\": context canceled"}},"hostname":"b2-30-node-ba218f","level":"error","msg":"Internal Error","time":"2024-04-16T06:43:47.574Z","version":"6.5.11"}
E0416 06:43:47.575044       1 writers.go:122] apiserver was unable to write a JSON response: http: Handler timeout
E0416 06:43:47.575062       1 status.go:71] apiserver received an error that is not an metav1.Status: &errors.errorString{s:"http: Handler timeout"}: http: Handler timeout
E0416 06:43:47.576369       1 writers.go:135] apiserver was unable to write a fallback JSON response: http: Handler timeout
I0416 06:43:47.576481       1 trace.go:236] Trace[1123356241]: "List" accept:application/json, */*,audit-id:0ff0ecfd-3511-4122-a980-bef3cd4e5f1a,client:10.10.100.1,api-group:actions.kio.kasten.io,api-version:v1alpha1,name:,subresource:,namespace:,protocol:HTTP/2.0,resource:migratefcdactions,scope:cluster,url:/apis/actions.kio.kasten.io/v1alpha1/migratefcdactions,user-agent:dashboardbff-server/v0.0.0 (linux/amd64) kubernetes/$Format,verb:LIST (16-Apr-2024 06:43:17.685) (total time: 29891ms):
Trace[1123356241]: [29.891320516s] [29.891320516s] END
E0416 06:43:47.576602       1 timeout.go:142] post-timeout activity - time-elapsed: 1.70787ms, GET "/apis/actions.kio.kasten.io/v1alpha1/migratefcdactions" result: <nil>
{"File":"kasten.io/k10/kio/aggapi/store/catalog.go","Function":"kasten.io/k10/kio/aggapi/store.CatalogStorage.List","Line":64,"cluster_name":"5d6ac71e-7d7d-4209-9399-8b85148be5cd","hostname":"b2-30-node-ba218f","key":"/actions.kio.kasten.io/exportactions","level":"debug","msg":"API List","time":"2024-04-16T06:43:49.233Z","version":"6.5.11"}
{"File":"kasten.io/k10/kio/aggapi/apps/cr_cache_updater.go","Function":"kasten.io/k10/kio/aggapi/apps.(*crCacheUpdater).handleCRDEvent","Line":147,"cluster_name":"5d6ac71e-7d7d-4209-9399-8b85148be5cd","eventType":"MODIFIED","hostname":"b2-30-node-ba218f","level":"debug","msg":"Skipping crd modified event","time":"2024-04-16T06:43:51.408Z","version":"6.5.11"}
{"File":"kasten.io/k10/kio/aggapi/apps/cr_cache_updater.go","Function":"kasten.io/k10/kio/aggapi/apps.(*crCacheUpdater).handleCRDEvent","Line":147,"cluster_name":"5d6ac71e-7d7d-4209-9399-8b85148be5cd","eventType":"MODIFIED","hostname":"b2-30-node-ba218f","level":"debug","msg":"Skipping crd modified event","time":"2024-04-16T06:43:51.418Z","version":"6.5.11"}
{"File":"kasten.io/k10/kio/aggapi/store/catalog.go","Function":"kasten.io/k10/kio/aggapi/store.CatalogStorage.List","Line":64,"cluster_name":"5d6ac71e-7d7d-4209-9399-8b85148be5cd","hostname":"b2-30-node-ba218f","key":"/actions.kio.kasten.io/importactions","level":"debug","msg":"API List","time":"2024-04-16T06:43:52.243Z","version":"6.5.11"}
{"File":"kasten.io/k10/kio/aggapi/store/catalog.go","Function":"kasten.io/k10/kio/aggapi/store.CatalogStorage.List","Line":64,"cluster_name":"5d6ac71e-7d7d-4209-9399-8b85148be5cd","hostname":"b2-30-node-ba218f","key":"/actions.kio.kasten.io/exportactions","level":"debug","msg":"API List","time":"2024-04-16T06:43:55.261Z","version":"6.5.11"}

We use Istio as Service Mesh. Maybe any known issues when using Istio?

E
Userlevel 3
Badge +1

Hello @mario.t 

 

Thanks for providing the logs. After looking over the logs it look to be possibly a issue with accessing Catalog. The rule of thumb with Istio is K10 has to be outside of management of Istio. Please take a look into this link.

 

https://istio.io/latest/docs/setup/additional-setup/sidecar-injection/

 

Thanks

Emmanuel

M

@EBrockman I disabled Istio on my kasten-io namespace and restarted all the pods. No Envoy Proxy is now involved in communication between the Kasten components.

I still get following error:

I0417 05:12:09.659878       1 trace.go:236] Trace[1991674505]: "List" accept:application/json, */*,audit-id:8700ae52-71ac-4b6b-b4d3-81c607c33b97,client:10.10.100.4,api-group:actions.kio.kasten.io,api-version:v1alpha1,name:,subresource:,namespace:,protocol:HTTP/2.0,resource:backupactions,scope:cluster,url:/apis/actions.kio.kasten.io/v1alpha1/backupactions,user-agent:dashboardbff-server/v0.0.0 (linux/amd64) kubernetes/$Format,verb:LIST (17-Apr-2024 05:11:39.658) (total time: 30000ms):
Trace[1991674505]: [30.000863509s] [30.000863509s] END
{"File":"kasten.io/k10/kio/aggapi/store/catalog.go","Function":"kasten.io/k10/kio/aggapi/store.CatalogStorage.List","Line":64,"cluster_name":"5d6ac71e-7d7d-4209-9399-8b85148be5cd","hostname":"b2-30-node-4934b6","key":"/actions.kio.kasten.io/backupactions","level":"debug","msg":"API List","time":"2024-04-17T05:12:09.694Z","version":"6.5.11"}
Log message dropped (buffer): {"File":"kasten.io/k10/kio/aggapi/util/errors.go","Function":"kasten.io/k10/kio/aggapi/util.WrapError","Level":"error","Line":25,"Message":"Internal Error","Time":"2024-04-17T05:12:09.659530441Z","cluster_name":"5d6ac71e-7d7d-4209-9399-8b85148be5cd","error":{"message":"Could not retrieve artifacts for prefix search","function":"kasten.io/k10/kio/rest/clients.FetchArtifactsForSearchPrefix","linenumber":65,"file":"kasten.io/k10/kio/rest/clients/catalogclient.go:65","fields":[{"name":"config","value":{}},{"name":"retries","value":1}],"cause":{"message":"Get \"http://10.3.107.173:8000/v0/artifacts/prefixSearch?key=api-list-prefix-key-backupactions.actions.kio.kasten.io\u0026limit=300\u0026reverseOrder=true\u0026value=%40\": context deadline exceeded"}},"hostname":"b2-30-node-4934b6","version":"6.5.11"}
{"File":"kasten.io/k10/kio/aggapi/store/catalog.go","Function":"kasten.io/k10/kio/aggapi/store.CatalogStorage.List","Level":"debug","Line":64,"Message":"API List","Time":"2024-04-17T05:12:09.69401627Z","cluster_name":"5d6ac71e-7d7d-4209-9399-8b85148be5cd","hostname":"b2-30-node-4934b6","key":"/actions.kio.kasten.io/backupactions","version":"6.5.11"}
 Error: {"message":"Fluentbit connection error","function":"kasten.io/k10/kio/log/hooks/fluentbit.(*Hook).handle","linenumber":97,"file":"kasten.io/k10/kio/log/hooks/fluentbit/fluentbit.go:97","cause":{"message":"Fluentbit connection problem","function":"kasten.io/k10/kio/log/hooks/fluentbit.(*Hook).dial","linenumber":88,"file":"kasten.io/k10/kio/log/hooks/fluentbit/fluentbit.go:88","cause":{"message":"dial tcp 10.3.119.86:24224: i/o timeout"}}}
{"File":"kasten.io/k10/kio/aggapi/store/catalog.go","Function":"kasten.io/k10/kio/aggapi/store.CatalogStorage.List","Line":64,"cluster_name":"5d6ac71e-7d7d-4209-9399-8b85148be5cd","hostname":"b2-30-node-4934b6","key":"/actions.kio.kasten.io/exportactions","level":"debug","msg":"API List","time":"2024-04-17T05:12:11.702Z","version":"6.5.11"}
Log message dropped (buffer): {"File":"kasten.io/k10/kio/aggapi/store/catalog.go","Function":"kasten.io/k10/kio/aggapi/store.CatalogStorage.List","Level":"debug","Line":64,"Message":"API List","Time":"2024-04-17T05:12:11.702586376Z","cluster_name":"5d6ac71e-7d7d-4209-9399-8b85148be5cd","hostname":"b2-30-node-4934b6","key":"/actions.kio.kasten.io/exportactions","version":"6.5.11"}
 Error: {"message":"Fluentbit connection error","function":"kasten.io/k10/kio/log/hooks/fluentbit.(*Hook).handle","linenumber":97,"file":"kasten.io/k10/kio/log/hooks/fluentbit/fluentbit.go:97","cause":{"message":"Fluentbit connection problem","function":"kasten.io/k10/kio/log/hooks/fluentbit.(*Hook).dial","linenumber":88,"file":"kasten.io/k10/kio/log/hooks/fluentbit/fluentbit.go:88","cause":{"message":"dial tcp 10.3.119.86:24224: i/o timeout"}}}
{"File":"kasten.io/k10/kio/aggapi/apps/cr_cache_updater.go","Function":"kasten.io/k10/kio/aggapi/apps.(*crCacheUpdater).handleCRDEvent","Line":147,"cluster_name":"5d6ac71e-7d7d-4209-9399-8b85148be5cd","eventType":"MODIFIED","hostname":"b2-30-node-4934b6","level":"debug","msg":"Skipping crd modified event","time":"2024-04-17T05:12:13.732Z","version":"6.5.11"}
{"File":"kasten.io/k10/kio/aggapi/apps/cr_cache_updater.go","Function":"kasten.io/k10/kio/aggapi/apps.(*crCacheUpdater).handleCRDEvent","Line":147,"cluster_name":"5d6ac71e-7d7d-4209-9399-8b85148be5cd","eventType":"MODIFIED","hostname":"b2-30-node-4934b6","level":"debug","msg":"Skipping crd modified event","time":"2024-04-17T05:12:13.745Z","version":"6.5.11"}

 

This is the log of  the catalog-svc:

17T05:15:21.633Z | {"File":"kasten.io/k10/rest/srv/catalogserver/kio_catalog_handler.go","Function":"kasten.io/k10/rest/srv/catalogserver.configureHandlers.ConfigureHandlersFunc.func1.10","Line":238,"cluster_name":"5d6ac71e-7d7d-4209-9399-8b85148be5cd","found_count":183,"hostname":"catalog-svc-69f565f64b-xshc8","level":"debug","msg":"Search artifacts result","search_key":"manifest-policy","search_value":"kasten-io-k10-system-reports-policy","time":"2024-04-17T05:15:21.633Z","version":"6.5.11"}
2024-04-17T05:15:21.652Z | {"File":"kasten.io/k10/rest/srv/catalogserver/kio_catalog_handler.go","Function":"kasten.io/k10/rest/srv/catalogserver.configureHandlers.ConfigureHandlersFunc.func1.10","Line":238,"cluster_name":"5d6ac71e-7d7d-4209-9399-8b85148be5cd","found_count":2,"hostname":"catalog-svc-69f565f64b-xshc8","level":"debug","msg":"Search artifacts result","search_key":"manifest-policy","search_value":"onlineaccunl-onlineaccunl-stage-backup","time":"2024-04-17T05:15:21.652Z","version":"6.5.11"}
2024-04-17T05:15:21.653Z | {"File":"kasten.io/k10/rest/srv/catalogserver/kio_catalog_handler.go","Function":"kasten.io/k10/rest/srv/catalogserver.configureHandlers.ConfigureHandlersFunc.func1.10","Line":238,"cluster_name":"5d6ac71e-7d7d-4209-9399-8b85148be5cd","found_count":2,"hostname":"catalog-svc-69f565f64b-xshc8","level":"debug","msg":"Search artifacts result","search_key":"manifest-policy","search_value":"onlinebatterienat-onlinebatterienat-stage-backup","time":"2024-04-17T05:15:21.653Z","version":"6.5.11"}
2024-04-17T05:15:21.654Z | {"File":"kasten.io/k10/rest/srv/catalogserver/kio_catalog_handler.go","Function":"kasten.io/k10/rest/srv/catalogserver.configureHandlers.ConfigureHandlersFunc.func1.10","Line":238,"cluster_name":"5d6ac71e-7d7d-4209-9399-8b85148be5cd","found_count":2,"hostname":"catalog-svc-69f565f64b-xshc8","level":"debug","msg":"Search artifacts result","search_key":"manifest-policy","search_value":"onlinebatteriende-onlinebatteriende-live-backup","time":"2024-04-17T05:15:21.653Z","version":"6.5.11"}
2024-04-17T05:15:21.654Z | {"File":"kasten.io/k10/rest/srv/catalogserver/kio_catalog_handler.go","Function":"kasten.io/k10/rest/srv/catalogserver.configureHandlers.ConfigureHandlersFunc.func1.10","Line":238,"cluster_name":"5d6ac71e-7d7d-4209-9399-8b85148be5cd","found_count":2,"hostname":"catalog-svc-69f565f64b-xshc8","level":"debug","msg":"Search artifacts result","search_key":"manifest-policy","search_value":"onlinebatteriende-onlinebatteriende-stage-backup","time":"2024-04-17T05:15:21.654Z","version":"6.5.11"}
2024-04-17T05:15:21.655Z | {"File":"kasten.io/k10/rest/srv/catalogserver/kio_catalog_handler.go","Function":"kasten.io/k10/rest/srv/catalogserver.configureHandlers.ConfigureHandlersFunc.func1.10","Line":238,"cluster_name":"5d6ac71e-7d7d-4209-9399-8b85148be5cd","found_count":2,"hostname":"catalog-svc-69f565f64b-xshc8","level":"debug","msg":"Search artifacts result","search_key":"manifest-policy","search_value":"solarspeicher-solarspeicher-stage-backup","time":"2024-04-17T05:15:21.655Z","version":"6.5.11"}
2024-04-17T05:15:23.528Z | {"File":"kasten.io/k10/kio/daemon/daemon.go","Function":"kasten.io/k10/kio/daemon.(*Daemon).run","Line":121,"cluster_name":"5d6ac71e-7d7d-4209-9399-8b85148be5cd","hostname":"catalog-svc-69f565f64b-xshc8","level":"debug","msg":"Daemon run completed","run_count":30,"time":"2024-04-17T05:15:23.527Z","version":"6.5.11"}
2024-04-17T05:15:23.528Z | {"File":"kasten.io/k10/kio/daemon/daemon.go","Function":"kasten.io/k10/kio/daemon.(*Daemon).run","Line":121,"cluster_name":"5d6ac71e-7d7d-4209-9399-8b85148be5cd","hostname":"catalog-svc-69f565f64b-xshc8","level":"debug","msg":"Daemon run completed","run_count":30,"time":"2024-04-17T05:15:23.528Z","version":"6.5.11"}

 

M

My authentication is made with OIDC. For testing purpose, I changed to basic auth. Now actions are showed.

When I try to run a policy once, it runs but the window does not disappear and I get 503 again.

I also cannot see action details. Maybe some problems with authentication?

M

I really dont know why, but after I disabled siem.logging.cloud.awsS3

  siem:
    logging:
      cluster:
        enabled: true
      cloud:
        path: k10audit/
        awsS3:
          enabled: false

in my chart values, it worked.

I saw many fluentbit errors and since I have no fluentbit instance running, I searched for any logging key and deactivated it. (Dont know if this is the corrosponding key for fluentbit exporting. If so, it maybe needs to be declared like this..)

Maybe this key should be deactivated by default, if more than the half of the users not using AWS or it should be more clear (if this was is my final solution).

E
Userlevel 3
Badge +1

Hello @mario.t 

 

Fluentbit comes with K10 by default. This error is normal initially but if it is consistent then possibly a port could be blocking Fluentbit access. I would also note that by default K10 works with no network policies. If you are using any you would need to set these up with K10 in mind.

 

Thanks

Emmanuel

M

@EBrockman ok i see, the logging-svc pod is the fluentbit target.

Now after restarting all pods in the kasten-io namespace, i had the problem again. (fluentbit errors + 504 errors in the dashboard)

I restarted only the aggregatedapi-svc and now it works again. Its a bit random behavior, sometimes the aggregatedapi-svc can connect, sometimes not. 

Same problem with the aggregatedapi-svc → jobs-svc. It fails to lock the kopia repository because the connection cannot be established.

I deactivated the networkpolicies for now.

E
Userlevel 3
Badge +1

Hello @mario.t 

 

Definitely sounds like some strange network problem in the cluster. Due note Network problems will create Fluentbit error’s this is a side effrect of the networking issue accessing between the pods. 

 

Thanks

Emmanuel

Comment


Terms & Conditions