Question

Kasten K10 Gateway has error "Shut down with error error: too many open files


I get the following error in my gateway-pod:

level=error msg="shut down with error error: too many open files" func=github.com/emissary-ingress/emissary/v3/pkg/busy.Main file="/go/pkg/busy/busy.go:87" CMD=entrypoint PID=1

 

The whole log output: 

time="2022-08-15 08:33:35.8226" level=info msg="Started Ambassador (Version 3.1.0)" func=github.com/emissary-ingress/emissary/v3/cmd/entrypoint.Main file="/go/cmd/entrypoint/entrypoint.go:96" CMD=entrypoint PID=1
time="2022-08-15 08:33:35.8526" level=info msg="AMBASSADOR_CLUSTER_ID=822467de-b20d-5916-8300-8dc2a87072b8" func=github.com/emissary-ingress/emissary/v3/cmd/entrypoint.Main file="/go/cmd/entrypoint/entrypoint.go:110" CMD=entrypoint PID=1
time="2022-08-15 08:33:35.8541" level=info msg="started command [\"diagd\" \"/ambassador/snapshots\" \"/ambassador/bootstrap-ads.json\" \"/ambassador/envoy/envoy.json\" \"--notices\" \"/ambassador/notices.json\" \"--port\" \"8004\" \"--kick\" \"kill -HUP 1\"]" func="github.com/datawire/dlib/dexec.(*Cmd).Start" file="/go/vendor/github.com/datawire/dlib/dexec/cmd.go:183" CMD=entrypoint PID=1 THREAD=/diagd dexec.pid=18
time="2022-08-15 08:33:35.8541" level=info msg="not logging input read from file \"/dev/stdin\"" func="github.com/datawire/dlib/dexec.(*Cmd).Start" file="/go/vendor/github.com/datawire/dlib/dexec/cmd.go:185" CMD=entrypoint PID=1 THREAD=/diagd dexec.pid=18 dexec.stream=stdin
time="2022-08-15 08:33:35.8542" level=info msg="not logging output written to file \"/dev/stdout\"" func="github.com/datawire/dlib/dexec.(*Cmd).Start" file="/go/vendor/github.com/datawire/dlib/dexec/cmd.go:188" CMD=entrypoint PID=1 THREAD=/diagd dexec.pid=18 dexec.stream=stdout
time="2022-08-15 08:33:35.8542" level=info msg="not logging output written to file \"/dev/stderr\"" func="github.com/datawire/dlib/dexec.(*Cmd).Start" file="/go/vendor/github.com/datawire/dlib/dexec/cmd.go:191" CMD=entrypoint PID=1 THREAD=/diagd dexec.pid=18 dexec.stream=stderr
time="2022-08-15 08:33:35.8551" level=info msg="Ambex 3.1.0 starting, snapdirPath /ambassador/snapshots" func=github.com/emissary-ingress/emissary/v3/pkg/ambex.Main file="/go/pkg/ambex/main.go:633" CMD=entrypoint PID=1 THREAD=/ambex
time="2022-08-15 08:33:35.8551" level=error msg="goroutine \"/ambex\" exited with error: too many open files" func="github.com/datawire/dlib/dgroup.(*Group).goWorkerCtx.func1.1" file="/go/vendor/github.com/datawire/dlib/dgroup/group.go:380" CMD=entrypoint PID=1 THREAD=/ambex
time="2022-08-15 08:33:35.8553" level=info msg="sending SIGINT" func="github.com/datawire/dlib/dexec.(*Cmd).Start.func1" file="/go/vendor/github.com/datawire/dlib/dexec/cmd.go:211" CMD=entrypoint PID=1 THREAD=/diagd
time="2022-08-15 08:33:35.8556" level=info msg="shutting down (gracefully)..." func="github.com/datawire/dlib/dgroup.(*Group).launchSupervisors.func1" file="/go/vendor/github.com/datawire/dlib/dgroup/group.go:238" CMD=entrypoint PID=1 THREAD=":shutdown_logger"
time="2022-08-15 08:33:35.8559" level=info msg="finished with error: signal: interrupt" func="github.com/datawire/dlib/dexec.(*Cmd).Wait" file="/go/vendor/github.com/datawire/dlib/dexec/cmd.go:257" CMD=entrypoint PID=1 THREAD=/diagd dexec.pid=18
time="2022-08-15 08:33:35.8560" level=error msg="goroutine \"/diagd\" exited with error: signal: interrupt" func="github.com/datawire/dlib/dgroup.(*Group).goWorkerCtx.func1.1" file="/go/vendor/github.com/datawire/dlib/dgroup/group.go:380" CMD=entrypoint PID=1 THREAD=/diagd
time="2022-08-15 08:33:35.8561" level=info msg="Memory Usage 0.00Gi\n PID 1, 0.06Gi: busyambassador entrypoint " func="github.com/emissary-ingress/emissary/v3/pkg/memory.(*MemoryUsage).Watch" file="/go/pkg/memory/memory.go:43" CMD=entrypoint PID=1 THREAD=/memory
time="2022-08-15 08:33:35.8576" level=warning msg="Warning, unable to watch devportals.v3alpha1.getambassador.io, unknown kind." func=github.com/emissary-ingress/emissary/v3/cmd/entrypoint.GetInterestingTypes file="/go/cmd/entrypoint/interesting_types.go:160" CMD=entrypoint PID=1 THREAD=/watcher
time="2022-08-15 08:33:35.8576" level=warning msg="Warning, unable to watch httproutes.v1alpha1.networking.x-k8s.io, unknown kind." func=github.com/emissary-ingress/emissary/v3/cmd/entrypoint.GetInterestingTypes file="/go/cmd/entrypoint/interesting_types.go:160" CMD=entrypoint PID=1 THREAD=/watcher
time="2022-08-15 08:33:35.8576" level=warning msg="Warning, unable to watch filterpolicies.v3alpha1.getambassador.io, unknown kind." func=github.com/emissary-ingress/emissary/v3/cmd/entrypoint.GetInterestingTypes file="/go/cmd/entrypoint/interesting_types.go:160" CMD=entrypoint PID=1 THREAD=/watcher
time="2022-08-15 08:33:35.8578" level=warning msg="Warning, unable to watch filters.v3alpha1.getambassador.io, unknown kind." func=github.com/emissary-ingress/emissary/v3/cmd/entrypoint.GetInterestingTypes file="/go/cmd/entrypoint/interesting_types.go:160" CMD=entrypoint PID=1 THREAD=/watcher
time="2022-08-15 08:33:35.8578" level=warning msg="Warning, unable to watch modules.v3alpha1.getambassador.io, unknown kind." func=github.com/emissary-ingress/emissary/v3/cmd/entrypoint.GetInterestingTypes file="/go/cmd/entrypoint/interesting_types.go:160" CMD=entrypoint PID=1 THREAD=/watcher
time="2022-08-15 08:33:35.8578" level=warning msg="Warning, unable to watch kubernetesendpointresolvers.v3alpha1.getambassador.io, unknown kind." func=github.com/emissary-ingress/emissary/v3/cmd/entrypoint.GetInterestingTypes file="/go/cmd/entrypoint/interesting_types.go:160" CMD=entrypoint PID=1 THREAD=/watcher
time="2022-08-15 08:33:35.8578" level=warning msg="Warning, unable to watch kubernetesserviceresolvers.v3alpha1.getambassador.io, unknown kind." func=github.com/emissary-ingress/emissary/v3/cmd/entrypoint.GetInterestingTypes file="/go/cmd/entrypoint/interesting_types.go:160" CMD=entrypoint PID=1 THREAD=/watcher
time="2022-08-15 08:33:35.8578" level=warning msg="Warning, unable to watch logservices.v3alpha1.getambassador.io, unknown kind." func=github.com/emissary-ingress/emissary/v3/cmd/entrypoint.GetInterestingTypes file="/go/cmd/entrypoint/interesting_types.go:160" CMD=entrypoint PID=1 THREAD=/watcher
time="2022-08-15 08:33:35.8579" level=warning msg="Warning, unable to watch gateways.v1alpha1.networking.x-k8s.io, unknown kind." func=github.com/emissary-ingress/emissary/v3/cmd/entrypoint.GetInterestingTypes file="/go/cmd/entrypoint/interesting_types.go:160" CMD=entrypoint PID=1 THREAD=/watcher
time="2022-08-15 08:33:35.8579" level=warning msg="Warning, unable to watch authservices.v3alpha1.getambassador.io, unknown kind." func=github.com/emissary-ingress/emissary/v3/cmd/entrypoint.GetInterestingTypes file="/go/cmd/entrypoint/interesting_types.go:160" CMD=entrypoint PID=1 THREAD=/watcher
time="2022-08-15 08:33:35.8579" level=warning msg="Warning, unable to watch consulresolvers.v3alpha1.getambassador.io, unknown kind." func=github.com/emissary-ingress/emissary/v3/cmd/entrypoint.GetInterestingTypes file="/go/cmd/entrypoint/interesting_types.go:160" CMD=entrypoint PID=1 THREAD=/watcher
time="2022-08-15 08:33:35.8579" level=warning msg="Warning, unable to watch mappings.v3alpha1.getambassador.io, unknown kind." func=github.com/emissary-ingress/emissary/v3/cmd/entrypoint.GetInterestingTypes file="/go/cmd/entrypoint/interesting_types.go:160" CMD=entrypoint PID=1 THREAD=/watcher
time="2022-08-15 08:33:35.8580" level=warning msg="Warning, unable to watch tlscontexts.v3alpha1.getambassador.io, unknown kind." func=github.com/emissary-ingress/emissary/v3/cmd/entrypoint.GetInterestingTypes file="/go/cmd/entrypoint/interesting_types.go:160" CMD=entrypoint PID=1 THREAD=/watcher
time="2022-08-15 08:33:35.8580" level=warning msg="Warning, unable to watch gatewayclasses.v1alpha1.networking.x-k8s.io, unknown kind." func=github.com/emissary-ingress/emissary/v3/cmd/entrypoint.GetInterestingTypes file="/go/cmd/entrypoint/interesting_types.go:160" CMD=entrypoint PID=1 THREAD=/watcher
time="2022-08-15 08:33:35.8580" level=warning msg="Warning, unable to watch ratelimitservices.v3alpha1.getambassador.io, unknown kind." func=github.com/emissary-ingress/emissary/v3/cmd/entrypoint.GetInterestingTypes file="/go/cmd/entrypoint/interesting_types.go:160" CMD=entrypoint PID=1 THREAD=/watcher
time="2022-08-15 08:33:35.8580" level=warning msg="Warning, unable to watch tcpmappings.v3alpha1.getambassador.io, unknown kind." func=github.com/emissary-ingress/emissary/v3/cmd/entrypoint.GetInterestingTypes file="/go/cmd/entrypoint/interesting_types.go:160" CMD=entrypoint PID=1 THREAD=/watcher
time="2022-08-15 08:33:35.8581" level=warning msg="Warning, unable to watch hosts.v3alpha1.getambassador.io, unknown kind." func=github.com/emissary-ingress/emissary/v3/cmd/entrypoint.GetInterestingTypes file="/go/cmd/entrypoint/interesting_types.go:160" CMD=entrypoint PID=1 THREAD=/watcher
time="2022-08-15 08:33:35.8581" level=warning msg="Warning, unable to watch listeners.v3alpha1.getambassador.io, unknown kind." func=github.com/emissary-ingress/emissary/v3/cmd/entrypoint.GetInterestingTypes file="/go/cmd/entrypoint/interesting_types.go:160" CMD=entrypoint PID=1 THREAD=/watcher
time="2022-08-15 08:33:35.8581" level=warning msg="Warning, unable to watch tracingservices.v3alpha1.getambassador.io, unknown kind." func=github.com/emissary-ingress/emissary/v3/cmd/entrypoint.GetInterestingTypes file="/go/cmd/entrypoint/interesting_types.go:160" CMD=entrypoint PID=1 THREAD=/watcher
time="2022-08-15 08:33:35.8602" level=info msg="shutting down (gracefully)..." func="github.com/datawire/dlib/dgroup.(*Group).launchSupervisors.func1" file="/go/vendor/github.com/datawire/dlib/dgroup/group.go:238" CMD=entrypoint PID=1 THREAD="/watcher:shutdown_logger"
time="2022-08-15 08:33:35.9172" level=info msg=" final goroutine statuses:" func=github.com/datawire/dlib/dgroup.logGoroutineStatuses file="/go/vendor/github.com/datawire/dlib/dgroup/group.go:84" CMD=entrypoint PID=1 THREAD=":shutdown_status"
time="2022-08-15 08:33:35.9173" level=info msg=" /ambex : exited with error" func=github.com/datawire/dlib/dgroup.logGoroutineStatuses file="/go/vendor/github.com/datawire/dlib/dgroup/group.go:95" CMD=entrypoint PID=1 THREAD=":shutdown_status"
time="2022-08-15 08:33:35.9173" level=info msg=" /diagd : exited with error" func=github.com/datawire/dlib/dgroup.logGoroutineStatuses file="/go/vendor/github.com/datawire/dlib/dgroup/group.go:95" CMD=entrypoint PID=1 THREAD=":shutdown_status"
time="2022-08-15 08:33:35.9173" level=info msg=" /envoy : exited without error" func=github.com/datawire/dlib/dgroup.logGoroutineStatuses file="/go/vendor/github.com/datawire/dlib/dgroup/group.go:95" CMD=entrypoint PID=1 THREAD=":shutdown_status"
time="2022-08-15 08:33:35.9174" level=info msg=" /external_snapshot_server: exited without error" func=github.com/datawire/dlib/dgroup.logGoroutineStatuses file="/go/vendor/github.com/datawire/dlib/dgroup/group.go:95" CMD=entrypoint PID=1 THREAD=":shutdown_status"
time="2022-08-15 08:33:35.9175" level=info msg=" /healthchecks : exited without error" func=github.com/datawire/dlib/dgroup.logGoroutineStatuses file="/go/vendor/github.com/datawire/dlib/dgroup/group.go:95" CMD=entrypoint PID=1 THREAD=":shutdown_status"
time="2022-08-15 08:33:35.9175" level=info msg=" /memory : exited without error" func=github.com/datawire/dlib/dgroup.logGoroutineStatuses file="/go/vendor/github.com/datawire/dlib/dgroup/group.go:95" CMD=entrypoint PID=1 THREAD=":shutdown_status"
time="2022-08-15 08:33:35.9175" level=info msg=" /snapshot_server : exited without error" func=github.com/datawire/dlib/dgroup.logGoroutineStatuses file="/go/vendor/github.com/datawire/dlib/dgroup/group.go:95" CMD=entrypoint PID=1 THREAD=":shutdown_status"
time="2022-08-15 08:33:35.9176" level=info msg=" /watcher : exited without error" func=github.com/datawire/dlib/dgroup.logGoroutineStatuses file="/go/vendor/github.com/datawire/dlib/dgroup/group.go:95" CMD=entrypoint PID=1 THREAD=":shutdown_status"
time="2022-08-15 08:33:35.9176" level=error msg="shut down with error error: too many open files" func=github.com/emissary-ingress/emissary/v3/pkg/busy.Main file="/go/pkg/busy/busy.go:87" CMD=entrypoint PID=1

 

What is wrong with it?


2 comments

Userlevel 3
Badge +1

Hello Pievee,

 

I recommend updating K10 to the latest version this looks to be an API/CRD issue where It's unable to access the ambassador API/CRD and possible networking API (this maybe partially related to ambassador). Please let me know the results, and we will continue are troubleshooting if need be

 

Thanks

Emmanuel

Userlevel 6
Badge +2

@Pievee 

The error message with respect to the open files can be resolved by increasing the Ulimits in your nodes.

Please take a look at the issues raised in emissary repository https://github.com/emissary-ingress/emissary/issues/1317

Comment