Kasten K10 Gateway has error "Shut down with error error: too many open files

Question

I get the following error in my gateway-pod:level=error msg="shut down with error error: too many open files" func=github.com/emissary-ingress/emissary/v3/pkg/busy.Main file="/go/pkg/busy/busy.go:87" CMD=entrypoint PID=1 The whole log output: time="2022-08-15 08:33:35.8226" level=info msg="Started Ambassador (Version 3.1.0)" func=github.com/emissary-ingress/emissary/v3/cmd/entrypoint.Main file="/go/cmd/entrypoint/entrypoint.go:96" CMD=entrypoint PID=1time="2022-08-15 08:33:35.8526" level=info msg="AMBASSADOR_CLUSTER_ID=822467de-b20d-5916-8300-8dc2a87072b8" func=github.com/emissary-ingress/emissary/v3/cmd/entrypoint.Main file="/go/cmd/entrypoint/entrypoint.go:110" CMD=entrypoint PID=1time="2022-08-15 08:33:35.8541" level=info msg="started command ["diagd" "/ambassador/snapshots" "/ambassador/bootstrap-ads.json" "/ambassador/envoy/envoy.json" "--notices" "/ambassador/notices.json" "--port" "8004" "--kick" "kill -HUP 1"]" func="github.com/datawire/dlib/dexec.(*Cmd).Start" file="/go/vendor/github.com/datawire/dlib/dexec/cmd.go:183" CMD=entrypoint PID=1 THREAD=/diagd dexec.pid=18time="2022-08-15 08:33:35.8541" level=info msg="not logging input read from file "/dev/stdin"" func="github.com/datawire/dlib/dexec.(*Cmd).Start" file="/go/vendor/github.com/datawire/dlib/dexec/cmd.go:185" CMD=entrypoint PID=1 THREAD=/diagd dexec.pid=18 dexec.stream=stdintime="2022-08-15 08:33:35.8542" level=info msg="not logging output written to file "/dev/stdout"" func="github.com/datawire/dlib/dexec.(*Cmd).Start" file="/go/vendor/github.com/datawire/dlib/dexec/cmd.go:188" CMD=entrypoint PID=1 THREAD=/diagd dexec.pid=18 dexec.stream=stdouttime="2022-08-15 08:33:35.8542" level=info msg="not logging output written to file "/dev/stderr"" func="github.com/datawire/dlib/dexec.(*Cmd).Start" file="/go/vendor/github.com/datawire/dlib/dexec/cmd.go:191" CMD=entrypoint PID=1 THREAD=/diagd dexec.pid=18 dexec.stream=stderrtime="2022-08-15 08:33:35.8551" level=info msg="Ambex 3.1.0 starting, snapdirPath /ambassador/snapshots" func=github.com/emissary-ingress/emissary/v3/pkg/ambex.Main file="/go/pkg/ambex/main.go:633" CMD=entrypoint PID=1 THREAD=/ambextime="2022-08-15 08:33:35.8551" level=error msg="goroutine "/ambex" exited with error: too many open files" func="github.com/datawire/dlib/dgroup.(*Group).goWorkerCtx.func1.1" file="/go/vendor/github.com/datawire/dlib/dgroup/group.go:380" CMD=entrypoint PID=1 THREAD=/ambextime="2022-08-15 08:33:35.8553" level=info msg="sending SIGINT" func="github.com/datawire/dlib/dexec.(*Cmd).Start.func1" file="/go/vendor/github.com/datawire/dlib/dexec/cmd.go:211" CMD=entrypoint PID=1 THREAD=/diagdtime="2022-08-15 08:33:35.8556" level=info msg="shutting down (gracefully)..." func="github.com/datawire/dlib/dgroup.(*Group).launchSupervisors.func1" file="/go/vendor/github.com/datawire/dlib/dgroup/group.go:238" CMD=entrypoint PID=1 THREAD=":shutdown_logger"    time="2022-08-15 08:33:35.8559" level=info msg="finished with error: signal: interrupt" func="github.com/datawire/dlib/dexec.(*Cmd).Wait" file="/go/vendor/github.com/datawire/dlib/dexec/cmd.go:257" CMD=entrypoint PID=1 THREAD=/diagd dexec.pid=18time="2022-08-15 08:33:35.8560" level=error msg="goroutine "/diagd" exited with error: signal: interrupt" func="github.com/datawire/dlib/dgroup.(*Group).goWorkerCtx.func1.1" file="/go/vendor/github.com/datawire/dlib/dgroup/group.go:380" CMD=entrypoint PID=1 THREAD=/diagdtime="2022-08-15 08:33:35.8561" level=info msg="Memory Usage 0.00Gi
    PID 1, 0.06Gi: busyambassador entrypoint " func="github.com/emissary-ingress/emissary/v3/pkg/memory.(*MemoryUsage).Watch" file="/go/pkg/memory/memory.go:43" CMD=entrypoint PID=1 THREAD=/memorytime="2022-08-15 08:33:35.8576" level=warning msg="Warning, unable to watch devportals.v3alpha1.getambassador.io, unknown kind." func=github.com/emissary-ingress/emissary/v3/cmd/entrypoint.GetInterestingTypes file="/go/cmd/entrypoint/interesting_types.go:160" CMD=entrypoint PID=1 THREAD=/watchertime="2022-08-15 08:33:35.8576" level=warning msg="Warning, unable to watch httproutes.v1alpha1.networking.x-k8s.io, unknown kind." func=github.com/emissary-ingress/emissary/v3/cmd/entrypoint.GetInterestingTypes file="/go/cmd/entrypoint/interesting_types.go:160" CMD=entrypoint PID=1 THREAD=/watchertime="2022-08-15 08:33:35.8576" level=warning msg="Warning, unable to watch filterpolicies.v3alpha1.getambassador.io, unknown kind." func=github.com/emissary-ingress/emissary/v3/cmd/entrypoint.GetInterestingTypes file="/go/cmd/entrypoint/interesting_types.go:160" CMD=entrypoint PID=1 THREAD=/watchertime="2022-08-15 08:33:35.8578" level=warning msg="Warning, unable to watch filters.v3alpha1.getambassador.io, unknown kind." func=github.com/emissary-ingress/emissary/v3/cmd/entrypoint.GetInterestingTypes file="/go/cmd/entrypoint/interesting_types.go:160" CMD=entrypoint PID=1 THREAD=/watchertime="2022-08-15 08:33:35.8578" level=warning msg="Warning, unable to watch modules.v3alpha1.getambassador.io, unknown kind." func=github.com/emissary-ingress/emissary/v3/cmd/entrypoint.GetInterestingTypes file="/go/cmd/entrypoint/interesting_types.go:160" CMD=entrypoint PID=1 THREAD=/watchertime="2022-08-15 08:33:35.8578" level=warning msg="Warning, unable to watch kubernetesendpointresolvers.v3alpha1.getambassador.io, unknown kind." func=github.com/emissary-ingress/emissary/v3/cmd/entrypoint.GetInterestingTypes file="/go/cmd/entrypoint/interesting_types.go:160" CMD=entrypoint PID=1 THREAD=/watchertime="2022-08-15 08:33:35.8578" level=warning msg="Warning, unable to watch kubernetesserviceresolvers.v3alpha1.getambassador.io, unknown kind." func=github.com/emissary-ingress/emissary/v3/cmd/entrypoint.GetInterestingTypes file="/go/cmd/entrypoint/interesting_types.go:160" CMD=entrypoint PID=1 THREAD=/watchertime="2022-08-15 08:33:35.8578" level=warning msg="Warning, unable to watch logservices.v3alpha1.getambassador.io, unknown kind." func=github.com/emissary-ingress/emissary/v3/cmd/entrypoint.GetInterestingTypes file="/go/cmd/entrypoint/interesting_types.go:160" CMD=entrypoint PID=1 THREAD=/watchertime="2022-08-15 08:33:35.8579" level=warning msg="Warning, unable to watch gateways.v1alpha1.networking.x-k8s.io, unknown kind." func=github.com/emissary-ingress/emissary/v3/cmd/entrypoint.GetInterestingTypes file="/go/cmd/entrypoint/interesting_types.go:160" CMD=entrypoint PID=1 THREAD=/watchertime="2022-08-15 08:33:35.8579" level=warning msg="Warning, unable to watch authservices.v3alpha1.getambassador.io, unknown kind." func=github.com/emissary-ingress/emissary/v3/cmd/entrypoint.GetInterestingTypes file="/go/cmd/entrypoint/interesting_types.go:160" CMD=entrypoint PID=1 THREAD=/watchertime="2022-08-15 08:33:35.8579" level=warning msg="Warning, unable to watch consulresolvers.v3alpha1.getambassador.io, unknown kind." func=github.com/emissary-ingress/emissary/v3/cmd/entrypoint.GetInterestingTypes file="/go/cmd/entrypoint/interesting_types.go:160" CMD=entrypoint PID=1 THREAD=/watchertime="2022-08-15 08:33:35.8579" level=warning msg="Warning, unable to watch mappings.v3alpha1.getambassador.io, unknown kind." func=github.com/emissary-ingress/emissary/v3/cmd/entrypoint.GetInterestingTypes file="/go/cmd/entrypoint/interesting_types.go:160" CMD=entrypoint PID=1 THREAD=/watchertime="2022-08-15 08:33:35.8580" level=warning msg="Warning, unable to watch tlscontexts.v3alpha1.getambassador.io, unknown kind." func=github.com/emissary-ingress/emissary/v3/cmd/entrypoint.GetInterestingTypes file="/go/cmd/entrypoint/interesting_types.go:160" CMD=entrypoint PID=1 THREAD=/watchertime="2022-08-15 08:33:35.8580" level=warning msg="Warning, unable to watch gatewayclasses.v1alpha1.networking.x-k8s.io, unknown kind." func=github.com/emissary-ingress/emissary/v3/cmd/entrypoint.GetInterestingTypes file="/go/cmd/entrypoint/interesting_types.go:160" CMD=entrypoint PID=1 THREAD=/watchertime="2022-08-15 08:33:35.8580" level=warning msg="Warning, unable to watch ratelimitservices.v3alpha1.getambassador.io, unknown kind." func=github.com/emissary-ingress/emissary/v3/cmd/entrypoint.GetInterestingTypes file="/go/cmd/entrypoint/interesting_types.go:160" CMD=entrypoint PID=1 THREAD=/watchertime="2022-08-15 08:33:35.8580" level=warning msg="Warning, unable to watch tcpmappings.v3alpha1.getambassador.io, unknown kind." func=github.com/emissary-ingress/emissary/v3/cmd/entrypoint.GetInterestingTypes file="/go/cmd/entrypoint/interesting_types.go:160" CMD=entrypoint PID=1 THREAD=/watchertime="2022-08-15 08:33:35.8581" level=warning msg="Warning, unable to watch hosts.v3alpha1.getambassador.io, unknown kind." func=github.com/emissary-ingress/emissary/v3/cmd/entrypoint.GetInterestingTypes file="/go/cmd/entrypoint/interesting_types.go:160" CMD=entrypoint PID=1 THREAD=/watchertime="2022-08-15 08:33:35.8581" level=warning msg="Warning, unable to watch listeners.v3alpha1.getambassador.io, unknown kind." func=github.com/emissary-ingress/emissary/v3/cmd/entrypoint.GetInterestingTypes file="/go/cmd/entrypoint/interesting_types.go:160" CMD=entrypoint PID=1 THREAD=/watchertime="2022-08-15 08:33:35.8581" level=warning msg="Warning, unable to watch tracingservices.v3alpha1.getambassador.io, unknown kind." func=github.com/emissary-ingress/emissary/v3/cmd/entrypoint.GetInterestingTypes file="/go/cmd/entrypoint/interesting_types.go:160" CMD=entrypoint PID=1 THREAD=/watchertime="2022-08-15 08:33:35.8602" level=info msg="shutting down (gracefully)..." func="github.com/datawire/dlib/dgroup.(*Group).launchSupervisors.func1" file="/go/vendor/github.com/datawire/dlib/dgroup/group.go:238" CMD=entrypoint PID=1 THREAD="/watcher:shutdown_logger"time="2022-08-15 08:33:35.9172" level=info msg="  final goroutine statuses:" func=github.com/datawire/dlib/dgroup.logGoroutineStatuses file="/go/vendor/github.com/datawire/dlib/dgroup/group.go:84" CMD=entrypoint PID=1 THREAD=":shutdown_status"time="2022-08-15 08:33:35.9173" level=info msg="    /ambex                   : exited with error" func=github.com/datawire/dlib/dgroup.logGoroutineStatuses file="/go/vendor/github.com/datawire/dlib/dgroup/group.go:95" CMD=entrypoint PID=1 THREAD=":shutdown_status"time="2022-08-15 08:33:35.9173" level=info msg="    /diagd                   : exited with error" func=github.com/datawire/dlib/dgroup.logGoroutineStatuses file="/go/vendor/github.com/datawire/dlib/dgroup/group.go:95" CMD=entrypoint PID=1 THREAD=":shutdown_status"time="2022-08-15 08:33:35.9173" level=info msg="    /envoy                   : exited without error" func=github.com/datawire/dlib/dgroup.logGoroutineStatuses file="/go/vendor/github.com/datawire/dlib/dgroup/group.go:95" CMD=entrypoint PID=1 THREAD=":shutdown_status"time="2022-08-15 08:33:35.9174" level=info msg="    /external_snapshot_server: exited without error" func=github.com/datawire/dlib/dgroup.logGoroutineStatuses file="/go/vendor/github.com/datawire/dlib/dgroup/group.go:95" CMD=entrypoint PID=1 THREAD=":shutdown_status"time="2022-08-15 08:33:35.9175" level=info msg="    /healthchecks            : exited without error" func=github.com/datawire/dlib/dgroup.logGoroutineStatuses file="/go/vendor/github.com/datawire/dlib/dgroup/group.go:95" CMD=entrypoint PID=1 THREAD=":shutdown_status"time="2022-08-15 08:33:35.9175" level=info msg="    /memory                  : exited without error" func=github.com/datawire/dlib/dgroup.logGoroutineStatuses file="/go/vendor/github.com/datawire/dlib/dgroup/group.go:95" CMD=entrypoint PID=1 THREAD=":shutdown_status"time="2022-08-15 08:33:35.9175" level=info msg="    /snapshot_server         : exited without error" func=github.com/datawire/dlib/dgroup.logGoroutineStatuses file="/go/vendor/github.com/datawire/dlib/dgroup/group.go:95" CMD=entrypoint PID=1 THREAD=":shutdown_status"time="2022-08-15 08:33:35.9176" level=info msg="    /watcher                 : exited without error" func=github.com/datawire/dlib/dgroup.logGoroutineStatuses file="/go/vendor/github.com/datawire/dlib/dgroup/group.go:95" CMD=entrypoint PID=1 THREAD=":shutdown_status"time="2022-08-15 08:33:35.9176" level=error msg="shut down with error error: too many open files" func=github.com/emissary-ingress/emissary/v3/pkg/busy.Main file="/go/pkg/busy/busy.go:87" CMD=entrypoint PID=1 What is wrong with it?

jaiganeshjk · Answer

@PieveeThe error message with respect to the open files can be resolved by increasing the Ulimits in your nodes.Please take a look at the issues raised in emissary repositoryhttps://github.com/emissary-ingress/emissary/issues/1317

Comment

Sign up

Login to the community

Scanning file for viruses.

This file cannot be downloaded