EMFILE and EBUSY errors in node 20

[enricoschaaf]

Describe the bug
When switching from node 18 to node 20 we noticed a lot of EMFILE and EBUSY that we never had before.

To Reproduce
This is really hard to reproduce it only happens sometimes when making network requests.

Expected behavior
No error to appear

Logs

{"errorType":"Error","errorMessage":"getaddrinfo EBUSY track-eu.customer.io","code":"EBUSY","errno":-16,"syscall":"getaddrinfo","hostname":"track-eu.customer.io","stack":["Error: getaddrinfo EBUSY track-eu.customer.io","    at __node_internal_captureLargerStackTrace (node:internal/errors:497:5)","    at __node_internal_ (node:internal/errors:716:10)","    at GetAddrInfoReqWrap.onlookupall [as oncomplete] (node:dns:118:26)"]}

{"status":"error","message":"datadog:Failed to flush extension. connect EMFILE 127.0.0.1:8124 - Local (undefined:undefined)"}

Platform (please complete the following information):
node20 x86_64

Additional context
Not sure if this an issue with the image or the lambda runtime so please let me know if another repo makes more sense

[jokalli2]

We have seen the same issue. Using Datadog layers as you seem to be using. We are trying to debug, I'll update this ticket if we get any more insight.

[jokalli2]

Datadog has released new version of lambda instrumentation, which fixed the issue for us.

[skilbjo]

Thanks for reporting this, I was meaning to report this too.

The EMFILE error appears to be due to npm install using more than 1,024 file descriptors, which is the AWS Lambda hard limit (source: https://docs.aws.amazon.com/lambda/latest/dg/gettingstarted-limits.html).

Two paths I found, that are unrealistic:

• get AWS to increase the file descriptor limit for Lambda / Codebuild with Lambda execution engines
• get npm install to reduce the number of open files it uses

Hope this can get someone kickstarted on the issue, as would be great to use Codebuild with Lambda execution engine to do the build