SNOW-992395: The lib leaks keepalive sockets when uploading files + using S3 client

[GabrielLomba]

Hi! I have a service running that uploads lots of files to a Snowflake stage to then insert the data into tables. The problem, however, is that there is a memory leak created by how the Snowflake SDK uses AWS S3 clients. There are many places where the SDK creates temporary clients (to name a few: here, here and here) but the SDK never calls client.destroy() on them (defined in the AWS SDK here).

Each S3 client creates a http agent with keepalive sockets and those are just never destroyed, creating increasing memory usage over the time. While Node docs state that destroying an agent is usually not needed, for keepalive sockets (this case), it's best to do it explicitly and, in our experience, keepalive sockets pointing at AWS infrastructure stay alive for a really long time, enough to cause increasing memory usage.

I also noticed the SDK creates clients (for instance, here) and set them in meta but they're never reused in the respective util files when they could. PR to fix this has been created.

1. What version of NodeJS driver are you using?
   v1.9.0

2. What operating system and processor architecture are you using?
   Ubuntu, x86_64

3. What version of NodeJS are you using?
   v14.19.0

4. What are the component versions in the environment (npm list)?
   NA

5. Server version:* E.g. 1.90.1
   NA

6. What did you do?
   My service uploads lots of data files to Snowflake stage which is then inserted into tables.

7. What did you expect to see?
   Controlled memory usage over a greater period of time.

8. What is your Snowflake account identifier, if any? (Optional)
   NA

[sfc-gh-dszmolka]

hi - thank you for raising this issue and especially for providing a possible solution for it; the contribution is much appreciated! we'll take a look

[sfc-gh-pmotacki]

Thank you @GabrielLomba. It looks like a bug in our our sdk. I will take a look on your PR and add fix to next version.

[sfc-gh-dszmolka]

checked why this issue was closed; apparently its PR was released in snowflake-sdk 1.9.3

[GabrielLomba]

@sfc-gh-pmotacki , I see the changes related to this issue were reverted in PR #747 (both client reuse and destruction) and are not present in the latest SDK version. The PR does not have any description so I have no idea why that was necessary. Can we reopen the issue in that case? Since the memory leak described still exists.