Memory leak in function responseInterceptor of file handlers/response-interceptor.ts

[heby281]

Checks

• I understand project setup issues should be asked on StackOverflow or in GitHub Discussions.
• I updated to latest http-proxy-middleware.

Describe the bug (be clear and concise)

Memory leak is found in function responseInterceptor of file handlers/response-interceptor.ts:
// concat data stream
_proxyRes.on('data', (chunk) => (buffer = Buffer.concat([buffer, chunk])));

The buffer cannot be released especially when the response data is very large.

Step-by-step reproduction instructions

1. Prepare 2 applications: proxy server, real server.
1. Set options in proxy server
    target: {
      protocol: 'http',
      host: '127.0.0.1',
      port: 8080,
    },
    changeOrigin: true,
    router: {
      url: 'http://127.0.0.1:8080'
    },
    agent: new https.Agent({ keepAlive: true }),
    ws: false,
    selfHandleResponse: true,
    onProxyRes: responseInterceptor((responseBuffer, proxyRes, req, res) => {
      return responseBuffer;
    }
2. Send a request, and the real server respond a large response, for example, 38MB.
3. Repeat the step 2.
4. OOM will be triggered.

Expected behavior (be clear and concise)

No memory leak.
Working code:

let buffers: Buffer[] = [];
let bufferLength = 0;
_proxyRes.on('data', (chunk) => {
    bufferLength += chunk.length;
    buffers.push(chunk);
});

_proxyRes.on('end', async () => {
// ... other code
let buffer = Buffer.from('', 'utf8');
if (bufferLength > 0) {
  buffer = Buffer.concat(buffers, bufferLength);
}
const interceptedBuffer = Buffer.from(await interceptor(buffer, originalProxyRes, req, res));
// ... other code
// clear buffers in the end.
  buffers = [];
  bufferLength = 0;
});

  _proxyRes.on('error', (error) => {
  // clear buffers when error.
  buffers = [];
  bufferLength = 0;
  res.end(`Error fetching proxied request: ${error.message}`);
});

How is http-proxy-middleware used in your project?

*** /opt/***/lib/node_modules/****
`-- [email protected]

What http-proxy-middleware configuration are you using?

target: {
      protocol: 'http',
      host: '127.0.0.1',
      port: 8080,
    },
    changeOrigin: true,
    router: {
      url: 'http://127.0.0.1:8080'
    },
    agent: new https.Agent({ keepAlive: true }),
    ws: false,
    selfHandleResponse: true,
    onProxyRes: responseInterceptor((responseBuffer, proxyRes, req, res) => {
      return responseBuffer;
    }

What OS/version and node/version are you seeing the problem?

npx: installed 1 in 1.045s

  System:
    OS: Linux 5.14 SLES 15-SP4
    CPU: (16) x64 Intel(R) Xeon(R) Silver 4214R CPU @ 2.40GHz
    Memory: 14.22 GB / 31.33 GB
    Container: Yes
    Shell: 4.4.23 - /bin/bash
  Binaries:
    Node: 18.12.1 - /opt/miep/bin/node
    npm: 6.14.15 - /opt/miep/bin/npm
  Managers:
    RubyGems: 2.7.6.3 - /usr/bin/gem
  Utilities:
    Curl: 8.0.1 - /usr/bin/curl
  IDEs:
    Vim: 9.0 - /bin/vim
  Languages:
    Bash: 4.4.23 - /bin/bash
    Java: 1.8.0_372 - /usr/bin/javac
    Perl: 5.26.1 - /usr/bin/perl
    Python: 3.6.15 - /opt/miep/bin/python
    Python3: 3.6.15 - /usr/bin/python3
    Ruby: 2.5.9 - /usr/bin/ruby
  Databases:
    SQLite: 3.39.3 - /usr/bin/sqlite3

Additional context (optional)

No response

[teeohhem]

Bump. We are also running into this.

@heby281 in your "working code" above, is that a patch? Do you have a solution/PR for it yet?