Endlessly spawning loaders in node 20.0.0 leading to large memory leak

[Jamesernator]

Search Terms

Node 20.0.0

Problem

As of Node 20.0.0 using the ts-node/esm loader has a problem where the loader is endlessly loaded over and over, for example if we run the following program:

import readline from "node:readline";

function input(prompt: string = ""): Promise<string> {
    return new Promise((resolve) => {
        const rl = readline.createInterface({
            input: process.stdin,
            output: process.stdout,
        });
        rl.question(prompt, (answer) => {
            resolve(answer);
            rl.close();
        });
    });
}

await input(`[Continue] `);

Running it with the esm loader:

node --loader ts-node/esm ./test.ts

we get:

(node:181357) ExperimentalWarning: Custom ESM Loaders is an experimental feature and might change at any time
(Use `node --trace-warnings ...` to show where the warning was created)
(node:181357) ExperimentalWarning: Custom ESM Loaders is an experimental feature and might change at any time
(Use `node --trace-warnings ...` to show where the warning was created)
(node:181357) ExperimentalWarning: Custom ESM Loaders is an experimental feature and might change at any time
(Use `node --trace-warnings ...` to show where the warning was created)
[Continue] (node:181357) ExperimentalWarning: Custom ESM Loaders is an experimental feature and might change at any time
(Use `node --trace-warnings ...` to show where the warning was created)
(node:181357) ExperimentalWarning: Custom ESM Loaders is an experimental feature and might change at any time
(Use `node --trace-warnings ...` to show where the warning was created)
(node:181357) ExperimentalWarning: Custom ESM Loaders is an experimental feature and might change at any time

These warnings will continue ad-infinitum until within a few seconds the Node process consumes all of the system memory and is eventually killed by the OS.

Now I'm not entirely sure if this problem is something ts-node can fix, or if something is broken in Node's loader implementation. I did create a little toy loader to see if Node was unconditionally broken and it didn't seem to be.

I haven't really even been able to debug this as --inspect-brk doesn't stop loader code so the memory exhaustion just happens within a few seconds so opening devtools and the like doesn't help.

My speculation here is that ts-node or typescript itself is forking a child process which winds up with the same loader.

This speculation is based on the fact that a simple dummy loader that forks a child process exhibits similar behaviour:

import childProcess from "node:child_process";

childProcess.fork("./worker.js");

console.log(`Loaded loader`);

export function resolve(specifier, context, nextResolve) {
    console.log(`resolve: `, [specifier, context, nextResolve]);
    return nextResolve(specifier, context);
}

Specifications

ts-node v10.9.1
node v20.0.0
compiler v5.0.4

• tsconfig.json, if you're using one:

{
    "include": [
        "./**/*.ts",
        "./**/*.mts",
        "./**/*.cts",
        "./**/*.json"
    ],
    "exclude": [
        "./**/*.d.ts",
        "./**/*.d.mts",
        "./**/*.d.cts"
    ],
    "compilerOptions": {
        "resolveJsonModule": true,
        "composite": true,
        "declaration": true,
        "declarationMap": true,
        "downlevelIteration": true,
        "module": "NodeNext",
        "moduleResolution": "nodenext",
        "rootDir": "./",
        "sourceMap": true,
        "target": "ES2022",
        "newLine": "lf",
        "useDefineForClassFields": true,
        "strict": true,
        "alwaysStrict": true,
        "exactOptionalPropertyTypes": true,
        "noImplicitOverride": true,
        "noUncheckedIndexedAccess": true,
        "forceConsistentCasingInFileNames": true,
        // "verbatimModuleSyntax": true
    },
    "ts-node": {
        "transpileOnly": true   
    }
}

• package.json:

{
  "name": "playground",
  "type": "module",
  "private": true,
  "exports": {
    "./*": "./*"
  },
  "scripts": {
    "build": "tsc --build",
    "lint": "eslint **/*.ts **/*.mts **/*.cts",
    "lint:fix": "eslint --fix **/*.ts **/*.mts **/*.cts"
  },
  "devDependencies": {
    "@jamesernator/eslint-config": "^9.5.0",
    "prettier": "^2.8.4"
  },
  "dependencies": {
    "chalk": "^5.2.0",
    "core-js": "^3.30.0",
    "playwright": "^1.32.3",
    "reblessed": "^0.2.1",
    "ts-node": "^10.9.1",
    "typescript": "^5.0.4"
  }
}

• Operating system and version: Linux 5.19.0-38-generic #39~22.04.1-Ubuntu SMP PREEMPT_DYNAMIC Fri Mar 17 21:16:15 UTC 2 x86_64 x86_64 x86_64 GNU/Linux

[Jamesernator]

I checked some more, I am basically certain the cause is from typescript using childProcess.fork somewhere. I've raised an issue on Node itself as this basically breaks the library within any loader.

[merceyz]

You're probably running into nodejs/node#47566, childProcess.fork seems unrelated.

[Jamesernator]

You're probably running into nodejs/node#47566, childProcess.fork seems unrelated.

Removing childProcess.fork in my replication removes the problem, I don't even observe the behaviour from your example in your linked issue (it just prints one warning as expected).

Regardless of the other issue, having this behaviour happen with childProcess.fork isn't even unexpected per se (as separate processes mean separate loaders). But it is problematic as common libraries might well be using fork (as typescript already is).

Curiously the successive loaders with typescript never actually resolve anything (though with my minimal reproduction they do), so at the very least such loaders can be lazy loaded.

Unless childProcess.fork can forward loaders, then the easier solution would just be not to pass the loader to the forked process (well with chained loaders this can just be all but the last).

[merceyz]

I don't even observe the behaviour from your example in your linked issue (it just prints one warning as expected).

Seems the reproduction was broken, I've updated it.

[pieterbeulque]

I have this issue as well. If I change my script from node --loader=ts-node/esm to ts-node-esm I get this error on Node 20:

TypeError [ERR_UNKNOWN_FILE_EXTENSION]: Unknown file extension ".ts" for ~/my-project/index.ts
    at new NodeError (node:internal/errors:399:5)
    at Object.getFileProtocolModuleFormat [as file:] (node:internal/modules/esm/get_format:99:9)
    at defaultGetFormat (node:internal/modules/esm/get_format:139:38)
    at defaultLoad (node:internal/modules/esm/load:83:20)
    at nextLoad (node:internal/modules/esm/hooks:781:28)
    at load (~/my-project/node_modules/.pnpm/[email protected]_@[email protected][email protected]/node_modules/ts-node/dist/child/child-loader.js:19:122)

So maybe running node --loader also throws this error, but it keeps trying to start?

The same script works fine on 18.x.

[smcenlly]

We had a similar issue. The problem was caused by importing a commonjs module from within a ES Module loader using [email protected]. This would cause the module loader to recursively load itself, with large CPU increase and memory increase in the node process.

In our case, we were importing arg, similar to what is being done here in ts-node.

[nickserv]

This issue has already been fixed in Node 20.1.0.

[Jamesernator]

This issue has already been fixed in Node 20.1.0.

Yes this can be closed now.