macOS: Error starting FSEvents stream with many folders

[bpasero]

It seems there is a maximum of folders that can be watched. Anywhere from 500 paths, I see an false return from the call to FSEventStreamStart.

~/Desktop/parcel-test> node index.js
node:internal/process/promises:391
    triggerUncaughtException(err, true /* fromPromise */);
    ^

[Error: Error starting FSEvents stream]

Node.js v20.17.0

I was not able to figure out where this limit is coming from so far.

Code to reproduce:

const watcher = require('@parcel/watcher');
const path = require('path');
const fs = require('fs');

async function main() {

    for (let i = 0; i < 1000; i++) {
        const dirPath = path.join(process.cwd(), `folder_${i}`);
        fs.mkdirSync(dirPath, { recursive: true });
        const filePath = path.join(dirPath, `file_${i}.txt`);
        fs.writeFileSync(filePath, `This is file ${i}`);
    }

    for (let i = 0; i < 1000; i++) {
        const dirPath = path.join(process.cwd(), `folder_${i}`);
        let subscription = await watcher.subscribe(dirPath, (err, events) => {
            console.log(events);
        });
    }
}

main();

[bpasero]

Interesting note from watchman: https://github.com/facebook/watchman/blob/cb0b42bae13a4276ec01d6bf1457cab883137b17/website/docs/troubleshooting.md#L192-L221

FSEventStreamStart: register_with_server: ERROR: f2d_register_rpc() => (null) (-21)
Nobody outside of Apple is sure what precisely this means, but it indicates that the fsevents service has gotten in a bad state. Possible reasons for this may include:

There are too many event stream clients
One or more event stream clients has gotten in a bad state and is somehow impacting the fsevents service
To resolve this issue, you may wish to try the following, which are progressively more invasive:

Avoid establishing multiple overlapping watches within the same filesystem tree, especially for large trees. We recommend watching only the root of a project or repo and not watching sub-trees within that tree. Organizations with large trees may wish to deploy the root_restrict_files configuration option so that watchman will only allow watching project roots.
Close or restart other applications that are using fsevents. Some examples are:
editors such as Sublime Text and TextMate.
Many nodejs packages and Grunt style workflows make use of fsevents. Make sure that you upgrade nodejs to at least version v0.11.14. If possible, configure your nodejs packages to use either sane or fb-watchman for file watching as this will consolidate the number of fsevents watches down to just the set maintained by watchman.
Restart the fsevents service: sudo pkill -9 -x fseventsd
Restart your computer

[devongovett]

Do you watch overlapping folders? I was thinking at some point of automatically avoiding that in the watcher library itself but maybe for now you can do that yourself in vscode?

[bpasero]

@devongovett yeah, thats a great point and we have mechanisms in place to try to reduce overlapping watches in VS Code. As a workaround I will improve our de-duplication to be more aggressive and force recursive watchers on a higher root to avoid this.

I still wanted to report this to learn if anyone has an idea about the internal (maybe not documented) limits in macOS with fsevents.

[shellscape]

Wanted to add that I'm running into some of the same concerns using this for jsx-email. I'm making sure the list of directory paths I'm watching is unique, but there's no overlap detection. Would prefer not to have to handle this manually for each implementation, and could see this as a real value-add for the package.