Too many files in src() leads to stack overflow

[yocontra]

Case: 8072 regular files in 207 dirs

[sanfords]

Has there been any movement on this? It bit me too.

[transitive-bullshit]

+1 running into this with src of a complicated set of node_modules

[anaran]

Have any of you lost file content due to this?
I ran a gulp-replace recursively on .js files in a clean gecko-dev git clone (just to be on the safe side).
After it said it only took a few ms it went on for many minutes, eventually hitting a stack overflow.

The big deal was that is produced a lot 0 byte .js files along the way.

Is this a known issue?

[yocontra]

If you npm update to the latest does this still happen?

[yocontra]

Ping - does this happen on 0.3.0?

[henvic]

+1

[henvic]

@contra,

I'm having the same issue.

When I try to do something such as
vinyl.src(['**']);

If I've many files (the limit is with regard to the file descriptor limit which on Unix you can check with ulimit -n) on the directory an error happens:

vinyl.src(['**'], {buffer: false});
stream.js:94
      throw er; // Unhandled stream error in pipe.
            ^
Error: EMFILE, readdir '/Users/henvic/projects/liferay/alloy-ui/demos/tree/assets/article0.png'

It'd be nice to find a proper way to solve this, but for now I'm going to verify the ulimit, and if low than a threshold, increase it.

@sindresorhus has had this issue many times: sindresorhus/guides#8
and I guess most of them was when using the vinylfs.src method when using gulp.src.

Do you guys think vinyl-fs should do something to avoid this limit and, if so, what?

[sindresorhus]

and I guess most of them was when using the vinylfs.src method when using gulp.src.

Nope, most my issues has been with any globbing/filewatcher lib out there, read grunt, gulp, glob, etc.

This is a general super annoying problem with OS X...

[sindresorhus]

Good news is that this is fixed in Node 0.12: nodejs/node-v0.x-archive@6820054

If that ever ships though... :P

[henvic]

Nice!

I'm going to use a simple workaround until then:
https://gist.github.com/henvic/f7fe64d74df148962a88

[henvic]

I was asked to create a npm module for this: osx-ulimit.

[yocontra]

Can anyone try to reproduce this with the new set of tooling? Just npm update - The lower highWaterMark on through2 should help this out

[yocontra]

Closing due to no response and not able to reproduce

[mnpenner]

I got this today. I've got ~45,941 files. Running node v0.10.36 on Ubuntu 14.04. [email protected].

[joshstrange]

Not sure if this is the same issue but I have an array of file paths (1 glob all the rest full paths) that I am passing to src and it appears that only the first 30 array elements are being respected. If I change the order of the array it's only the first 30 array elements (globs only count as 1). This might not be related, I'm about to dig into the source code to try and figure it out now.

EDIT: I was using the src function twice in my stream (I'm using gulp) which may have contributed to my issues. I'm using merge-stream now and it's working.