yarn bundle throwing error ENOENT: no such file or directory

[rhlsthrm]

Describe the bug
I have a yarn workspace, I am trying to use this plugin to bundle and create docker images. When I try to bundle a package to be dockerized, I get an error:

Internal Error: ENOENT: no such file or directory, stat '/private/var/folders/0x/cml8jk85195dd8p2f1swfrb00000gn/T/xfs-764eefc3/node_modules/@connext/nxtp-router'
    at Object.statSync (node:fs:1536:3)
    at ar.statSync (/Users/rahul/.node/corepack/yarn/3.2.0/yarn.js:319:2861)
    at Ht.removeEmptyDirectories (/Users/rahul/connext/repos/nxtp/.yarn/plugins/@ojkelly/plugin-all.cjs:44:4895)
    at Ht.removeEmptyDirectories (/Users/rahul/connext/repos/nxtp/.yarn/plugins/@ojkelly/plugin-all.cjs:44:4992)
    at async Ht.removeEmptyDirectories (/Users/rahul/connext/repos/nxtp/.yarn/plugins/@ojkelly/plugin-all.cjs:44:4981)
    at async Ht.removeEmptyDirectories (/Users/rahul/connext/repos/nxtp/.yarn/plugins/@ojkelly/plugin-all.cjs:44:4981)
    at async Ht.removeExcluded (/Users/rahul/connext/repos/nxtp/.yarn/plugins/@ojkelly/plugin-all.cjs:44:5630)
    at async t (/Users/rahul/connext/repos/nxtp/.yarn/plugins/@ojkelly/plugin-all.cjs:45:1313)
    at async ar.mktempPromise (/Users/rahul/.node/corepack/yarn/3.2.0/yarn.js:319:63472)
    at async Ht.execute (/Users/rahul/connext/repos/nxtp/.yarn/plugins/@ojkelly/plugin-all.cjs:45:2653)

To Reproduce
Steps to reproduce the behavior:

1. Checkout this repo on branch amarok: https://github.com/connext/nxtp/tree/amarok
2. Run yarn bundle in packages/agents/sequencer
3. See error

Expected behavior
Bundle happens.

Screenshots

Desktop (please complete the following information):

• OS: macOS

Additional context
Yarn version: 3.2.0

[ojkelly]

Hey @rhlsthrm,

That error usually occurs because either the package @connext/nxtp-router is either not listed in that bundled package's dependencies.

Looking at the package.json, I can't see it listed. https://github.com/connext/nxtp/blob/amarok/packages/agents/sequencer/package.json#L47

I added the following to dependencies

    "@connext/nxtp-router": "workspace:*",
    "@connext/nxtp-txservice": "workspace:*",
    "@connext/nxtp-utils": "workspace:*",
    "@connext/nxtp-subgraph": "workspace:*",

And then it bundled.

If you're still experiencing issues with it, please re-open this issue.

[rhlsthrm]

@ojkelly but those packages are not supposed to be part of the sequencer package's dependencies. The sequencer only uses a subset of the packages.

My assumption is that the point of this is only to use the workspace packages where this is a dependency.

[ojkelly]

@rhlsthrm fair enough, something else must be at play then.

I just checked .yarnrc.yml and the project is set to nodeLinker: node-modules. While that doesn't strictly block yarn bundle, it's less predictable than nodeLinker: pnp.

Bundling works after switching to pnp linking. I don't know if there's other reason's you can't use pnp, so it may not be the option here. Though, for bundling I do recommend using pnp if you can.

Let's re-open to see if we can find the cause in this case.

[rhlsthrm]

We have not tried pnp yet. I'm worried about breaking some other downstream workflow using pnp but I will try it and see if everything else works.

[SamKosky]

Hey @ojkelly,

I've encountered this same issue with trying to bundle a package that exists within a monorepo with nodeLinker set to node-modules in .yarnrc.yml. We have a React Native project in our monorepo, which is why we've avoided using pnp.

I've tried excluding the React Native project via --exclude and a .bundleignore file but it produces the same error, even though we have no dependencies on our react-native project.

Error output:

❯ yarn bundle --exclude "@project/mobile" --ignore-file .bundleignore
➤ YB1000: ┌ Prepare /Users/.../project/apps/server for bundling
➤ YB1000: │ Preparing temporary directory
➤ YB1000: │ Copying repo to temporary directory
Internal Error: ENOENT: no such file or directory, chmod '/private/var/folders/38/63ftm4nj1_b89c4r9crxm39h0000gn/T/xfs-ce8f7901/apps/mobile/ios/Pods/Headers/Private/RNFBApp/RCTConvert+FIRApp.h'

Does --exclude explicitly exclude those packages or file paths from being copied to the temp directory?

[ojkelly]

Nothing prevents copying all the files at the moment, as we're using an os file copy to setup the tmp dir.

.bundleignore prevents those files from hanging around in the final zip or in the tmp folder (usually done for docker).

It sounds like you might be onto the right solution. If you're interested in making a PR the relevant area is here https://github.com/ojkelly/yarn.build/blob/main/packages/plugins/plugin-bundle/src/commands/bundle/index.ts#L355

Otherwise I think I can get to solving this in the next few weeks.

I think an ideal approach is to take the --exclude folders, and subtract them from a list of files to copy instead of taking a full copy of the whole repo.

[mlewando-cp]

Hi :)
any updates on that?

[mgara]

Unfortunately, I came across the same issue ... I tried multiple recipes ...also as you already mentioned the nodeLinker: pnp works except that I can not use PnP (which is amazing) I must stick with node-modules as I'm using prisma in my project, and I'm pretty sure many fellas out there are using Prisma ...
In the meantime, I was looking at the stack trace I've found that it usually happens within the removeEmptyDirectories function.

[ojkelly]

Unfortunately, I came across the same issue ... I tried multiple recipes ...also as you already mentioned the nodeLinker: pnp works except that I can not use PnP (which is amazing) I must stick with node-modules as I'm using prisma in my project, and I'm pretty sure many fellas out there are using Prisma ...
In the meantime, I was looking at the stack trace I've found that it usually happens within the removeEmptyDirectories function.

I think what we probably need to do here to get this unblocked is add error handling around that area to catch, and probably ignore the errors thrown.

The proper fix sounds like ensuring we check if a file/folder exists before deleting it.

then again, if we try to delete and get ENOENT: no such file or directory it’s basically the same.

[mgara]

My only concern, again it's your code, that if you do that it might break (or not) the integrity of the operation: why is it attempting to delete something that doesn't exist, in my case I think it was a symbolic link pointing to my workspace packages.

[ojkelly]

My only concern, again it's your code, that if you do that it might break (or not) the integrity of the operation:

Yep fair. In this case the only point of that function removeEmptyDirectories is to help the [removeExcluded](https://github.com/ojkelly/yarn.build/blob/f0bace2ef13d0e36cf54a739c43db3b76cd4fcf8/packages/plugins/plugin-bundle/src/commands/bundle/index.ts#L257) in reducing the size of the bundle.

This error appeared around this code in the past #120 which fixed #119.

On Windows, when traversing directories to delete empty directories, the bundle command incorrectly joined paths with path.join when ppath.join should have been used. As a result, the removeEmptyDirectories recursed into invalid paths and the check for whether the invalid path is a directory failed due to ENOENT: no such file or directory.

It's preferable for the bundle to be marginally bigger, than break in some circumstances.

why is it attempting to delete something that doesn't exist

I think #142 is likely related to this, because it appears we've followed a path, and tried to delete something that does not exist.

If you're able to provide a link to a repro I think I know where we need to look now. Otherwise, if you have a bit of time, I'd drop some console.log on everything in

yarn.build/packages/plugins/plugin-bundle/src/commands/bundle/index.ts

Line 228 in f0bace2

.

And see what it's getting tripped up on.

Again, that functions job is just to reduce the bundle size.

Also see there's two uses of removeExcluded, the first is for each package which doesn't clear empty dirs, and the second is for the global excluded list which does clear empty dirs

yarn.build/packages/plugins/plugin-bundle/src/commands/bundle/index.ts

Line 542 in f0bace2

await this.removeExcluded(ctx, {

in my case I think it was a symbolic link pointing to my workspace packages.

Can you expand on this? is it just like the packages/**/package/node_modules or something in the top node_modules?

We run yarn install before reducing the bundle size to let yarn itself remove any unused dependencies (of which there may be some, as we have removed your other local packages that are needed).

I feel like we're finally getting closer to resolving this bug.

[FlipABit]

Was any progress ever made on this? I’m running into the same issue.

[nikitar]

This also happens if you have python code in your project. In our case, it's a top-level "jobs" folder which is not connected at all to yarn/node/js, but gets copied just by being in the same repo.

➤ YB1000: │ Preparing temporary directory
➤ YB1000: │ Copying repo to temporary directory
Internal Error: EPERM: operation not permitted, chmod '/private/var/folders/8f/t4ltlkgs3p7d5g0s9p_j_k_h0000gn/T/xfs-b41fd866/jobs/annotation-history/venv/bin/python3'
Error: EPERM: operation not permitted, chmod '/private/var/folders/8f/t4ltlkgs3p7d5g0s9p_j_k_h0000gn/T/xfs-b41fd866/jobs/annotation-history/venv/bin/python3'

(venv/bin/python3 is a link to /usr/bin/python3)

I was really surprised that yarn bundle has to copy everything at all. For large projects, I imagine it'd be even more of an issue.