[Bug?]: Yarn downloads irrelevant optional dependencies (e.g. for a different OS)

[evanw]

Self-service

• I'd be willing to implement a fix

Describe the bug

I'm the main developer behind esbuild and I'm working on changing the installation strategy for the binary executable from an install script to a list of optional dependencies, one for each supported OS and architecture combination. This works fine with npm and pnpm but doesn't work well with Yarn because Yarn always downloads all optional dependencies instead of only downloading the optional dependency for the current OS and architecture. So switching installation strategies would presumably make installing esbuild with Yarn a lot slower than it should be.

The motivation for doing this is to move the complexity of figuring out what to download from where out of the esbuild package and into the package manager instead. That way installing esbuild should "just work" in complex scenarios including when offline, with a custom registry, with a custom proxy, or with limited file system permissions. Some of these scenarios don't work with esbuild's current install script and it's sometimes very hard to debug these issues since it's hard to replicate a user's network environment. See evanw/esbuild#789 (comment) and the surrounding thread for more context.

To reproduce

These specific instructions are for macOS:

$ echo '{ "dependencies": { "esbuild-experimental-optdeps": "0.12.3" } }' > package.json
$ touch yarn.lock
$ rm -f ~/.yarn/berry/cache/esbuild-*
$ yarn set version berry
$ yarn
$ ls ~/.yarn/berry/cache/esbuild-*
esbuild-darwin-64-npm-0.12.3-3bce268eff-8.zip
esbuild-darwin-arm64-npm-0.12.3-2067324df1-8.zip
esbuild-experimental-optdeps-npm-0.12.3-da58fe24de-8.zip
esbuild-freebsd-64-npm-0.12.3-b28b30bfa6-8.zip
esbuild-freebsd-arm64-npm-0.12.3-5601b669f5-8.zip
esbuild-linux-32-npm-0.12.3-182249b5a7-8.zip
esbuild-linux-64-npm-0.12.3-4067979d42-8.zip
esbuild-linux-arm-npm-0.12.3-91320e7a3b-8.zip
esbuild-linux-arm64-npm-0.12.3-ef73807905-8.zip
esbuild-linux-mips64le-npm-0.12.3-daf3dd1009-8.zip
esbuild-linux-ppc64le-npm-0.12.3-81866b1f7b-8.zip
esbuild-windows-32-npm-0.12.3-778762815e-8.zip
esbuild-windows-64-npm-0.12.3-5ac80b68d1-8.zip

The esbuild-experimental-optdeps package is one I have made specifically to test this optional dependency installation strategy. As you can see here, all 13 packages have been downloaded instead of only downloading the 2 relevant packages like both npm and pnpm. See evanw/esbuild#789 (comment) for more detail about npm and pnpm.

Environment

System:
  OS: macOS 11.5.1
  CPU: (12) x64 Intel(R) Core(TM) i7-9750H CPU @ 2.60GHz
Binaries:
  Node: 16.5.0 - /private/var/folders/j6/np400cw17sz0n5ljd67byrzw0000gn/T/xfs-0b02617d/node
  Yarn: 3.0.1 - /private/var/folders/j6/np400cw17sz0n5ljd67byrzw0000gn/T/xfs-0b02617d/yarn
  npm: 7.20.6 - ~/.npm-local/bin/npm

Additional context

Yarn's inefficient behavior here isn't a deal-breaker for me since it isn't a correctness issue, just a performance issue. But at some point I may move forward with this different package installation strategy for esbuild, at which point installing esbuild with Yarn will become much slower than with npm and pnpm.

[evanw]

Forgive my lack of understanding (I don't know what the formal semantics of optionalDependencies are supposed to be), but I'm assuming what npm is doing is correct and when npm installs [email protected], it appears to download the metadata for all packages and construct a package-lock.json file with all packages but it appears to only actually download one package. This is different from Yarn which appears to actually download all packages.

Each of my packages is much bigger than its metadata (metadata is ~0.5mb, package is ~8mb), so I was hoping to use optional dependencies to have the install process still be relatively efficient despite having many dependencies. In other words while npm appears to be downloading approximately 0.5mb * 13 + 8mb = 14.5mb, Yarn appears to be downloading approximately 0.5mb * 13 + 8mb * 12 = 102.5mb making the install process slower, especially over slow connections. Edit: actually those sizes are uncompressed instead of compressed, but it's the same idea.

Is the difference in observed behavior here an indication that npm is doing something wrong? Or perhaps does way Yarn's internals work make it not viable to use the package metadata without downloading the package contents? I was hoping that this approach would be a good broadly-compatible path forward for packages with prebuilt binary executables, but I understand very little about how package managers actually work so I could be totally off. I really appreciate you taking the time to respond to my questions!

[arcanis]

I just dig a little deeper - it appears you're correct, and other package managers skip installing this package. Strangely it goes against what the old Yarn doc stated, perhaps that's where I got confused. Since I'm wrong on that I'll be a little bit less passionate in my arguments, although I'm not quite sure we can/should change this behaviour yet 😄

One important thing about Yarn is that we try hard to provide a development environment that's the same from one person to the other. Of course native packages are kinda on their own, but at least we want to enforce that the package resolution is stable regardless of the system. In the case of node_modules installs it's perhaps even more important than PnP since different packages can lead to different hoisting. So not installing optional deps not only affects them, but also may change how the whole installed tree looks like (unless we do something like not hoisting optional deps' deps?).

I was wondering how that works. It looks like npm only downloads the package for the current platform so I assumed npm's cross-platform caching use case had to work out somehow. But maybe npm's caching is actually broken in this area?

I might be mistaken, but npm's cache isn't intended to be cross platform. By contrast Yarn has always offered a fairly strong set of caching guarantees via the Offline Mirror. Afaik npm never implemented this (although they came close once, with npm archive).

With 2+ we even introduced Zero-Installs, which push that even further. But native deps are already a problem with that, so adding optional deps to the mix may not change things much (although it'd lead to different .pnp.cjs files depending on the system, which isn't a problem currently ... could be adressed by encoding all the dependencies and filtering the right alternatives at runtime, but it starts to become quite complex).

Isn't that something that yarnpkg/berry#2751 would have to solve as well though? Or does that proposal still always download all packages for all platforms? If so, then the package variants feature wouldn't be helpful for esbuild's use case after all since I'm trying to avoid always downloading all packages for all platforms. I'm mentioning this since you asked about that proposal in relation to esbuild in #1154.

#2751 doesn't have this problem - by default it would only install for the current platform, but you could list other ones to be fetched and cached using your configuration (see the section called "Cache integration").

By writing this I notice however that #2751 would fall as well in the case where the resolution will necessarily be different depending on the systems. So perhaps my prejudices against skipping optional dependencies are a little bit too extreme and will have to be addressed sooner or later.

[larixer]

In the case of node_modules installs it's perhaps even more important than PnP since different packages can lead to different hoisting

I don't see how it's more important for node_modules installs, to be fair. The package for your system will be available on some spot inside node_modules, otherwise, the hoisting layout will be the same on all the systems.

[evanw]

Heads up that esbuild’s package installation now works this way as of version 0.13.0.

[merceyz]

@evanw Thanks for the heads up, seems our e2e test isn't happy with the 0.13.0 release https://github.com/yarnpkg/berry/runs/3680543550#step:5:53

[evanw]

Ah, whoops. I have a Yarn PnP workaround that needs to copy the binary executable out of the zip file before it needs to be run and it looks like I forgot a mkdir -p before writing to the file. My testing worked on my machine because that directory already existed. I'll push out a fix for that now.

[merceyz]

Since esbuild has a postinstall it is already out of the zip file so you could place it there instead

[evanw]

Thanks for the pointers! I like the sound of preferUnplugged: true. I think I'll take that approach since it's the simplest.

[merceyz]

I edited it away as it will unpack all the versions when only one of them will be used, what you're doing now is more efficient disk usage. It would actually be preferable if you set preferUnplugged: false on them so they wont be unpacked unnecessarily

[FezVrasta]

Related question here #3872

Why isn't Yarn downloading all the versions in cache? I see it skips linking them because of platform incompatibility, but I still expect them all to be in the cache so that zero-installs work across different machines.