Creating packages consumable through npm install

[AshishShanker]

I am new to Yarn 2/3, and wanted to find out if there is a prescribed way to make Yarn monorepos available to be consumed by non-Yarn packages. Ideally, I would like to be able to produce a tarball that another Node project could install through npm install. I am not looking to publish the monorepo workspaces to npm registry (public or private)

I have looked at yarn npm publish but that appears to be geared towards actually publishing to the npm registry, with seemingly no opportunity to make it quit after generating a publishable package. In fact, it will not proceed without npm registry credentials. I could potentially set up a localhost registry but hoping I don't have to.

I have looked at yarn pack which creates a compressed archive of all the files in the monorepo, but the resulting .tgz is not directly usable by npm because npm doesn't understand the workspace: protocol, which is how I do internal referencing between monorepo packages. It also doesn't include any of the dependencies from .yarn/cache. I was hoping that yarn pack would either generate good old node_modules or would produce the right scaffolding so npm can resolve dependencies using .pnp.cjs

Lastly, I have also looked at yarn.build (https://github.com/ojkelly/yarn.build), but that doesn't appear to produce a package that can be directly installed by npm install path/to/.tgz

[ojkelly]

Yarn and npm are similar but different ways to manage a project. While they're often converging on features, they aren't interchangeable.

If you limit your project to the small subset of features that work across both you might be able to get this to work, but it wouldn't make sense. It's a lot of work to restrict yourself (and team) for no real value.

If you want to use a yarn monorepo to develop (I do!) you can't use npm with it.

I'm the author of yarn.build. The bundle command is essentially a way of getting a nodejs artefact that you can run on your server. You don't technically need yarn or npm to run it (with pnp mode), and can just run node entrypoint.js (you need to set a package.json#main script in your workspace). That script bootstraps pnp and reexports your packages main file.

You can do the same by running node --require ./pnp.cjs path/to/you/script.js. You can also just run yarn node path/to/you/script.js

A key thing to note here is that with either yarn by itself, or the zip you get from yarn bundle if you're using pnp mode you don't need to install anything. All the dependencies are/should be vendored into your repo.

I think what you want to do it quite possible, just you don't need npm for it at all.

[AshishShanker]

Hi @ojkelly thanks for taking a look at my query. I was looking to use monorepo as well, and the dev experience of putting together the application went really well.

However, the consumers of my package use npm to manage their application. As such, I needed a way to deliver some code artefact preferably through a git+https link that they could incorporate as a dependency in their package.json and simply do npm install on their end to pull my package.

In order for that use case to be supported, I reckoned I would need to transpile my ts to js + .d.ts, and offer them an artefact with a package.json that npm can comprehend. One of the key issues is that when you have workspace: protocol based dependencies within the packages of the monorepo, that protocol is not understood by npm. There were a few other things that convinced me supporting npm consumable artefacts from a yarn2 driven dev process is not going to be straightforward.

I think the entrypoint.js approach will certainly help yarn intercept node calls for packages and fulfill them quickly, but yarn pack or yarn.build didn't really bundle in .yarn/cache out of the box, so not clear how that would happen. Note that I only want direct dependencies in, not devDependencies.

Hopefully, this clarifies where I'm coming from. I'm not looking to use npm for my work, but still need to support others' workflows that utilize npm

[ojkelly]

yarn.build does bundle .yarn/cache, you need to be using the pnp linker so there's files there.

Even though their using npm to manage their application, if you're delivering a node app in it's entirety and they just need to run it npm isn't needed. They just need node.

The only reasonable ways for you to support npm will be to either publish to npm (public or private) using workspace:^ as the version for your local packages (that gets converted to the published version with yarn publish). Or to use a compiler like Webpack to bundle everything including your dependencies into a single file, such that the package manager isn't needed.

If you have the time I'd suggest trying the following because it should work:

• setup your yarn monorepo (with yarn 3)
• in the package that you run your app from, set package.json#main to the compiled js script, usually something like dist/index.js (this tells yarn.build to create the entrypoint.js file
• run yarn bundle in the workspace (ie next to package.json) that runs your application
• send the zip to your consumers with the following instructions

1. unzip the bundle.zip into a new folder
2. run node entrypoint.js

Thats it, it should pickup the pnp dependencies and run provided they have node installed.

Here's an example with the e2e example in yarn.build. It has a the following in the root package.json:

  "scripts": {
    "start": "node entrypoint.js"
  },

And the package that we run has it's main file described in it's package.json.

TO bundle it, we run yarn bundle in that workspace, so this directory yarn.build/e2e/lambda-project/packages/lambda.

Extracting that zip to a separate folder, we can then run the entrypoint.js script with either node, yarn or npm.

# print out the contents of unzipped bundle
~/t/test ❯❯❯ la
total 1176
-rw-r--r--  1 owenkelly  staff   110B 26 Oct 22:27 .gitignore
-rwxr-xr-x  1 owenkelly  staff   558K 28 Oct 11:39 .pnp.cjs
drwxr-xr-x  7 owenkelly  staff   224B 28 Oct 11:39 .yarn
-rw-r--r--  1 owenkelly  staff   155B 26 Oct 22:27 .yarnrc.yml
-rw-r--r--  1 owenkelly  staff   318B 28 Oct 11:39 entrypoint.js
-rw-r--r--  1 owenkelly  staff   163B 26 Oct 22:27 jest.config.js
-rw-r--r--  1 owenkelly  staff   2.7K 27 Oct 15:52 lambda-project.integration.ts
-rw-r--r--  1 owenkelly  staff   210B 28 Oct 11:39 package.json
drwxr-xr-x  4 owenkelly  staff   128B 26 Oct 02:26 packages
-rw-r--r--  1 owenkelly  staff   1.0K 28 Oct 11:39 yarn.lock

# run with node
~/t/test ❯❯❯ node entrypoint.js
{"statusCode":200,"body":"\"function foobar(){} // MUGLIFIED\""}

# run with yarn
~/t/test ❯❯❯ yarn start
{"statusCode":200,"body":"\"function foobar(){} // MUGLIFIED\""}

# run with npm
~/t/test ❯❯❯ npm start

> start
> node entrypoint.js

{"statusCode":200,"body":"\"function foobar(){} // MUGLIFIED\""}

---

The only reason this wouldn't work is if you're shipping them a package not a node app. If they need to import your app into theirs then you need to treat your app a a library. In that case something like a private npm package, or other means to distribute the package will work. This is also independent of yarn, and you could continue to develop with yarn.

But you would be best to look at how open source packages are distributed instead of how nodejs apps are distributed. If you do need to share a package, the underlying package manager you developed it with shouldn't matter. You'll need to make all the other local packages available as well, if you import them.

yarn.build is for nodejs apps (for you to run).

yarn publish is for publishing packages (for someone else to run).

[AshishShanker]

Thanks for the additional pointers. I do need to send them a package that they need to include in their code as a dependency, not a node app. Ideally, what package manager I use shouldn't matter, but in case of Yarn 2/3, it doesn't let me put out a package consumable by npm either by putting out node_modules or putting together adequate machinery for the package to work in the npm world.

I had started looking at Rollup as a bundler but couldn't make it work quickly. So in the interest of time, I had to roll back to Yarn 1.

Yarn publish would be a great option for my use case if there could be some way to make it --dry-run where it would create the exact bundle it is expecting to publish to an npm registry, but without actually doing the publish. Unfortunately, yarn publish is a non-starter because it asks for npm registry credentials as the first step, instead of last step.

I have built npm packages for publishing in the past in a different context, and used Verdaccio for local testing, but in the current context, I haven't had the luxury of time to set that up, run yarn publish and then examine the tgz file it handed to npm.

Thanks a lot for all your help. I will continue pushing on making the Yarn 2/3 developer experience work. I'll let you know if I make any significant advances.