Upgrade to Yarn Modern v4 #6446
Conversation
| - run: yarn install --production=false --frozen-lockfile --ignore-scripts | ||
| - run: yarn install --immutable |
There was a problem hiding this comment.
--frozen-lockfile is replaced by --immutable. This means the install will abort with an error exit code if the lockfile was to be modified by installing third party libraries.
There was a problem hiding this comment.
--ignore-scripts is replaced by enableScripts: false in .yarnrc.yml. This disables post-installation scripts from third party libraries.
There was a problem hiding this comment.
--production=false was originally used to ensure devDeps were not installed during ci-builds. However, it's not a flag that applies to Yarn Modern, and Yarn will not install any package listed in devDependencies if the NODE_ENV environment variable is set to production, which is the case in many of our build scripts anyway.
There was a problem hiding this comment.
Turns out that --immutable always installs all dependencies. An install can't be limited to production dependencies without changing the lockfile: yarnpkg/berry#2253 (comment)
--production=false should have ensured that devDeps were installed, though. Otherwise the subsequent build would fail, without the build tools.
| # Based on https://github.com/github/gitignore/blob/master/Node.gitignore | ||
| # Yarn https://yarnpkg.com/getting-started/qa#which-files-should-be-gitignored |
There was a problem hiding this comment.
This root .gitignore file was generated years ago and contained a lot of "extras" that don't apply to FEM. I've cleaned it up and included the recommended Yarn files.
| @@ -1,9 +1,27 @@ | |||
| # Based on https://github.com/github/gitignore/blob/master/Node.gitignore | |||
|
|
|||
| # https://yarnpkg.com/getting-started/qa#which-files-should-be-gitignored | |||
There was a problem hiding this comment.
Same as comment.
| @@ -0,0 +1,3 @@ | |||
| nodeLinker: node-modules | |||
| yarnPath: .yarn/releases/yarn-4.5.1.cjs | |||
| enableScripts: false | |||
There was a problem hiding this comment.
Docs for this are here: https://yarnpkg.com/configuration/yarnrc#enableScripts. We don't want to run post-install scripts from third party libraries, and this config replaces --ignore-scripts from Yarn Classic.
| [](https://travis-ci.com/zooniverse/front-end-monorepo) | ||
| [](https://coveralls.io/github/zooniverse/front-end-monorepo?branch=master) | ||
| [](https://pullreminders.com?ref=badge) | ||
|
|
||
| [](https://lernajs.io/) | ||
| [](https://github.com/zooniverse/front-end-monorepo/blob/master/LICENSE.md) | ||
|  | ||
|
|
||
| ️Take a look at [our roadmap](https://trello.com/b/yg0r4dG5/front-end-rebuild-roadmap)! 🛣️ | ||
|
|
There was a problem hiding this comment.
The deleted links no longer exist.
| @@ -39,7 +34,7 @@ Node, git, and yarn can be installed through [homebrew](https://brew.sh/) on Mac | |||
|
|
|||
| This monorepo is managed with [Yarn Workspaces](https://yarnpkg.com/lang/en/docs/workspaces/). | |||
|
|
|||
| Yarn Workspaces allow us to maintain package modularity for javascript projects that have interdependency. Organizationally, they allows us to track issues, pull requests, and progress for all related packages in one place. | |||
| Yarn Workspaces allow us to maintain package modularity for javascript projects that have interdependency. Organizationally, they allows us to track issues, pull requests, and progress for all related packages in one place. See documentation for [Corepack](https://yarnpkg.com/corepack), too. | |||
There was a problem hiding this comment.
You might need to use Corepack with Yarn Modern.
| printf 'Installing dependencies...\n' | ||
| yarn install --immutable | ||
| printf '\n' |
There was a problem hiding this comment.
The same concept as #6393. Install once, and then build the rest of the packages. This brings the bootstrap file up to date with the CI build scripts.
| "node": ">=16.18" | ||
| "node": ">=20.5" |
There was a problem hiding this comment.
Updated to match package.json in the individual workspaces. I don't think this one matters much because the ci builds reference the files in .github, and each package has its own package.json.
| "devDependencies": { | ||
| "@babel/cli": "^7.25.9", |
There was a problem hiding this comment.
Had to install @babel/cli in a couple of our libraries because they use the babel command in the build scripts.
|
Review (attempt) currently in progress. PSA: you may need to run "yarn install" first.When I switch to this branch and try running (Full version)❗ It appears that I first need to run Additional notes:
Additional debugging notes
|
PR Review (ongoing)Manual testing update
|
There was a problem hiding this comment.
PR Review
Packages: (root) / (everything)
This PR updates the monorepo to use Yarn v4.
- This is purely a dev-side/build-side update. No features are added/removed/changed, and the development & build processes themselves should look basically the same.
Code changes look fine, as far as I can tell, and basic testing indicates that most major packages build & run fine locally. There's a new caveat about running yarn install, but see the 'Status' section at the end for full details.
Testing (CLI)
Basic pass/fail criteria for this review is "can I run some of FEM's major packages on the local dev server?"
Base setup:
- yarn installed on local zsh CLI: 4.5.1
- yarn corepack enabled?: yes
Testing (app-project):
- Switch to branch:
git checkout yarn-modern - Initialise: on root, run 🆕
yarn install, andyarn bootstrap - Run app-project locally:
cd packages/app-project,yarn dev - Open test project: https://local.zooniverse.org:3000/projects/darkeshard/test-project-2021
- Choose a random workflow, classify, and submit.
✅ Tests look good. app-project appears to work on local.
Testing (Docker)
Basic pass/fail criteria for this review is the same as above.
- Initialise: on root, run
docker-compose build - Run app-project locally:
cd packages/app-project,docker-compose up - Open test project: https://local.zooniverse.org:3000/projects/darkeshard/test-project-2021
❓ Tests look odd - the server runs, but on HTTP (https is blocked), and all I'm getting are 404 pages. Which means the local dev server is running and the app-project code is there, but there's probably something wonky in a config, or my test methods need revision. (I rarely, if ever, use Docker for local dev. I'm only testing this to be thorough.)
🤷 I'm marking this as a "OK, this is fine, I guess" since I'm not too worried about Docker, but we may need to revisit this if another developer uses docker as their main development method.
Testing (CLI, Tabula Rasa Edition)
Basic pass/fail criteria for this review is "can I run some of FEM's major packages on the local dev server, if I install the Front End Monorepo from scratch ?"
Tabula Rasa Policy:
- delete existing
/front-end-monorepofolder, or move it elsewhere. - Install a fresh copy of FEM:
git clone [email protected]:zooniverse/front-end-monorepo.git
Testing (app-project):
- Proceed with the same steps as Testing (CLI) above.
✅ Tests still look good. This tabula rasa test confirms the necessity of the 🆕 yarn install step during initialisation.
Status
Changes LGTM, but I'd like to highlight a new "required step" on the dev side that seems to have appeared as a result of the Yarn 4 update.
❗ 🆕 yarn install is now required before running yarn scripts
- Information for Zooniverse devs: when switching to this branch for the first time (and after this PR is merged), you may need to run
yarn install, at least once, before runningyarn bootstrap.- Once installed, you can run yarn bootstrap with no issue, as many times as you like.
- You also need to run
yarn installonce after runningyarn panic. - You know we need to run yarn install when: you attempt to run yarn panic or yarn bootstrap, but instead see the error message: "Usage Error: Couldn't find the node_modules state file - running an install might help (findPackageLocation)"
- Actions to be taken: The README needs to be updated with this information. (@goplayoutside3 : this can be part of this PR, or I can make the doc update a separate PR - whatever seems cleaner.)
Documentation Update
### Debugging the production release
Make sure you have pulled the latest production version.
"""sh <== change these to tildes
git pull --tags -f
""" <== change these to tildes
Check out the latest production release.
"""sh <== change these to tildes
git checkout production-release
""" <== change these to tildes
// NEW
You'll need to run yarn install _once_ for the next yarn script to work. Otherwise, you may encounter the error message, _"Usage Error: Couldn't find the node_modules state file - running an install might help (findPackageLocation)"_
// NEW
"""sh <== change these to tildes
yarn install
""" <== change these to tildes
Run the bootstrap script to build all the libraries and apps. You can use `bootstrap:es6` here for a faster build if you don't want to run the tests.
"""sh <== change these to tildes
yarn bootstrap
""" <== change these to tildes
### With Node and yarn
Alternatively, you can install Node 20 and yarn and build the monorepo packages.
"""sh <== change these to tildes
git clone [email protected]:zooniverse/front-end-monorepo.git
cd front-end-monorepo
yarn install <== NEW
yarn bootstrap
""" <== change these to tildes
The `bootstrap` script will install the dependencies and build any local packages used as dependencies.
// NEW
`yarn install` is required to be run _once_ for any yarn script to work. If you see an error message like _"Usage Error: Couldn't find the node_modules state file - running an install might help (findPackageLocation)",_ it means you haven't run `yarn install` yet.
The Docker build is a production build, so staging projects will 404, if that helps. front-end-monorepo/docker-compose.yml Lines 17 to 21 in b4008d2 |
|
Thanks! That's helpful to know. I want to continue a discussion here that was started in #6465 (that PR is about to be merged) @eatyourgreens you mentioned the size of the FEM image.
In this PR,
😞 Might need to look into yarn workspaces focus. |
|
I think Yarn 4 will automatically ignore dev dependencies, as long as The build is a multi-stage build, with two stages:
|
|
I could also be totally wrong about Yarn 4 production installs and |
I think you're right about this because I remember reading about it when first creating this PR, but now I can't find it documented 🤔 Will continue to investigate... |
|
I'm also happy to answer any other questions that come up about the Docker builds, (if I know the answers.) |
|
Ugh using Docker Desktop to build an image of this branch is even larger 🫠 This screenshot is from And strangely enough, with |
|
This looks more like what I'd expect from
jimodonnell@Jims-MacBook-Pro front-end-monorepo % docker compose run --rm prod-shell
[+] Creating 1/0
✔ Network front-end-monorepo_default Created 0.0s
/usr/src # du -sh *
790.3M node_modules
4.0K package.json
641.7M packages
824.0K yarn.lock
/usr/src # |
|
I'm not sure why Yarn 4 is creating 2GB builds. I don't think those would break deploys, but they'd definitely slow down uploads/downloads to GitHub Container Repository. I don't know if GHCR has any usage limits either. I vaguely remember that, about a year ago, the monorepo ran into errors in GitHub Actions because the production images got really large. |
|
@goplayoutside3 I've opened #6469 to try and make it easier to debug the production build. One thing I noticed from the old docker compose setup: front-end-monorepo/docker-compose.yml Lines 4 to 5 in 958e806 |
|
Here's a recent tutorial for multi-stage Docker builds, including examples for Next.js. https://labs.iximiuz.com/tutorials/docker-multi-stage-builds |
|
#6469 is now included in this PR, and I can successfully deploy this branch. By using Docker Desktop and this branch locally, I can see that Storybook is the biggest contributor to both The prod build is still way larger than the dev build, and I don't want to merge this Yarn upgrade until I understand why that is. I can list the Dockerfile commands that contribute the largest file transfers, but I've reached my Docker x Yarn skills limit. I'll need to research and come back to this PR.
|
This is 344MB for the project app, 312MB for the root app and ~15MB for all the other libraries.
The I found a useful post, which goes through how to deploy standalone Next.js builds with Docker. It would need to be modified to work with a monorepo, but should give smaller images. |
|
With Yarn 4, you can also look for this in the build output (which tells you that the production build is installing dev dependencies 😞 .) YN0000: ┌ Fetch step
=> => # ➤ YN0013: │ 2413 packages were added to the project (+ 866.81 MiB).
=> => # ➤ YN0000: └ Completed in 29s 875ms |
|
After you've deployed this branch, does the image run?
jimodonnell@Jims-MacBook-Pro front-end-monorepo % docker-compose up
[+] Running 5/5
✔ Network front-end-monorepo_default Created 0.1s
✔ Container front-end-monorepo-fe-project-1 Created 0.5s
✔ Container front-end-monorepo-prod-shell-1 Created 0.5s
✔ Container front-end-monorepo-dev-shell-1 Created 0.5s
✔ Container front-end-monorepo-fe-root-1 Created 0.5s
Attaching to dev-shell-1, fe-project-1, fe-root-1, prod-shell-1
prod-shell-1 exited with code 0
dev-shell-1 exited with code 0
fe-root-1 | Usage Error: Couldn't find the node_modules state file - running an install might help (findPackageLocation)
fe-root-1 |
fe-root-1 | $ yarn run [--inspect] [--inspect-brk] [-T,--top-level] [-B,--binaries-only] [--require #0] <scriptName> ...
fe-project-1 | Usage Error: Couldn't find the node_modules state file - running an install might help (findPackageLocation)
fe-project-1 |
fe-project-1 | $ yarn run [--inspect] [--inspect-brk] [-T,--top-level] [-B,--binaries-only] [--require #0] <scriptName> ...
fe-root-1 exited with code 1
fe-project-1 exited with code 1
jimodonnell@Jims-MacBook-Pro front-end-monorepo % I can also reproduce that error by manualling running jimodonnell@Jims-MacBook-Pro front-end-monorepo % docker-compose run --rm prod-shell
/usr/src # ls
node_modules package.json packages yarn.lock
/usr/src # yarn start
Usage Error: Couldn't find the node_modules state file - running an install might help (findPackageLocation)
$ yarn run [--inspect] [--inspect-brk] [-T,--top-level] [-B,--binaries-only] [--require #0] <scriptName> ...
/usr/src # EDIT: seems to be my local setup. |
Thanks I'll take a look! |
|
Zach and I chatted about this upgrade today in the office. We discussed reasons why there's unexpected Docker image size differences between the master branch and this branch, as well as why the One of the possible causes is that @eatyourgreens it would be helpful to know if you have some legacy knowledge about why the Dockerfile RUN commands include the following flags: |
|
Those are mounting a build cache. See #2309 This should be the docs for the Basically, those flags should be caching This Stack Overflow answer has some examples of Buildkit's |
I was, in fact, completely wrong. Yarn 4 has no concept of an immutable production install. The first reply here has a detailed (though annoying) explanation. This feature request talks about how production installs might work in Yarn 4. |
|
Here's another explanation of cache mounts, taken from #2309. https://vsupalov.com/docker-buildkit-features/#using-a-cache-mount |
You can check this by looking at the logs for a build on this branch. https://github.com/zooniverse/front-end-monorepo/actions/runs/12019431323/job/33505979000 The yarn install for the production image is at line 518. 
|
|
~~TBH, #2351 is also useful for understanding how the Docker builds are using BuildKit, and caching. |
|
We're running free software engineering training for Oxford Physics in Jan/Feb 2025, including a day on Docker and containers. If anyone from Zooniverse is interested, there should be a full list on https://train.rse.ox.ac.uk. |
|
I've rewritten the Dockerfile, in #6515, to use the base image suggestion from Node.js 20, Yarn 4, and Next.js on Docker (and fix #6513.) |
|
@goplayoutside3 here's the documentation for Yarn 1 uses a global cache, but I think Yarn 4 handles caching in a completely different manner, with local caching being the default, not global. The default location of the Yarn 1 global cache depends on the OS. This is a monorepo build with Yarn 1.22, on Alpine Linux: jimodonnell@Jims-MBP-2 front-end-monorepo % docker compose run --rm dev-shell
[+] Creating 1/0
✔ Network front-end-monorepo_default Created 0.0s
/usr/src $ yarn cache dir
/home/node/.cache/yarn/v6On MacOS, the Yarn cache dir will probably be in your user directory soemwhere. jimodonnell@Jims-MBP-2 front-end-monorepo % yarn cache dir
/Users/jimodonnell/Library/Caches/Yarn/v6With Yarn 4, the default cache directory is |
|
Here's a link to the BuildKit docs, via zooniverse/talk-api#271. BuildKit was experimental when the original PRs were written, but it's now the default Docker build system. https://docs.docker.com/build/buildkit/ I'm not 100% sure if BuildKit cache mounts are supported by GitHub Actions. Zach probably knows more about this than me. If not, it might be worth reaching out to Cam for help. You can see another example of cache mounts in the Talk API here: |
|
Found this in the Docker docs:
https://docs.docker.com/build/ci/github-actions/cache/#cache-mounts So I would guess those cache mounts aren't being used in CI ie. Yarn is making ~2400 HTTP requests to download packages for every GitHub Actions build? My best guess for the increased image size with Yarn 4 would be downloaded packages in There's also a note in the Yarn 4 docs about committing the cache to Git, for reproducible builds.
https://yarnpkg.com/features/caching If the Yarn cache were checked out with the Git repo, then the build won't need cache mounts. |
|
Just following up on this to say that the details of our Docker training, including how to read and write Docker files, are up now: https://train.rse.ox.ac.uk/event/29 Oxford Zooniverse team members can register for free via https://www.rse.ox.ac.uk/training |
Linked Issue and/or Talk Post
Yarn Migration docs: https://yarnpkg.com/migration/guide
Toward: #5970
Closes: #1470
Closes: #6391
Describe your changes
--frozen-lockfilecommand to--immutableper the migration guide.enableScripts: falsein the yarn config. This replaces the legacy--ignore-scriptsflag..gitignoreand remove some unrelated rules from it.How to Review
There's a lot of changes required to use Yarn Modern, so please see my comments below on individual file changes.
I deployed this branch to the
fe-project-branchand verified a successful deploy and I could see a project at that url. I might overwritefe-project-branchwith a different PR, but the successful deploy GH Action is linked here.To review:
yarn panicandyarn bootstrap.Checklist
PR Creator - Please cater the checklist to fit the review needed for your code changes.
PR Reviewer - Use the checklist during your review. Each point should be checkmarked or discussed before PR approval.
General
yarn panic && yarn bootstrapordocker-compose up --buildand FEM works as expectedGeneral UX
Example Staging Project: i-fancy-cats
Maintenance