Add yarn-berry3 and yarn-berry4 to fetch and use version 2 yarn.lock files #355053
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,4 @@ | ||
| Please be _very_ careful when changing the content of `yarn-berry-supported-archs.json`. | ||
| This file is used to instruct `yarn-berry` which packages to fetch and install into our | ||
| offline cache. If this file is changed, _ALL_ cached offline contents for `yarn-berry` | ||
| are invalidated and will need their hashes to be updated! |
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -14,6 +14,8 @@ | |
| prefetch-yarn-deps, | ||
| fixup-yarn-lock, | ||
| yarn, | ||
| yarn-berry_3, | ||
| yarn-berry_4, | ||
| makeSetupHook, | ||
| cacert, | ||
| callPackage, | ||
|
|
@@ -96,6 +98,9 @@ in | |
| }; | ||
| }; | ||
|
|
||
| # TODO: Move from `yarnLock` environment variable to the actual `src` argument for | ||
| # `yarn` and `yarn-berry`. This should work for both versions in place, because both | ||
| # versions will look for a yarn.lock file in the current working directory (`src`) | ||
| fetchYarnDeps = | ||
| let | ||
| f = | ||
|
|
@@ -104,6 +109,7 @@ in | |
| src ? null, | ||
| hash ? "", | ||
| sha256 ? "", | ||
| yarnVersion ? 1, | ||
| ... | ||
| }@args: | ||
| let | ||
|
|
@@ -132,22 +138,71 @@ in | |
| dontInstall = true; | ||
|
|
||
| nativeBuildInputs = [ | ||
| ( | ||
| { | ||
| "1" = prefetch-yarn-deps; | ||
| "3" = yarn-berry_3; | ||
| "4" = yarn-berry_4; | ||
| } | ||
| .${builtins.toString yarnVersion} | ||
| ) | ||
| cacert | ||
| ]; | ||
| GIT_SSL_CAINFO = "${cacert}/etc/ssl/certs/ca-bundle.crt"; | ||
| NODE_EXTRA_CA_CERTS = "${cacert}/etc/ssl/certs/ca-bundle.crt"; | ||
|
|
||
| configurePhase = lib.optionalString (yarnVersion > 1) '' | ||
| runHook preConfigure | ||
|
|
||
| export HOME="$NIX_BUILD_TOP" | ||
| export YARN_ENABLE_TELEMETRY=0 | ||
| yarnLock=''${yarnLock:=$PWD/yarn.lock} | ||
| DIR=$(dirname $yarnLock) | ||
| if [ $DIR = "${builtins.storeDir}" ]; then | ||
| echo "Only a yarn.lock file has been passed (e.g. yarnLock = ./yarn.lock;)" | ||
| echo "Please change this so a directory can be deduced (e.g. yarnLock = finalAttrs.src + "/yarn.lock";)" | ||
| exit 1 | ||
| fi | ||
| if [ ! -f $DIR/package.json ]; then | ||
| echo "No package.json has been found in the same directory as the yarn.lock file" | ||
| exit 1 | ||
| fi | ||
| if ! grep -q "__metadata:" $DIR/yarn.lock; then | ||
| echo "This yarn.lock file appears to be a version 1 yarn.lock file" | ||
| echo "Please use fetchYarnDeps with yarnVersion = 1" | ||
| exit 1 | ||
| fi | ||
| cp $DIR/yarn.lock $DIR/package.json $HOME/ | ||
| chmod +w $HOME/yarn.lock $HOME/package.json | ||
| yarn config set enableGlobalCache false | ||
| yarn config set cacheFolder $out | ||
| yarn config set supportedArchitectures --json "$(cat ${./yarn-berry-supported-archs.json})" | ||
|
|
||
| runHook postConfigure | ||
| ''; | ||
|
|
||
| buildPhase = | ||
| '' | ||
| runHook preBuild | ||
|
|
||
| mkdir -p $out | ||
| '' | ||
| + lib.optionalString (yarnVersion > 1) '' | ||
doronbehar marked this conversation as resolved.
Show resolved
Hide resolved
|
||
| yarn install --immutable --mode skip-build | ||
|
There was a problem hiding this comment. This will break the FODs almost instantly if Yarn changes anything in how they pull deps, and also is probably not reproducible between platforms when platform-specific dependencies are used. There was a problem hiding this comment. Valid point. This is partly why there are two yarn versions. Also this could be said for all external package managers, not just yarn-berry. I'm uncertain about your point about platform specific dependencies. I'll have to look into it. There was a problem hiding this comment. Platform specific dependencies should be made reproducible by explicitly setting There was a problem hiding this comment. @winterqt what is your reply on the arguments laid out above? Your "request for changes" is blocking the merge of the PR, and this is the only review comment left really open. |
||
| '' | ||
| + lib.optionalString (yarnVersion == 1) '' | ||
| yarnLock=''${yarnLock:=$PWD/yarn.lock} | ||
| if grep -q "__metadata:" $yarnLock; then | ||
| echo "This yarn.lock file does not appear to be a version 1 yarn.lock file" | ||
| echo "Please use fetchYarnDeps with eiher yarnVersion = 3 or yarnVersion = 4" | ||
| exit 1 | ||
| fi | ||
| (cd $out; prefetch-yarn-deps --verbose --builder $yarnLock) | ||
| '' | ||
| + '' | ||
| runHook postBuild | ||
| ''; | ||
|
|
||
| outputHashMode = "recursive"; | ||
| } | ||
| // hash_ | ||
|
|
@@ -174,6 +229,28 @@ in | |
| }; | ||
| } ./yarn-config-hook.sh; | ||
|
|
||
| yarnBerry3ConfigHook = makeSetupHook { | ||
|
There was a problem hiding this comment. Personally I like how in This way you can just: let
yarn = yarn-berry_3;
in
{
yarnOfflineCache = yarn.fetchDeps { ... };
nativeBuildInputs = [ yarn yarn.configHook ];
}Just an observation, not necessarily a blocker. There was a problem hiding this comment. I like this idea. However this would mean a major refactoring of how the There was a problem hiding this comment.
Small correction:
Furthermore, this suggestion can ease the refactoring: We could make There was a problem hiding this comment. I mean we could easily just make these as aliases. But agree that this couls be done in another PR. There was a problem hiding this comment. Ok, lets target that for another PR |
||
| name = "yarn-config-hook"; | ||
| propagatedBuildInputs = [ yarn-berry_3 ]; | ||
| substitutions = { | ||
| yarn = lib.getExe yarn-berry_3; | ||
| }; | ||
| meta = { | ||
| description = "Install nodejs dependencies from an offline yarn berry cache (version 3)"; | ||
| }; | ||
| } ./yarn-berry-config-hook.sh; | ||
|
|
||
| yarnBerry4ConfigHook = makeSetupHook { | ||
| name = "yarn-config-hook"; | ||
| propagatedBuildInputs = [ yarn-berry_4 ]; | ||
| substitutions = { | ||
| yarn = lib.getExe yarn-berry_4; | ||
| }; | ||
| meta = { | ||
| description = "Install nodejs dependencies from an offline yarn berry cache (version 4)"; | ||
| }; | ||
| } ./yarn-berry-config-hook.sh; | ||
|
|
||
| yarnBuildHook = makeSetupHook { | ||
| name = "yarn-build-hook"; | ||
| meta = { | ||
|
|
||
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,7 @@ | ||
| { | ||
| "name": "yarn-testing", | ||
| "packageManager": "[email protected]", | ||
| "dependencies": { | ||
| "lit-html": "^3.2.1" | ||
| } | ||
| } |
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,30 @@ | ||
| # This file is generated by running "yarn install" inside your project. | ||
| # Manual changes might be lost - proceed with caution! | ||
|
|
||
| __metadata: | ||
| version: 6 | ||
| cacheKey: 10c0 | ||
|
|
||
| "@types/trusted-types@npm:^2.0.2": | ||
| version: 2.0.7 | ||
| resolution: "@types/trusted-types@npm:2.0.7" | ||
| checksum: 4c4855f10de7c6c135e0d32ce462419d8abbbc33713b31d294596c0cc34ae1fa6112a2f9da729c8f7a20707782b0d69da3b1f8df6645b0366d08825ca1522e0c | ||
| languageName: node | ||
| linkType: hard | ||
|
|
||
| "lit-html@npm:^3.2.1": | ||
| version: 3.2.1 | ||
| resolution: "lit-html@npm:3.2.1" | ||
| dependencies: | ||
| "@types/trusted-types": "npm:^2.0.2" | ||
| checksum: 31c02df2148bf9a73545570cbe57aae01c4de1d9b44060b6ff13641837d38e39e6b1abcf92e13882cc84f5fee37605ed79602b91ad479728549014462808118e | ||
| languageName: node | ||
| linkType: hard | ||
|
|
||
| "yarn-testing@workspace:.": | ||
| version: 0.0.0-use.local | ||
| resolution: "yarn-testing@workspace:." | ||
| dependencies: | ||
| lit-html: ^3.2.1 | ||
| languageName: unknown | ||
| linkType: soft |
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,7 @@ | ||
| { | ||
| "name": "yarn-testing", | ||
| "packageManager": "[email protected]", | ||
| "dependencies": { | ||
| "lit-html": "^3.2.1" | ||
| } | ||
| } |
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,30 @@ | ||
| # This file is generated by running "yarn install" inside your project. | ||
| # Manual changes might be lost - proceed with caution! | ||
|
|
||
| __metadata: | ||
| version: 8 | ||
| cacheKey: 10c0 | ||
|
|
||
| "@types/trusted-types@npm:^2.0.2": | ||
| version: 2.0.7 | ||
| resolution: "@types/trusted-types@npm:2.0.7" | ||
| checksum: 10c0/4c4855f10de7c6c135e0d32ce462419d8abbbc33713b31d294596c0cc34ae1fa6112a2f9da729c8f7a20707782b0d69da3b1f8df6645b0366d08825ca1522e0c | ||
| languageName: node | ||
| linkType: hard | ||
|
|
||
| "lit-html@npm:^3.2.1": | ||
| version: 3.2.1 | ||
| resolution: "lit-html@npm:3.2.1" | ||
| dependencies: | ||
| "@types/trusted-types": "npm:^2.0.2" | ||
| checksum: 10c0/31c02df2148bf9a73545570cbe57aae01c4de1d9b44060b6ff13641837d38e39e6b1abcf92e13882cc84f5fee37605ed79602b91ad479728549014462808118e | ||
| languageName: node | ||
| linkType: hard | ||
|
|
||
| "yarn-testing@workspace:.": | ||
| version: 0.0.0-use.local | ||
| resolution: "yarn-testing@workspace:." | ||
| dependencies: | ||
| lit-html: "npm:^3.2.1" | ||
| languageName: unknown | ||
| linkType: soft |
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,31 @@ | ||
| yarnBerryConfigHook(){ | ||
| echo "Executing yarnBerryConfigHook" | ||
|
|
||
| # Use a constant HOME directory, don't use $NIX_BUILD_TOP because of | ||
| # https://github.com/NixOS/nixpkgs/issues/189691 | ||
| mkdir -p "$TMP/home" | ||
| export HOME="$TMP/home" | ||
| if [[ -n "$yarnOfflineCache" ]]; then | ||
| offlineCache="$yarnOfflineCache" | ||
| fi | ||
| if [[ -z "$offlineCache" ]]; then | ||
| echo yarnConfigHook: No yarnOfflineCache or offlineCache were defined\! >&2 | ||
| exit 2 | ||
| fi | ||
|
|
||
| export YARN_ENABLE_TELEMETRY=0 | ||
| yarn config set enableGlobalCache false | ||
| yarn config set enableScripts false | ||
| yarn config set cacheFolder "$offlineCache" | ||
|
|
||
| yarn install --immutable --immutable-cache | ||
|
|
||
| # TODO: Check if this is really needed | ||
| patchShebangs node_modules | ||
|
|
||
| echo "finished yarnConfigHook" | ||
| } | ||
|
|
||
| if [[ -z "${dontYarnInstallDeps-}" ]]; then | ||
| postConfigureHooks+=(yarnBerryConfigHook) | ||
| fi |
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,25 @@ | ||
| { | ||
|
There was a problem hiding this comment. We should add a comment somewhere so that people avoid touching this file/are very careful when doing it. Ideally we'd be able to use JSON with comments... There was a problem hiding this comment. This file means that we can never expand our platform support without mass FOD breaks across the tree? That seems very bad. There are already platforms you can boot NixOS on that would be ruled out by this file. At the very least we should try to list every value already used in npm here so that it’s relatively safe to expand in future? There was a problem hiding this comment. Since JSON and comments don't agree (and we also read the file directly), I added a There was a problem hiding this comment.
yes
We did add all the relevant platforms, cpu's and libc's (see https://yarnpkg.com/configuration/yarnrc#supportedArchitectures) that are available. The only one missing is There was a problem hiding this comment. If it’s everything Yarn supports then I guess it’s not so bad – assuming we will be able to add additional platforms if Yarn does support more in future without invalidating every FOD? I do think we should include Windows. You can use Nixpkgs to cross‐build a lot of stuff for Windows today, and people are working on native bring‐up. What’s the harm? There was a problem hiding this comment. I thought the idea is that packages can’t include any platforms other than the ones Yarn supports, so no existing FOD should change as long as we keep up with Yarn? Or can packages on npm reference other platforms and Yarn just can’t support fetching them? There was a problem hiding this comment. done. There was a problem hiding this comment.
Yes, I think your right. My guess is that adding another platform or compiler will a huge undertaking anyway, so maybe they will publish yet another big version upgrade, which would make it easier for us (e.g. There was a problem hiding this comment. Unfortunately, I think I was correct here:
The npm docs imply that the fields are quite freeform; e.g. the The correct solution here would be to fetch everything independent of platform – theoretically Yarn could add Because we’re adding fundamental packaging infrastructure here that is part of our public interface and will surely be relied upon by out‐of‐tree packages, and changing FOD hashes are very difficult to spot and debug, I don’t think we’ll have the option of just breaking it in the future, unless we can show that the backwards incompatibility would be entirely theoretical in the existing public corpus. There was a problem hiding this comment.
Actually, it is not a limitation of the configuration, as it can be adapted to include everything node supports. https://yarnpkg.com/configuration/yarnrc#supportedArchitectures says:
which states:
So we can include all of those for our
Yes, writing our own fetcher could be a solution. This will be quite a task to parse the
There seems to be an option (
Totally agree. This is why we added versions here. Not only because upstream has two supported versions (3 and 4), but also because their |
||
| "os": [ | ||
| "darwin", | ||
| "linux", | ||
| "win32" | ||
| ], | ||
| "cpu": [ | ||
| "arm", | ||
| "arm64", | ||
| "ia32", | ||
| "x64", | ||
| "loong64", | ||
| "mips", | ||
| "mipsel", | ||
| "ppc", | ||
| "ppc64", | ||
| "riscv64", | ||
| "s390", | ||
| "s390x" | ||
| ], | ||
| "libc": [ | ||
| "glibc", | ||
| "musl" | ||
| ] | ||
| } | ||
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Or maybe the comment should go here. (Or we can just explicitly pin versions right here.)
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I added it at the package level, because this is where we would do our update