Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix non-deterministic inherit printing #11123

Merged
merged 1 commit into from
Jul 20, 2024

Conversation

infinisil
Copy link
Member

@infinisil infinisil commented Jul 17, 2024

Motivation

In very rare cases (I had about 7 cases out of 32200 files!), the order of how inherit-from bindings are printed when using nix-instantiate --parse gets messed up.

The cause of this seems to be because the std::map the bindings are placed in is keyed on a pointer, which then uses an implementation-defined strict total order.

The fix here is to key the bindings on their displacement instead, which maintains the same order as they appear in the file.

Unfortunately I wasn't able to make a reproducible test for this in the source, there's something about the local environment that makes it unreproducible for me.

However I was able to make a reproducible test in a Nix build on a Nix version from a very recent master:

nix build github:infinisil/non-det-nix-parsing-repro

which shows the difference (prettified for easy viewing):

diff --git a/a.nix b/b.nix
index 6f0637998..a5fd9ebfa 100644
--- a/a.nix
+++ b/b.nix
@@ -1,6 +1,6 @@
 ({ crun, emptyDirectory, haskell, haskellPackages, lib, makeWrapper, stdenv }: (
-  let inherit (lib) makeBinPath;
-  inherit ((haskell).lib.compose) addBuildTools appendConfigureFlags justStaticExecutables overrideCabal;
+  let inherit ((haskell).lib.compose) addBuildTools appendConfigureFlags justStaticExecutables overrideCabal;
+  inherit (lib) makeBinPath;
   bundledBins = ((lib).optional (stdenv).isLinux crun);
   overrides = (old: {
     hercules-ci-agent = (overrideCabal (o: { buildTarget = "lib:hercules-ci-agent hercules-ci-agent-unit-tests"; configureFlags = ((o).configureFlags or ([ ]) ++ [ (("--bindir=" + emptyDirectory + "/hercules-ci-built-without-binaries/no-bin")) ]); enableSharedExecutables = false; isExecutable = false; isLibrary = true; postInstall = ""; }) (old).hercules-ci-agent);

Context

I've found this when I tried to verify that nix-instantiate --parse doesn't change upon applying a formatter on Nixpkgs. Turns out it does change at least because of this.

Another unrelated instance of this is #11120

Priorities and Process

This work is sponsored by Antithesis

Add 👍 to pull requests you find important.

The Nix maintainer team uses a GitHub project board to schedule and track reviews.

@infinisil infinisil requested a review from edolstra as a code owner July 17, 2024 00:25
@infinisil infinisil changed the title Fix non-deterministic parser printing Fix non-deterministic inherit parser printing Jul 17, 2024
Copy link
Member

@roberth roberth left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

❤️

@@ -105,7 +105,7 @@ void ExprAttrs::showBindings(const SymbolTable & symbols, std::ostream & str) co
}
for (const auto & [from, syms] : inheritsFrom) {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This could be considered the root cause.
Indexing by a pointer is not necessarily wrong, but enumerating items ordered by addresses is non-deterministic.

src/libexpr/nixexpr.cc Show resolved Hide resolved
@roberth roberth changed the title Fix non-deterministic inherit parser printing Fix non-deterministic inherit printing Jul 17, 2024
In _very_ rare cases (I had about 7 cases out of 32200 files!),
the order of how inherit-from bindings are printed when using
`nix-instantiate --parse` gets messed up.

The cause of this seems to be because the std::map the bindings are
placed in is keyed on a _pointer_, which then uses an
[implementation-defined strict total order](https://en.cppreference.com/w/cpp/language/operator_comparison#Pointer_total_order).

The fix here is to key the bindings on their displacement instead,
which maintains the same order as they appear in the file.

Unfortunately I wasn't able to make a reproducible test for this in the
source, there's something about the local environment that makes it
unreproducible for me.

However I was able to make a reproducible test in a Nix build on a Nix
version from a very recent master:

    nix build github:infinisil/non-det-nix-parsing-repro

Co-authored-by: Robert Hensing <[email protected]>
@infinisil infinisil force-pushed the fix-non-det-parser-printing branch from a77cb49 to a6dccae Compare July 18, 2024 22:42
@infinisil
Copy link
Member Author

Applied and rebased with your suggestion

@roberth roberth merged commit 56757e1 into NixOS:master Jul 20, 2024
11 checks passed
@roberth
Copy link
Member

roberth commented Jul 20, 2024

Thanks!

@infinisil infinisil deleted the fix-non-det-parser-printing branch July 21, 2024 01:23
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants