Formatting unquotes interpolation

[mmarx]

Consider foo.nix:

''
  O=''\'''${A}'
''

Formatting changes the semantics of this, turning it into an interpolation:

''
  O='''${A}'
''

foo.nix: Parses differently after formatting. This is a bug in nixfmt. Please report it at https://github.com/NixOS/nixfmt

Before formatting:
Term (IndentedString (Ann [] [[TextPart "O='''${A}'"],[]] Nothing))

After formatting:
Term (IndentedString (Ann [] [[TextPart "O='''",Interpolation Term (Token (Ann [] (Identifier "A") Nothing)),TextPart "'"],[]] Nothing))

[piegamesde]

This language is cursed beyond repair, and I see no hope in fixing this without a proper formal specification of its string syntax and semantics.

[nixos-discourse]

This issue has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/formatting-team-meeting-2024-06-25/47661/1

[infinisil]

Bumping this higher, since it's a correctness bug and is kind of blocking for NixOS/nixpkgs#322537. @dasJ looked into it in todays meeting, I'll copy the notes here too:

• We discovered the relevant bits from the Nix lexer here
• These should be relatively directly portable to Haskell (current code around here)
• @Sereja313 recommended this tutorial on Megaparsec by Mark Karpov

[dasJ]

More details from the debugging session: It's probably the parser. The example string can be minimized to ''\'''${A}. Nixfmt parses this as:

Term
    ( IndentedString
        ( Ann []
            [
                [ TextPart "'''${A}" ]
            , []
            ] Nothing
        )
    )

while nix parses it as

"'\${A}\n"

It probably dies because the first 4 characters (''\') get converted to ' and the rest gets converted to \${A} (at least this is what nix does) while nixfmt converts the second part to ''${A}. Nixfmt doesn't escape the first single quote, resulting in 3 single quotes which does not escape the ${.