Nixops / python2.7-cffi fails to build on unstable

[aij]

Describe the bug

Can't build nixops on nixos-unstable due to build error in python2.7-cffi
A clear and concise description of what the bug is.

To Reproduce
Steps to reproduce the behavior:

1. git checkout nixos-unstable
2. nix-build . -A pkgs.nixops

Alternately:
2. nix-build . -A pkgs.python2Packages.cffi

Alternately, if you are already on the latest nixos-unstable

1. nix-shell -p nixops

Expected behavior
Build should succeed.

Screenshots

testing/cffi0/test_verify2.py::test_arithmetic_in_cdef
  /build/cffi-1.14.4/cffi/vengine_cpy.py:192: UserWarning: reimporting '_cffi__x8f5c8caex68dfdccc' might overwrite older definitions
    % (self.verifier.get_module_name()))

testing/cffi0/test_verify2.py::test_passing_large_list
  /build/cffi-1.14.4/cffi/vengine_cpy.py:192: UserWarning: reimporting '_cffi__x2a9aab6xd8ac7f7f' might overwrite older definitions
    % (self.verifier.get_module_name()))

-- Docs: https://docs.pytest.org/en/latest/warnings.html
= 32 failed, 1908 passed, 96 skipped, 1 deselected, 4 xfailed, 200 warnings in 538.72 seconds =
builder for '/nix/store/7591wry4fp0m6da1jxxr1993rxkpxisp-python2.7-cffi-1.14.4.drv' failed with exit code 1
error: build of '/nix/store/7591wry4fp0m6da1jxxr1993rxkpxisp-python2.7-cffi-1.14.4.drv' on 'ssh://bob@m5' failed: builder for '/nix/store/7591wry4fp0m6da1jxxr1993rxkpxisp-python2.7-cffi-1.14.4.drv' failed with exit code 1
builder for '/nix/store/7591wry4fp0m6da1jxxr1993rxkpxisp-python2.7-cffi-1.14.4.drv' failed with exit code 1
cannot build derivation '/nix/store/7gfy8pcl9nzmfyyjqiff5hpfnnld5z9j-python2.7-cryptography-3.3.1.drv': 1 dependencies couldn't be built
cannot build derivation '/nix/store/hiqwi24i9vk3vzfxw0awnfq7xz3fidy3-python2.7-PyJWT-1.7.1.drv': 1 dependencies couldn't be built
cannot build derivation '/nix/store/rwwc11fdnwdwpglv7ryhc6akmc2djycl-python2.7-urllib3-1.26.2.drv': 1 dependencies couldn't be built
cannot build derivation '/nix/store/yjf8909746qci0i2snwhb149n84zy36f-python2.7-adal-1.2.5.drv': 1 dependencies couldn't be built
cannot build derivation '/nix/store/zdnwxyny3ryy98wk3q9vddddzvzha5g8-python2.7-botocore-1.19.51.drv': 1 dependencies couldn't be built
cannot build derivation '/nix/store/4fw99l9i1h6nkl9var5jzpsap2z32anc-python2.7-requests-2.25.0.drv': 1 dependencies couldn't be built
cannot build derivation '/nix/store/vhm9skck8fj9fzlkybx48asiw0yf9win-nixops-1.7.drv': 1 dependencies couldn't be built
error: build of '/nix/store/vhm9skck8fj9fzlkybx48asiw0yf9win-nixops-1.7.drv' failed

real    9m9.912s
user    0m0.523s
sys     0m0.158s

Additional context

Given that Python2 is deprecated, but Nixops 2.0 is not ready, yet, I'm not sure whether it's worth trying to fix python2Modules.cffi or work around it somehow until nixops can be upgraded.

Related issues:

• Python2 migration Convert remaining python2 applications over to python3 #101964
• Nixops testing Add NixOps test #97166
• Nixops 2.0 Release 2.0 old tracking issue nixops#1242

Notify maintainers

cffi: @domenkozar @LnL7

nixops: @AmineChikhaoui @eelco @rob @domenkozar

Metadata
Please run nix-shell -p nix-info --run "nix-info -m" and paste the result.

• system: "x86_64-linux"
• host os: Linux 5.4.84, NixOS, 21.03.git.6ea2fd15d88 (Okapi)
• multi-user?: yes
• sandbox: yes
• version: nix-env (Nix) 2.3.10
• channels(root): "nixos-20.03pre208171.2e8fc97dbfa"
• nixpkgs: /etc/nixos/aij/unstable

Maintainer information:

# a list of nixpkgs attributes affected by the problem
attribute: 
    - pkgs.nixops
    - pkgs.python2Packages.cffi
# a list of nixos modules affected by the problem
module:

[sloane-shark]

@aij you don't happen to have a workaround available do you?

[aij]

My current "workaround" is to use an older version of nixpkgs, but that's not sustainable for long 56bb1b0 was the last nixos-unstable build that worked, ed4d6a8 was the last nixos-unstable-small that worked.

This error appears to be caused by the gcc9 -> gcc10 transition, per git bisect run nix-build . -A pkgs.python2Packages.cffi

git bisect start
# good: [29f9a107883a28f4330943689cea8debbb3f1713] python3Packages.cffi: 1.14.3 -> 1.14.4
git bisect good 29f9a107883a28f4330943689cea8debbb3f1713
# bad: [499792889db010e29213963544b0de7f5af66fe0] Merge pull request #108198 from lblasc/luajit-2020-12-29
git bisect bad 499792889db010e29213963544b0de7f5af66fe0
# good: [ed4d6a8eff74b3b6e372778b64d9a90e6c8c93aa] linux-hardened: Switch to 5.10
git bisect good ed4d6a8eff74b3b6e372778b64d9a90e6c8c93aa
# bad: [22daf1c8ff31dc203bb71c2920fc43fe03af7075] mp4v2: fix build on gcc10
git bisect bad 22daf1c8ff31dc203bb71c2920fc43fe03af7075
# good: [d87de5dd7d19ae873bc8ba265da317912b795519] Merge staging-next into staging
git bisect good d87de5dd7d19ae873bc8ba265da317912b795519
# good: [7f8fcb20844ca6bae59e518fca6d616a292c94ef] Merge pull request #102723 from r-ryantm/auto-update/poppler-glib
git bisect good 7f8fcb20844ca6bae59e518fca6d616a292c94ef
# bad: [b1b3ca79073ba82b551906fd11cc0ca086f81894] gcc: make -fcommon default on gcc10
git bisect bad b1b3ca79073ba82b551906fd11cc0ca086f81894
# good: [ed85c4f3c87c32cebfc74506773b1604cbaf2167] Merge staging-next into staging
git bisect good ed85c4f3c87c32cebfc74506773b1604cbaf2167
# good: [0fb4e8c4c3429399431ba90bf89f51422f3f80a3] gdk-pixbuf: enable debug info
git bisect good 0fb4e8c4c3429399431ba90bf89f51422f3f80a3
# good: [3a3b682793042046233421a60deb71e4d0750bb6] Merge pull request #103674 from chkno/lcms-src-fix
git bisect good 3a3b682793042046233421a60deb71e4d0750bb6
# good: [2d9193f9f032c52594d90eb64b8c4ffa8c615354] Merge staging-next into staging
git bisect good 2d9193f9f032c52594d90eb64b8c4ffa8c615354
# bad: [af802654b91e150fdaa8e32c29a0eefbdf953597] use gcc10 instead of gcc9
git bisect bad af802654b91e150fdaa8e32c29a0eefbdf953597
# first bad commit: [af802654b91e150fdaa8e32c29a0eefbdf953597] use gcc10 instead of gcc9

[B4dM4n]

The issue is caused by the recent change to gcc10. The python2 function ctypes.util.find_library uses gcc to search for available libraries. During it's tests, the cffi module tries to load the c library.
With gcc9, it finds libc.so.6 from the python executable RUNPATH, which is fine.
With gcc10, it finds libc.so from the python executable RUNPATH, but this file is broken:

$ cat /nix/store/rqrklqsvw4ydpcg5kdcvn506fhcbqxk2-glibc-2.32-10/lib/libc.so
/* GNU ld script
   Use the shared library, but some functions are only in
   the static library, so try that secondarily.  */
OUTPUT_FORMAT(elf64-x86-64)
GROUP ( /nix/store/rqrklqsvw4ydpcg5kdcvn506fhcbqxk2-glibc-2.32-10/lib/libc.so.6 /nix/store/rqrklqsvw4ydpcg5kdcvn506fhcbqxk2-glibc-2.32-10/lib/libc_nonshared.a  AS_NEEDED ( /nix/store/rqrklqsvw4ydpcg5kdcvn506fhcbqxk2-glibc-2.32-10/lib/ld-linux-x86-64.so.2 ) )

There are 2 other libraries in glibc which are linker scripts:

$ file /nix/store/rqrklqsvw4ydpcg5kdcvn506fhcbqxk2-glibc-2.32-10/lib/* | grep ASCII
/nix/store/rqrklqsvw4ydpcg5kdcvn506fhcbqxk2-glibc-2.32-10/lib/libc.so: ASCII text
/nix/store/rqrklqsvw4ydpcg5kdcvn506fhcbqxk2-glibc-2.32-10/lib/libgcc_s.so: ASCII text
/nix/store/rqrklqsvw4ydpcg5kdcvn506fhcbqxk2-glibc-2.32-10/lib/libm.so: ASCII text

This should fixed in the glibc build.

[B4dM4n]

Looking at the glibc build made me doubt about what needs to be changed. The build explicitly notes that libgcc_s.so is a linker script and Debian handles it in exactly the same way.

nixpkgs/pkgs/development/libraries/glibc/default.nix

Lines 76 to 77 in 187af93

	# the .so It used to be a symlink, but now it is a script
	cp -a ${stdenv.cc.cc}/lib/libgcc_s.so $out/lib/libgcc_s.so

So this looks like this needs to be fixed in Python-2.7, which is EOL 😔

CC @FRidh

[FRidh]

this issue is resolved on master 349585e