Issues lurking with `gobject-introspection` #196

ocaisa · 2022-11-08T13:11:17Z

With something that was using Meson I was seeing

/bin/bash: /lib64/libc.so.6: version `GLIBC_2.33' not found (required by /cvmfs/pilot.eessi-hpc.org/versions/2021.12/compat/linux/x86_64/usr/lib64/libtinfo
.so.6)

which I'm guessing meant it was picking the host ld and not the EasyBuild wrapper. Looking at the configure step I saw

C compiler for the host machine: gcc (gcc 10.3.0 "gcc (GCC) 10.3.0")
C linker for the host machine: gcc ld.gold 2.37

which would imply it is using the ld.gold used by gcc (which seems right since it reports 2.37, the version in the prefix, not 2.30 the version in the host).

Anyway, it seemed like a good idea to set the linker directly (as per the docs) with

preconfigopts = "export CC_LD=$(which ld) &&"

but actually this throws an error

== 2022-11-08 12:59:53,619 build_log.py:169 ERROR EasyBuild crashed with an error (at easybuild/software/EasyBuild/4.6.0/lib/python3.9/site-packages/easybuild/base/exceptions.py:124 in __init__): cmd "export CC_LD=$(which ld) && meson --prefix /project/def-sponsor00/easybuild/software/at-spi2-core/2.40.2-GCCcore-10.3.0 --libdir lib  /project/60005/easybuild/build/atspi2core/2.40.2/GCCcore-10.3.0/at-spi2-core-2.40.2/" exited with exit code 1 and output:
The Meson build system
Version: 0.58.0
Source dir: /project/60005/easybuild/build/atspi2core/2.40.2/GCCcore-10.3.0/at-spi2-core-2.40.2
Build dir: /project/60005/easybuild/build/atspi2core/2.40.2/GCCcore-10.3.0/easybuild_obj
Build type: native build
Project name: at-spi2-core
Project version: 2.40.2

../at-spi2-core-2.40.2/meson.build:1:0: ERROR: Unsupported linker, only bfd, gold, and lld are supported, not /tmp/eb-nf5k_pyz/tmp1a3n724j/rpath_wrappers/ld_wrapper/ld.

The text was updated successfully, but these errors were encountered:

ocaisa · 2022-11-08T13:17:35Z

Urgh, this may not be an easy fix mesonbuild/meson#6662

ocaisa · 2022-11-08T13:22:39Z

I don't see how we can give a full path to a linker: https://github.com/mesonbuild/meson/blob/master/mesonbuild/compilers/mixins/gnu.py#L311-L316

ocaisa · 2022-11-08T13:31:51Z

Ah, you are meant to set the linker flavour with the environment variables...not sure if that means the problem is something else or not

ocaisa · 2022-11-08T14:00:41Z

This may be a false flag, I think introspection is up to something, disabling that with

meson --prefix /project/def-sponsor00/easybuild/software/at-spi2-core/2.40.2-GCCcore-10.3.0 -Dintrospection=no --libdir lib

allowed the package to install

ocaisa · 2022-11-08T14:01:40Z

It is worth noting though that EasyBuild does not currently create compiler wrappers for ld.bfd and ld.gold

eli-schwartz · 2022-11-09T04:57:49Z

Ah, you are meant to set the linker flavour with the environment variables...not sure if that means the problem is something else or not

The environment variable tries to map -fuse-ld values, which expects to see a flavor.

If you want to set a linker that GCC doesn't specifically know about, you probably want gcc -B/path/to/ld_wrapper/ (set via LDFLAGS).

ocaisa · 2022-12-01T16:45:35Z

The full error with gobject-introspection is

[49/62] Generating Atspi-2.0.gir with a custom command
FAILED: atspi/Atspi-2.0.gir 
/project/def-sponsor00/easybuild/software/GObject-Introspection/1.68.0-GCCcore-10.3.0/bin/g-ir-scanner --no-libtool --namespace=Atspi --nsversion=2.0 --warn-all --output atspi/Atspi-2.0.gir --c-include=atspi/atspi.h --pkg-export=atspi-2 --quiet -I/project/60005/easybuild/build/atspi2core/2.40.2/GCCcore-10.3.0/at-spi2-core-2.40.2/atspi -I/project/60005/easybuild/build/atspi2core/2.40.2/GCCcore-10.3.0/easybuild_obj/atspi -I./. -I../at-spi2-core-2.40.2/. -I./registryd -I../at-spi2-core-2.40.2/registryd -I./. -I../at-spi2-core-2.40.2/. --filelist=/project/60005/easybuild/build/atspi2core/2.40.2/GCCcore-10.3.0/easybuild_obj/atspi/libatspi.so.0.0.1.p/Atspi_2.0_gir_filelist --include=DBus-1.0 --include=GLib-2.0 --include=GObject-2.0 --symbol-prefix=atspi --identifier-prefix=Atspi --cflags-begin -D_POSIX_C_SOURCE=200809L -D_DEFAULT_SOURCE -I./. -I../at-spi2-core-2.40.2/. -I./registryd -I../at-spi2-core-2.40.2/registryd -I/cvmfs/pilot.eessi-hpc.org/versions/2021.12/compat/linux/x86_64/usr/include/dbus-1.0 -I/cvmfs/pilot.eessi-hpc.org/versions/2021.12/compat/linux/x86_64/usr/lib64/dbus-1.0/include -I/project/def-sponsor00/easybuild/software/GLib/2.68.2-GCCcore-10.3.0/include/glib-2.0 -I/project/def-sponsor00/easybuild/software/GLib/2.68.2-GCCcore-10.3.0/lib/glib-2.0/include -I/cvmfs/pilot.eessi-hpc.org/versions/2021.12/compat/linux/x86_64/usr/include '-DG_LOG_DOMAIN="dbind"' -I/project/def-sponsor00/easybuild/software/GObject-Introspection/1.68.0-GCCcore-10.3.0/include/gobject-introspection-1.0 -I/project/def-sponsor00/easybuild/software/X11/20210518-GCCcore-10.3.0/include -I/project/def-sponsor00/easybuild/software/GLib/2.68.2-GCCcore-10.3.0/include -I/project/def-sponsor00/easybuild/software/GObject-Introspection/1.68.0-GCCcore-10.3.0/include --cflags-end -I./. -I../at-spi2-core-2.40.2/. --add-include-path=/project/def-sponsor00/easybuild/software/GObject-Introspection/1.68.0-GCCcore-10.3.0/share/gir-1.0 --add-include-path=./. --add-include-path=../at-spi2-core-2.40.2/. -L./atspi --library atspi -L/project/def-sponsor00/easybuild/software/X11/20210518-GCCcore-10.3.0/lib64 -L/project/def-sponsor00/easybuild/software/X11/20210518-GCCcore-10.3.0/lib -L/project/def-sponsor00/easybuild/software/GLib/2.68.2-GCCcore-10.3.0/lib64 -L/project/def-sponsor00/easybuild/software/GLib/2.68.2-GCCcore-10.3.0/lib -L/project/def-sponsor00/easybuild/software/GObject-Introspection/1.68.0-GCCcore-10.3.0/lib64 -L/project/def-sponsor00/easybuild/software/GObject-Introspection/1.68.0-GCCcore-10.3.0/lib -L/project/def-sponsor00/easybuild/software/Meson/0.58.0-GCCcore-10.3.0/lib64 -L/project/def-sponsor00/easybuild/software/Meson/0.58.0-GCCcore-10.3.0/lib -L/cvmfs/pilot.eessi-hpc.org/versions/2021.12/software/linux/x86_64/amd/zen2/software/GCCcore/10.3.0/lib64 -L/cvmfs/pilot.eessi-hpc.org/versions/2021.12/software/linux/x86_64/amd/zen2/software/GCCcore/10.3.0/lib -L/cvmfs/pilot.eessi-hpc.org/versions/2021.12/compat/linux/x86_64/usr/lib64 --extra-library=dbus-1 -L/project/def-sponsor00/easybuild/software/GLib/2.68.2-GCCcore-10.3.0/lib --extra-library=gobject-2.0 --extra-library=glib-2.0 -L/project/def-sponsor00/easybuild/software/X11/20210518-GCCcore-10.3.0/lib --extra-library=X11 --extra-library=Xtst --extra-library=Xi -L/project/def-sponsor00/easybuild/software/GObject-Introspection/1.68.0-GCCcore-10.3.0/lib --extra-library=girepository-1.0 --sources-top-dirs /project/60005/easybuild/build/atspi2core/2.40.2/GCCcore-10.3.0/at-spi2-core-2.40.2/subprojects/ --sources-top-dirs /project/60005/easybuild/build/atspi2core/2.40.2/GCCcore-10.3.0/easybuild_obj/subprojects/
/bin/bash: /lib64/libc.so.6: version `GLIBC_2.33' not found (required by /cvmfs/pilot.eessi-hpc.org/versions/2021.12/compat/linux/x86_64/usr/lib64/libtinfo.so.6)
linking of temporary binary failed: Command '['gcc', '-o', '/project/60005/easybuild/build/atspi2core/2.40.2/GCCcore-10.3.0/easybuild_obj/tmp-introspect07cjsv_q/Atspi-2.0', '-I/project/def-sponsor00/easybuild/software/X11/20210518-GCCcore-10.3.0/include', '-I/project/def-sponsor00/easybuild/software/GLib/2.68.2-GCCcore-10.3.0/include', '-I/project/def-sponsor00/easybuild/software/GObject-Introspection/1.68.0-GCCcore-10.3.0/include', '-O2', '-ftree-vectorize', '-march=native', '-fno-math-errno', '/project/60005/easybuild/build/atspi2core/2.40.2/GCCcore-10.3.0/easybuild_obj/tmp-introspect07cjsv_q/Atspi-2.0.o', '-L.', '-Wl,-rpath,.', '-Wl,--no-as-needed', '-L./atspi', '-L/project/def-sponsor00/easybuild/software/X11/20210518-GCCcore-10.3.0/lib64', '-Wl,-rpath,/project/def-sponsor00/easybuild/software/X11/20210518-GCCcore-10.3.0/lib64', '-L/project/def-sponsor00/easybuild/software/X11/20210518-GCCcore-10.3.0/lib', '-Wl,-rpath,/project/def-sponsor00/easybuild/software/X11/20210518-GCCcore-10.3.0/lib', '-L/project/def-sponsor00/easybuild/software/GLib/2.68.2-GCCcore-10.3.0/lib64', '-Wl,-rpath,/project/def-sponsor00/easybuild/software/GLib/2.68.2-GCCcore-10.3.0/lib64', '-L/project/def-sponsor00/easybuild/software/GLib/2.68.2-GCCcore-10.3.0/lib', '-Wl,-rpath,/project/def-sponsor00/easybuild/software/GLib/2.68.2-GCCcore-10.3.0/lib', '-L/project/def-sponsor00/easybuild/software/GObject-Introspection/1.68.0-GCCcore-10.3.0/lib64', '-Wl,-rpath,/project/def-sponsor00/easybuild/software/GObject-Introspection/1.68.0-GCCcore-10.3.0/lib64', '-L/project/def-sponsor00/easybuild/software/GObject-Introspection/1.68.0-GCCcore-10.3.0/lib', '-Wl,-rpath,/project/def-sponsor00/easybuild/software/GObject-Introspection/1.68.0-GCCcore-10.3.0/lib', '-L/project/def-sponsor00/easybuild/software/Meson/0.58.0-GCCcore-10.3.0/lib64', '-Wl,-rpath,/project/def-sponsor00/easybuild/software/Meson/0.58.0-GCCcore-10.3.0/lib64', '-L/project/def-sponsor00/easybuild/software/Meson/0.58.0-GCCcore-10.3.0/lib', '-Wl,-rpath,/project/def-sponsor00/easybuild/software/Meson/0.58.0-GCCcore-10.3.0/lib', '-L/cvmfs/pilot.eessi-hpc.org/versions/2021.12/software/linux/x86_64/amd/zen2/software/GCCcore/10.3.0/lib64', '-Wl,-rpath,/cvmfs/pilot.eessi-hpc.org/versions/2021.12/software/linux/x86_64/amd/zen2/software/GCCcore/10.3.0/lib64', '-L/cvmfs/pilot.eessi-hpc.org/versions/2021.12/software/linux/x86_64/amd/zen2/software/GCCcore/10.3.0/lib', '-Wl,-rpath,/cvmfs/pilot.eessi-hpc.org/versions/2021.12/software/linux/x86_64/amd/zen2/software/GCCcore/10.3.0/lib', '-L/cvmfs/pilot.eessi-hpc.org/versions/2021.12/compat/linux/x86_64/usr/lib64', '-Wl,-rpath,/cvmfs/pilot.eessi-hpc.org/versions/2021.12/compat/linux/x86_64/usr/lib64', '-L/project/def-sponsor00/easybuild/software/GLib/2.68.2-GCCcore-10.3.0/lib', '-Wl,-rpath,/project/def-sponsor00/easybuild/software/GLib/2.68.2-GCCcore-10.3.0/lib', '-L/project/def-sponsor00/easybuild/software/X11/20210518-GCCcore-10.3.0/lib', '-Wl,-rpath,/project/def-sponsor00/easybuild/software/X11/20210518-GCCcore-10.3.0/lib', '-L/project/def-sponsor00/easybuild/software/GObject-Introspection/1.68.0-GCCcore-10.3.0/lib', '-Wl,-rpath,/project/def-sponsor00/easybuild/software/GObject-Introspection/1.68.0-GCCcore-10.3.0/lib', '-latspi', '-ldbus-1', '-lgobject-2.0', '-lglib-2.0', '-lX11', '-lXtst', '-lXi', '-lgirepository-1.0', '-L/project/def-sponsor00/easybuild/software/GLib/2.68.2-GCCcore-10.3.0/lib', '-lgio-2.0', '-lgobject-2.0', '-Wl,--export-dynamic', '-lgmodule-2.0', '-pthread', '-lglib-2.0', '-L/project/def-sponsor00/easybuild/software/X11/20210518-GCCcore-10.3.0/lib64', '-L/project/def-sponsor00/easybuild/software/X11/20210518-GCCcore-10.3.0/lib', '-L/project/def-sponsor00/easybuild/software/GLib/2.68.2-GCCcore-10.3.0/lib64', '-L/project/def-sponsor00/easybuild/software/GLib/2.68.2-GCCcore-10.3.0/lib', '-L/project/def-sponsor00/easybuild/software/GObject-Introspection/1.68.0-GCCcore-10.3.0/lib64', '-L/project/def-sponsor00/easybuild/software/GObject-Introspection/1.68.0-GCCcore-10.3.0/lib', '-L/project/def-sponsor00/easybuild/software/Meson/0.58.0-GCCcore-10.3.0/lib64', '-L/project/def-sponsor00/easybuild/software/Meson/0.58.0-GCCcore-10.3.0/lib', '-L/cvmfs/pilot.eessi-hpc.org/versions/2021.12/software/linux/x86_64/amd/zen2/software/GCCcore/10.3.0/lib64', '-L/cvmfs/pilot.eessi-hpc.org/versions/2021.12/software/linux/x86_64/amd/zen2/software/GCCcore/10.3.0/lib']' returned non-zero exit status 1.

specifically, we probably need to take a look inside the python script g-ir-scanner and see where things are going wrong.

ocaisa · 2022-12-02T15:07:12Z

I think this has something to do with the use of /bin/bash in EasyBuild which is allowing us to escape the prefix environment (and means we start to use the system linker)

boegel · 2022-12-02T15:16:44Z

@ocaisa Can you check if you see the same problem when using python3 -O -m easybuild.main ... instead of eb ...?

ocaisa · 2022-12-02T15:34:04Z

@boegel It didn't help, and I checked the bash shell at the time the build command is called:

== 2022-12-02 15:31:57,205 build_log.py:169 ERROR EasyBuild crashed with an error (at easybuild/software/EasyBuild/4.6.0/lib/python3.9/site-packages/easyb
uild/base/exceptions.py:124 in __init__): cmd "echo $0 && ninja -j 2 " exited with exit code 1 and output:
/cvmfs/pilot.eessi-hpc.org/versions/2021.12/compat/linux/x86_64/bin/bash
...

so I'm not sure where exactly things go wrong.

boegel · 2024-01-02T15:42:15Z

see also #428 (comment), which suggests that ldd is the culprit here

As discussing during today's support meeting, we should check whether the hardcoded /bin/bash in ldd is deliberate; it seems like a bug to me in Gentoo Prefix?

The problem here seems to be that LD_LIBRARY_PATH is being set to the location of libraries in the compat layer, which breaks the host's /bin/bash...

Any ideas here @amadio?

Possible workaround could be:

runtime patch to replace call to ldd to inject unset LD_LIBRARY_PATH;
call ldd via the bash of the compat layer (effectively skipping the hardcoded /bin/bash);

amadio · 2024-01-22T09:25:29Z

What is setting LD_LIBRARY_PATH? Gentoo prefix should always be used with an empty LD_LIBRARY_PATH. If something in the prefix is setting it, it should be considered a bug. Just using /bin/bash should not be a problem, although enforcing <prefix>/bin/bash is a good idea in any case.

amadio · 2024-01-22T09:25:58Z

Sorry for the late reply, I was on vacations during the end of year break.

bedroge · 2024-01-23T15:07:44Z

What is setting LD_LIBRARY_PATH? Gentoo prefix should always be used with an empty LD_LIBRARY_PATH. If something in the prefix is setting it, it should be considered a bug. Just using /bin/bash should not be a problem, although enforcing <prefix>/bin/bash is a good idea in any case.

The issue pops up at the installation of at-spi2-core, which calls the g-ir-scanner tool from GObject-Introspection, and this tool tries to compile some stuff. At some point, I think, it tries to check if the generated binary can find its dependencies, and for this it sets $LD_LIBRARY_PATH to some known directories and runs ldd on the binary. The $LD_LIBRARY_PATH includes $EPREFIX/usr/lib64, which breaks /bin/bash in some environments, e.g. in our build container. And since the ldd in the Prefix installation has a /bin/bash shebang, it means that ldd breaks as well. We now worked around it by basically telling g-ir-scanner to use another (dummy) variable instead of $LD_LIBRARY_PATH, but still we're wondering why the shebang of ldd is not eprefixified (as that also would solve the problem)?

amadio · 2024-01-23T16:05:04Z

There are a few places in glibc where the shebang is set and that's not prefixified. You will probably observe similar behavior when calling system("command") in code with the prefix glibc. That's something that has been on my radar for a while, but I've never had time to fix:

glibc $ git grep 'define.*bin/sh'
manual/process.texi:#define SHELL "/bin/sh"
sysdeps/generic/paths.h:#define _PATH_BSHELL    "/bin/sh"
sysdeps/posix/system.c:#define  SHELL_PATH      "/bin/sh"       /* Path of the shell.  */
sysdeps/unix/sysv/linux/paths.h:#define _PATH_BSHELL    "/bin/sh"

It only ever showed to be a problem in some very specific testing setups for me, so it was not that high priority...

bedroge · 2024-01-23T16:10:05Z

There are a few places in glibc where the shebang is set and that's not prefixified. You will probably observe similar behavior when calling system("command") in code with the prefix glibc. That's something that has been on my radar for a while, but I've never had time to fix:
glibc $ git grep 'define.*bin/sh'
manual/process.texi:#define SHELL "/bin/sh"
sysdeps/generic/paths.h:#define _PATH_BSHELL    "/bin/sh"
sysdeps/posix/system.c:#define  SHELL_PATH      "/bin/sh"       /* Path of the shell.  */
sysdeps/unix/sysv/linux/paths.h:#define _PATH_BSHELL    "/bin/sh"
It only ever showed to be a problem in some very specific testing setups for me, so it was not that high priority...

So, just to be sure, in principle it would be safe if we would just patch our ldd during/after the Prefix installation and use $EPREFIX/bin/bash in the shebang? I.e. there's no good reason for using /bin/bash there?

bedroge · 2024-07-19T07:39:08Z

I'm reopening this issue, as we're now seeing similar issues for Pango (#644) and Paraver (#570). We should really fix this in GObject-Introspection itself, maybe using something like the proposed fix in #570 (comment).

bedroge · 2024-07-19T08:27:02Z

Also GTK3/3.24.39-GCCcore-13.2.0 has the same issue, see #644 (comment).

ocaisa · 2024-07-19T08:57:39Z

I like your fix at the end of #570 (comment), it's simple and can be EESSI-specific if we do it via a hook

ocaisa · 2024-07-19T09:01:12Z

To make it easier to fix via sed in a hook, it may be better to target https://gitlab.gnome.org/GNOME/gobject-introspection/-/blob/1.76.1/giscanner/ccompiler.py?ref_type=tags#L224 (only include LD_LIBRARY_PATH in the list if it is set in the environment)

ocaisa · 2024-07-19T09:04:39Z

Maybe we can test this in a single PR? A hook to patch gobject-introspection, remove all the hooks that work around this issue and rebuild all the associated packages

bedroge · 2024-07-19T09:28:14Z

Yep, good idea, I'll try to do that.

ocaisa · 2024-07-22T11:58:42Z

This has been resolved via #646 (and #647 for Zen4)

boegel · 2024-08-26T08:50:46Z

Also fixed for A64FX now via #683

ocaisa changed the title ~~Meson does not like the ld wrapper used by EasyBuild~~ Issues lurking with gobject-introspection Dec 1, 2022

ocaisa mentioned this issue Feb 16, 2023

add OpenFOAM v9 to EESSI pilot 2021.12 #195

Closed

10 tasks

ocaisa mentioned this issue Aug 9, 2023

Avoid hardcoding /bin/bash easybuilders/easybuild-framework#4320

Closed

ocaisa mentioned this issue Dec 21, 2023

{2023.06}[foss/2023a] R-bundle-CRAN 2023.12 #428

Merged

bedroge mentioned this issue Jan 16, 2024

{2023.06}[GCCcore/12.3.0] at-spi2-core 2.49.91 #454

Merged

boegel closed this as completed in #454 Jan 16, 2024

bedroge reopened this Jul 19, 2024

bedroge mentioned this issue Jul 19, 2024

prevent GObject-Introspection from setting EB-filtered environment variables #646

Merged

ocaisa closed this as completed in #646 Jul 19, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Issues lurking with `gobject-introspection` #196

Issues lurking with `gobject-introspection` #196

ocaisa commented Nov 8, 2022

ocaisa commented Nov 8, 2022

ocaisa commented Nov 8, 2022

ocaisa commented Nov 8, 2022

ocaisa commented Nov 8, 2022

ocaisa commented Nov 8, 2022

eli-schwartz commented Nov 9, 2022

ocaisa commented Dec 1, 2022

ocaisa commented Dec 2, 2022

boegel commented Dec 2, 2022

ocaisa commented Dec 2, 2022

boegel commented Jan 2, 2024 •

edited

Loading

amadio commented Jan 22, 2024

amadio commented Jan 22, 2024

bedroge commented Jan 23, 2024 •

edited

Loading

amadio commented Jan 23, 2024

bedroge commented Jan 23, 2024

bedroge commented Jul 19, 2024

bedroge commented Jul 19, 2024

ocaisa commented Jul 19, 2024

ocaisa commented Jul 19, 2024 •

edited

Loading

ocaisa commented Jul 19, 2024

bedroge commented Jul 19, 2024

ocaisa commented Jul 22, 2024

boegel commented Aug 26, 2024

Issues lurking with gobject-introspection #196

Issues lurking with gobject-introspection #196

Comments

ocaisa commented Nov 8, 2022

ocaisa commented Nov 8, 2022

ocaisa commented Nov 8, 2022

ocaisa commented Nov 8, 2022

ocaisa commented Nov 8, 2022

ocaisa commented Nov 8, 2022

eli-schwartz commented Nov 9, 2022

ocaisa commented Dec 1, 2022

ocaisa commented Dec 2, 2022

boegel commented Dec 2, 2022

ocaisa commented Dec 2, 2022

boegel commented Jan 2, 2024 • edited Loading

amadio commented Jan 22, 2024

amadio commented Jan 22, 2024

bedroge commented Jan 23, 2024 • edited Loading

amadio commented Jan 23, 2024

bedroge commented Jan 23, 2024

bedroge commented Jul 19, 2024

bedroge commented Jul 19, 2024

ocaisa commented Jul 19, 2024

ocaisa commented Jul 19, 2024 • edited Loading

ocaisa commented Jul 19, 2024

bedroge commented Jul 19, 2024

ocaisa commented Jul 22, 2024

boegel commented Aug 26, 2024

Issues lurking with `gobject-introspection` #196

Issues lurking with `gobject-introspection` #196

boegel commented Jan 2, 2024 •

edited

Loading

bedroge commented Jan 23, 2024 •

edited

Loading

ocaisa commented Jul 19, 2024 •

edited

Loading