The executable files in the sysroot/sbin and sysroot/usr/bin directories of the multilib toolchain are random

[Cooper-Qu]

Problem Description

When building a multilib toolchain, there are multiple copies of libraries distributed in the sysroot/libxxx directory, but only one copy of executable files in the sysroot/sbin and sysroot/usr/bin directories. According to the rules for compiling glibc in the Makefile, glibc for different arch and abi combinations can be compiled in parallel. Therefore, the arch and abi of the executable files in sysroot/sbin and sysroot/usr/bin depend on the last glibc installed, it will overwrite the previously installed glibc executable files.

Potential Impact

When using a pre-built multilib toolchain and Yocto to build a Linux distribution, there may be check errors or program runtime errors due to ABI inconsistencies.

[TommyMurphyTM1234]

Are you saying that this issue is specific to the use of make linux -jn --enable-multilib?
So a workaround is to omit -jn?

[Cooper-Qu]

Are you saying that this issue is specific to the use of make linux -jn --enable-multilib?
So a workaround is to omit -jn?

The confusion caused by -jn is just a symptom, not the root cause of the problem. The real issue lies in the fact that the sysroot directory contains multiple copies of libraries, each with different arch and abi specifications, while there is only one copy of the glibc program in the sbin or usr/bin directory.
Currently, Linux's multilib tools offer four types of libraries: rv32imac-ilp32, rv32imafdc-ilp32d, rv64imac-lp64, and rv64imafdc-lp64d. The specific glibc program in the sbin or usr/bin directory will depend on the last installed glibc type during the toolchain construction process.
In most cross-compiling scenarios, sbin or usr/bin programs are not used, so there is no problem. However, when using pre-built toolchain to build a Linux distribution, these programs are checked and utilized. For example, if the target is rv64imafdc-lp64d while the sbin or usr/bin program is rv32imac-ilp32, an error will occur, requiring a non-multilib toolchain to be rebuilt to solve the issue rather than a one-size-fits-all toolchain.
One solution is to change the directory structure of sysroot, allowing multiple copies of sbin or usr/bin programs, but the details of how to store them need further consideration.

[Cooper-Qu]

Any update on this issue?

[TommyMurphyTM1234]

One solution is to change the directory structure of sysroot, allowing multiple copies of sbin or usr/bin programs, but the details of how to store them need further consideration.

Do you know if any other multi-arch toolchain (Arm perhaps?) face this issue and deal with it that way?

[TommyMurphyTM1234]

Perhaps this issue is what this project is attempting to address?

• https://github.com/arsv/riscv-toolchain

The toolchain is built with a single set of tools (ld, as, gcc etc)
and several independent sysroots, one per target. Each target gets properly
prefixed commands (riscv64-unknown-gnu-ld, riscv64-unknown-gnu-gcc etc)
pairing the tools with the right sysroot for the target.

Seems to me that this issue is one that probably needs to be raised/logged and dealt with in the relevant upstream project(s?) - e.g. gcc, binutils, gdb - where ongoing maintenance (including RISC-V related changes) is now done rather than here?

[TommyMurphyTM1234]

Perhaps this issue is what this project is attempting to address?

• https://github.com/arsv/riscv-toolchain

The toolchain is built with a single set of tools (ld, as, gcc etc)
and several independent sysroots, one per target. Each target gets properly
prefixed commands (riscv64-unknown-gnu-ld, riscv64-unknown-gnu-gcc etc)
pairing the tools with the right sysroot for the target.

I tried that repo but hit build problems:

• https://github.com/arsv/riscv-toolchain/issues

Seems to me that this issue is one that probably needs to be raised/logged and dealt with in the relevant upstream project(s?) - e.g. gcc, binutils, gdb - where ongoing maintenance (including RISC-V related changes) is now done rather than here?

I'm not sure what upstream projects and build scripts construct the sysroot but it's not the riscv-gnu-toolchain ones:

• https://github.com/riscv-collab/riscv-gnu-toolchain/blob/master/Makefile.in

I think that the relevant projects/build scripts would need to be identified and this issue logged upstream against them.

[TommyMurphyTM1234]

I tried crosstool-NG to see if that did something different with the organization of the multilib sysroot but it doesn't - presumably because whatever is done in that context is in the upstream gcc et. al project build scripts/makefiles?