What is crossdev
----------------

crossdev is a cross-compiler environment generator for Gentoo.

It is useful for various purposes:

- build cross-compiler toolchain for an operating system
- build cross-compiler toolchain for embedded target (bare metal)
- cross-compile whole Gentoo on a new (or existing) target
- cross-compile your favourite tool for every target out there
  just to make sure it still compiles and works. Countless bugs
  were found and fixed like that :)

Crossdev nano HOWTO
-------------------

So you want to cross-compile a Gentoo package (say, busybox to s390x):

  # crossdev -t s390x-unknown-linux-gnu
  # (optional) ARCH=s390 PORTAGE_CONFIGROOT=/usr/s390x-unknown-linux-gnu eselect profile set default/linux/s390/17.0/s390x
  # USE=static s390x-unknown-linux-gnu-emerge -v1 busybox
  # file /usr/s390x-unknown-linux-gnu/bin/busybox
  /usr/s390x-unknown-linux-gnu/bin/busybox: ELF 64-bit MSB executable, IBM S/390, version 1 (GNU/Linux), statically linked, for GNU/Linux 3.2.0, stripped

Done!

You can use qemu-user to run this binary:

  $ qemu-s390x -L /usr/s390x-unknown-linux-gnu/ /usr/s390x-unknown-linux-gnu/bin/busybox uname -m
  s390x

or even chroot to the /usr/s390x-unknown-linux-gnu directory!

https://wiki.gentoo.org/wiki/Crossdev_qemu-static-user-chroot

Supported platforms
-------------------

Cross-compilation is fairly well supported to linux targets.
Windows is not too broken either.

Be prepared for rough corners. This doc will try to help you
understand what crossdev does and does not do.

A few examples of targets that worked today (produce running
executables or kernels if applies):

 aarch64-gentoo-linux-musl
 aarch64-unknown-linux-gnu
 alpha-unknown-linux-gnu
 arm-none-eabi
 armv5tel-softfloat-linux-gnueabi
 armv6zk-unknown-linux-musleabihf
 armv7a-unknown-linux-gnueabihf
 avr
 hppa-unknown-linux-gnu
 hppa2.0-unknown-linux-gnu
 hppa64-unknown-linux-gnu
 i686-pc-gnu
 i686-w64-mingw32
 ia64-unknown-linux-gnu
 loongarch64-unknown-linux-gnu
 m68k-unknown-linux-gnu
 mips-unknown-linux-gnu
 mips64-unknown-linux-gnu
 mips64el-unknown-linux-gnu
 mipsel-unknown-linux-gnu
 mmix
 msp430-elf
 nios2-unknown-linux-gnu
 or1k-linux-musl
 powerpc-unknown-linux-gnu
 powerpc64-unknown-linux-gnu
 powerpc64le-unknown-linux-gnu
 s390-unknown-linux-gnu
 s390x-unknown-linux-gnu
 sh4-unknown-linux-gnu
 sparc-unknown-linux-gnu
 sparc64-unknown-linux-gnu
 vax-unknown-linux-gnu
 x86_64-HEAD-linux-gnu
 x86_64-UNREG-linux-gnu
 x86_64-pc-linux-gnu
 x86_64-w64-mingw32
 xtensa-esp32-elf

A few more targets are likely to Just Work.
And many more can be made to work with a litle touch.

How crossdev works (high-level overview)
----------------------------------------

crossdev is a tiny shell wrapper around emerge tool. The wrapper
overrides a few variables to aim emerge at another target.

Crossdev leverages the following features of portage (and ::gentoo
ebulds):

- ability to override ROOT=/usr/<target> to install cross-compiled
  packages into a new root on a filesystem to avoid cluttering host.

- ability to override PORTAGE_CONFIGROOT=/usr/<target> to untangle
  from host's /etc/portage/ configuration. Namely crossdev populates
      /usr/<target>/etc/portage/
  with defaults suitable for cross-compiling (ARCH, KERNEL, ELIBC
  variables and so on). You can change all of them.

- set CBUILD/CHOST/CTARGET variables accordingly to force build
  system into cross-compiling mode. For autotools-based system
  it means running ./configure script using following options:
    ./configure --build=${CBUILD} --host=${CHOST} --target=${CTARGET} ...

If toolchains were simple programs crossdev would be a one-liner script:

  ARCH=...    \
  CBUILD=...  \
  CHOST=...   \
  CTARGET=... \
  ROOT=...    \
      emerge "$@"

Unfortunately today's toolchains have cycles/loops in their build-time dependencies:

- cross-compiler itself normally needs a libc built for <target> because
  libc defines various aspects of userland ABI and features provided.
- and libc is written in C and thus needs a cross-compiler to be built for
  <target>.

That's where crossdev comes in useful. It unties this vicious compiler<->libc
circle by carefully running the following emerge commands (assume s390x
example).

Here is what crossdev actually does:

1. create an overlay with new ebuilds (symlinks to existing ebuilds)
2. build cross-binutils:
   $ emerge cross-s390x-unknown-linux-gnu/binutils
3. Install system headers (kernel headers and libc headers):
   $ USE="headers-only" emerge cross-s390x-unknown-linux-gnu/linux-headers
   $ USE="headers-only" emerge cross-s390x-unknown-linux-gnu/glibc
4. Build minimal GCC without libc support (not able to link final
   executables yet)
   $ USE="-*" emerge cross-s390x-unknown-linux-gnu/gcc
5. Build complete libc (gcc will need crt.o files)
   $ emerge cross-s390x-unknown-linux-gnu/linux-headers
   $ emerge cross-s390x-unknown-linux-gnu/glibc
6. Build full GCC (able to link final binaries for C and C++)
   $ emerge cross-s390x-unknown-linux-gnu/gcc

Done!

How crossdev works (more details)
---------------------------------

This section contains more details on what actually happens (what crossdev
does for you).

Here we elaborate on each step outlined in previous section:

1. create an overlay with new ebuilds (symlinks to existing ebuilds)
   <skipping numerous mkdir and ln commands>. After this step the
   outcomes are:

   - overlay layout is formed in cross-overlay/:

     $ ls -l cross-overlay/cross-s390x-unknown-linux-gnu
     binutils -> /gentoo-ebuilds/gentoo/sys-devel/binutils
     gcc -> /gentoo-ebuilds/gentoo/sys-devel/gcc
     glibc -> /gentoo-ebuilds/gentoo/sys-libs/glibc
     linux-headers -> /gentoo-ebuilds/gentoo/sys-kernel/linux-headers

   - /usr/cross-s390x-unknown-linux-gnu (aka $SYSROOT) layout is set:

     $ ls -l /usr/s390x-unknown-linux-gnu/etc/portage/
     make.conf
     make.profile -> /gentoo-ebuilds/gentoo/profiles/embedded
     profile/

     Here we override ARCH, LIBC, KERNEL, CBUILD, CHOST, CTARGET and a
     few other variables.

   - a few convenience wrappers are created:

     /usr/bin/s390x-unknown-linux-gnu-emerge -> cross-emerge
     /usr/bin/s390x-unknown-linux-gnu-pkg-config -> cross-pkg-config
     /usr/bin/s390x-unknown-linux-gnu-fix-root -> cross-fix-root

   This way we share ebuild code and still can install cross-compilers
   independently. Each with it's own version of libc.

2. build cross-binutils
   # emerge cross-s390x-unknown-linux-gnu/binutils

   This way we can install the same version of binutils aiming at
   a new target. As a result we get tools like:
   s390x-unknown-linux-gnu-ar (static library archiver)
   s390x-unknown-linux-gnu-as (assembler)
   s390x-unknown-linux-gnu-ld (linker)
   ... <many others>

   Nothing special here.

3. Install minimal set of system headers (kernel and libc)
   $ USE="headers-only" emerge cross-s390x-unknown-linux-gnu/linux-headers
   $ USE="headers-only" emerge cross-s390x-unknown-linux-gnu/glibc

   As we don't have cross-compiler yet ebuilds just copy a bunch of
   header files into
       /usr/s390x-unknown-linux-gnu/usr/include
   and setup symlinks like:
       /usr/s390x-unknown-linux-gnu/sys-include -> usr/include
   to make cross-gcc happy.

   These include symlinks are target-dependent. A few unusual examples:

   - windows (mingw): /usr/x86_64-w64-mingw32/mingw -> usr
   - hurd: /usr/i686-pc-gnu/include -> usr/include
   - DOS: /usr/i686-pc-gnu/dev/env/DJDIR/include -> ../../../usr/include

   Side note: we could have omited symlink creation completely
   and build gcc with parameter:
     --with-native-system-header-dir=${EPREFIX}/usr/include
   That way ${SYSROOT} directory contents would be even more like normal
   root. Worth a try! TODO: actually do it.

4. Build minimal GCC without libc support (not able to link final executables
   yet)
   # USE="-*" emerge cross-s390x-unknown-linux-gnu/gcc

   Here gcc uses headers from step [3.] to find out what target libc can do:

   - POSIX support
   - trigonometry functions
   - threading
   - vital constants

   As a result we only get C code generator. No knowledge of how to link
   executables or shared libraries as those require bits of libc.

   For tiniest targets (bare-metal) this can be a final step to get basic
   C toolchain.

5. Build complete libc
   # emerge cross-s390x-unknown-linux-gnu/linux-headers
   # emerge cross-s390x-unknown-linux-gnu/glibc

   Here we build full libc against system headers. As a result we get C
   startup files (crt.o) and can now link full C programs!

6. Build full GCC (able to link final binaries for C and C++)
   # USE="" emerge cross-s390x-unknown-linux-gnu/gcc

   Here we get full C++ support, various default flags enabled (pie,
   sanitizers, stack protectors and others).

   The final result is ready for large-scale operations.

Various notes (AKA dirty little tricks)
---------------------------------------

- config.site

  Some ./configure scripts rely on runtime feature testing. We would
  still like to enable things even in cross-environment.

  crossdev installs /usr/share/config.site.d/80crossdev.conf with a bunch of cache
  variables preset for targets. It might be a nice place to drop
  more things into. Or it could be a source of all your cross-compilation
  problems if variables set incorrect values.

- eclass importing

  To find out various things about the target, crossdev loads multilib.eclass
  and tries to find out the default ABI supported by the target.

- crossdev is just a tiny shell script around emerge :)

  Its full source code is comparable to the size of this README.

- USE=headers-only

  Many toolchain ebuilds (libcs and kernel headers) are aware of
  headers-only install specifically for crossdev and similar tools
  to be able to build cross-toolchains.

- How to test crossdev layout generation:

  $ mkdir -p foo
  $ PORTAGE_CONFIGROOT=$(pwd)/foo EPREFIX=$(pwd)/foo PORT_LOGDIR=$(pwd)/foo ./crossdev -t mmix -P -p

  This needs some local patching. TODO: fix it to Just Work (perhaps with
  additional --test options).

Happy cross-compiling!

Testing
-------

This repository provides a script for testing crossdev in a container. It can
be used to verify whether crossdev works for various host profiles:

  $ ./scripts/container_test.sh --target aarch64-unknown-linux-musl
  $ ./scripts/container_test.sh --tag musl --target aarch64-unknown-linux-musl

--tag determines the container tag of the docker.io/gentoo/stage3 image used in
the script and can be used to pick the appropriate host stage3/profile to use.

This script is used by the CI job to test the whole matrix of supported host
profiles and targets.