Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Add instructions for cross compilation #17

Open
mfreeborn opened this issue Feb 18, 2023 · 13 comments
Open

Add instructions for cross compilation #17

mfreeborn opened this issue Feb 18, 2023 · 13 comments

Comments

@mfreeborn
Copy link
Contributor

I think a sizeable use-case for this library is to develop libcamera applications for raspberry pi devices - at least that is my use case. The problem is that these devices can be very underpowered with regards to software development and compiling programs.

I have therefore been spending the last couple of days on and off trying to get cross compilation working - I think I'm getting pretty close now.

I am using this repo to provide the cross compilation toolchain binaries, as well as a basic sysroot directory. It is missing a couple of things:

  1. GLIBC is v2.27, but we need at least v2.30. I think this is just a case of updating the configs and re-running the github action on the above-linked repo.

  2. It doesn't have an up to date libcamera compiled within it. I have currently circumvented this problem by compiling libcamera on my raspberry pi 0 and copy the appropriate files to the sysroot of the above toolchain.

There is one more oddity that I have come across, regarding stream.h. The static assertion on line 24 static_assert(sizeof(struct libcamera_stream_configuration) == sizeof(libcamera_stream_configuration_t)); fails (112 == 88)... and I'm not sure why (something to do with host being 64-bit and target being 32-bit?) . Naively commenting the line out at least lets me compile camera_c_api, before getting the errors numbered above. I don't know why that static assertion is required in the first place, or why those two sizes would be different.
@chemicstry
Copy link
Contributor

Usually with Rust cross-compilation is quite easy:

  • rustup target add aarch64-unknown-linux-gnu
  • cargo build --target aarch64-unknown-linux-gnu

bindgen and cc should support cross-compilation out of the box, but the problem is libcamera. I couldn't find any info on cross-compiling except this issue. But for some reason libcamera doesn't want to use different toolchain and still compiles using system gcc. Didn't investigate further.

There is one more oddity that I have come across, regarding stream.h. The static assertion on line 24 static_assert(sizeof(struct libcamera_stream_configuration) == sizeof(libcamera_stream_configuration_t)); fails (112 == 88)... and I'm not sure why (something to do with host being 64-bit and target being 32-bit?) . Naively commenting the line out at least lets me compile camera_c_api, before getting the errors numbered above. I don't know why that static assertion is required in the first place, or why those two sizes would be different.

Yes, this is likely due to 32-bit target. I guess we could add ifdefs to problematic headers. Btw, what raspberry are you using? Both RPi 3 and 4 support 64-bit architecture.

Also, could you explain your workflow? Do you cross-compile and then copy binaries over to raspberry? I only found rust-analyzer with vscode to be slow on RPi 4, but regular incremental builds for libcamera are just a couple of seconds.

@mfreeborn
Copy link
Contributor Author

mfreeborn commented Feb 18, 2023 via email

@kbingham
Copy link

libcamera uses meson as it's build system. If you want to cross compile libcamera to aarch64 you have to provide a suitable cross file and environment for that.

Meson documents this at:

@chemicstry
Copy link
Contributor

chemicstry commented Feb 21, 2023 •
edited
Loading

Thanks for the tip, I actually succeeded in cross-compiling libcamera, but didn't have time to update. Here are the steps on Ubuntu 20.04 if anyone is interested:

  • Install dependencies 
    sudo apt install gcc-aarch64-linux-gnu g++-aarch64-linux-gnu pkg-config-aarch64-linux-gnu
  • Create aarch64_cross.txt: 
    [binaries]
c = '/usr/bin/aarch64-linux-gnu-gcc'
cpp = '/usr/bin/aarch64-linux-gnu-g++'
ar = '/usr/bin/aarch64-linux-gnu-gcc-ar'
strip = '/usr/bin/aarch64-linux-gnu-strip'
pkgconfig = '/usr/bin/aarch64-linux-gnu-pkg-config'

[host_machine]
system = 'linux'
cpu_family = 'aarch64'
cpu = 'aarch64'
endian = 'little'
  • Build: 
    meson cross-build --cross-file aarch64_cross.txt
ninja -C cross-build

I got stuck on pkg-config and sysroot stuff though, because I have no idea how they work. Doing ninja -C cross-build install installs aarch64 binaries into system folders instead of under arch-specific subfolder. Some guides on the internet suggested setting up multi-arch system via apt, but I'm not sure if that is neccessary. The rust pkg-config lib gives the following error (hint):

Install a sysroot for the target platform and configure it via PKG_CONFIG_SYSROOT_DIR and PKG_CONFIG_PATH, or install a cross-compiling wrapper for pkg-config and set it via PKG_CONFIG environment variable.

I will update if I get any further.

@kbingham
Copy link

Remember if you're cross compiling like that - then it will only 'work' if you're cross compiling on ubuntu-20.04 on your 'host', and your target device is also running ubuntu-20.04. You can't expect to cross compile on ubuntu-20.04 (x86) and target raspberry-pi-bullseye like that for instance. There will be all sorts of conflicts with different library versions and other dragons.

You can however have a bullseye-arm64 root filesystem set as the sysroot to compile against if it matches what the target will have. That's what sysroots are for.

I had an intern tackle this with docker to handle the requirements in the past which might have something helpful/useful:

https://github.com/CactiChameleon9/Libcamera-RPiOS-Build-Enviroment

@mfreeborn
Copy link
Contributor Author

I've spent too long on this - I managed to get it working transiently, but I think the cost-benefit isn't there.

Cross compiling either rust or libcamera is doable enough, but I've run into endless brick walls trying to get it to all work together in a sufficiently simple manner.

At one stage I ended up juggling the Ubuntu build system's filesystem, a debootstrap filesystem, a sysroot from this cross compiler set up and finally the filesystem on the target machine. Throw in pkg-config and multiarch issues, dynamic linking of standard libraries from all over the place, the fact you are trying to compose meson cross compilation (and cross compilation of optional extras, rust cross compilation, building the C wrapper in rust (with cc) and generating the bindings to the C wrapper in rust and throw in the fact that debian armhf != Raspberry Pi armhf... I could go on.

So my conclusion is that whilst it's possible, it's not practical.

If one is targeting a higher end Raspberry Pi, just use something like VS Codes SSH remote development. If one is targeting e.g. a Raspberry Pi 0 (like me), then just develop the application on your development machine (I need to research how to use the vimc driver, or use a cheapo usb camera) and make a coffee whilst you do your one final native compilation on the Pi.

@kbingham
Copy link

for vimc ... just 'sudo modprobe vimc'.

@kbingham
Copy link

If you have energy left, It could still be worth retrying what you've done with the docker image from https://github.com/CactiChameleon9/Libcamera-RPiOS-Build-Enviroment and extending that instead of https://github.com/tttapa/docker-arm-cross-toolchain as the cross compiler we use is installed directly from the bullseye repo, where as the 'docker-arm-cross-toolchain' is custom creating something.

@mfreeborn
Copy link
Contributor Author

Thanks for the tip about vimc.

I believe that one of the issues that tttapa's setup solves (at least for 32-bit systems) is that debian armhf supports armv7 CPUs, whilst Raspberry Pi's opinion of armhf is armv6. Therefore, the default debian armhf builds are not necessarily compatible (I suspect they might work for a bit... until they don't).

@kbingham
Copy link

Ah, wait - are you trying to use the rpi-zero *original? Or the RPi zero 2 ?

@mfreeborn
Copy link
Contributor Author

mfreeborn commented Feb 21, 2023 via email

@twistedfall twistedfall mentioned this issue Apr 18, 2023
Merged
@fishrockz
Copy link

fishrockz commented Jul 2, 2023 •
edited
Loading

Hi

I have been doing some embedded projects and have cross compiled one of your examples in my experimental integration tool https://gitlab.com/pointswaves/girderstream-test-project/-/blob/264bd9a4ea87da9af8cd9ccdc724e153087ae85d/elements/libcamera-rs.grd (the only reason i have not build more of your examples is because my cross tool chain does not have a update enough rust for your other ones. Updating this is on my todo list lol)

I use yocto/buildstream/buildroot at work but i have done the above with my experimental project girderstream. I only mention this so i can show you my example while also noting that its not the common way but that i do do quite a lot of embedded.

My way of cross building libcamera-rs, in plain english, so you don't need to go look at my example in a unusual build file is as follows.

Place your previously cross-compiled libcamera and its dependencies in a directory, in this case /targetfs/

You will then need to configure pkgconfig to find your cross compiled libcamera, ie were is the libcamera.pc file etc

export PKG_CONFIG_PATH=/targetfs/usr/lib/pkgconfig/
export PKG_CONFIG_LIBDIR=/targetfs/usr/lib/pkgconfig/
export PKG_CONFIG_SYSROOT_DIR=/targetfs/

At this point pkgconfig can find everything so you should just need to tell cargo that you want to cross compile

First update your .cargo/config so it knows which linker to use etc

[target.aarch64-unknown-linux-gnu]
linker = "aarch64-unknown-linux-gnu-gcc"
rustflags = ["-Clink-args=-Xlinker -rpath-link=/targetfs/usr/lib/%{target-triplet}/"]

Tell cargo/rustc how to build the C wrapper in libcamera-rs-sys by telling it what c++ compiler to use.
export TARGET_CXX=your-cross-compiler-g++"

Then compile the example
cargo build --release --example version --target=aarch64-unknown-linux-gnu

The above is all pretty standard for cross building a libABC-rs-sys except for having to set the rpath

I have had to manually set the rpath, so the linker can find the .so files of the dependnecies in the /targetfs/ which is not on the default linker path

As you can see the version example links to many of the libcamera dependencies

/ # /lib/ld-linux-aarch64.so.1 --list /version
	linux-vdso.so.1 (0x0000ffffada53000)
	libcamera.so.0.0.5 => /usr/lib/aarch64-linuxgrd-gnu/libcamera.so.0.0.5 (0x0000ffffad7e0000)
	libstdc++.so.6 => /usr/lib/aarch64-linuxgrd-gnu/libstdc++.so.6 (0x0000ffffad5b0000)
	libgcc_s.so.1 => /usr/lib/aarch64-linuxgrd-gnu/libgcc_s.so.1 (0x0000ffffad580000)
	libc.so.6 => /usr/lib/aarch64-linux-gnu/libc.so.6 (0x0000ffffad3d0000)
	/lib/ld-linux-aarch64.so.1 (0x0000ffffada1b000)
	libcamera-base.so.0.0.5 => /usr/lib/aarch64-linuxgrd-gnu/libcamera-base.so.0.0.5 (0x0000ffffad390000)
	libgnutls.so.30 => /usr/lib/aarch64-linuxgrd-gnu/libgnutls.so.30 (0x0000ffffad140000)
	libyaml-0.so.2 => /usr/lib/aarch64-linuxgrd-gnu/libyaml-0.so.2 (0x0000ffffad110000)
	libm.so.6 => /usr/lib/aarch64-linux-gnu/libm.so.6 (0x0000ffffad070000)
	libunwind.so.8 => /usr/lib/aarch64-linuxgrd-gnu/libunwind.so.8 (0x0000ffffad030000)
	libnettle.so.8 => /usr/lib/aarch64-linuxgrd-gnu/libnettle.so.8 (0x0000ffffacfd0000)
	libhogweed.so.6 => /usr/lib/aarch64-linuxgrd-gnu/libhogweed.so.6 (0x0000ffffacf60000)
	libgmp.so.10 => /usr/lib/aarch64-linuxgrd-gnu/libgmp.so.10 (0x0000ffffaced0000)
/ # ./version 
libcamera: v0.0.5

The way i think this should work is that you should provide both the -L (linker) and the -I (include) paths instead of just the -I .clang_arg(format!("-I{}", libcamera_include_path.display()))

The docs for pkgconfig https://docs.rs/pkg-config/latest/pkg_config/struct.Library.html talk about the -I and -L paths.

I think the reason you don't normally need to set the link path is because up until now all the required .so files were on the default link paths, but if you ever tried to build libcamera-rs-sys on a system were the .so of the deps were not on the default link path, i think you would have this issue.

So would you like me to write some patches to:

  • Pass on the link path between pkgconfig and the libcamera-rs-sys bindgen builder?
  • Write up the above in to the readme?

@chemicstry
Copy link
Contributor

@fishrockz wow, thanks for the in depth explanation.

Pass on the link path between pkgconfig and the libcamera-rs-sys bindgen builder?

Sure

Write up the above in to the readme?

Maybe you could write this as a separate file (i.e. CROSSCOMPILING.md) and link it from the main README? Also, could you include how to cross-compile libcamera itself? Unless there is a howto somwhere else that we can link to.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
4 participants
@kbingham @chemicstry @fishrockz @mfreeborn