Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Raspberry Pi 4 support #63720

Closed
majewsky opened this issue Jun 24, 2019 · 172 comments
Closed

Raspberry Pi 4 support #63720

majewsky opened this issue Jun 24, 2019 · 172 comments

Comments

@majewsky
Copy link
Contributor

Raspberry Pi just announced immediate availability of the Model 4. It will probably be some time until the aarch64 maintainers can get their hands on these, but I figured this can go into the backlog already.

@domenkozar
Copy link
Member

It would be especially nice if hardware video acceleration would now finally work due to OSS video firmware

@duckfullstop
Copy link
Contributor

duckfullstop commented Jun 25, 2019

After a definitely-not-impulsive trek to Cambridge yesterday, myself and @nekomimiscience both have Pi 4 4GB's as of yesterday and we can spend some time writing / testing some stuff to get these working properly.

From her experimentation yesterday night, the current kernel shipped with the aarch64 image does not boot, but fudging it by copying the kernel from the new official image (which is now based on debian/buster) at least gets the system to start.

We'll probably need to bring over some of the kernel patches from their fork (see here for a summary, though it'll probably involve quite a lot of the stuff committed on June 21st here), then it will probably be a case of just pulling the firmware Pi3B+ style. Should we be mainlining these patches onto the main kernel tree here, or would it be saner to have a separate package?

@ScarletHg
Copy link
Member

Most of the work in getting the Pi 4B to at least boot (without hacks) I've scoped out so far is upstream (U-Boot, Linux kernel).

I'm going to see how far I can get in bringing up U-Boot in my spare time but I don't have too much time to devote to it until Sunday. No promises at this stage.

Otherwise, version bumps on Raspberry Pi related packages seem to be the near-term way forwards (in particular, raspberrypifw and linux_rpi). Then until mainline Linux supports the Pi4B we can at least build images that boot by selecting the linux_rpi kernel.

Extremely excited about the new OSS video firmware though.

@duckfullstop
Copy link
Contributor

There appear to be some blocking issues on the current kernel when built for aarch64 which I would argue block merging it into NixOS.

Some of these are tracked in the following issue: raspberrypi/linux#3032

From what I can ascertain, at the time of writing this comment:

  • there seems to be a problem addressing more than 1GB of memory when root is on the SD card, the workaround is to chainload from SD onto a USB drive
  • the current tagged version of the kernel (20190620-1) does not compile properly due to a problem with the BCM2835 video driver; this has been fixed in this issue by disabling it altogether, which is far from ideal (but hey, at least it boots and gets to X)

Someone has succeeded in getting Gentoo to boot with an aarch64 kernel, written up here - but that memory problem feels like a bit of a blocker. Keeping an eye on that issue to see if there's a resolution.

I guess once that's fixed, we could target a specific commit of the kernel instead of the 20190620-1 release - though I'm not sure merging that kind of potential instability is a good idea. Open to comment.

@JohnAZoidberg
Copy link
Member

If you get it to work, please add the instructions here: https://nixos.wiki/wiki/NixOS_on_ARM/Raspberry_Pi

@majewsky
Copy link
Contributor Author

I linked from the wiki page back to this issue so that users consulting the wiki can see what is going on over here.

@matthewbauer matthewbauer added this to the 20.03 milestone Aug 16, 2019
@tkerber
Copy link
Member

tkerber commented Aug 17, 2019

I've managed to get nixos fully booting on my new pi4 (and am posting this from it!). I built this fork of the rpi kernel before it was merged: raspberrypi/linux#3144. Compile with bcm2711_defconfig. Beyond that, I had to make the following changes:

  • Update wireless firmware.
  • Add a derivation to build armstub8-gic.bin. Tbh, I don't actually know what this does, except that it won't boot without it :p
  • Update nixos/system/boot/loader/raspberrypi to a) build and copy armstub8-gic.bin, and b) add the corresponding enable_gic=1 and armstub=armstub8-gic.bin lines to config.txt.

Right now, the most obvious deficits you can read about in e.g. the gentoo build, of sd cards, wifi, and RAM not working, are no longer an issue.

Beyond that, it seems like a pretty standard aarch64 build (though it's the first one I've done). I did not look into the sdcard image builder. (I did some messy hack for running the activation script on the chroot'd x86_64 host instead)

I'll try to do a more proper writeup, and make pull reqs for the nixos modules needed over the weekend.
nixos-rpi4

@jbaum98
Copy link
Contributor

jbaum98 commented Aug 17, 2019

@tkerber congrats! I'm excited for the write-up!

Did you get 4GB of ram working? My understanding is that the aarch64 kernel had a bug where it could only use 1GB of ram.

@tkerber
Copy link
Member

tkerber commented Aug 17, 2019

Not sure to be honest. I've compiled a kernel which I believe to patch the core issue, but I still had symptoms which look very much like those of running without limiting the RAM... I'll try a more up-to-date kernel tomorrow, and see if that persists.

@tkerber
Copy link
Member

tkerber commented Aug 17, 2019

This issue is known upstream: raspberrypi/linux#3157 - for now, I guess limiting to 1G is the best action to take for a stable system (maybe only while downloading with WiFi as the issue suggests - I wouldn't know, there is little else I do with machines :p).

@yaroslavros
Copy link

raspberrypi/linux#3159 should fix WiFi issue

I run arm64 gentoolto on RPI4 4G without any trouble now.

@tkerber
Copy link
Member

tkerber commented Aug 18, 2019

I've just made a pull request. Here's how to build a working config against it:

Example Config

Replace UUIDs as appropriate.

{ pkgs, ... }:

{
  # Tell the host system that it can, and should, build for aarch64.
  nixpkgs = rec {
    crossSystem = (import <nixpkgs> {}).pkgsCross.aarch64-multiplatform.stdenv.targetPlatform;
    localSystem = crossSystem;
  };

  fileSystems = {
    "/" = {
      fsType = "ext4";
      device = "/dev/disk/by-uuid/67739d77-fe40-46af-8e49-6e6facb79a87";
    };
    "/boot" = {
      fsType = "vfat";
      device = "/dev/disk/by-uuid/5F19-927F";
    };
  };

  services.xserver = {
    enable = true;
    displayManager.slim.enable = true;
    desktopManager.xterm.enable = false;
    windowManager.i3.enable = true;
    videoDrivers = [ "fbdev" ];
  };

  hardware.enableRedistributableFirmware = true;

  users.users.exampleuser = {
    isNormalUser = true;
    password = "badpassword";
  };
  
  # For the ugly hack to run the activation script in the chroot'd host below. Remove after sd card is set up.
  environment.etc."binfmt.d/nixos.conf".text = ":aarch64:M::\\x7fELF\\x02\\x01\\x01\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x02\\x00\\xb7\\x00:\\xff\\xff\\xff\\xff\\xff\\xff\\xff\\x00\\xff\\xff\\xff\\xff\\xff\\xff\\x00\\xff\\xfe\\xff\\xff\\xff:/run/binfmt/aarch64:";
  boot= {
    kernelPackages = pkgs.linuxPackages_rpi4;
    loader = {
      grub.enable = false;
      raspberryPi = {
        enable = true;
        version = 4;
      };
    };
  };
}

How to Build

This is what I did to set up my microsd. It's quite hacky, but it works. Any command below should be in a root shell (apart from the initial build).

  1. Set up your host system with qemu-aarch64 binfmt (https://nixos.wiki/wiki/NixOS_on_ARM#Crosscompiling).
  2. Partition your SD card with a small FAT32 boot partition, and a large ext4 root partition.
  3. Mount these to /mnt and /mnt/boot.
  4. Build your system: NIX_PATH="nixpkgs=path/to/repo" nix build '(import <nixpkgs/nixos> { configuration.imports = [ path/to/config.nix ]; }).system'
  5. Run nixos-install --system $(readlink result). Activation will fail.
  6. Run install -D /etc/binfmt.d/nixos.conf /mnt/etc/binfmt.d/nixos/conf, and install -D /run/binfmt/aarch64 /mnt/run/binfmt/aarch64
    This lets us run aarch64 executables in the chroot.
  7. Run the activation scripts:
    chroot /mnt /nix/var/nix/profiles/system/activate
    chroot /mnt /run/current-system/bin/switch-to-configuration boot
    This may still fail. Repeat step 6 if so (nixos really doesn't like that
    we're trying to force non-sanctioned files in /etc and /run).
  8. Find and run passwd as root: ls /mnt/nix/store/*shadow*/bin/passwd chroot /mnt /nix/store/..../bin/passwd to set your password.
  9. Unmount, insert into the pi, and enjoy!

@duckfullstop
Copy link
Contributor

duckfullstop commented Aug 18, 2019

The main reason @NekomimiScience and I held off on suggesting that nixpkgs merge the current upstream kernel is because it wasn't possible to address more than 1G of memory on aarch64 (which, in our opinion, was an indication that it probably wasn't merge ready). @tkerber, am I reading the previous comments right in that this is no longer a problem, or is it still not possible to use more than 1G?

@tkerber
Copy link
Member

tkerber commented Aug 18, 2019

Looking again at raspberrypi/linux#3159, it does seem it's not solved yet. I'm withdrawing my pull req until this gets a consistent upstream fix. I'll leave up the branch, as people can build against it for now. It does seem like it'll still be a few weeks before the kernel is stable, though.

@duckfullstop
Copy link
Contributor

@tkerber If you're able to use the full range of memory, then I'd say it can probably be merged with a "probably unstable" note attached to it. Probably worth a test!

@vikanezrimaya
Copy link
Member

I don't plan to use WiFi, since there is Gigabit Ethernet onboard, so if the WiFi is the only issue, I'm ok with it! Are Gigabit Ethernet and USB 3.0 ports working?

I hope to get an RPi4 ~soon-ish, so I might be able to pitch in and help with testing it (mostly in headless mode)

@tkerber
Copy link
Member

tkerber commented Aug 18, 2019

I can't personally attest to either at the moment. That said, I haven't seen them being brought up as an issue anywhere, and things seem to be moving fast to patch the issues that are remaining.

@yaroslavros
Copy link

FWIW, I have just submitted a fresh PR raspberrypi/linux#3164 to address the WIFI DMA issue.

@satmandu
Copy link

@tkerber FYI U-boot DOES work on the RPI4 using some work which doesn't look like it has been integrated into upstream yet. My RPI4 is able to boot using this: https://github.com/agherzan/u-boot/tree/ag/v2019.07-rpi4-wip

@lategoodbye
Copy link

@satmandu One of the reasons why these changes hasn't integrated into U-Boot upstream yet is the fact that the devicetree sources are under review during Linux upstreaming. It's not a good idea to adopt the downstream files directly.

https://marc.info/?l=linux-arm-kernel&m=156571347332483

@jbaum98
Copy link
Contributor

jbaum98 commented Aug 19, 2019

@tkerber I've tried to follow your instructions, but I don't have a computer with NixOS installed on it. I cloned your branch on an arm AWS instance and tried to build an sd card image, but I could only get a rainbow screen on the pi. Do you have any suggestions on getting the sd card image working?

@tkerber
Copy link
Member

tkerber commented Aug 19, 2019

@jbaum98 I'm not sure - I don't know enough about the sdcard making process to know what could go wrong. It sounds like an issue in early boot though, so likely the boot partition isn't set up correctly. The pi loads the first partition, which should be fat32, and does a bunch of stuff with it, before booting a specified kernel and initramfs. Sounds like this could be either a) this not being properly set up for whatever reason, or b) it straight up failing to execute the kernel.

@jbaum98
Copy link
Contributor

jbaum98 commented Aug 19, 2019

I got it working following your instructions! I just booted from a live USB of NixOS and it all worked great.

@numinit
Copy link
Contributor

numinit commented Aug 23, 2019

How do I add your git repo as a channel?

Edit: We got it working. You have to export nixpkgs=/path/to/repo in NIX_PATH before switching to the new config.

@wizeman
Copy link
Member

wizeman commented Sep 6, 2019

It seems like the major issues mentioned in this thread have been solved using the latest commit from the https://github.com/raspberrypi/linux kernel repo?

Specifically, I got the full 4GB of RAM to work fine (and it seems like the Wi-Fi issues have been solved as well, although I don't use Wi-Fi so I can't test it).

Anyone willing to make a PR?

@nixos-discourse
Copy link

This issue has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/has-any-one-succeeded-to-drive-all-devices-of-rpi4b-8gb/11850/1

@tfc
Copy link
Contributor

tfc commented Mar 17, 2021

Hi there, i have discovered the same problem with the raspberry image, and when trying to build a new one from master i additionally get this problem:

$ nix-build nixos/release.nix -A sd_image_raspberrypi4
...
root module: sun4i_drm
modprobe: FATAL: Module sun4i_drm not found in directory /nix/store/4p7kd78cdzb93zqvhs651nlqvwk94ry5-linux-5.4.79-1.20201201-modules/lib/modules/5.4.79
builder for '/nix/store/kpmyrhdgbxqka3zly1q0p55a5v9dxm6y-linux-5.4.79-1.20201201-modules-shrunk.drv' failed with exit code 1
...

when removing those modules from the file nixos/modules/installer/sd-card/sd-image-aarch64.nix, it builds again.

@mohe2015
Copy link
Contributor

mohe2015 commented Mar 17, 2021

Hi there, i have discovered the same problem with the raspberry image, and when trying to build a new one from master i additionally get this problem:

$ nix-build nixos/release.nix -A sd_image_raspberrypi4
...
root module: sun4i_drm
modprobe: FATAL: Module sun4i_drm not found in directory /nix/store/4p7kd78cdzb93zqvhs651nlqvwk94ry5-linux-5.4.79-1.20201201-modules/lib/modules/5.4.79
builder for '/nix/store/kpmyrhdgbxqka3zly1q0p55a5v9dxm6y-linux-5.4.79-1.20201201-modules-shrunk.drv' failed with exit code 1
...

when removing those modules from the file nixos/modules/installer/sd-card/sd-image-aarch64.nix, it builds again.

On which system are you trying to build? e.g. x86 or arm? Because if you are on x86 or so I think you need to add "--system aarch64-linux" or so. But maybe I just forgot and this is a different issue

@tfc
Copy link
Contributor

tfc commented Mar 17, 2021

@mohe2015 just tried, did not change anything unfortunately. Your assumptions are correct, i am building on an x86 box. although the raspi image boots just fine on the raspi4 without --system aarch64-linux when i deleted those sun4i_* things.

@mohe2015
Copy link
Contributor

@tfc See #111683 so it's expected behaviour that it doesn't work right now. The arg I posted isn't needed - nix detects it automatically that it has to build using emulation.

@tfc
Copy link
Contributor

tfc commented Mar 18, 2021

I just bisected the wifi problem. Wifi doesn't work on the rpi4 since this commit: d7c4ca2de61c27ccc

It updates the nonfree firmware...

@tfc
Copy link
Contributor

tfc commented Mar 18, 2021

With these two patches, the nixpkgs master successfully builds the sd image for the rpi4 with working wifi:

1.) remove the sun4i_* modules as known from #111683

index 96ebb7537da..dbeb4897ae0 100644
--- a/nixos/modules/installer/sd-card/sd-image-aarch64.nix
+++ b/nixos/modules/installer/sd-card/sd-image-aarch64.nix
@@ -21,8 +21,6 @@
   boot.initrd.availableKernelModules = [
     # Allows early (earlier) modesetting for the Raspberry Pi
     "vc4" "bcm2835_dma" "i2c_bcm2835"
-    # Allows early (earlier) modesetting for Allwinner SoCs
-    "sun4i_drm" "sun8i_drm_hdmi" "sun8i_mixer"
   ];
 
   sdImage = {

2.) update the nonfree linux firmware blobs again, since the pre-latest version that d7c4ca2 introduced does not work:

diff --git a/pkgs/os-specific/linux/firmware/firmware-linux-nonfree/default.nix b/pkgs/os-specific/linux/firmware/firmware-linux-nonfree/default.nix
index 6e7f8d89dde..b66c4e73084 100644
--- a/pkgs/os-specific/linux/firmware/firmware-linux-nonfree/default.nix
+++ b/pkgs/os-specific/linux/firmware/firmware-linux-nonfree/default.nix
@@ -2,12 +2,12 @@
 
 stdenvNoCC.mkDerivation rec {
   pname = "firmware-linux-nonfree";
-  version = "2021-02-08";
+  version = "2021-03-15";
 
   src = fetchgit {
     url = "https://git.kernel.org/pub/scm/linux/kernel/git/firmware/linux-firmware.git";
     rev = "refs/tags/" + lib.replaceStrings ["-"] [""] version;
-    sha256 = "0c85cd659312isfz1r87qswsgfhy0rljagcwspnvjljqrh9bsgzq";
+    sha256 = "sha256-BnYqveVFJk/tVYgYuggXgYGcUCZT9iPkCQIi48FOTWc=";
   };
 
   installFlags = [ "DESTDIR=$(out)" ];
@@ -17,7 +17,7 @@ stdenvNoCC.mkDerivation rec {
 
   outputHashMode = "recursive";
   outputHashAlgo = "sha256";
-  outputHash = "0l4xsgxdvjffad7a98n42nyqy3ihs6m6hy3qsfkqin9z10413x5n";
+  outputHash = "sha256-TzAMGj7IDhzXcFhHAd15aZvAqyN+OKlJTkIhVGoTkIs=";
 
   meta = with lib; {
     description = "Binary firmware collection packaged by kernel.org";

i'm going to upstream the blob thing another MR that does this was merged on monday already, so the raspi4 image has working wifi since monday again.

@cyntheticfox
Copy link
Contributor

cyntheticfox commented Mar 30, 2021

Is the Hydra builds failing a known issue? I tried following the Wiki Guide and updating to NixOS-Unstable, and my boot device can't find NIXOS_SD after rebooting.

Based on the most recent commits, it looks like the Allwinner stuff was removed, but the Hydra build logs show a failure looking for it.

Edit: I see now what's happening: #111683

@MarcoPolo
Copy link
Contributor

For getting usb only boot working I replaced the files in partition 1 with the versions from the raspberry pi os image. Not config.txt; and I didn't add any new files. I'm not sure what the difference is here, maybe some new change to support booting from USB? The original files booted the SD card fine, but failed when only the usb was present (no SD card).

@necrophcodr
Copy link
Contributor

It appears to still be broken. Booting the latest image from Hydra ( https://hydra.nixos.org/build/142828023 ) results in the image being unable to find /nix/.../init, and failing to continue booting from there.

@domenkozar
Copy link
Member

domenkozar commented May 11, 2021

I've written a draft for Raspberry Pi tutorial. I'd be happy if someone gives it a try and reports back if it works and if they got lost at some point. Bonus karma for a screencast :)

I've also opened NixOS-hardware PR.

@domenkozar
Copy link
Member

domenkozar commented May 12, 2021

I've moved GPU/acceleration issues into a separate thread: #122726

@kwaerner
Copy link

I've written a draft for Raspberry Pi tutorial. I'd be happy if someone gives it a try and reports back if it works and if they got lost at some point. Bonus karma for a screencast :)

I've also opened NixOS-hardware PR.

I just finished your tutorial - works great! Tried it on an 8GB Raspberry Pi 4B. Only issue I had was that I had to unplug my keyboard and plug it back in after each reboot for it to work. I also saw some messages about bluetooth at the login shell (found an ArchLinux post about this - you are probably aware of any issues).

Btw: completely new to NixOS, but I will try my best if there is something you want me to test on my installation. :)

@06kellyjac
Copy link
Member

@kwaerner welcome to NixOS/nix!

If you need any help there's the discourse, irc, or posting an issue here. And the docs + wiki are pretty solid. More details on the nixos.org website.

@domenkozar
Copy link
Member

That's warming to hear @kwaerner :)

The tutorial is now live at https://nix.dev/tutorials/installing-nixos-on-a-raspberry-pi

As @samueldr has aced #122726, I believe we can close this issue.

There are probably still some issues with configuring hardware, please open specific issues for those.

@n8henrie
Copy link
Contributor

If this issue is closed, should the asterisk be removed from the support column in the table here?

@samueldr
Copy link
Member

YES

@cmacrae
Copy link
Contributor

cmacrae commented Jul 5, 2021

Just want to run this by others who may still be following this before opening a specific issue: I don't seem to be able to get past Starting kernel... when booting any images generated by the Hydra build job (mentioned in the guide) - including images as recent as yesterday evening.

To be clear, this is straight from dd'ing the image to an SD card and trying to boot, not after a rebuild (I've seen this comment which suggests a fix for that scenario).

Is anyone else experiencing this?

@mohe2015
Copy link
Contributor

mohe2015 commented Jul 5, 2021

@cmacrae I would try a self-built image first if you have the resources to do so (qemu binfmt may work well). I booted a self-built image with boot.kernelPackages = pkgs.linuxPackages_latest; yesterday over USB on the 8GB edition so maybe that's also needed. AFAIR the Starting kernel... usually means the kernel has actually started (aah that's what the referenced comment says) but the display driver or whatever isn't able to boot. So maybe you also want to build an image with ssh if you would build one yourself.

@cmacrae
Copy link
Contributor

cmacrae commented Jul 5, 2021

Thanks @mohe2015 :) Alright, I'll give that a shot, then

@cmacrae
Copy link
Contributor

cmacrae commented Jul 7, 2021

Just want to run this by others who may still be following this before opening a specific issue: I don't seem to be able to get past Starting kernel... when booting any images generated by the Hydra build job (mentioned in the guide) - including images as recent as yesterday evening.

To be clear, this is straight from dd'ing the image to an SD card and trying to boot, not after a rebuild (I've seen this comment >which suggests a fix for that scenario).

Is anyone else experiencing this?

Just to update others who may have a similar issue to the one I experienced above: my problem was that I hadn't updated my Pi's firmware. After following the steps outlined here, I was able to boot the Hydra provided images :)

@mohe2015
Copy link
Contributor

mohe2015 commented Jul 7, 2021

Just to update others who may have a similar issue to the one I experienced above: my problem was that I hadn't updated my Pi's firmware. After following the steps outlined here, I was able to boot the Hydra provided images :)

Nice that you could fix it. I would've recommended you to check for firmware updates if you would've used USB boot but I didn't know it's also needed for SD card boot.

@domenkozar
Copy link
Member

domenkozar commented Jul 7, 2021

That's why I recommend updating firmware as an installation step.

Maybe that needs to move further up before even booting the image?

@domenkozar
Copy link
Member

Thanks @cmacrae - NixOS/nix.dev#166

@cmacrae
Copy link
Contributor

cmacrae commented Jul 7, 2021

Perfect, that's exactly what I was going to suggest :) Thanks @domenkozar 🙇

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests