secureboot x86_64 upgrade tests yielding grub OOM

[dustymabe]

Recently we added an upgrade test framework for testing upgrades from systems started on older released versions. See:

• tests/kola: add extended upgrade test fedora-coreos-config#2310
• tests/kola: a few updates for the upgrade test fedora-coreos-config#2335
• jobs: add kola-upgrade test fedora-coreos-pipeline#847

Now that we are running the tests we are noticing a fairly consistent UEFI+SecureBoot upgrade test failure where GRUB complains about running out of memory:

error: ../../grub-core/kern/mm.c:376:out of memory.

Then later during system boot you'll see a kernel panic because the system can't find the root device (presumable because it failed to fully load the initramfs into memory).

This is happening pretty consistently once the upgrading nodes get to the 37.20230322.2.0 update (this is on the testing stream) regardless of where they start.

• 37.20221106.2.1->37.20230401.2.0
  • kola-uefi-secure-b58ddc4d.console.txt
• 36.20220505.2.0->37.20230401.2.0
  • kola-uefi-secure-52d5bfb6.console.txt
• 35.20211029.2.0->37.20230401.2.0
  • kola-uefi-secure-fd77ad10.console.txt
• 34.20210427.2.0->37.20230401.2.0
  • kola-uefi-secure-32836152.console.txt

An easy workaround for this is to give the test VM more than 1G of memory, but we'd really like to understand what the underlying problem is too.

Note we've seen something like this in CI before (except not on upgrades) for secureboot: #1382

[jlebon]

I tested this locally and couldn't reproduce it, so there's possibly some outside-the-VM factor involved.

$ cosa kola run --tag extended-upgrade --build=37.20221225.2.1 --qemu-firmware=uefi-secure
kola -p qemu-unpriv --build 37.20221225.2.1 run --tag extended-upgrade --qemu-firmware=uefi-secure --output-dir tmp/kola
⚠️  Skipping kola test pattern "fcos.internet":
  👉 https://github.com/coreos/coreos-assembler/pull/1478
⚠️  Skipping kola test pattern "podman.workflow":
  👉 https://github.com/coreos/coreos-assembler/pull/1478
=== RUN   ext.config.upgrade.extended
--- PASS: ext.config.upgrade.extended (105.43s)

In case it matters, my pet container has:

$ rpm -q edk2-ovmf
edk2-ovmf-20230301gitf80f052277c8-1.fc37.noarch

which seems to be the latest currently.

[dustymabe]

Some more investigation:

It appears there were some memory allocation fixes in grub that went into 2.06-75.fc37. I have confirmed that starting at 37.20230122.2.0 yields no problems, where starting at or before 37.20230110.2.0 does yield the problem.

The package diff there looks like:

$ rpm-ostree --repo=./ db diff 5232fec18afc1dedde77c506dec7eaa2896176f48710593c2026cecd07cc84d7 74ff03e7e2999ffeed34540225b563e1c52bf9b527e59698f0406f7c4b3e8b09
ostree diff commit from: 5232fec18afc1dedde77c506dec7eaa2896176f48710593c2026cecd07cc84d7
ostree diff commit to:   74ff03e7e2999ffeed34540225b563e1c52bf9b527e59698f0406f7c4b3e8b09
Upgraded:
  NetworkManager 1:1.40.0-1.fc37 -> 1:1.40.10-1.fc37
  NetworkManager-cloud-setup 1:1.40.0-1.fc37 -> 1:1.40.10-1.fc37
  NetworkManager-libnm 1:1.40.0-1.fc37 -> 1:1.40.10-1.fc37
  NetworkManager-team 1:1.40.0-1.fc37 -> 1:1.40.10-1.fc37
  NetworkManager-tui 1:1.40.0-1.fc37 -> 1:1.40.10-1.fc37
  efivar-libs 38-5.fc37 -> 38-6.fc37
  flatpak-session-helper 1.14.1-1.fc37 -> 1.14.1-2.fc37
  git-core 2.39.0-1.fc37 -> 2.39.1-1.fc37
  glibc 2.36-8.fc37 -> 2.36-9.fc37
  glibc-common 2.36-8.fc37 -> 2.36-9.fc37
  glibc-minimal-langpack 2.36-8.fc37 -> 2.36-9.fc37
  grub2-common 1:2.06-72.fc37 -> 1:2.06-75.fc37
  grub2-efi-x64 1:2.06-72.fc37 -> 1:2.06-75.fc37
  grub2-pc 1:2.06-72.fc37 -> 1:2.06-75.fc37
  grub2-pc-modules 1:2.06-72.fc37 -> 1:2.06-75.fc37
  grub2-tools 1:2.06-72.fc37 -> 1:2.06-75.fc37
  grub2-tools-minimal 1:2.06-72.fc37 -> 1:2.06-75.fc37
  iptables-legacy 1.8.8-3.fc37 -> 1.8.8-4.fc37
  iptables-legacy-libs 1.8.8-3.fc37 -> 1.8.8-4.fc37
  iptables-libs 1.8.8-3.fc37 -> 1.8.8-4.fc37
  iptables-nft 1.8.8-3.fc37 -> 1.8.8-4.fc37
  iptables-services 1.8.8-3.fc37 -> 1.8.8-4.fc37
  iptables-utils 1.8.8-3.fc37 -> 1.8.8-4.fc37
  kernel 6.0.18-300.fc37 -> 6.1.6-200.fc37
  kernel-core 6.0.18-300.fc37 -> 6.1.6-200.fc37
  kernel-modules 6.0.18-300.fc37 -> 6.1.6-200.fc37
  lz4-libs 1.9.3-5.fc37 -> 1.9.4-1.fc37
  moby-engine 20.10.21-1.fc37 -> 20.10.22-1.fc37
  mozjs102 102.6.0-1.fc37 -> 102.7.0-1.fc37
  selinux-policy 37.17-1.fc37 -> 37.18-1.fc37
  selinux-policy-targeted 37.17-1.fc37 -> 37.18-1.fc37
  sudo 1.9.11-4.p3.fc37 -> 1.9.12-1.p2.fc37
  vim-data 2:9.0.1054-1.fc37 -> 2:9.0.1221-1.fc37
  vim-minimal 2:9.0.1054-1.fc37 -> 2:9.0.1221-1.fc37

[dustymabe]

closing this out since we are working around the problem on the testing side in coreos/fedora-coreos-pipeline#850 and there's not really much for us to do about it other than try to keep the bootloader up to date in the future.