tune2fs error causes ignition-ostree-uuid-boot.service to fail #735
Assignees
Labels
Comments
|
Can you provide the full logs for |
|
It's probably either |
|
@jlebon I could do so, if it would not continue to reboot ... Is there any magic key-combination or ignition option to boot to shel/ssh even with error occuring? |
|
@ArneCL you should get an emergency prompt in the initramfs, right after the failure. It can be activated by following the on-screen instructions (i.e. pressing Enter) within 5 minutes. If you don't see the prompt, you may first need to adjust the console settings according to your environment, see https://docs.fedoraproject.org/en-US/fedora-coreos/emergency-shell/. |
|
@lucab I do get a serial console - However I do not get a prompt to press enter :( However, in the GRUB-Edit section, i can see that two terminals are configured . Do I understand correctly that the emergency-console might be displayed on the "wrong" terminal, if two terminals are configured? |
|
@ArneCL Yes. You may need to remove one console from the kernel command line arguments. |
|
@ArneCL yes, it may happen if you only have a virtual console and no serial. In that case, dropping the wrong entry from the command-line (or even both, if kernel autodetection can handle your hardware) should give you back the missing prompt. |
|
@bgilbert @lucab Unfortunately, this is giving me a hard time: As stated I do have access to the serial console - However, removing the second console in the GRUB-Edit does not seem to get me an emergency console, only a "virtual console" . I can see that my changes to the GRUB-Edit take effect, i.e. i can also disable the console I'm accessing it with - Thus giving me a blank screen after hitting ctrl-x to boot.... Any Ideas? Can I provide you with more Input from my Side? |
|
@bgilbert @lucab Good news - I was able to obtain some additional debug info by routing the systemd-output to console by setting the kernel parameters and also |
|
I'm getting the same issue withe FCOS 33.20210301.3.1, see the console log below. |
|
@openstacker This looks like a different but serious issue. In your case, it looks like no restamping occurred unlike the original post. I've split it out into a separate issue: #784. |
|
@jlebon
Hopefully this provides enough information to fix this issue? If you need more details/logs/... feel free to ask! Ignition File |
|
Hey @ArneCL - I'm able to reproduce this on some local hardware I own. Unfortunately I think What does work for me is to change the UUID after install but before reboot. Something like: The We need to figure out how to make this problem go away, though. So that we don't need a workaround. |
|
I don't understand why this would only happen in some circumstances, or just on some hardware. |
Race condition maybe? In my case I'm using a local build of FCOS for aarch64 and trying to install to a Raspberry Pi4 targetting a external USB drive (2T). |
|
I believe I am experiencing this same issue on a different hypervisor (KVM/libvirt) and I'm using RHCOS. The commonality is I am using TrueNAS SCALE and the underlying VirtIO/AHCI device being virtualized is backed by a ZFS ZVOL. This might have nothing to do with the loop, but I thought it was interesting that I share this with the OP. I was attempting to run OCP 4.8 fc3 in my home lab to do some learning. However this boot loop was not resolved by the above instructions about tune2fs - even after e2fsck -yf, or, using tune2fs to mark the device as dirty and fscking again, tune2fs continued to refuse the -U random command, claiming the disk was not clean. Every other utility reported that the disk was clean. Is there any other work around you could suggest to get past the uuid service not starting? Are there any logs I could produce that would be helpful to the cause? |
|
BTW I've been having this same bootloop behavior with FCOS on the first boot, and I assumed I was doing something wrong with my ignition config. Once I hit this in OCP I figured out how to access my serial console for the emergency shell and did some deeper digging, which lead me to this issue. I can readily reproduce this with either FCOS or RHCOS, whichever helps. |
|
@bhale - are you using |
|
dustymabe
changed the title
|
OK we've been given a patched grub that allows us to use the
TL;DR It's a workaround for this problem and should prevent the need for us to run any filesystem checking before we run @ArneCL @openstacker @bhale - can I get you to test and report if the following dev builds help? |
dustymabe
added a commit
to dustymabe/fedora-coreos-config
that referenced
this issue
Jun 11, 2021
…ature We're hitting issues in some cases where users are seeing the `tune2fs -U random` on the /boot filesystem fail even though it's ran directly after an `e2fsck`. To prevent needing a filesystem check at all let's store the metadata checksum seed in the superblock which will allow us to change the UUID without needing any filesystem checks. This was not possible before but we recently got a grub fix so that having this filesystem feature enabled wouldn't cause grub2 to throw an error. - https://lists.gnu.org/archive/html/grub-devel/2021-06/msg00031.html Fixes: coreos/fedora-coreos-tracker#735
This was referenced Jun 11, 2021
Merged
Closed
|
@dustymabe That did the trick for me. Thank you for looking into this. |
|
OK Now I've got an easy reproducer for the problem, just setting the time to something in the not too distant past right before running the fsck and tune2fs. $ git diff HEAD^ overlay.d/05core/usr/lib/dracut/modules.d/40ignition-ostree/ignition-ostree-firstboot-uuid
diff --git a/overlay.d/05core/usr/lib/dracut/modules.d/40ignition-ostree/ignition-ostree-firstboot-uuid b/overlay.d/05core/usr/lib/dracut/modules.d/40ignition-ostree/ignition-ostree-firstboot-uuid
index 0027bed..c9d734f 100755
--- a/overlay.d/05core/usr/lib/dracut/modules.d/40ignition-ostree/ignition-ostree-firstboot-uuid
+++ b/overlay.d/05core/usr/lib/dracut/modules.d/40ignition-ostree/ignition-ostree-firstboot-uuid
@@ -27,10 +27,9 @@ esac
if [ "${TYPE}" == "${orig_type}" ] && [ "${UUID}" == "${orig_uuid}" ]; then
case "${TYPE}" in
- # For now we need to fsck first, see https://github.com/coreos/coreos-assembler/pull/1452
- # Basically we're not passing `metadata_csum_seed` as a mkfs.ext4 option
- # because grub2 barfs on it.
- ext4) e2fsck -fy "${target}" && tune2fs -U random "${target}" ;;
+ ext4)
+ date --set="2021-04-14 20:12:00";
+ e2fsck -fy "${target}" && tune2fs -U random "${target}" ;;
xfs) xfs_admin -U generate "${target}" ;;
*) echo "unexpected filesystem type ${TYPE}" 1>&2; exit 1 ;;
esac |
dustymabe
added a commit
to dustymabe/fedora-coreos-config
that referenced
this issue
Jun 14, 2021
…ature We're hitting issues in some cases where users are seeing the `tune2fs -U random` on the /boot filesystem fail even though it's ran directly after an `e2fsck`. To prevent needing a filesystem check at all let's store the metadata checksum seed in the superblock which will allow us to change the UUID without needing any filesystem checks. This was not possible before but we recently got a grub fix so that having this filesystem feature enabled wouldn't cause grub2 to throw an error. - https://lists.gnu.org/archive/html/grub-devel/2021-06/msg00031.html Fixes: coreos/fedora-coreos-tracker#735
dustymabe
added a commit
to dustymabe/fedora-coreos-config
that referenced
this issue
Jun 14, 2021
…t metadata_csum_seed It will be some time before we can support metadata_csum_seed in grub on all our target platforms. This commit adds a workaround for the case where grub doesn't support it yet. We specifically workaround the behavior of tune2fs that isn't sensitive to timing inconsistencies. See coreos/fedora-coreos-tracker#735 (comment) Fixes: coreos/fedora-coreos-tracker#735
dustymabe
added a commit
to dustymabe/fedora-coreos-config
that referenced
this issue
Jun 15, 2021
…t metadata_csum_seed It will be some time before we can support metadata_csum_seed in grub on all our target platforms. This commit adds a workaround for the case where grub doesn't support it yet. We specifically workaround the behavior of tune2fs that isn't sensitive to timing inconsistencies. See coreos/fedora-coreos-tracker#735 (comment) Fixes: coreos/fedora-coreos-tracker#735
jlebon
pushed a commit
to coreos/fedora-coreos-config
that referenced
this issue
Jun 15, 2021
…ature We're hitting issues in some cases where users are seeing the `tune2fs -U random` on the /boot filesystem fail even though it's ran directly after an `e2fsck`. To prevent needing a filesystem check at all let's store the metadata checksum seed in the superblock which will allow us to change the UUID without needing any filesystem checks. This was not possible before but we recently got a grub fix so that having this filesystem feature enabled wouldn't cause grub2 to throw an error. - https://lists.gnu.org/archive/html/grub-devel/2021-06/msg00031.html Fixes: coreos/fedora-coreos-tracker#735
jlebon
pushed a commit
to coreos/fedora-coreos-config
that referenced
this issue
Jun 15, 2021
…t metadata_csum_seed It will be some time before we can support metadata_csum_seed in grub on all our target platforms. This commit adds a workaround for the case where grub doesn't support it yet. We specifically workaround the behavior of tune2fs that isn't sensitive to timing inconsistencies. See coreos/fedora-coreos-tracker#735 (comment) Fixes: coreos/fedora-coreos-tracker#735
dustymabe
added
the
status/pending-testing-release
dustymabe
added a commit
to dustymabe/coreos-assembler
that referenced
this issue
Jun 15, 2021
Add support for a bootfs_metadata_csum_seed image.yaml knob that tells us to enabled the metadata_csum_seed ext4 filesystem feature. This is gated behind an image.yaml knob because support for this feature flag was only recently added to grub. https://lists.gnu.org/archive/html/grub-devel/2021-06/msg00031.html This helps us not have to run a filesystem check before running tune2fs to randomize the filesystem UUID on first boot. See coreos/fedora-coreos-tracker#735 for more context. The knob in image.yaml will look like: ``` bootfs_metadata_csum_seed: true ```
dustymabe
added a commit
to dustymabe/fedora-coreos-config
that referenced
this issue
Jun 15, 2021
This will tell COSA to set the metadata_csum_seed filesystem feature for the boot filesystem. This helps us not have to run a filesystem check before running tune2fs to randomize the filesystem UUID on first boot. See coreos/fedora-coreos-tracker#735 for more context.
dustymabe
added a commit
to dustymabe/fedora-coreos-config
that referenced
this issue
Jun 15, 2021
This will tell COSA to set the metadata_csum_seed filesystem feature for the boot filesystem. This helps us not have to run a filesystem check before running tune2fs to randomize the filesystem UUID on first boot. See coreos/fedora-coreos-tracker#735 for more context. Depends on coreos/coreos-assembler#2228
dustymabe
added a commit
to dustymabe/coreos-assembler
that referenced
this issue
Jun 16, 2021
Add support for a bootfs_metadata_csum_seed image.yaml knob that tells us to enabled the metadata_csum_seed ext4 filesystem feature. This is gated behind an image.yaml knob because support for this feature flag was only recently added to grub. https://lists.gnu.org/archive/html/grub-devel/2021-06/msg00031.html This helps us not have to run a filesystem check before running tune2fs to randomize the filesystem UUID on first boot. See coreos/fedora-coreos-tracker#735 for more context. The knob in image.yaml will look like: ``` bootfs_metadata_csum_seed: true ```
dustymabe
added a commit
to dustymabe/coreos-assembler
that referenced
this issue
Jun 16, 2021
Add support for a bootfs_metadata_csum_seed image.yaml knob that tells us to enable the metadata_csum_seed ext4 filesystem feature. This is gated behind an image.yaml knob because support for this feature flag was only recently added to grub. https://lists.gnu.org/archive/html/grub-devel/2021-06/msg00031.html This helps us not have to run a filesystem check before running tune2fs to randomize the filesystem UUID on first boot. See coreos/fedora-coreos-tracker#735 for more context. The knob in image.yaml will look like: ``` bootfs_metadata_csum_seed: true ```
darkmuggle
pushed a commit
to coreos/coreos-assembler
that referenced
this issue
Jun 16, 2021
Add support for a bootfs_metadata_csum_seed image.yaml knob that tells us to enable the metadata_csum_seed ext4 filesystem feature. This is gated behind an image.yaml knob because support for this feature flag was only recently added to grub. https://lists.gnu.org/archive/html/grub-devel/2021-06/msg00031.html This helps us not have to run a filesystem check before running tune2fs to randomize the filesystem UUID on first boot. See coreos/fedora-coreos-tracker#735 for more context. The knob in image.yaml will look like: ``` bootfs_metadata_csum_seed: true ```
dustymabe
added a commit
to coreos/fedora-coreos-config
that referenced
this issue
Jul 1, 2021
This will tell COSA to set the metadata_csum_seed filesystem feature for the boot filesystem. This helps us not have to run a filesystem check before running tune2fs to randomize the filesystem UUID on first boot. See coreos/fedora-coreos-tracker#735 for more context. Depends on coreos/coreos-assembler#2228
|
The fix for this went into testing stream release |
dustymabe
added
status/pending-stable-release
|
The fix for this went into stable stream release |
dustymabe
removed
the
status/pending-stable-release
ravanelli
pushed a commit
to ravanelli/coreos-assembler
that referenced
this issue
Aug 25, 2021
Add support for a bootfs_metadata_csum_seed image.yaml knob that tells us to enable the metadata_csum_seed ext4 filesystem feature. This is gated behind an image.yaml knob because support for this feature flag was only recently added to grub. https://lists.gnu.org/archive/html/grub-devel/2021-06/msg00031.html This helps us not have to run a filesystem check before running tune2fs to randomize the filesystem UUID on first boot. See coreos/fedora-coreos-tracker#735 for more context. The knob in image.yaml will look like: ``` bootfs_metadata_csum_seed: true ```
HuijingHei
pushed a commit
to HuijingHei/fedora-coreos-config
that referenced
this issue
Oct 10, 2023
…ature We're hitting issues in some cases where users are seeing the `tune2fs -U random` on the /boot filesystem fail even though it's ran directly after an `e2fsck`. To prevent needing a filesystem check at all let's store the metadata checksum seed in the superblock which will allow us to change the UUID without needing any filesystem checks. This was not possible before but we recently got a grub fix so that having this filesystem feature enabled wouldn't cause grub2 to throw an error. - https://lists.gnu.org/archive/html/grub-devel/2021-06/msg00031.html Fixes: coreos/fedora-coreos-tracker#735
HuijingHei
pushed a commit
to HuijingHei/fedora-coreos-config
that referenced
this issue
Oct 10, 2023
…t metadata_csum_seed It will be some time before we can support metadata_csum_seed in grub on all our target platforms. This commit adds a workaround for the case where grub doesn't support it yet. We specifically workaround the behavior of tune2fs that isn't sensitive to timing inconsistencies. See coreos/fedora-coreos-tracker#735 (comment) Fixes: coreos/fedora-coreos-tracker#735
HuijingHei
pushed a commit
to HuijingHei/fedora-coreos-config
that referenced
this issue
Oct 10, 2023
This will tell COSA to set the metadata_csum_seed filesystem feature for the boot filesystem. This helps us not have to run a filesystem check before running tune2fs to randomize the filesystem UUID on first boot. See coreos/fedora-coreos-tracker#735 for more context. Depends on coreos/coreos-assembler#2228
HuijingHei
pushed a commit
to HuijingHei/fedora-coreos-config
that referenced
this issue
Oct 10, 2023
…ature We're hitting issues in some cases where users are seeing the `tune2fs -U random` on the /boot filesystem fail even though it's ran directly after an `e2fsck`. To prevent needing a filesystem check at all let's store the metadata checksum seed in the superblock which will allow us to change the UUID without needing any filesystem checks. This was not possible before but we recently got a grub fix so that having this filesystem feature enabled wouldn't cause grub2 to throw an error. - https://lists.gnu.org/archive/html/grub-devel/2021-06/msg00031.html Fixes: coreos/fedora-coreos-tracker#735
HuijingHei
pushed a commit
to HuijingHei/fedora-coreos-config
that referenced
this issue
Oct 10, 2023
…t metadata_csum_seed It will be some time before we can support metadata_csum_seed in grub on all our target platforms. This commit adds a workaround for the case where grub doesn't support it yet. We specifically workaround the behavior of tune2fs that isn't sensitive to timing inconsistencies. See coreos/fedora-coreos-tracker#735 (comment) Fixes: coreos/fedora-coreos-tracker#735
HuijingHei
pushed a commit
to HuijingHei/fedora-coreos-config
that referenced
this issue
Oct 10, 2023
This will tell COSA to set the metadata_csum_seed filesystem feature for the boot filesystem. This helps us not have to run a filesystem check before running tune2fs to randomize the filesystem UUID on first boot. See coreos/fedora-coreos-tracker#735 for more context. Depends on coreos/coreos-assembler#2228
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Describe the bug
After installing FCOS on a BHyve VM, the VM gets stuck in a boot-loop. On the emergency-console, I can see the error
messages pasted below:
Failed to start Ignition OSTree: Generate filesystem UUID (boot). The error occurs with VirtIO disks as well as with AHCI Disks (UEFI).Please note that FCOS 31.20200517.3.0-live.x86_64.iso installs and runs fine, however FCOS >=32.20201104.3.0-live.x86_64.iso fails with the described error.
Reproduction steps
Steps to reproduce the behavior:
curl -LOsudo coreos-installer install /dev/vda --ignition fcos.ignExpected behavior
FCOS / VM booting to shell
A clear and concise description of what actually happened.
Actual behavior
FCOS rebooting due to error while creating Filesystem UUID
System details
Ignition config
Additional information
Add any other information about the problem here.
The text was updated successfully, but these errors were encountered: