Virtualbox builds failing on latest update

[ssplatt]

Description

Starting September 18, my Virtualbox image builds have failed with the initial configuration being left in 'gurumeditation' state. I tried updating Virtualbox, Vagrant, Packer, and all underlying software in the build script but none of it worked. I believe the issue is due to the kernel and/or hypervisor configuration in the Github Actions runner itself.

Platforms affected

• Azure DevOps
• GitHub Actions - Standard Runners
• GitHub Actions - Larger Runners

Runner images affected

• Ubuntu 20.04
• Ubuntu 22.04
• Ubuntu 24.04
• macOS 12
• macOS 13
• macOS 13 Arm64
• macOS 14
• macOS 14 Arm64
• Windows Server 2019
• Windows Server 2022

Image version and build link

Current runner version: '2.319.1'
Operating System
Ubuntu
22.04.5
LTS
Runner Image
Image: ubuntu-22.04
Version: 20240915.1.0
Included Software: https://github.com/actions/runner-images/blob/ubuntu22/20240915.1/images/ubuntu/Ubuntu2204-Readme.md
Image Release: https://github.com/actions/runner-images/releases/tag/ubuntu22%2F20240915.1
Runner Image Provisioner
2.0.384.1
GITHUB_TOKEN Permissions
Contents: read
Metadata: read
Packages: read

https://github.com/ssplatt/vagrant-box-centos-stream-9/actions/runs/10933205667/job/30351289562

Is it regression?

https://github.com/ssplatt/vagrant-box-centos-stream-9/actions/runs/10914285276

Expected behavior

virtual machine starts, runs, builds, and shuts down.

last good build:

Current runner version: '2.319.1'
Operating System
  Ubuntu
  [2](https://github.com/ssplatt/vagrant-box-centos-stream-9/actions/runs/10914285276/job/30292164221#step:1:2)2.04.4
  LTS
Runner Image
  Image: ubuntu-22.04
  Version: 20240908.1.0
  Included Software: https://github.com/actions/runner-images/blob/ubuntu22/20240908.1/images/ubuntu/Ubuntu2204-Readme.md
  Image Release: https://github.com/actions/runner-images/releases/tag/ubuntu22%2F20240908.1
Runner Image Provisioner
  2.0.[3](https://github.com/ssplatt/vagrant-box-centos-stream-9/actions/runs/10914285276/job/30292164221#step:1:3)84.1
GITHUB_TOKEN Permissions
  Contents: read
  Metadata: read
  Packages: read
 ...
 Run make build
  make build
  shell: /usr/bin/bash -e {0}
./scripts/build_box_virtualbox.sh
Installed plugin github.com/hashicorp/vagrant v1.1.5 in "/home/runner/.config/packer/plugins/github.com/hashicorp/vagrant/packer-plugin-vagrant_v1.1.5_x5.0_linux_amd64"
The configuration is valid.
==> vagrant.box: Creating a Vagrantfile in the build directory...
==> vagrant.box: Adding box using vagrant box add ...
    vagrant.box: (this can take some time if we need to download the box)
==> vagrant.box: Calling Vagrant Up (this can take some time)...
==> vagrant.box: Using SSH communicator to connect: 1[2](https://github.com/ssplatt/vagrant-box-centos-stream-9/actions/runs/10914285276/job/30292164221#step:5:2)7.0.0.1
==> vagrant.box: Waiting for SSH to become available...
==> vagrant.box: Connected to SSH!
==> vagrant.box: Provisioning with shell script: provision.sh
...
    vagrant.box: Complete!
==> vagrant.box: Created symlink /etc/systemd/system/multi-user.target.wants/vboxservice.service → /usr/lib/systemd/system/vboxservice.service.
==> vagrant.box: Removed "/etc/systemd/system/default.target".
==> vagrant.box: Created symlink /etc/systemd/system/default.target → /usr/lib/systemd/system/graphical.target.
==> vagrant.box: Packaging box...
==> vagrant.box: destroying Vagrant box...
Build 'vagrant.box' finished after 13 minutes 52 seconds.

==> Wait completed after 13 minutes 52 seconds

==> Builds finished. The artifacts of successful builds are:
--> vagrant.box: Vagrant box 'package.box' for 'virtualbox' provider

Actual behavior

Run vagrant --version
  vagrant --version
  packer --version
  vboxmanage --version
  shell: /usr/bin/bash -e {0}
Vagrant [2](https://github.com/ssplatt/vagrant-box-centos-stream-9/actions/runs/10933205667/job/30351289562#step:6:2).3.7
Packer v1.11.2
6.1.50_Ubuntur1610[3](https://github.com/ssplatt/vagrant-box-centos-stream-9/actions/runs/10933205667/job/30351289562#step:6:3)3


./scripts/build_box_virtualbox.sh
Installed plugin github.com/hashicorp/vagrant v1.1.[5](https://github.com/ssplatt/vagrant-box-centos-stream-9/actions/runs/10933205667/job/30351289562#step:7:6) in "/home/runner/.config/packer/plugins/github.com/hashicorp/vagrant/packer-plugin-vagrant_v1.1.5_x5.0_linux_amd64"
The configuration is valid.
==> vagrant.box: Creating a Vagrantfile in the build directory...
==> vagrant.box: Adding box using vagrant box add ...
    vagrant.box: (this can take some time if we need to download the box)
==> vagrant.box: Calling Vagrant Up (this can take some time)...
==> vagrant.box: Vagrant error: The guest machine entered an invalid state while waiting for it
==> vagrant.box: to boot. Valid states are 'starting, running'. The machine is in the
==> vagrant.box: 'gurumeditation' state. Please verify everything is configured
==> vagrant.box: properly and try again.
==> vagrant.box: 
==> vagrant.box: If the provider you're using has a GUI that comes with it,
==> vagrant.box: it is often helpful to open that and watch the machine, since the
==> vagrant.box: GUI often has more helpful error messages than Vagrant can retrieve.
==> vagrant.box: For example, if you're using VirtualBox, run `vagrant up` while the
==> vagrant.box: VirtualBox GUI is open.
==> vagrant.box: 
==> vagrant.box: The primary issue for this error is that the provider you're using
==> vagrant.box: is not properly configured. This is very rarely a Vagrant issue.
==> vagrant.box:
==> vagrant.box: Step "StepUp" failed, aborting...
==> vagrant.box: aborted: skipping cleanup of step "StepAddBox"
==> vagrant.box: aborted: skipping cleanup of step "StepCreateVagrantfile"
==> vagrant.box: aborted: skipping cleanup of step "StepOutputDir"
Build 'vagrant.box' errored after 54 seconds 54[7](https://github.com/ssplatt/vagrant-box-centos-stream-9/actions/runs/10933205667/job/30351289562#step:7:8) milliseconds: Vagrant error: The guest machine entered an invalid state while waiting for it
to boot. Valid states are 'starting, running'. The machine is in the
'gurumeditation' state. Please verify everything is configured
properly and try again.

If the provider you're using has a GUI that comes with it,
it is often helpful to open that and watch the machine, since the
GUI often has more helpful error messages than Vagrant can retrieve.
For example, if you're using VirtualBox, run `vagrant up` while the
VirtualBox GUI is open.

The primary issue for this error is that the provider you're using
is not properly configured. This is very rarely a Vagrant issue.


==> Wait completed after 54 seconds 547 milliseconds

==> Some builds didn't complete successfully and had errors:
--> vagrant.box: Vagrant error: The guest machine entered an invalid state while waiting for it
to boot. Valid states are 'starting, running'. The machine is in the
'gurumeditation' state. Please verify everything is configured
properly and try again.

If the provider you're using has a GUI that comes with it,
it is often helpful to open that and watch the machine, since the
GUI often has more helpful error messages than Vagrant can retrieve.
For example, if you're using VirtualBox, run `vagrant up` while the
VirtualBox GUI is open.

The primary issue for this error is that the provider you're using
is not properly configured. This is very rarely a Vagrant issue.


==> Builds finished but no artifacts were created.
make: *** [Makefile:4: build] Error 1
Error: Process completed with exit code 2.

Repro steps

start build, fail.

[benrobot]

I'm in a similar situation. My azure pipeline which starts a docker image as part of some unit tests was working fine on version 20240908.1.0 but stopped working on 20240915.1.0.

[danilapog]

Hey!
I also encountered this problem after updating the images to 20240915.1.0.
On images 20240908.1.0 all was fine.

[vidyasagarnimmagaddi]

Hi @ssplatt ,Thank you for bringing this issue to us. We are looking into this issue and will update you on this issue after investigating.

[ssplatt]

rolling back to ubuntu-20.04 does work, btw.

[dbast]

seeing this issue on ubuntu-latest (virtualbox 6) and ubuntu-24.04 (virtualbox 7) ... enabling video capture in packer via:

vboxmanage = [
    ["modifyvm", "{{.Name}}", "--recordingscreens", "all"],
    ["modifyvm", "{{.Name}}", "--recordingmaxsize", "500"],
    ["modifyvm", "{{.Name}}", "--recordingfile", "capture.webm"],
    ["modifyvm", "{{.Name}}", "--recording", "on"],

shows how the kernel boots, but then everything gets stuck with no error shown.

makes me wonder if those failures are subject to runner-image changes OR hypervisor changes OR issues with nested virtualisation?

[fallard84]

I'm facing the same issue since the release of runner image 20240915 . Using ubuntu-24.04 is also a workaround for those who can't use ubuntu-20.04.