-
Notifications
You must be signed in to change notification settings - Fork 34
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
resources: Add documentation about extending disk images #148
base: stable
Are you sure you want to change the base?
Changes from all commits
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,149 @@ | ||
--- | ||
|
||
layout: documentation | ||
title: Extending Pre-Built Disk Images Using Packer | ||
parent: gem5-standard-library | ||
doc: gem5 documentation | ||
permalink: /documentation/gem5-stdlib/extending-disk-images | ||
author: Harshil Patel | ||
|
||
--- | ||
|
||
gem5 Resources provides pre-built generic Ubuntu 24.04 disk images that you can extend and modify using Packer. | ||
|
||
### Steps | ||
|
||
1. **Download the Base Disk Image** | ||
|
||
Obtain the base disk image from gem5 Resources. | ||
These disk images are used to run x86, ARM, and RISC-V simulations in gem5 and are not related to the host system ISA. | ||
Links to the base images are available for [x86](https://resources.gem5.org/resources/x86-ubuntu-24.04-img), [ARM](https://resources.gem5.org/resources/arm-ubuntu-24.04-img), and [RISC-V](https://resources.gem5.org/resources/riscv-ubuntu-24.04-img). Download the image from the `versions` tab. | ||
|
||
2. **Unzip the Downloaded Image** | ||
|
||
Unzip the downloaded image using `gunzip`: | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. This is just a comment on a gripe of mine and not a request for any change but I've heard you say this before so feel I should mention it in case you don't already know; You don't need gnuzip to decompress a gzipped file: gzip disk.iso # creates a compressed file 'disk.iso.gz'
gzip -d disk.iso.gz # decompresses 'disk.iso.gz', creating a new 'disk.iso' I know gunzip is popular but I always found it odd to have these as two separate applications when one does just fine (FYI: gunzip is literally just gzip with the |
||
|
||
```sh | ||
gunzip /path/to/image | ||
``` | ||
|
||
3. **Calculate the `sha256sum` of the Unzipped Image** | ||
|
||
Packer requires the sha256sum of the base image we are using as a field called `iso_checksum`. | ||
Calculate the `sha256sum` of the unzipped image: | ||
|
||
```sh | ||
sha256sum /path/to/unzipped/image | ||
``` | ||
|
||
4. **Create a Packer Script** | ||
|
||
We use packer to automate the process of creating a disk image. | ||
Packer used qemu to make the disk. | ||
It will make the disk image and boot it with the arguments provided. | ||
It then connects to the image via ssh and runs any commands or instructions it is provided. | ||
Below is an example Packer script that extends a base image with placeholders: | ||
|
||
```hcl | ||
packer { | ||
required_plugins { | ||
qemu = { | ||
source = "github.com/hashicorp/qemu" | ||
version = "~> 1" | ||
} | ||
} | ||
} | ||
|
||
variable "image_name" { | ||
type = string | ||
default = "x86-ubuntu-24-04" # Update with your desired image name | ||
} | ||
|
||
variable "ssh_password" { | ||
type = string | ||
default = "12345" # Update if different for the base image, all 22.04 and 24.04 images on gem5 resources have this password | ||
} | ||
|
||
variable "ssh_username" { | ||
type = string | ||
default = "gem5" # Update if different for the base image, all 22.04 and 24.04 images on gem5 resources have this username | ||
} | ||
|
||
source "qemu" "initialize" { | ||
accelerator = "kvm" | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. So I need to have a KVM enabled machine? Does this work even if my disk image is X86 but my host is ARM? There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. It is set by the creator of the base image. I edited the comments to mention that the 22.04 and 24.04 images on gem5 resources have this user and password. |
||
boot_command = [ | ||
"<wait120>", // waiting for the disk image to boot, this will vary depending the host system building the image | ||
"gem5<enter><wait>", // loging in | ||
"12345<enter><wait>", // loging in | ||
"sudo mv /etc/netplan/50-cloud-init.yaml.bak /etc/netplan/50-cloud-init.yaml<enter><wait>", // re enable network as it is disabled by default | ||
"12345<enter><wait>", // password for sudo | ||
"sudo netplan apply<enter><wait>", // applying netplan changes | ||
"<wait>" | ||
] # This boot command logs in and re-enables the network so that Packer can connect via SSH | ||
cpus = "4" | ||
disk_size = "5000" | ||
format = "raw" | ||
headless = "true" | ||
disk_image = "true" | ||
iso_checksum = "sha256:# sha256sum of the base image" | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Do you need this? It's rather annoying they need to get the md5 of the disk image just to put it here. There's no way to disable the checksum? There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. it serves as a check that the file is not curropted. |
||
iso_urls = [""] # Path to the base image | ||
memory = "8192" | ||
output_directory = "" # Output directory path | ||
qemu_binary = "/usr/bin/qemu-system-x86_64" | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. I assume you need to update this for each of the 3 base images? There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. yes |
||
qemuargs = [["-cpu", "host"], ["-display", "none"]] | ||
shutdown_command = "echo '${var.ssh_password}'|sudo -S shutdown -P now" | ||
ssh_password = "${var.ssh_password}" | ||
ssh_username = "${var.ssh_username}" | ||
ssh_wait_timeout = "60m" | ||
vm_name = "${var.image_name}" | ||
ssh_handshake_attempts = "1000" | ||
} | ||
|
||
build { | ||
sources = ["source.qemu.initialize"] | ||
|
||
Harshil2107 marked this conversation as resolved.
Show resolved
Hide resolved
Comment on lines
+102
to
+104
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. I think it'd be worth explaining what quemu is and in what capacity you're using it here to work with the disk images There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. I added a small description in the begining of the section. |
||
provisioner "file" { # You can use this provisioner if you want to transfer files from your host machine to the disk image | ||
source = "# path to the file that you want to put on the disk image" | ||
destination = "# path (on the disk image) where you want to put the file" | ||
} | ||
|
||
provisioner "shell" { | ||
execute_command = "echo '${var.ssh_password}' | {{ .Vars }} sudo -E -S bash '{{ .Path }}'" | ||
scripts = ["# path to the post-install script that will extend the disk image"] | ||
expect_disconnect = true | ||
} | ||
} | ||
Comment on lines
+110
to
+115
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. This is the most important part of the whole thing and you really brushed over it. I'd make this more tutorially and have a good example here of what a script can do here to setup a disk image. Like, I have an application i want to be in the disk image. How do I put it there? How to I setup it up correctly so it runs when the US finishes booting? |
||
``` | ||
|
||
After modifying the above Packer script with the necessary information, you can run the Packer file. | ||
|
||
5. **Run the Packer File** | ||
|
||
To run the Packer file, use the following commands: | ||
|
||
```sh | ||
sudo apt-get update && sudo apt-get install packer # Installing Packer | ||
Comment on lines
+124
to
+125
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Don't you need to install qemu and stuff like |
||
|
||
packer init /path/to/packer/script | ||
packer build /path/to/packer/script | ||
``` | ||
|
||
**Note:** If you want to view the terminal output of the disk image during the build process, you can use a VNC viewer. Packer will provide a VNC port during the build, which you can connect to using a VNC viewer. | ||
|
||
## Important Notes | ||
|
||
- If you are extending the RISC-V base image, please note that you would need to remount the file system as read/write upon logging in before enabling network. | ||
The reason we need to do this is because qemu cant not handle `m5 exit` during the boot process properly so it boots into read only for safety. | ||
The boot command will look something like this: | ||
|
||
```hcl | ||
boot_command =["<wait45>", | ||
"gem5<enter><wait>", | ||
"12345<enter><wait>", | ||
"sudo mount -o remount,rw /<enter><wait>", // remounting file system | ||
"12345<enter><wait>", | ||
"sudo mv /etc/netplan/50-cloud-init.yaml.bak /etc/netplan/50-cloud-init.yaml<enter><wait>", | ||
"sudo netplan apply<enter><wait>", | ||
"<wait>" | ||
] | ||
``` |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Why's this bundled in with the standard library? I don't think that's the right parent.