spike: Create a kairos "installer"

[jimmykarily]

Right now, the way to create a Kairos "derivative" image is to either feed a base image to our dockerfiles of take one of our images and use it as a base image for another dockerfile.

Our dockerfiles are rather complex and the logic very hard to follow. This makes them very difficult to extend and test.
Also, using dockerfiles, makes docker a hard dependency (or some other tool that can build dockerfiles anyway).

We could move all the dockerfile logic in a go binary (let's call it the "installer" for now but we need a better name to avoid confusion with the agent that installs Kairos on the machines). By doing that, one would simply run the binary inside an OS and all the dependencies to make it a Kairos OS would be installed with no additional dependencies.

[bencorrado]

I would love to see a factory provisioning step added for this, ideally this would be a yip/cloudconfig file that is loaded at installation to augment the installation process that is not persisted to the system. I have gotten called out on security audits for leaving installation scripts on deployed devices as it gives more information than is needed to a potential hacker about how the system is bootstrapped.

Currently we run things in

  kairos-uki-install.after:
    - commands:
      - |

which allow us to add unique keys, names, capture device identities, etc.

and then at the end of that stage, before the first boot, we actually remove that whole block like this:

  # Cleanup unwanted stages from YAML file
  echo "Cleaning up the bootstrapping instructions"
  /oem/yq eval -i 'del(.stages."kairos-uki-install.after")' /oem/90_custom.yaml
  rm /oem/yq

It works, but it feels crude and fragile. If I could just add another cloud-config to the install process (we use a PiKVM to mount a unique disks to each device, one the Kairos installer, the other the user-data with the yip/cloud-config) It could be nice to be able to use an installer-data file too.

[mudler]

We need to first spend a bit of time in designing an architecture (high-level) which satisfies our current needs before jumping directly into implementing a solution. Moving to a spike

[tbrasser]

I was thinking of trying https://zarf.dev for k3s and up, might also be interesting for kairos?

[Itxaka]

https://github.com/Itxaka/kairos-init

[Itxaka]

I was thinking of trying zarf.dev for k3s and up, might also be interesting for kairos?

had a look at this and while its nice and it fills some of our gaps (single binary, yummy!) its also heavily skewed towards runnig as part of a k8s cluster :(

Ideally we want to have something like this:

FROM quay.io/kairos/kairos-init:v1.0.0 AS kairos-init

FROM ubuntu:24.04
COPY --from=kairos-init kairos-init /kairos-init
RUN /kairos-init
RUN rm /kairos-init

And that would build a container with all the bits and blops inside to be either used to build an iso with AuroraBoot or use it directly as upgrade source for a existing kairos system.

So lowing the barrier of converting an existing container into a kairos-ready container.

kairos-init should:

• Work in different Os (fedora, ubuntu, rocky, opensuse, etc...) with no distinction
• install required packages (cryptsetup for encryption, disk stuff for partitioning, init system, etc...)
• install kairos framework (immucore, agent, default cloud configs)
• prepare the kernel and regenerate initramfs (build initramfs with immucore+agent on it, softlink kernel and initrd to expected places)

Those seem like the minimal things we need to get a kairos ready container.

Nice things to have:

• Everything as a feature. Apart from immutability with immucore and agent for day2 operations, more things should be not hardcoded. As an example, maybe expanding disks on boot is not necessary, or XFS/LVM filesystem support so those should be toggled. If not required, packages should be removed, immucore should not require those in the initramfs, some steps need to check before enabling or running those steps like yip and such. This would allow us more control over the feature set and help reduce the footprint.
• kernel selection. Instead of just installing the default kernel, a more filtered selection could be done. Modules, firmware, maybe even versions if the system has several to choose from.
• a trigger system. Installing a new kernel or a new feature needs a rebuild of the initrd, so a trigger system needs to be implemented that can trigger rebuild of other features when needed. It needs to be simple enough to use and nto retrigger things twice, so a priority system may be needed? (i.e. initrd needs to be rebuild at the end of everything)
• Maybe instead of the above, a DAG needs to be implemented to build the system, so we can depend on one step to the other and some steps can run parallel.
• Extensions or plugins need to be implemented, so its easy to add your own features to the init thing. I.E. I need a specific configuration or some special packages, I can drop a plugin that installs them and runs at the proper place with depends and triggers
• some type of config to expose extra things as well. Maybe users dont want to create a plugin to install some extra packages, so we have to think about how to expose that config so they can just drop a config file
• Aurora integration. Could be nice to have it as simple as a command for aurora to build an iso and just pass the base image and get a final iso. Ideally this would be the entrypoint for the future of kairos instead of earthfiles, just passing a base image and aurora would output the given thing, iso, uki, container, netboot, whatever.
• hooks before and after every step! like the agent has hooks, this should have as well. Yip configs that can be run per step easily.

[jimmykarily]

Nice things to have:

* Everything as a feature. Apart from immutability with immucore and agent for day2 operations, more things should be not hardcoded. As an example, maybe expanding disks on boot is not necessary, or XFS/LVM filesystem support so those should be toggled. If not required, packages should be removed, immucore should not require those in the initramfs, some steps need to check before enabling or running those steps like yip and such. This would allow us more control over the feature set and help reduce the footprint.

* kernel selection. Instead of just installing the default kernel, a more filtered selection could be done. Modules, firmware, maybe even versions if the system has several to choose from.

This would be awesome. This way, maybe we can implement the complex process of installing the "firmware" package and then removing all unnecessary drivers from uki images in go so we can create smaller bootable ISOs for trusted boot.

* a trigger system. Installing a new kernel or a new feature needs a rebuild of the initrd, so a trigger system needs to be implemented that can trigger rebuild of other features when needed. It needs to be simple enough to use and nto retrigger things twice, so a priority system may be needed? (i.e. initrd needs to be rebuild at the end of everything)

Is this meant to be used in CI? It's not a bad idea but I would postpone it a bit to first get the basic functionality.

* Maybe instead of the above, a DAG needs to be implemented to build the system, so we can depend on one step to the other and some steps can run parallel.

Most of the things kairos-init will do, will involve installing packages and that can't run in parallel (apt et al, only allow one instance to be running). Unless it really is a graph or dependent steps, I would avoid making it look like one.

* Extensions or plugins need to be implemented, so its easy to add your own features to the init thing. I.E. I need a specific configuration or some special packages, I can drop a plugin that installs them and runs at the proper place with depends and triggers

If people need to do additional stuff on top of our base images, shouldn't they just create their own Dockerfiles? Is kairos-init meant to be used for derivative images too? Maybe we should limit its use to "official" Kairos base images and let everyone else work with Dockerfiles instead of implementing plugins for kairos-init?

* some type of config to expose extra things as well. Maybe users dont want to create a plugin to install some extra packages, so we have to think about how to expose that config so they can just drop a config file

See my previous comment.

* Aurora integration. Could be nice to have it as simple as a command for aurora to build an iso and just pass the base image and get a final iso. Ideally this would be the entrypoint for the future of kairos instead of earthfiles, just passing a base image and aurora would output the given thing, iso, uki, container, netboot, whatever.

Relevant: #1546

* hooks before and after every step! like the agent has hooks, this should have as well. Yip configs that can be run per step easily.

I'm not sure if I like this or not. I was thinking that this tool would replace Dockerfiles making it easier to understand what we install/run/whatever. This suggestion here, goes towards a declarative "build recipe". So instead of the Go code, the steps will be defined in a config which is yet another moving part. To be honest, I'd rather have everything hardcoded in Go than having yamls modifying the process.

[Itxaka]

• a trigger system. Installing a new kernel or a new feature needs a rebuild of the initrd, so a trigger system needs to be implemented that can trigger rebuild of other features when needed. It needs to be simple enough to use and nto retrigger things twice, so a priority system may be needed? (i.e. initrd needs to be rebuild at the end of everything)

Is this meant to be used in CI? It's not a bad idea but I would postpone it a bit to first get the basic functionality.

No but the idea of enabling disable things in a more dynamic nature, means that things need to be ordered so tey trigger at the correct time :D

* Maybe instead of the above, a DAG needs to be implemented to build the system, so we can depend on one step to the other and some steps can run parallel.

Most of the things kairos-init will do, will involve installing packages and that can't run in parallel (apt et al, only allow one instance to be running). Unless it really is a graph or dependent steps, I would avoid making it look like one.

Well, maybe but its a nice way of ordering things and if we expect in the future to have toggle features that you can enabled/disable maybe its the way to go?

* Extensions or plugins need to be implemented, so its easy to add your own features to the init thing. I.E. I need a specific configuration or some special packages, I can drop a plugin that installs them and runs at the proper place with depends and triggers

If people need to do additional stuff on top of our base images, shouldn't they just create their own Dockerfiles? Is kairos-init meant to be used for derivative images too? Maybe we should limit its use to "official" Kairos base images and let everyone else work with Dockerfiles instead of implementing plugins for kairos-init?

I have no idea, but I was thinking also about our usecase. Plugins could work for things like different arches or different boards, without needing to fully change the code.

* some type of config to expose extra things as well. Maybe users dont want to create a plugin to install some extra packages, so we have to think about how to expose that config so they can just drop a config file

See my previous comment.

* Aurora integration. Could be nice to have it as simple as a command for aurora to build an iso and just pass the base image and get a final iso. Ideally this would be the entrypoint for the future of kairos instead of earthfiles, just passing a base image and aurora would output the given thing, iso, uki, container, netboot, whatever.

Relevant: #1546

* hooks before and after every step! like the agent has hooks, this should have as well. Yip configs that can be run per step easily.

I'm not sure if I like this or not. I was thinking that this tool would replace Dockerfiles making it easier to understand what we install/run/whatever. This suggestion here, goes towards a declarative "build recipe". So instead of the Go code, the steps will be defined in a config which is yet another moving part. To be honest, I'd rather have everything hardcoded in Go than having yamls modifying the process.

Im just saying that at one point somebody is gonna ask for that. Because they need to do X or they need do Y and we will need to add it sooner or later. Just thinking about having it in place and getting it out of the way looool

[jimmykarily]

I'm not sure if I like this or not. I was thinking that this tool would replace Dockerfiles making it easier to understand what we install/run/whatever. This suggestion here, goes towards a declarative "build recipe". So instead of the Go code, the steps will be defined in a config which is yet another moving part. To be honest, I'd rather have everything hardcoded in Go than having yamls modifying the process.

Im just saying that at one point somebody is gonna ask for that. Because they need to do X or they need do Y and we will need to add it sooner or later. Just thinking about having it in place and getting it out of the way looool

The discussion goes back to the scope of this tool. If it's meant to help people build Kairos derivatives, then people might ask for such things, yes. But if it's only meant to create our official images, then we don't need any hooks. We make it part of the regular steps and we are done.

If we allow dynamically extending what the tool does, we are in danger of implementing our own generic alternative of Dockerfiles which was not the goal. What we want, is an alternative to Dockerfiles, only for one specific case, ours.
What I mean is, in our case, in which we are building many different flavors, using different base images and other input (arch, target device, boot manager etc), Dockerfiles fell short.
Most of our users though, they start with just one such combination and they only want to extend that one. For example, a user will want to extend the Kairos base image for [ubuntu, 24.10, arm, rpi4, core]. For that, a Dockerfile is probably enough and there is no reason why they would expect their changes should be possible through kairos-init.

[Itxaka]

I'm not sure if I like this or not. I was thinking that this tool would replace Dockerfiles making it easier to understand what we install/run/whatever. This suggestion here, goes towards a declarative "build recipe". So instead of the Go code, the steps will be defined in a config which is yet another moving part. To be honest, I'd rather have everything hardcoded in Go than having yamls modifying the process.

Im just saying that at one point somebody is gonna ask for that. Because they need to do X or they need do Y and we will need to add it sooner or later. Just thinking about having it in place and getting it out of the way looool

The discussion goes back to the scope of this tool. If it's meant to help people build Kairos derivatives, then people might ask for such things, yes. But if it's only meant to create our official images, then we don't need any hooks. We make it part of the regular steps and we are done.

If we allow dynamically extending what the tool does, we are in danger of implementing our own generic alternative of Dockerfiles which was not the goal. What we want, is an alternative to Dockerfiles, only for one specific case, ours. What I mean is, in our case, in which we are building many different flavors, using different base images and other input (arch, target device, boot manager etc), Dockerfiles fell short. Most of our users though, they start with just one such combination and they only want to extend that one. For example, a user will want to extend the Kairos base image for [ubuntu, 24.10, arm, rpi4, core]. For that, a Dockerfile is probably enough and there is no reason why they would expect their changes should be possible through kairos-init.

OK cool, that makes it clearer then! I was thinking that this may be used like the factory to generate kairos images for anybody, but indeed, if its used inside a dockerfile then it makes sense that people will build the derivatives with docker as they do now.

[jimmykarily]

@kairos-io/maintainers ^ thoughts? Let's agree on the scope of the tool before we proceed.

[jimmykarily]

btw, I wouldn't even create releases of the tool. We can build it from source (specific git hash) in the dockerfile in a separate stage (FROM golang AS builder etc). It should be easy for us to make changes and test them, without tagging, releasing etc. The same way a bash script or a dockerfile would work. Anyway, that's technical details for later.

[mudler]

I was thinking of trying zarf.dev for k3s and up, might also be interesting for kairos?

had a look at this and while its nice and it fills some of our gaps (single binary, yummy!) its also heavily skewed towards runnig as part of a k8s cluster :(

Ideally we want to have something like this:

FROM quay.io/kairos/kairos-init:v1.0.0 AS kairos-init

FROM ubuntu:24.04
COPY --from=kairos-init kairos-init /kairos-init
RUN /kairos-init
RUN rm /kairos-init

And that would build a container with all the bits and blops inside to be either used to build an iso with AuroraBoot or use it directly as upgrade source for a existing kairos system.

So lowing the barrier of converting an existing container into a kairos-ready container.

💯

kairos-init should:

* Work in different Os (fedora, ubuntu, rocky, opensuse, etc...) with no distinction

* install required packages (cryptsetup for encryption, disk stuff for partitioning, init system, etc...)

* install kairos framework (immucore, agent, default cloud configs)

* prepare the kernel and regenerate initramfs (build initramfs with immucore+agent on it, softlink kernel and initrd to expected places)

On top of my mind I would add:

• fallback to a generic configuration if the flavor isn't detected/supported officially (with a big warning displayed)
• handle installation profile: e.g. we can encapsulate "fips-mode" "uki-mode" (which could be combined) in specific profiles that we maintain in the binary. How we maintain these in the binary can be in whatever way we want (YAML files? Golang files?)

Those seem like the minimal things we need to get a kairos ready container.

Nice things to have:

* Everything as a feature. Apart from immutability with immucore and agent for day2 operations, more things should be not hardcoded. As an example, maybe expanding disks on boot is not necessary, or XFS/LVM filesystem support so those should be toggled. If not required, packages should be removed, immucore should not require those in the initramfs, some steps need to check before enabling or running those steps like yip and such. This would allow us more control over the feature set and help reduce the footprint.

mh albeit useful, this is going to make things really complex, so we should limit it in a way or another.

My take here is to introduce e.g. the concept of "profiles" that encapsulates a ruleset, and avoid completely free-form configuration.
For now we could have e.g. --profile flag to select a combination of configurations.

* kernel selection. Instead of just installing the default kernel, a more filtered selection could be done. Modules, firmware, maybe even versions if the system has several to choose from.

good point here. Kernel selection and system packages can be tricky. I would provide a flag instead to piggyback that to the user if possible these (e.g. --no-kernel-install, --no-firmware-install, etc.) or even assume all the packages are already installed (e.g. to drop only static files).

* a trigger system. Installing a new kernel or a new feature needs a rebuild of the initrd, so a trigger system needs to be implemented that can trigger rebuild of other features when needed. It needs to be simple enough to use and nto retrigger things twice, so a priority system may be needed? (i.e. initrd needs to be rebuild at the end of everything)

I think this should be part of the kairos init command, and we should assume that is run at the end of the Dockerfile always.

At that point I expect no other action to be made (e.g. rebuilding initrd).

* Maybe instead of the above, a DAG needs to be implemented to build the system, so we can depend on one step to the other and some steps can run parallel.

No strong preference here, but if we see that many of the steps can be run in parallel then would make totally sense

* Extensions or plugins need to be implemented, so its easy to add your own features to the init thing. I.E. I need a specific configuration or some special packages, I can drop a plugin that installs them and runs at the proper place with depends and triggers

I'm having a bit of mixed feelings on this: I would rather simply let the user define any other change/configuration before invoking kairos init, and we assume that is the last command they should run in the Dockerfile (as it would need to probably rebuild initrd, and do some operations that have to be "conclusive")

* some type of config to expose extra things as well. Maybe users dont want to create a plugin to install some extra packages, so we have to think about how to expose that config so they can just drop a config file

ditto on the above

* Aurora integration. Could be nice to have it as simple as a command for aurora to build an iso and just pass the base image and get a final iso. Ideally this would be the entrypoint for the future of kairos instead of earthfiles, just passing a base image and aurora would output the given thing, iso, uki, container, netboot, whatever.

That'd be super-useful!

* hooks before and after every step! like the agent has hooks, this should have as well. Yip configs that can be run per step easily.

I'm leaning towards @jimmykarily 's points here. I'd probably stick with some concept like "build profiles". This is because mainly kairos-init isn't meant to run during runtime, but only as a build-only command.

btw, I wouldn't even create releases of the tool. We can build it from source (specific git hash) in the dockerfile in a separate stage (FROM golang AS builder etc). It should be easy for us to make changes and test them, without tagging, releasing etc. The same way a bash script or a dockerfile would work. Anyway, that's technical details for later.

It definitely should and this is one of the most important parts: we want replace instructions that redirect the user to select framework images when going over the current BYOI flow. We should provide then an option to just download a binary of a specific version, and run a command on it instead of have to mangle with Dockerfile or Earthfiles and pinning framework images.

If not part of the agent, the kairos init command should be a standalone binary and be treated as a first-class citizen.

[Itxaka]

ok, a basic form of it is now under https://github.com/Itxaka/kairos-init

• It can build for ubuntu 20.04/22.04/24.04/24.10
• It passes most of the tests https://github.com/kairos-io/kairos/actions/runs/12127513510
• The Dockerfile to transform an image is:

FROM golang AS build
WORKDIR /app
COPY go.mod go.sum .
RUN go mod download
COPY . .
ENV CGO_ENABLED=0
RUN go build -o /app/kairos-init .

FROM ubuntu:24.04 AS base-kairos
COPY --from=build /app/kairos-init /kairos-init
RUN /kairos-init -l debug -f all
RUN rm /kairos-init

Another example of a multi-distro Dockerfile can be found at the kairos PR #3044

ARG FLAVOR=24.04

FROM golang AS build
WORKDIR /app
RUN git clone https://github.com/Itxaka/kairos-init.git .
RUN go mod download
ENV CGO_ENABLED=0
RUN go build -o /app/kairos-init .

FROM ubuntu:${FLAVOR} AS kairos
COPY --from=build /app/kairos-init /kairos-init
RUN /kairos-init -l debug -f all
RUN rm /kairos-init

That can generate different container images based on the FLAVOR arg so you can do:

• docker build -t kairos-24.10 --build-arg=FLAVOR=24.10 .
• docker build -t kairos-20.04 --build-arg=FLAVOR=20.04 .
• etc...

And it will build with the proper base ubuntu image.

As this was only a spike and we got other cards to investigate other tools, I think this can be closed.

Takeaways:

• Seems nicer to have all packages for all distros/arches/versions in a single file
• Doing it in code allows more flexibility than dockerfiles for different paths
• Doing version ranges and such seems much easier here that having to handle different targets in a dockerfile
• Maintenance can be a problem, but it currently can be (not the first time we miss packages for distro X)
• Could allow us to track packages better.
• Could allow us to try-our-best (TM) for non-supported distros
• Still, lot of work to substitute our current Dockerfiles
• Can allow us to finegrain more stuff like kernel versions or dependencies
• Can allow us to provide extra features like kdump or fips directly without too much user intervention
• Users can still build derivatives with Dockerfiles based on what we publish, that wont go away
• Less visibility into our build process, kind of magic
• Lot of work still to do

If other tools can provide something similar to this we should jump on it. Pulumi, cuelang, packaer, etc.. will provide the building blocks against this, so if our side was config only, it would be much nicer that doing it from scratch. On the other side, building it ourselves allows us to have very specific things (like kernel firmware removal, kdump enblement and such) which may be difficult to plug into this third party tools.

[mauromorales]

Overall it looks great to me, I can see many of the benefits it will bring. I'm still hesitant to see how the final maintenance of this will be once all the different flavors, boards, variation, uki/non-uki, etc. are introduced on top of the MVP, but I can see why the benefits might still outweigh the drawbacks on that.

Great job everyone, and sorry to jump on the discussion so late

[Itxaka]

Follow up on #3045