Setup Hydra build farm for CI

[rvolosatovs]

Currently we rely on free tier of https://www.cachix.org/ for binary cache, which stores 5 GB of most recently used artifacts and we use default Github Action runners to build and push to these caches.

There are a few issues with that approach:

1. The free tier Cachix cache is already garbage collecting our artifacts from around 2 months ago, once we start pushing kernels into the cache, this period will shorten and long-term this will not be enough and our CI jobs won't be able to fully benefit from caching
2. Github Action runners are pretty slow and e.g. the kernel build takes around 2 hours.
3. Binary cache is managed by a third party, that's not ideal - it would be more secure to have our own managed binary cache with our own public key.

To speed up our CI and for improved caching and security, I propose setting up a Hydra instance on one of Equinix boxes. https://github.com/NixOS/hydra
Example of what it looks like can be found here https://hydra.nixos.org/
Hydra supports building for multiple platforms and there's a nice web interface, where status of each job can be inspected and artifacts can be directly downloaded manually, if so desired. For example, here are the latest master builds of https://github.com/NixOS/nix
Hydra also supports Github webhooks https://hydra.nixos.org/build/182892026/download/1/hydra/webhooks.html
And it's extremely simple to set up on NixOS https://nixos.wiki/wiki/Hydra#Installation

We would then have a centralized location, where build of each branch could be directly inspected and binary artifacts could be downloaded and, of course, it also serves as the binary cache for nix.

Note, that any machine with nix installed can be used for distributed building https://nixos.wiki/wiki/Distributed_build and as a binary cache https://nixos.wiki/wiki/Binary_Cache, that's where the most value lies, Hydra is not required for any of that, it only provides a a nice web-based user interface and automation functionality. It also means we wouldn't need to provision SSH keys in our Github Action builders to "push" builds to the remote builder, but rather the remote will "pull" and report the status when done.

[rvolosatovs]

We may also investigate:

• https://hercules-ci.com
• https://nixbuild.net/
• https://garnix.io/

[rvolosatovs]

Unfortunately, KVM is not supported on Github Actions, which means we cannot build our images (e.g. Amazon VHD) in CI currently. We should choose a CI provider, which does support KVM

[rvolosatovs]

For a marginal improvement and KVM builds, https://github.com/nixbuild/nixbuild-action could be the simplest to set up
Garnix has M1 available https://garnix.io/blog/may-14-release-notes