Skip to content

Operating a Builder

Jump to bottom
Cole Helbling edited this page May 7, 2020 · 16 revisions

Basic Setup

We recommend you create a special user for ofborg operation. This user should have a name and email configured with git in order to allow non-interactive merges. For example:

$ git config --global user.name "ofborg"
$ git config --global user.email "[email protected]"

To get started, run:

$ nix-shell ./shell.nix
$ cd ofborg
$ cargo build

Next, copy example.config.json to config.json and edit its contents as you see fit. Set nix.remote to an empty string if you are not using the daemon.

You can then run the builder like so:

$ ./target/debug/builder ./config.json

config.public.json contains ofborg's public configuration for the Nixpkgs repository, which is then merged with a private configuration file of credentials. These files contain some special keys like trusted users and log storage and are only used in the backend processing tasks. Thus, there is no need for them on builders.

Make sure that the ofborg user has access to /dev/kvm, as it is necessary for running tests.

If you want to run multiple builder instances on the same physical machine please make sure they use different configurations with different instance identities (same username/password is OK) and different repository paths. Running two builders with the same configuration risks data corruption.

Multiple Builders

Note if you run multiple builders (which you can) they must each have a separately configured identity, and probably should also have separate build roots at this time.

Service

This is a service I use:

let
  grahamcofborg = builtins.storePath
    (builtins.fromJSON
      (builtins.readFile
        (builtins.fetchurl
          "https://hydra.nixos.org/job/ofborg/release/ofborg.rs.x86_64-linux/latest/api/get-info"
        )
      )
    ).outPath;
  # or if you'd prefer run out of a checkout:
  # grahamcofborg = (import ./ofbborg/default.nix).ofborg.rs;
in {
users.users.gc-of-borg = {
  description = "GC Of Borg Workers";
  home = "/var/lib/gc-of-borg";
  createHome = true;
  group = "gc-of-borg";
  uid = 402;
};
users.groups.gc-of-borg.gid = 402;

systemd.services.grahamcofborg-builder = {
  enable = true;
  after = [ "network.target" "network-online.target" ];
  wants = [ "network-online.target" ];
  wantedBy = [ "multi-user.target" ];

  path = with pkgs; [
    nix
    git
    curl
    bash
  ];
  serviceConfig = {
    User = "gc-of-borg";
    Group = "gc-of-borg";
    PrivateTmp = true;
    WorkingDirectory = "/var/lib/gc-of-borg";
    Restart = "always";
    RestartSec = "10s";
  };

  preStart = ''
    mkdir -p ./nix-test
  '';
  script = ''
    export HOME=/var/lib/gc-of-borg;
    export NIX_REMOTE=daemon;
    git config --global user.email "[email protected]"
    git config --global user.name "GrahamCOfBorg"
    export RUST_BACKTRACE=1
    ${grahamcofborg}/bin/builder /path/to/myconfig.json
  '';
};

Please avoid putting the builder config in to the Nix store.

Debugging

If you get:

Protocol("Unexpected method frame: connection.close, expected: connection.open-ok")

it means your credentials don't have enough permissions.

If you get:

"error: access to path '/private/var/lib/ofborg/checkout/repo/af830982ebfb48a0b27598205be6f7ea/builder/lnl7-mac/default.nix' is forbidden in restricted mode

it means your checkout.root uses /var instead of the full canonical path /private/var/lib/ofborg/checkouts

Best Practices

Macs and Sandboxing

Ensure you have sandboxing enabled on your Mac's Nix. Do this by using a "recent enough" Nix Unstable as of 2018-01-06, nixUnstable in Nixpkgs for 17.09 and master are new enough:

ndndx:nixpkgs graham$ nix-build . -A nixUnstable

...

/nix/store/chl6llrafsyfqyf7q199b6smc4fv9abk-nix-unstable-1.12pre5810_5d5b931f

ndndx:nixpkgs graham$ sudo su -

ndndx:~ root# /nix/store/chl6llrafsyfqyf7q199b6smc4fv9abk-nix-unstable-1.12pre5810_5d5b931f/bin/nix-env --uninstall nix
uninstalling 'nix-1.11.16'
building '/nix/store/6ban3wlcl8y9n6y11wl64ssgpxvsz195-user-environment.drv'...
created 1 symlinks in user environment

ndndx:~ root# /nix/store/chl6llrafsyfqyf7q199b6smc4fv9abk-nix-unstable-1.12pre5810_5d5b931f/bin/nix-env --install /nix/store/chl6llrafsyfqyf7q199b6smc4fv9abk-nix-unstable-1.12pre5810_5d5b931f
installing 'nix-unstable-1.12pre5810_5d5b931f'
building '/nix/store/x2cybarskjbcfm3lmivv9bpajm66i1ck-user-environment.drv'...
created 8 symlinks in user environment

ndndx:~ root# nix --version
nix (Nix) 1.12pre5810_5d5b931f

ndndx:~ root# cat /Library/LaunchDaemons/org.nixos.nix-daemon.plist  
<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
  <dict>
    <key>Label</key>
    <string>org.nixos.nix-daemon</string>
    <key>RunAtLoad</key>
    <true/>
    <key>Program</key>
    <string>/nix/store/chl6llrafsyfqyf7q199b6smc4fv9abk-nix-unstable-1.12pre5810_5d5b931f/bin/nix-daemon</string>
    <key>StandardErrorPath</key>
    <string>/var/log/nix-daemon.log</string>
    <key>StandardOutPath</key>
    <string>/dev/null</string>
  </dict>
</plist>

Then make sure your /etc/nix/nix.conf has:

build-use-sandbox = true

and make sure your nix.conf does not have:

pre-build-hook =

ndndx:~ root# launchctl stop org.nixos.nix-daemon
ndndx:~ root# pgrep nix
ndndx:~ root# launchctl start org.nixos.nix-daemon
ndndx:~ root# pgrep nix
55039

Test your sandboxing works via:

let pkgs = import <nixpkgs> {}; in pkgs.runCommand "test" {} "ls /Users"
ndndx:~ root# nix-build ./test.nix 
these derivations will be built:
  /nix/store/sqanh64mvgsijgppwkz7rxzbzzy5xbvw-test.drv
building '/nix/store/sqanh64mvgsijgppwkz7rxzbzzy5xbvw-test.drv'...
ls: cannot access '/Users': Operation not permitted
builder for '/nix/store/sqanh64mvgsijgppwkz7rxzbzzy5xbvw-test.drv' failed with exit code 2
error: build of '/nix/store/sqanh64mvgsijgppwkz7rxzbzzy5xbvw-test.drv' failed

Security and Privacy of your builder

Running a builder will inherently expose some information about your system publicly in GitHub comments. Efforts are taken to minimize the risk and exposure, but it cannot be perfect. Please evaluate the risks for yourself when deciding to run a builder.

Builder Identity

The builder's identity is considered public information, and efforts are not taken to hide it.

Home Directory Leakage

For example, one operator was unhappy that the builder exposed their home directory in an error log:

error: while evaluating the attribute ‘args’ of the derivation ‘hello-1.12.0’ at /Users/theiruser/.nix-test-rs/repo/commit-hash/builder/their-builder-identity/pkgs/stdenv/generic/make-derivation.nix:98:11:
couldn't change to directory of ‘/nix/var/nix/daemon-socket/socket’: No such file or directory

If you'd rather your home directory / user account information not exposed, ensure the build root is outside of your home directory.

Clone this wiki locally