Recommended way to protect/store/share deployment state files

[3noch]

I'm pretty terrified of losing one of my state files and losing all ability to interact with my deployments via nixops. What is the recommended strategy for keeping the state files safe and sharing them so that I and others on the team can deploy now and forever into the future?

I've seen some people use git + git-crypt to store the state in the repository. However, I can't imagine that doing merges on a state file is a good idea. Some have mentioned the export feature. However, keeping your export data in sync with the actual state file is pretty complex and error-prone (it would be much better if nixops could write directly to an export file while it makes changes to a deployment).

[3noch]

The reason I made this an issue is because I really think this is such a common problem that it should be documented clearly, and even better, built right into nixops itself.

[3noch]

Here's a script I'm using for the task: https://gist.github.com/3noch/4ee83fedb56d7aa71d23948e363ca9e2

[domenkozar]

Relevant #464

[kamilchm]

Terraform Remote State with different backends https://www.terraform.io/docs/state/remote/ looks like one of possible implementations.

[domenkozar]

Since we're using key/values for state we could refactor the code to support more than just sqlite. Someone just needs to do it :)

[earldouglas]

It'd be nice if I could version control (most of) the state as JSON (a la --export, but as primary storage), and keep sensitive values somewhere like Vault, or even just as environment variables.

How much of the state should be considered sensitive? Aside from the SSH private key, is it all pretty low-risk?

[3noch]

@earldouglas Right now I have a script (linked above) that will sync your state data to exported files (JSON). I use git-crypt to keep the files safe in the repository.

[eqyiel]

See also the discussion here:
http://lists.science.uu.nl/pipermail/nix-dev/2016-September/021657.html
https://blog.wearewizards.io/how-to-use-nixops-in-a-team

[3noch]

I've recently reworked my script and it no longer exports to JSON. The reason is that in order to truly store deployment state in the repo we need to make sure that users never run commands against an outdated view of the state. Exporting to JSON is slow...but importing is even slower. To truly use JSON we'd need to do the following every time someone wanted to touch the deployment:

1. Create a new temporary deployment state file (You cannot safely use an old one because you cannot import a deployment on top of another and nixops delete is completely broken in many cases).
2. Import the JSON state into the state file.
3. Run the command.
4. Export the JSON state back to the original JSON file.
5. Delete the temporary state file.

This would add 5-10 seconds of overhead to every command. As such I've just scrapped the use of JSON and now keep SQLite state files encrypted in the repo. This comes with several downsides:

1. The SQLite database changes every time you run nixops deploy (possibly due to some non-determinism in how SQLite stores data or due to some sort of timestamp data). This is not true of the exported JSON. If you run nixops deploy back-to-back the exported JSON does not change after the first deploy, because the state is idempotent. Thus using SQLite data makes the repo history messier and more cumbersome since you must always commit these trivial file changes just in case they are meaningful.
2. Obviously, the SQLite database is entirely impenetrable for diffing purposes.

Due to all these considerations, I would like to propose that nixops scrap SQLite format and just use a normal human-readable format like JSON from the get-go. This would mostly obviate the need for import/export and make keeping deployment state safely much easier. If there are worries about contention, then I further recommend that each deployment be kept in a separate file. This is much better for VCS use-cases anyway and it alleviates the need to strive for atomicity since multiple users should never actually deploy the same deployment simultaneously.

[ip1981]

@3noch nixops update machine path (/nix/store/....) on every deploy. That's why state is amost always changing.

[3noch]

@ip1981 Not when you run nixops deploy without any changes.

However, I've since discovered that nixops can cope with some level of outdated fields in the state data. For example, if you deploy some minor changes (i.e. not adding or removing machines, or changing keys, etc.) and accidentally revert the state file then nixops will still be able to deploy without issue. I haven't "stress tested" this, but it's at least a tad bit comforting. ;)

[ip1981]

Yes. I've stressed it quite well, over a year already. Actually, only IP and root ssh key matters.

P. S. Who runs deploy without changes? ;)

[3noch]

@ip1981 Ah good to know. Can you point me to some documentation on that kind of thing?

P.S. Haha...my point is that the changes happening to deployment state are very opaque to most of us, especially when deploying no changes actually changes the state file.

[ip1981]

I don't have any docs, but we keep state in JSON. So every time I can see what is changed and why, undo changes, share with others, etc. Nobody has complained so far :)

Though I believe nixops makes calls to see if provisioned infra has changed. This is a bit annoying.

[3noch]

@ip1981 Would you be willing to describe your process in a bit more detail? I'm quite interested how you go about making deployments and securely/safely sharing state with others.

[ip1981]

This wrapper and git-crypt.

#!/usr/bin/env bash

set -euo pipefail

NIXOPS=${NIXOPS:-nixops}
export NIX_PATH=.

usage () {
cat <<USAGE
Usage: $0 <nixops command> <realm/spec.nix> [nixops options]
Examples:
  $0 deploy realms/vbox.nix
  $0 info realms/vbox.nix
  $0 deploy realms/dumpoo-ec2.nix --build-only
  $0 destroy realms/cats.nix --include slothcat
USAGE
}

fatal () {
  echo '** ERROR:' "$@" >&2
  usage >&2
  exit 1
}

if [ $# -lt 2 ]; then
  fatal "missing agruments."
fi

CMD="$1"; shift
REALM_NIX="$1"; shift

case "$REALM_NIX" in
  *realms/*.nix) REALM=$(basename "$REALM_NIX" .nix);;
  *) fatal "invalid realm spec: $REALM_NIX";;
esac

cd "$(dirname "$0")"

state="secrets/nixops-${REALM}.json"
db=$(mktemp -u "secrets/tmp.${REALM}.XXXXXX.nixops")

save() {
  if [ -f "$db" ]; then
    "$NIXOPS" export -s "${db}" > "${state}.tmp"
    mv "${state}.tmp" "${state}"
  fi
}

clean() {
    rm -f "$db"*
}

create() {
  "$NIXOPS" create -s "$db" -d "$REALM" "<realms/${REALM}.nix>"
}

case "$CMD" in
  create)
    trap 'clean' EXIT
    [ ! -f "$state" ] || fatal "\`$state' already exists."
    create && save
    ;;
  *)
    if [[ "$@" == *'--build-only'* ]]; then
      trap 'clean' EXIT
      create
    else
      trap 'save && clean' EXIT
      [ -f "$state" ] || fatal "\`$state' does not exists."
      "$NIXOPS" import -s "${db}" < "$state"
    fi
    "$NIXOPS" "$CMD" -s "$db" -d "$REALM" "$@"
    ;;
esac

[ip1981]

There are few things I don't like with that. But I'd not fix them in bash ;)

[ip1981]

Actually, it would be perfect if nixops worked with json directly, stop writing unnecessary data like machine top closure, keys, etc. That should be derived anyway.

[3noch]

@ip1981 Great thanks! This looks very similar to my setup and even closer to my previous setup. However, I found that export/import on every command call was far too slow. Therefore I fully agree that nixops should work directly with JSON (or some other human-readable format).

[corpix]

Just lost all my state files(because I thought NixOps will not recreate machines if they are already exist).
Now I need manage state files(we are trying NixOs in a team) or use ssh deployment. I am "surprised".

Bash scripts look like unnecessary complication and scares me(and my team), what if NixOps could get rid of this state files and become stateless?

[3noch]

@corpix It could never be completely stateless. It needs at least to have keys to the servers.

[corpix]

@3noch Good point.

[sorki]

This seems to be rather hot topic and looks like a major pain point when you want (need) to share configuration with other people rather then deploying via single machine.

According to @ip1981 @3noch "IP and root ssh key matters" - I would love to have a mode that would allow specifying a single ssh key for all machines or specifying keys per-machine like you have to do with IPs with none backend (could alleviate #676 as well).

If I understand all this correctly such mode could work in a stateless manner for people who only need none backend, right?

Also found https://github.com/grafted-in/nixops-manager

Related tickets: #2 #250 #600

[lpil]

It's been some time since the previous discussion, what's the recommended approach now?

[grahamc]

#1264 implements state backends, and those backends could implement locking, encryption, or remote storage.