Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Example NixOps State Backends #1264

Merged
merged 80 commits into from
Jul 7, 2021
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
80 commits
Select commit Hold shift + click to select a range
61d39d7
Remove flake support
adisbladis Jul 15, 2020
6a8289b
Add deploy without creating a state file
adisbladis Mar 12, 2020
e9426b0
add_common_{modify,deployment}_options: add types
grahamc Mar 22, 2020
2292ee3
Begin to describe an API for storage backends
grahamc Mar 23, 2020
29388ae
script_defs: accept a network file, not a state file
grahamc Mar 23, 2020
ba716a4
evaluation: parse out storage provider data
grahamc Mar 23, 2020
3ee765d
deployment: HACK: make the network.nix always part of the nix exprs
grahamc Mar 24, 2020
5515e92
network_state: use a storage backend
grahamc Mar 24, 2020
952beb7
LegacyBackend: init
grahamc Mar 24, 2020
81717db
MemoryBackend: init
grahamc Mar 24, 2020
bb64023
fixup d6b34b2251c9b30da3377b970cb617c73ef208ab: storage API: use Mapp…
grahamc Apr 23, 2020
41f658a
fixup d6b34b2251c9b30da3377b970cb617c73ef208ab: storage API: drop the…
grahamc Apr 23, 2020
8a59a58
eval network: use ImmutableValidatedObject for good errors
grahamc Apr 28, 2020
97c4a41
nixops.exceptions: add NixError for Nix-related problems
grahamc Apr 28, 2020
b3c0fa5
eval_network, eval_attrs: handle missing 'network'
grahamc Apr 28, 2020
bfa6ffc
eval_network: check the network property is a dictionary
grahamc Apr 28, 2020
c2335f2
add a test for when the network.nix has no network attr
grahamc Apr 28, 2020
9a59a3e
add a test for when the network.nix's network is of the wrong type
grahamc Apr 28, 2020
f45fcf9
Add a test for when the network's value is an empty attrset
grahamc Apr 28, 2020
1b45714
util: ImmutableValidatedObject: don't set _frozen(?)
grahamc Apr 28, 2020
2ff99b5
storage backends: strengthen typecheck, making T invariant
grahamc Apr 29, 2020
1e8aa36
storage backends: simplify options() type: just handle kwargs, return…
grahamc Apr 29, 2020
d85ba2c
locks: init protocol with a noop
grahamc Apr 29, 2020
76ff3a3
evaluation.nix: handle Lock configuration too
grahamc Apr 29, 2020
7202a10
Use the network's lock when accessing its state
grahamc Apr 29, 2020
ad87a56
Squash with 869d2765a
adisbladis Jul 15, 2020
98ea4cc
Squash with 869d2765a
adisbladis Jul 15, 2020
310bada
storage: fixup handling of edge casesin network parsing
grahamc Apr 29, 2020
524a2d5
ImmutableValidatedObject: support validating Unions with nested Immut…
grahamc Apr 29, 2020
1304da4
Add and improve slome storage test expressions
grahamc Apr 29, 2020
f3f6d4d
Add some test expressions for locks
grahamc Apr 29, 2020
0f1d687
fixup d2f845a9f5c47d2e1f5bcbdf30583c4f5d655bb5 with no locking indica…
grahamc Apr 29, 2020
a2e25f1
fixup f52369f6b4c1cb94c7cba244ef737e814537ba3b^C
grahamc Apr 29, 2020
87b64d8
Split argument parser into it's own file
adisbladis Jul 15, 2020
a389d39
Fix internal plugin
adisbladis Jul 15, 2020
e385752
Memory storage: Run modify_deployment() when creating internal state
adisbladis Jul 15, 2020
2543248
Remove template support
adisbladis Jul 15, 2020
f7d6219
nit: Use os.path.join instead of stringly concat
adisbladis Jul 15, 2020
0a68459
Remove the positional nix_exprs argument from modify commands
adisbladis Jul 15, 2020
2120542
op_list: Fix calculation of number of machines & types
adisbladis Jul 15, 2020
bc25d83
locks/storage plugins: access via Manager, eliminate global vars
grahamc Jul 10, 2020
d754007
Add command to unlock the deployment lock
adisbladis Jul 15, 2020
08d9d81
Use git to find project root in development shell
adisbladis Jul 16, 2020
737e8bd
Remove xml
adisbladis Jul 16, 2020
e64611e
Remove non-json option show
adisbladis Jul 16, 2020
4ae8d5d
Unify nix-instantiate calls
adisbladis Jul 16, 2020
7cd5184
Move all evaluation logic to evaluation.py
adisbladis Jul 16, 2020
3df656c
evaluation.py: Handle network arguments
adisbladis Jul 17, 2020
80bb68f
evaluation: Completely unify evaluation in the `eval` python function
adisbladis Jul 17, 2020
db696e2
Re-introduce flakes support
adisbladis Jul 17, 2020
5f54070
Fix unification of evaluation logic
adisbladis Jul 17, 2020
26ad1c6
Move more evaluation logic from deployment to evaluation module
adisbladis Jul 17, 2020
9451aae
Use evaluation module for machine evaluation
adisbladis Jul 17, 2020
92f5fd5
Convert bootstrap network eval to use the eval method
adisbladis Jul 17, 2020
5379e0c
Fix deployment when network is a flake
adisbladis Jul 17, 2020
33ada85
Fix show arguments for flakes
adisbladis Jul 17, 2020
ccfa30f
Always set a network expression
adisbladis Jul 17, 2020
80049f1
Default is_flake to false
adisbladis Jul 17, 2020
e4873c3
Remove obsolete tests
adisbladis Jul 17, 2020
2cad90d
Add a plugin entry point
adisbladis Jul 17, 2020
47cf418
nixops/__main__.py: Add more static typing
adisbladis Jul 18, 2020
b05052e
nixops/backends/__init__.py: Add more static typing
adisbladis Jul 18, 2020
b879324
nixops/backends/none.py: Add more static typing
adisbladis Jul 18, 2020
8256550
nixops/deployment.py: Add more static typing
adisbladis Jul 18, 2020
9687d74
nixops/nix_expr.py: Add more static typing
adisbladis Jul 18, 2020
a92ebfe
nixops/plugins/__init__.py: Add more static typing
adisbladis Jul 18, 2020
b3bad1c
nixops/plugins/manager.py: Add more static typing
adisbladis Jul 18, 2020
7ce24cf
nixops/resources/__init__.py: Add more static typing
adisbladis Jul 18, 2020
e48422a
nixops/resources/ssh_keypair.py: Add more static typing
adisbladis Jul 18, 2020
9feab77
nixops/script_defs.py: Add more static typing
adisbladis Jul 18, 2020
6bcdee4
nixops/plugins/__init__.py: Add more static typing
adisbladis Jul 18, 2020
0c83259
nixops/plugins/manager.py: Add more static typing
adisbladis Jul 18, 2020
5f542a0
nixops/util.py: Add more static typing
adisbladis Jul 18, 2020
7d81a21
Reduce nix-instantiate calls from 3 to 2
adisbladis Jul 19, 2020
4ea1728
Remove all traces of s3 storage/locking implementations
adisbladis Feb 8, 2021
bc31429
Rename `network.nix` to `nixops.nix`
adisbladis Feb 8, 2021
d87cf56
Add migration step docs
adisbladis Jul 7, 2021
b2b79b4
Add state migration docs
adisbladis Jul 7, 2021
e14c835
Fix output of migration error message to not include internal paramet…
adisbladis Jul 7, 2021
3f7d77a
Remove TODO from migration error message output
adisbladis Jul 7, 2021
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion doc/guides/deploy-without-root.rst
Original file line number Diff line number Diff line change
Expand Up @@ -61,7 +61,7 @@ the next step.
Configuring the NixOps Network
******************************

Edit your network.nix to specify the machine's
Edit your nixops.nix to specify the machine's
``deployment.targetUser``:

.. code-block:: nix
Expand Down
76 changes: 76 additions & 0 deletions doc/manual/migrating.rst
Original file line number Diff line number Diff line change
@@ -0,0 +1,76 @@
.. _chap-overview:

Overview
========

This chapter aims to provide guidelines on migrating from NixOps 1.x to 2.0.

.. _sec-layout:

Code layout changes
-------------------

Using NixOps 1.0 multiple deployments spread out over the file and deployed
from any working directory with the ``--deployment (-d)`` parameter.

NixOps 2 however requires a file relative to the invocation working directory.
It needs to be called either ``nixops.nix`` for a traditional deployment or
``flake.nix`` for the as of yet experimental
`flakes support <https://github.com/tweag/rfcs/blob/flakes/rfcs/0049-flakes.md>`.

.. _sec-state-location:

State location
--------------

In NixOps 1.0 deployment state such as provisioned resources are stored in a
SQLite database located in ``~/.nixops``.

NixOps 2 however has pluggable state backends, meaning that you will have to
make a choice where to store this state.

To implement the old behaviour of loading deployment state from the SQLite
database located in ``~/.nixops`` add the following snippet to your deployment:

::
{
network = {
storage.legacy = {};
};
}

To implement a fire-and-forget strategy use this code snippet:

::
{
network = {
storage.memory = {};
};
}

For additional state storage strategies see the various NixOps plugins.

.. _sec-state-migration:

State migration
---------------

Migrating to any non-legacy backend from a previous deployment requires a
manual migration step.

#. Start by configuring the legacy backend as such::
{
network = {
storage.legacy = {};
};
}

#. Then export the current state::
nixops export > state.json

#. Now go ahead and configure your desired state backend.

#. And finally import the old state::
nixops import < state.json

#. Make sure to remove ``state.json`` as it may contain deployment secrets.
4 changes: 3 additions & 1 deletion flake.nix
Original file line number Diff line number Diff line change
Expand Up @@ -38,7 +38,9 @@
] ++ (builtins.attrValues linters);

shellHook = ''
export PATH=${builtins.toString ./scripts}:$PATH
git_root=$(${pkgs.git}/bin/git rev-parse --show-toplevel)
export PYTHONPATH=$git_root:$PYTHONPATH
export PATH=$git_root/scripts:$PATH
'';
};

Expand Down
13 changes: 11 additions & 2 deletions nix/eval-machine-info.nix
Original file line number Diff line number Diff line change
Expand Up @@ -17,6 +17,8 @@ let
zipAttrs = set: builtins.listToAttrs (
map (name: { inherit name; value = builtins.catAttrs name set; }) (builtins.concatMap builtins.attrNames set));

flakeExpr = (builtins.getFlake flakeUri).outputs.nixopsConfigurations.default;

networks =
let
getNetworkFromExpr = networkExpr:
Expand All @@ -31,7 +33,7 @@ let
in
map ({ key }: getNetworkFromExpr key) networkExprClosure
++ optional (flakeUri != null)
((call (builtins.getFlake flakeUri).outputs.nixopsConfigurations.default) // { _file = "<${flakeUri}>"; });
((call flakeExpr) // { _file = "<${flakeUri}>"; });

network = zipAttrs networks;

Expand Down Expand Up @@ -260,6 +262,11 @@ in rec {
in
[ f ] ++ map getRequires requires;

exprToArgs = nixopsExpr: f:
if builtins.isFunction nixopsExpr then
map (a: { "${a}" = builtins.toString f; } ) (builtins.attrNames (builtins.functionArgs nixopsExpr))
else [];

fileToArgs = f:
let
nixopsExpr = import f;
Expand All @@ -270,6 +277,8 @@ in rec {

getNixOpsArgs = fs: lib.zipAttrs (lib.unique (lib.concatMap fileToArgs (getNixOpsExprs fs)));

nixopsArguments = getNixOpsArgs networkExprs;
nixopsArguments =
if flakeUri == null then getNixOpsArgs networkExprs
else lib.listToAttrs (builtins.map (a: {name = a; value = [ flakeUri ];}) (lib.attrNames (builtins.functionArgs flakeExpr)));

}
Empty file removed nix/templates/.keep
Empty file.
Loading