Key value state + tests refactoring #969

srghma · 2018-06-14T07:12:20Z

This pr consumes:

Tests done:

TODOs:

each test has its own store file (for tests parallelization)
python2 tests.py tests.functional.test_cloning_clones -vv once in 5 times throws error https://pastebin.com/fe7njGJH
tests.functional.test_stopping_stops with ec2 issue https://pastebin.com/BJwKMWti (fix 898107c). I suppose this bug occured in this test because aws returns termination responce faster if instance already stopped, just after that nixops tries to remove security group, and fails, try remove fix and you will see it youself
tests.functional.test_stopping_stops with libvirt issue https://pastebin.com/e4VLG8Cc
tests.functional.test_ec2_rds_dbinstance not working tests.functional.test_ec2_rds_dbinstance not working #970
python2 tests.py tests.functional.test_vpc:test_deploy_vpc -vv no working with json state file https://pastebin.com/N8E90uke (fix e1b9158)
python2 tests.py tests.functional.test_vpc:test_deploy_vpc_machine -vv not working (https://pastebin.com/hqYBi7hF)

CONSIDER (dont mind it 😄, just thoughts):

not use StateDict at all because it makes return value type vague, you can put array but get string, use nixops.utils.attr_property instead (also StateDict should be named ResourceStateDict, because only ResourceState can use it)
substitute every isinstance(XXX, basestring) with isinstance(XXX, six.text_type)
all nixops.util.attr_property should be wrapped in with self.db, maybe highlight their name like self.state_prop_XXX
in tests.functional.shared.destroy_deployments_and_remove_state_file dont ignore 'VirtualBoxState' object has no attribute 'get_backups' error https://pastebin.com/W228z0rN

Needed to faclitate multiple backends (like sqlite3/other remote database)

… it in state file implementation to __db

For now it's just the underlying DB, but it resembles something that can roll back the current transaction.

tests/functional/shared/deployment_run_command.py

tests/functional/test_deploys_nixos/__init__.py

coretemp · 2018-11-05T08:27:05Z

tests/functional/test_ec2_backups/__init__.py

+ while backups[backup_id]['status'] == "running":
+ time.sleep(10)
+ backups = deployment.get_backups()
+ deployment_run_command(deployment, "rm {}/back-me-up".format(path))


Please run any commands via an API that uses arrays to process commands. E.g. foo(["ls","-l"]) vs foo("ls -l").

@coretemp

cons:

more characters to print

harder to use .format method with multiple {}. e.g.

deployment_run_command(deployment, "rm {}/back-me-up {}/and-me".format(path))

vs

deployment_run_command(deployment, ["rm", "{}/back-me-up".format(path), "{}/and-me".format(path)])

I don't agree, but perhaps others do. I.e., if it was my code base, it wouldn't get in, but it's not my play ground.

tests/functional/test_ec2_backups/__init__.py

coretemp · 2018-11-05T08:29:18Z

tests/functional/test_ec2_rds_dbinstance/__init__.py

+ # If your account doesn't support EC2-Classic, you will get an error:
+ # `VPC DB Security Groups cannot be modified with this API version.
+ # Please use an API version between 2012-01-15 and 2012-10-31 to modify this group.`
+ # TODO: remove it?


Removing it seems reasonable or otherwise it should check for EC2-Classic support.

@AmineChikhaoui what do you think, remove this comment and ec2-rds-dbinstance-with-sg.nix file OR leave commented and ec2-rds-dbinstance-with-sg.nix file for legacy?

I'm asking because you are author of this test which I can't run https://github.com/NixOS/nixops/blob/aa3602d03cd03fc748cdcf1fc25d97ef863474f0/tests/functional/ec2-rds-dbinstance-with-sg.nix

Checking for classic support is less work then asking him.

coretemp · 2018-11-05T08:29:42Z

tests/functional/test_invalid_indentifier/__init__.py

+from tests.functional.shared.create_deployment import create_deployment
+from tests.functional.shared.using_unique_state_file import using_unique_state_file
+
+parent_dir = path.dirname(__file__)


Same comments as before about __file__

tests/functional/test_rebooting_reboots/__init__.py

srghma · 2018-11-10T17:19:22Z

NOTE: how to run test

to show all tests:

python2 tests.py --collect-only -v

to run only one test:

# for test named
# test_device_name_to_boto_expected (tests.unit.test_device_name_to_boto_expected.TestDeviceNameToBotoExpected) ... ok

python2 tests.py tests.unit.test_device_name_to_boto_expected

# for tests generated by @parametrized (there is no way to run only one test of these)

# tests.functional.test_send_keys_sends_keys.test_send_keys_sends_keys('json', ('vbox', ['/home/srghma/projects/nixops/tests/functional/shared/nix_expressions/logical_base.nix', '/home/srghma/projects/nixops/tests/functional/shared/nix_expressions/vbox_base.nix'])) ... ok
# tests.functional.test_send_keys_sends_keys.test_send_keys_sends_keys('json', ('libvirtd', ['/home/srghma/projects/nixops/tests/functional/shared/nix_expressions/logical_base.nix', '/home/srghma/projects/nixops/tests/functional/shared/nix_expressions/libvirtd_base.nix'])) ... ok
# tests.functional.test_send_keys_sends_keys.test_send_keys_sends_keys('json', ('ec2', ['/home/srghma/projects/nixops/tests/functional/shared/nix_expressions/logical_base.nix', '/home/srghma/projects/nixops/tests/functional/shared/nix_expressions/ec2_base.nix'])) ... ok
# .....

python2 tests.py tests.functional.test_send_keys_sends_keys.test_send_keys_sends_keys

Proof, that tests, that changed after `git merge` are working

python2 tests.py tests.functional.test_ec2_with_nvme_device_mapping: nixops store, json store
python2 tests.py tests.functional.test_ec2_backups.test_ec2_backups_simple_xd https://pastebin.com/S9vrXZxB
python2 tests.py tests.functional.test_ec2_backups.test_ec2_backups_simple_nvme https://pastebin.com/qC8a9AuJ
python2 tests.py tests.functional.test_ec2_backups.test_ec2_backups_raid_xd - https://pastebin.com/rrLL88DC
python2 tests.py tests.functional.test_ec2_backups.test_ec2_backups_raid_nvme - https://pastebin.com/y3qkqRNt

coretemp · 2018-11-12T09:00:56Z

Great that you show a limited form proof that the feature is working. 👍

I think I will leave the rest to others (because I don't have merge rights anyway).

Mic92 · 2018-11-16T22:23:13Z

This quite a massive pull request. I wonder we should split it up into smaller ones so the nixops maintainer do not run away, when having to review this.

Mic92 · 2018-11-17T10:13:44Z

If your pull request contains
new libraries they can go to this file:

https://github.com/NixOS/nixops/blob/master/setup.cfg

If you want to run the typecheck on your local checkout either run:

$ nix-build --quiet release.nix -A build.x86_64-linux

or

$ nix run nixpkgs.mypy -c mypy nixops

srghma · 2018-11-18T14:51:58Z

@Mic92 tnx for the suggestion, added mypy-parametrized to setup.cfg

…v wrong change

srghma · 2018-11-18T15:19:50Z

@rbvermaa could you please review this PR

tbenst · 2019-12-04T15:03:02Z

@srghma anything I or others can do to help with this PR? Since there’s interest from core devs Would be awesome to get this merged. I’m guessing the new plugin architecture means this needs to be (possibly non-trivially) rebased?

srghma · 2019-12-05T07:00:40Z

for and revert prev wrong change

@@ -0,0 +1 @@
+fix: coretemp review -> fix typo -> change of -


remove file

srghma · 2019-12-05T07:08:48Z

@srghma anything I or others can do to help with this PR? Since there’s interest from core devs Would be awesome to get this merged. I’m guessing the new plugin architecture means this needs to be (possibly non-trivially) rebased?

hi, I'm sorry, I won't participate, just have no time

@moretea , since this is basically your PR (I just added test matrix and made sure everything is working) , what do you think about rebasing your commits?

grahamc · 2020-03-26T19:23:34Z

Hello!

Thank you for this PR.

In the past several months, some major changes have taken place in
NixOps:

Backends have been removed, preferring a plugin-based architecture.
Here are some of them:
NixOps Core has been updated to be Python 3 only, and at the
same time, MyPy type hints have been added and are now strictly
required during CI.

This is all accumulating in to what I hope will be a NixOps 2.0
release. There is a tracking issue for that:
#1242 . It is possible that
more core changes will be made to NixOps for this release, with a
focus on simplifying NixOps core and making it easier to use and work
on.

My hope is that by adding types and more thorough automated testing,
it will be easier for contributors to make improvements, and for
contributions like this one to merge in the future.

However, because of the major changes, it has become likely that this
PR cannot merge right now as it is. The backlog of now-unmergable PRs
makes it hard to see which ones are being kept up to date.

If you would like to see this merge, please bring it up to date with
master and reopen it. If the or mypy type checking fails, please
correct any issues and then reopen it. I will be looking primarily at
open PRs whose tests are all green.

Thank you again for the work you've done here, I am sorry to be
closing it now.

Graham

moretea added 30 commits March 13, 2017 19:42

Move statefile into state.file module

7035abe

Introduce different state state schemes.

6da031f

Needed to faclitate multiple backends (like sqlite3/other remote database)

Rename statefile->state in deployment.py

8269650

Move file based deployment lock to file state implementation.

c73773f

Fix indenting

8a4e34a

Bite the bullet; remove _db references from business objects + rename…

bb056ea

… it in state file implementation to __db

move get_resources_for

6cc171a

move set_deployment_attrs

813a231

move del_deployment_attr

7b58934

move get_deployment_attr

7ed3fd3

move create_resource

0db686d

Move most logic in export() to get_all_deployment_attrs in statefile.

97e0714

Make import use an atomic transaction thingy.

584d658

For now it's just the underlying DB, but it resembles something that can roll back the current transaction.

move clone_deployment

97bda1a

move part of delete_resource

77e9b93

missed a few _'s

4d33f93

move part of delete to _delete_deployment

5649e39

use state atomic

4075901

Move part of logic of rename to _rename_resource

5f39278

oh yeah, I did rename _db to __db!

6605128

Use atomic instead of _db

5881c06

Use atomic instead of _db

de9d700

move set_resource_attrs

ff23c1c

move del_resource_attr

eb2a24a

move get_resource_attr

47b3a43

move part of export to get_all_resource_attrs

6835823

Use atomic instead of _db

3739051

oh yeah, I did rename _db to __db!

66dee1d

simple typo's

6254ccb

Need to pass in object; too much coupling to refactor right now

2d1a822

coretemp reviewed Nov 5, 2018

View reviewed changes

tests/functional/shared/deployment_run_command.py Outdated Show resolved Hide resolved

coretemp reviewed Nov 5, 2018

View reviewed changes

tests/functional/test_deploys_nixos/__init__.py Show resolved Hide resolved

coretemp reviewed Nov 5, 2018

View reviewed changes

tests/functional/test_ec2_backups/__init__.py Outdated Show resolved Hide resolved

coretemp reviewed Nov 5, 2018

View reviewed changes

tests/functional/test_rebooting_reboots/__init__.py Show resolved Hide resolved

Merge remote-tracking branch 'upstream/master' into kv-state

c3ec96b

srghma added 5 commits November 10, 2018 19:27

refactor: coretemp review -> change "echo -n" to "printf"

d912e7d

refactor: coretemp review -> fix typo -> revert "backend" to "backed"

b65546f

refactor: coretemp review -> fix typo -> "of" to "for"

2ceecef

refactor: coretemp review -> json_file, _find_deployment method

b63914b

refactor: coretemp review -> fix typo -> "michine" to "machine"

b5ee915

Merge branch 'master' into kv-state

cb411dd

srghma added 2 commits November 18, 2018 16:44

fix: build errors

c4eecfb

feat: mypy -> add mypy-parametrized

e0a34e8

fix: coretemp review -> fix typo -> change "of -> for" and revert pre…

16e0b6c

…v wrong change

srghma force-pushed the kv-state branch from 7d2105c to 16e0b6c Compare November 18, 2018 15:19

johnalotoski mentioned this pull request Jul 9, 2019

Nixops Core, Plugin Restructuring PR #1179

Merged

srghma commented Dec 5, 2019

View reviewed changes

for and revert prev wrong change

@@ -0,0 +1 @@

fix: coretemp review -> fix typo -> change of -

Copy link

Contributor Author

srghma Dec 5, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

remove file

grahamc closed this Mar 26, 2020

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Key value state + tests refactoring #969

Key value state + tests refactoring #969

srghma commented Jun 14, 2018 •

edited

Loading

coretemp Nov 5, 2018

srghma Nov 10, 2018

coretemp Nov 12, 2018

coretemp Nov 5, 2018

srghma Nov 10, 2018

coretemp Nov 12, 2018

coretemp Nov 5, 2018

srghma commented Nov 10, 2018

coretemp commented Nov 12, 2018

Mic92 commented Nov 16, 2018

Mic92 commented Nov 17, 2018

srghma commented Nov 18, 2018

srghma commented Nov 18, 2018

tbenst commented Dec 4, 2019

srghma Dec 5, 2019

srghma commented Dec 5, 2019 •

edited

Loading

grahamc commented Mar 26, 2020

		@@ -0,0 +1 @@
		fix: coretemp review -> fix typo -> change of -

Key value state + tests refactoring #969

Key value state + tests refactoring #969

Conversation

srghma commented Jun 14, 2018 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

srghma commented Nov 10, 2018

NOTE: how to run test

Proof, that tests, that changed after git merge are working

coretemp commented Nov 12, 2018

Mic92 commented Nov 16, 2018

Mic92 commented Nov 17, 2018

srghma commented Nov 18, 2018

srghma commented Nov 18, 2018

tbenst commented Dec 4, 2019

Choose a reason for hiding this comment

srghma commented Dec 5, 2019 • edited Loading

grahamc commented Mar 26, 2020

srghma commented Jun 14, 2018 •

edited

Loading

Proof, that tests, that changed after `git merge` are working

srghma commented Dec 5, 2019 •

edited

Loading