Voting Key generation using typescript and multi file support #177

fboucquez · 2021-03-02T13:53:34Z

Update:

This PR now supports both the original catapult tools' voting key file generation and the native typescript. The original is the one used by default. You can enable the new native typescript with the useExperimentalNativeVotingKeyGeneration: true custom preset.

I'm also working on multi-voting file support. The voting file metadata (public key, start, and end epoch) is loaded directly from the .dat files.

Voting file upgrade idea:

Bootstrap knows the network "current" finalizationEpoch (by either a manually upgraded custom preset value for offline setup or asking the network for online)
If the node's current voting file has an end epoch close to the current epoch (close to expiring), --upgrade will create 'private_key_tree2.dat' that continues the current file. I defined "Close to expiring" when the epoch is in the upper half of the voting file. If the file's epoch length is 360, close to expiring will be 180+. Our nodes are 180 epochs, current epoch is 104 so they are close to expiring.
When a new voting file is created, bootstrap will advise running the link command again. The link command detects what has changed and announces the required transactions (linking the new voting key). The voting key files metadata (public key, start, and end epochs) will be in addresses.yml so it can also be linked using the symbol CLI if desired.

For the end-user, the upgrade process is pretty much the same:

$ npm install -g symbol-bootstrap
$ symbol-bootstrap start -p mainnet -a dual -c custom-preset.yml --upgrde
>>> New Voting File Created!
$ symbol-bootstrap link

We can also add a new command "upgradeVotingFiles" that does the --upgrade of voting files explicitly

TODOs:

Test how the node behaves when transitioning from one file to another
Double-check if we need to unlink already expired voting keys in the network. I think there is a limit about how many public keys can be linked. We may only need the current voting key and the future voting key
Double-check want to do with old voting key files. Do we need to remove them? If we need to remove them, we need to "shift them", if voting file 1 is removed, voting file 2 should be renamed to voting key 1, 3 to voting key 2, etc

Slack chat: https://nemgroup-workspace.slack.com/archives/C015AEQ9ZL2/p1616410897191200

Related to #149

Original:
This is my first implementation of voting key file generation done in Javascript/Typescript.

I think the code is ok, the main issue is that the SDK is really slow generating keys and signing data. The SDK uses tweetnacl, which is small but slow. These are some numbers on my dev machine when creating the voting key file:

(votingKeyEndEpoch - votingKeyStartEpoch + 1 -> TIME)

1 epoch -> 50 MS
5 epochs -> 114 MS
1000 epochs -> 10 seconds
10000 epochs -> 1 minute 41 seconds
26280 epochs -> 4 Minutes 25 seconds

@gimer , could you validate the logic in https://github.com/nemtech/symbol-bootstrap/blob/native_voting_key_file/src/service/VotingUtils.ts#L24?

As unit testing, I'm comparing files generates with catapult tools against files generated using the native implementation. Checking that the headers and sizes are the same. I would add more assertions validating the internal items (private key and signatures).

To speed up the implementation we could:

Change the SDK crypto library. It could make the SDK faster but it would make it heavier, it may not work on Web, it could have secondary effects. Not Recommended at this stage.
Change the crypto lib Bootstrap uses for the voting key implementation but keep the SDK for the rest of the key generation. Maybe something like monocypher

Before doing any improvement, I want to double-check the logic is fine.

I'm pushing hard for this feature not because I want to reimplement the tool. This will allow me to make Bootstrap config offline without docker and OS independent (catapult tools wouldn't need to be compiled and installed in the offline machine). Supernodes will have fewer excuses to not use Bootstrap to setup the configuration (when runing barebone or with docker compose in the final online node machine)

Fixes #169
Related to #170

Update:
noble-ed25519 takes ~40 seconds for the 26280 epochs

src/service/VotingUtils.ts

gimre-xymcity · 2021-03-02T14:18:45Z

10000 - I hope you remember that for publicnet limit is 1460, so the tool should probably check that limit somewhere

fboucquez · 2021-03-02T14:55:12Z

10000 - I hope you remember that for publicnet limit is 1460, so the tool should probably check that limit somewhere

Thanks. We also use bootstrap for private networks, epoch length would be large in this case. I can add the limit the public/testnet nodes

fboucquez · 2021-03-02T18:47:25Z

Update, I've added multi implementation (all implementations pass the vector tests). The fastest one by far is noble-ed25519. It's the current default.
Bootstrap doesn't need catapult tools anymore when creating the node configuration. (Nemgen is still used but for new networks)

src/service/VotingUtils.ts

Added multi implementation, best one is noble-ed25519 Updated VotingService.ts to use the native one instead of the catapult tools

# Conflicts: # cmds/config-testnet-supernode.sh # package-lock.json # package.json # presets/testnet/network.yml # src/model/Addresses.ts # src/service/BootstrapUtils.ts # src/service/ConfigLoader.ts # test/service/BootstrapUtils.test.ts # test/supernode.yml

# Conflicts: # presets/shared.yml

…otstrap into native_voting_key_file # Conflicts: # presets/shared.yml

# Conflicts: # CHANGELOG.md # src/service/LinkService.ts # src/service/RewardProgramService.ts

# Conflicts: # CHANGELOG.md # presets/shared.yml # src/service/ConfigService.ts # src/service/VotingService.ts

Wayonb · 2021-06-07T03:06:47Z

docs/upgradeVotingKeys.md

+
+Voting file upgrade:
+- If the node's current voting file has an end epoch close to the current epoch ("close to expiring") this command creates a new 'private_key_treeX.dat' that continues the current file.
+- "Close to expiring" happens when the epoch is in the upper half of the voting file. If the file's epoch length is 720, close to expiring will be 360+.


I think the default should be to allow the user to upgrade 1 month before keys expire.
Allow them to override with a warning.

I think at this stage allowing the user to upgrade 3 months in advance is fine, especially because this is the first upgrade and people need to get the process in place (like all nodes). We could reduce the default values in a later voting key upgrade

I think a month is more than enough time to run the update command.
Allnode could create a cron job that runs every 6 months to generate new key files. If they want to run it early, then nothing stopping them since there is an override option.
We are kind thinking similar except your suggestion would require an update later to the bootstrap.

Ok, I will upgrade the preset. Defaults will be:

votingKeyDesiredLifetime: 6 months in epochs (how long the file last)
votingKeyDesiredFutureLifetime: 1 month in epochs (1 month before the current file expires bootstrap will create a new file)

Wayonb · 2021-06-07T14:44:02Z

README.md

@@ -277,6 +277,7 @@ General users should install this tool like any other node module.
 * [`symbol-bootstrap run`](docs/run.md) - It boots the network via docker using the generated `docker-compose.yml` file and configuration. The config and compose methods/commands need to be called before this method. This is just a wrapper for the `docker-compose up` bash call.
 * [`symbol-bootstrap start`](docs/start.md) - Single command that aggregates config, compose and run in one line!
 * [`symbol-bootstrap stop`](docs/stop.md) - It stops the docker-compose network if running (symbol-bootstrap started with --detached). This is just a wrapper for the `docker-compose down` bash call.
+* [`symbol-bootstrap upgradeVotingKeys`](docs/upgradeVotingKeys.md) - It upgrades the voting keys and files when required.


upgradeVotingKeys not sure if command name. Not really upgrading the voting keys. Maybe registerVotingKeys

Unsure, the command creates the file when necessary, it doesn't register in the network. For that, the link command is required (or symbol cli)

hehe...maybe register is not a good name either.
upgradeVotingKeys doesnt make much sense to me since voting key/files are not upgradable.
@segfaultxavi thoughts?
maybe addVotingKeys or createVotingKeys

renewVotingKeys?

Options:

updateVotingKeys

createVotingKeys

addVotingKeys

renewVotingKeys

maybe 3 ? @segfaultxavi @Wayonb

This command, together with link, does two things, right? Creates new keys AND removes (or unregisters) the old ones. I think add and create both fail to convey the second action. I can see @Wayonb's point that the keys are not actually "updated", so I suggested the renew option :)
But this command will be very seldom used (twice a year!) so let's go with whatever :D

I've talked to @Wayonb , updateVotingKeys is the name.

presets/mainnet/network.yml

src/service/LinkService.ts

src/service/VotingService.ts

Wayonb · 2021-06-07T17:01:38Z

src/service/VotingService.ts

+            const votingUtils = new VotingUtils();
+            await BootstrapUtils.deleteFile(join(votingKeysFolder, 'metadata.yml'));
+            let newFileCreated = false;
+            while (true) {


Not sure why while(true)? maybe I am missing something.
Are multiple files getting created? I would assume when the loop is finish either it creates a voting key file or not.

if you have a short votingKeyDesiredLifetime and a large votingKeyDesiredFutureLifetime in your custom preset, the service generates multiple files. Most people would not change this, but you have the option of having smaller files at once if you like

ok. I would suggest not generating keys that are not been used.

Fair enough. I have removed the while and only one file can be created at a given time. Also, votingKeyDesiredFutureLifetime cannot be greater than votingKeyDesiredLifetime.

src/service/VotingService.ts

src/service/VotingUtils.ts

# Conflicts: # CHANGELOG.md # package-lock.json

segfaultxavi · 2021-06-15T07:29:21Z

src/commands/updateVotingKeys.ts

+
+Voting file update:
+- If the node's current voting file has an end epoch close to the current epoch ("close to expiring") this command creates a new 'private_key_treeX.dat' that continues the current file.
+- By default, "Close to expiring" happens the voting file in the . By default, bootstrap will allow creating new files once the current file reaches its last month.


I don't know what are you trying to say here, the first sentence makes no sense.

Yeap, what about now?

segfaultxavi

👍

Wayonb · 2021-06-15T14:56:23Z

CHANGELOG.md

@@ -7,6 +7,9 @@ The changelog format is based on [Keep a Changelog](https://keepachangelog.com/e
 ## [1.0.7] - NEXT
 **Milestone**: Mainnet(1.0.1.0)

+-   Added multi voting key file support.
+-   Added `upgradeVotingKeys` command.
+


updateVotingKeys?

fboucquez requested review from gimre-xymcity, Wayonb and rg911 March 2, 2021 14:07

gimre-xymcity reviewed Mar 2, 2021

View reviewed changes

src/service/VotingUtils.ts Outdated Show resolved Hide resolved

src/service/VotingUtils.ts Outdated Show resolved Hide resolved

fboucquez changed the title ~~WIP Voting Key generation using typescript~~ Voting Key generation using typescript Mar 2, 2021

fboucquez requested a review from gimre-xymcity March 2, 2021 18:46

fboucquez mentioned this pull request Mar 3, 2021

Bootstrap Offline (Post-Launch) #178

Open

gimre-xymcity reviewed Mar 4, 2021

View reviewed changes

src/service/VotingUtils.ts Outdated Show resolved Hide resolved

src/service/VotingUtils.ts Show resolved Hide resolved

src/service/VotingUtils.ts Show resolved Hide resolved

src/service/VotingUtils.ts Show resolved Hide resolved

fboucquez changed the title ~~Voting Key generation using typescript~~ Voting Key generation using typescript (Post-Launch) Mar 4, 2021

fboucquez added 6 commits March 30, 2021 19:45

WIP Voting Key generation

fa72519

Using root private key to sign the item

2c50839

Added Gimre's vector tests

c0a6d94

Added multi implementation, best one is noble-ed25519 Updated VotingService.ts to use the native one instead of the catapult tools

Updated CHANGELOG.md

c9eb9c0

Added 2.test-sign.json vector tests

f4a0457

Multi voting file support

c631e13

fboucquez force-pushed the native_voting_key_file branch from c735043 to c631e13 Compare March 30, 2021 22:51

fboucquez changed the title ~~Voting Key generation using typescript (Post-Launch)~~ Voting Key generation using typescript and multi file support Mar 31, 2021

fboucquez added 5 commits May 11, 2021 07:28

Updated improvements

070c7cc

fixed unit test

a51bd07

Link command improvement. Unlinking old and linking up to 3 files

181d971

votingKeyDesiredLifetime rename

0e1fad9

segfaultxavi mentioned this pull request May 14, 2021

Explain how to link multiple voting keys symbol/symbol-docs#608

Closed

fboucquez added 4 commits May 19, 2021 18:06

Merge remote-tracking branch 'origin/dev' into native_voting_key_file

2500a78

# Conflicts: # presets/shared.yml

Prettier to ignore generated files

11d3b0a

Merge branch 'dev' into native_voting_key_file

9a3157e

Merge branch 'native_voting_key_file' of github.com:nemtech/symbol-bo…

3b62d86

…otstrap into native_voting_key_file # Conflicts: # presets/shared.yml

fboucquez force-pushed the native_voting_key_file branch from e3a8623 to 3b62d86 Compare May 21, 2021 16:06

fboucquez added 4 commits May 24, 2021 07:47

Merge remote-tracking branch 'origin/dev' into native_voting_key_file

5c46791

# Conflicts: # CHANGELOG.md # src/service/LinkService.ts # src/service/RewardProgramService.ts

Merge remote-tracking branch 'origin/dev' into native_voting_key_file

e128b6c

# Conflicts: # CHANGELOG.md # presets/shared.yml # src/service/ConfigService.ts # src/service/VotingService.ts

Merge remote-tracking branch 'origin/dev' into native_voting_key_file

36e2b4c

Merge remote-tracking branch 'origin/dev' into native_voting_key_file

8416e08

fboucquez force-pushed the native_voting_key_file branch 3 times, most recently from 1f65bd3 to 8e8d422 Compare June 3, 2021 15:53

Epoch is not resolved from the network when possible

45a9dac

fboucquez force-pushed the native_voting_key_file branch from 8e8d422 to 45a9dac Compare June 3, 2021 16:02

Wayonb suggested changes Jun 7, 2021

View reviewed changes

fboucquez requested a review from Wayonb June 8, 2021 13:48

fboucquez added 2 commits June 8, 2021 11:46

Merge remote-tracking branch 'origin/dev' into native_voting_key_file

cd51e8a

# Conflicts: # CHANGELOG.md # package-lock.json

Feedback fixes

fa5fed1

segfaultxavi suggested changes Jun 15, 2021

View reviewed changes

Updated docs

a7207a3

fboucquez requested a review from segfaultxavi June 15, 2021 11:46

segfaultxavi previously approved these changes Jun 15, 2021

View reviewed changes

Wayonb previously approved these changes Jun 15, 2021

View reviewed changes

fboucquez dismissed stale reviews from Wayonb and segfaultxavi via ea7c963 June 15, 2021 16:33

Update CHANGELOG.md

ea7c963

fboucquez merged commit 8f889b6 into dev Jun 15, 2021

fboucquez deleted the native_voting_key_file branch June 15, 2021 16:34

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Voting Key generation using typescript and multi file support #177

Voting Key generation using typescript and multi file support #177

fboucquez commented Mar 2, 2021 •

edited

Loading

gimre-xymcity commented Mar 2, 2021

fboucquez commented Mar 2, 2021 •

edited

Loading

fboucquez commented Mar 2, 2021 •

edited

Loading

Wayonb Jun 7, 2021

fboucquez Jun 8, 2021

Wayonb Jun 9, 2021

fboucquez Jun 14, 2021

Wayonb Jun 7, 2021

fboucquez Jun 8, 2021

Wayonb Jun 9, 2021

segfaultxavi Jun 9, 2021

fboucquez Jun 14, 2021 •

edited

Loading

segfaultxavi Jun 14, 2021

fboucquez Jun 14, 2021

Wayonb Jun 7, 2021

fboucquez Jun 8, 2021

Wayonb Jun 9, 2021

fboucquez Jun 14, 2021

segfaultxavi Jun 15, 2021

fboucquez Jun 15, 2021

segfaultxavi left a comment

Wayonb Jun 15, 2021

Voting Key generation using typescript and multi file support #177

Voting Key generation using typescript and multi file support #177

Conversation

fboucquez commented Mar 2, 2021 • edited Loading

gimre-xymcity commented Mar 2, 2021

fboucquez commented Mar 2, 2021 • edited Loading

fboucquez commented Mar 2, 2021 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

fboucquez Jun 14, 2021 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

segfaultxavi left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

fboucquez commented Mar 2, 2021 •

edited

Loading

fboucquez commented Mar 2, 2021 •

edited

Loading

fboucquez commented Mar 2, 2021 •

edited

Loading

fboucquez Jun 14, 2021 •

edited

Loading