Add high watermark to slashing database #896

siladu · 2023-08-31T20:43:00Z

Add high_watermark_epoch and high_watermark_slot to metadata db table
CRUD operations in MetadataDao
- find
- update (assumes GVR metadata is already inserted)
- delete
Add trigger to prevent updating high_watermark when it is <= low_watermark (not strictly required since we assume high_watermark is set in the future but it's a cheap constraint to enforce to give some protection)
Add trigger to prevent updating low_watermark when it is > high_watermark. Can occur when:
- explicitly setting low_watermark using watermark-repair
- implicitly via slashing import of a block/attestation that is > high_watermark, including via key manager API.

Log example error when using watermark-repair violates the condition:

$ web3signer --config-file=tmp/config.yaml eth2 watermark-repair --epoch=8 --slot=8
...
Failed to initialize Web3Signer
Cause: org.postgresql.util.PSQLException: ERROR: Insert/Update violates constraint: low_watermark slot must be less than high_watermark_slot in the metadata table
  Where: PL/pgSQL function check_low_watermarks() line 10 at RAISE [statement:"INSERT INTO low_watermarks (validator_id, slot) VALUES (:validator_id, :slot) ON CONFLICT (validator_id) DO UPDATE set slot = :slot where ((low_watermarks.slot <= :slot) OR low_watermarks.slot IS NULL)", arguments:{positional:{}, named:{validator_id:1,slot:8}, finder:[]}]
...
Caused by: org.postgresql.util.PSQLException: ERROR: Insert/Update violates constraint: low_watermark slot must be less than high_watermark_slot in the metadata table
  Where: PL/pgSQL function check_low_watermarks() line 10 at RAISE
...

Log example of partially successful slashing import:

$ web3signer --config-file=tmp/config.yaml eth2 import --from=import.json
...
2023-09-05 10:46:28.407+10:00 | main | INFO  | DbSlashingProtection | Importing slashing protection database
2023-09-05 10:46:28.506+10:00 | main | ERROR | InterchangeV5Importer | Failed importing slashing protection data for validator 3 caused by:org.postgresql.util.PSQLException: ERROR: Insert/Update violates constraint: low_watermark slot must be less than high_watermark_slot in the metadata table
  Where: PL/pgSQL function check_low_watermarks() line 10 at RAISE [statement:"INSERT INTO low_watermarks (validator_id, slot) VALUES (:validator_id, :slot) ON CONFLICT (validator_id) DO UPDATE set slot = :slot where ((low_watermarks.slot <= :slot) OR low_watermarks.slot IS NULL)", arguments:{positional:{}, named:{validator_id:1,slot:11}, finder:[]}]
2023-09-05 10:46:28.513+10:00 | ForkJoinPool.commonPool-worker-4 | ERROR | InterchangeV5Importer | Failed importing slashing protection data for validator 4 caused by:org.postgresql.util.PSQLException: ERROR: Insert/Update violates constraint: low_watermark slot must be less than high_watermark_slot in the metadata table
  Where: PL/pgSQL function check_low_watermarks() line 10 at RAISE [statement:"INSERT INTO low_watermarks (validator_id, slot) VALUES (:validator_id, :slot) ON CONFLICT (validator_id) DO UPDATE set slot = :slot where ((low_watermarks.slot <= :slot) OR low_watermarks.slot IS NULL)", arguments:{positional:{}, named:{validator_id:1,slot:12}, finder:[]}]
2023-09-05 10:46:28.519+10:00 | ForkJoinPool.commonPool-worker-2 | INFO  | InterchangeV5Importer | Imported slashing protection data for validator 0xb0adbf87af490e5bf94574125d20ccb29fd2466e2d667316290520b75cd76878b7cbdc4f6caceadced1e0aa172635bec
2023-09-05 10:46:28.534+10:00 | ForkJoinPool.commonPool-worker-1 | INFO  | InterchangeV5Importer | Imported slashing protection data for validator 0xb0adbf87af490e5bf94574125d20ccb29fd2466e2d667316290520b75cd76878b7cbdc4f6caceadced1e0aa172635bec
2023-09-05 10:46:28.542+10:00 | ForkJoinPool.commonPool-worker-5 | INFO  | InterchangeV5Importer | Imported slashing protection data for validator 0xb0adbf87af490e5bf94574125d20ccb29fd2466e2d667316290520b75cd76878b7cbdc4f6caceadced1e0aa172635bec
2023-09-05 10:46:28.543+10:00 | ForkJoinPool.commonPool-worker-3 | INFO  | InterchangeV5Importer | Imported slashing protection data for validator 0xb0adbf87af490e5bf94574125d20ccb29fd2466e2d667316290520b75cd76878b7cbdc4f6caceadced1e0aa172635bec
2023-09-05 10:46:28.544+10:00 | main | INFO  | DbSlashingProtection | Import complete

Log example of failed slashing import via key import endpoint:

2023-09-05 11:07:50.961+10:00 | vert.x-worker-thread-12 | INFO  | DbSlashingProtection | Importing slashing protection database for keys: 0xa37ed0e245653917901d5c35a9cc456bf5680abf69cd00c52c3b66e271c10d5306505bafb3c1ef75982ae13b9a8f5409
2023-09-05 11:07:50.992+10:00 | vert.x-worker-thread-12 | DEBUG | BlockImporter | Updating Block slot low watermark to 2
2023-09-05 11:07:51.010+10:00 | vert.x-worker-thread-12 | ERROR | InterchangeV5Importer | Failed importing slashing protection data for validator 0 caused by:org.postgresql.util.PSQLException: ERROR: Insert/Update violates constraint: low_watermark slot must be less than high_watermark_slot in the metadata table
  Where: PL/pgSQL function check_low_watermarks() line 10 at RAISE [statement:"INSERT INTO low_watermarks (validator_id, slot) VALUES (:validator_id, :slot) ON CONFLICT (validator_id) DO UPDATE set slot = :slot where ((low_watermarks.slot <= :slot) OR low_watermarks.slot IS NULL)", arguments:{positional:{}, named:{validator_id:2,slot:2}, finder:[]}]
2023-09-05 11:07:51.010+10:00 | vert.x-worker-thread-12 | INFO  | DbSlashingProtection | Import complete
2023-09-05 11:07:51.014+10:00 | pool-2-thread-1 | INFO  | DefaultArtifactSignerProvider | Loaded new signer for identifier '0xa37ed0e245653917901d5c35a9cc456bf5680abf69cd00c52c3b66e271c10d5306505bafb3c1ef75982ae13b9a8f5409'
2023-09-05 11:07:51.016+10:00 | vert.x-worker-thread-12 | INFO  | RegisteredValidators | Validators registered successfully in database:1

Documentation

I thought about documentation and added the doc-change-required label to this PR if updates are required.

Changelog

I thought about adding a changelog entry, and added one if I deemed necessary.

Testing

I thought about testing these changes in a realistic/non-local environment.

- high_watermark_epoch and high_watermark_slot - CRUD operations in MetadataDao - find - update (assumes GVR metadata is already inserted) - delete

siladu · 2023-08-31T23:42:41Z

Blocked - Ready for review but won't merge this until after Holesky support is merged/released.

jframe · 2023-09-01T03:22:02Z

...ing-protection/src/main/java/tech/pegasys/web3signer/slashingprotection/dao/MetadataDao.java

+        .findFirst();
+  }
+
+  public int updateHighWatermark(final Handle handle, final HighWatermark highWatermark) {


Does this need a constraint on the high watermark being height then all of the low watermarks?

I didn't add anything around that because this AC simplifies it:

Web3signer operator is responsible for setting a good high-watermark that is in the future

...but: even though you could still set a HWM such that LWM < HWM < latestSignedData, it's probably not a bad (and cheap?) check to have in place. I will look into it.

...ing-protection/src/main/java/tech/pegasys/web3signer/slashingprotection/dao/MetadataDao.java

… greater than low_watermarks

…ess than or equal to high_watermarks

…k-metadata Signed-off-by: Simon Dudley <[email protected]>

...g-protection/src/main/resources/migrations/postgresql/V00012__add_highwatermark_metadata.sql

siladu · 2023-09-07T22:12:54Z

Tested performance with 100,000 validators:

Used AT to generate 100K random BLS keys and load into a signer
For each key, sent one block and one attestation signature request
This gave me a low_watermarks table of 100,000 rows containing both slots and epochs
Ran the following EXPLAIN ANALYZE query to simulate adding a high watermark and exercising the check_before_insert_or_update_high_watermarks trigger.

Min: 27ms
Max: 49ms

explain analyze update metadata set high_watermark_epoch = 131477, high_watermark_slot = 4214365;
                                                 QUERY PLAN
------------------------------------------------------------------------------------------------------------
 Update on metadata  (cost=0.00..17.80 rows=0 width=0) (actual time=27.678..27.679 rows=0 loops=1)
   ->  Seq Scan on metadata  (cost=0.00..17.80 rows=780 width=46) (actual time=0.020..0.021 rows=1 loops=1)
 Planning Time: 0.600 ms
 Trigger check_before_insert_or_update_high_watermarks: time=23.080 calls=1
 Execution Time: 27.946 ms



web3signer=# web3signer=# explain analyze update metadata set high_waternull, high_watermark_slot = null;
                                                 QUERY PLAN
------------------------------------------------------------------------------------------------------------
 Update on metadata  (cost=0.00..17.80 rows=0 width=0) (actual time=49.197..49.197 rows=0 loops=1)
   ->  Seq Scan on metadata  (cost=0.00..17.80 rows=780 width=46) (actual time=0.420..0.421 rows=1 loops=1)
 Planning Time: 0.539 ms
 Trigger check_before_insert_or_update_high_watermarks: time=48.524 calls=1
 Execution Time: 49.351 ms

siladu · 2023-09-08T03:06:44Z

No meaningful difference in time for slashing data import. This test exercises the check_before_insert_or_update_low_watermarks trigger upon insert of low_watermarks.

Tested slashing data import of 100,000 validators (one block, one attestation each)

Control (master branch):
- Run 1: 50.63s user 26.98s system 71% cpu 1:48.94 total
- Run 2: 50.38s user 26.91s system 73% cpu 1:45.23 total
- Run 3: 53.33s user 28.19s system 71% cpu 1:54.60 total

No high watermark set:
- Run 1: 54.19s user 27.88s system 64% cpu 2:06.50 total
- Run 2:  52.06s user 27.20s system 77% cpu 1:42.51 total
- Run 3: 50.59s user 26.52s system 77% cpu 1:39.00 total

With high watermark set:
- Run 1: 53.38s user 25.33s system 63% cpu 2:03.14 total
- Run 2: 52.51s user 27.29s system 74% cpu 1:46.83 total
- Run 3: 50.86s user 27.31s system 74% cpu 1:45.55 total

With high watermark set to 1, all but one validator violates constraint:
- Run 1: 44.15s user 15.31s system 79% cpu 1:14.73 total

jframe

LGTM

Signed-off-by: Simon Dudley <[email protected]>

Add high_watermark data to metadata db table

69b62c6

- high_watermark_epoch and high_watermark_slot - CRUD operations in MetadataDao - find - update (assumes GVR metadata is already inserted) - delete

siladu marked this pull request as ready for review August 31, 2023 23:41

siladu added the TeamCerberus Under active development by TeamCerberus @Consensys label Aug 31, 2023

siladu marked this pull request as draft August 31, 2023 23:42

siladu added the blocked label Aug 31, 2023

siladu self-assigned this Aug 31, 2023

jframe reviewed Sep 1, 2023

View reviewed changes

siladu added 2 commits September 1, 2023 18:09

Add PL/pgSQL triggers as constraints for checking high_watermarks are…

24f0dc6

… greater than low_watermarks

Add PL/pgSQL trigger as constraints for checking low_watermarks are l…

bab8bb2

…ess than or equal to high_watermarks

siladu changed the title ~~Add high_watermark data to metadata db table~~ Add high watermark to slashing database Sep 5, 2023

Merge remote-tracking branch 'upstream/master' into add-high-watermar…

cd69f99

…k-metadata Signed-off-by: Simon Dudley <[email protected]>

siladu mentioned this pull request Sep 6, 2023

Prevent signing at or beyond high watermark siladu/web3signer#1

Closed

3 tasks

jframe reviewed Sep 7, 2023

View reviewed changes

siladu removed the blocked label Sep 7, 2023

siladu marked this pull request as ready for review September 7, 2023 23:23

jframe approved these changes Sep 8, 2023

View reviewed changes

Merge branch 'master' into add-high-watermark-metadata

00bed32

Signed-off-by: Simon Dudley <[email protected]>

siladu enabled auto-merge (squash) September 8, 2023 03:21

siladu merged commit 79d8053 into Consensys:master Sep 8, 2023
2 checks passed

siladu deleted the add-high-watermark-metadata branch September 17, 2023 21:54

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add high watermark to slashing database #896

Add high watermark to slashing database #896

siladu commented Aug 31, 2023 •

edited

Loading

siladu commented Aug 31, 2023 •

edited

Loading

jframe Sep 1, 2023

siladu Sep 1, 2023 •

edited

Loading

siladu commented Sep 7, 2023 •

edited

Loading

siladu commented Sep 8, 2023 •

edited

Loading

jframe left a comment

Add high watermark to slashing database #896

Add high watermark to slashing database #896

Conversation

siladu commented Aug 31, 2023 • edited Loading

Documentation

Changelog

Testing

siladu commented Aug 31, 2023 • edited Loading

jframe Sep 1, 2023

Choose a reason for hiding this comment

siladu Sep 1, 2023 • edited Loading

Choose a reason for hiding this comment

siladu commented Sep 7, 2023 • edited Loading

siladu commented Sep 8, 2023 • edited Loading

jframe left a comment

Choose a reason for hiding this comment

siladu commented Aug 31, 2023 •

edited

Loading

siladu commented Aug 31, 2023 •

edited

Loading

siladu Sep 1, 2023 •

edited

Loading

siladu commented Sep 7, 2023 •

edited

Loading

siladu commented Sep 8, 2023 •

edited

Loading