Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

dcrpg: Remove duplicates for votes, misses and treasury table #1979

Open
wants to merge 1 commit into
base: master
Choose a base branch
from

Conversation

ukane-philemon
Copy link
Collaborator

Closes #1978

  1. Duplicates for votes removed successfully:
2024-01-03 21:40:00.003 [INF] PSQL: Block import elapsed: 0.30 minutes, 145 blocks (8.11 blocks/s)
2024-01-03 21:40:00.003 [INF] PSQL: Beginning SYNC STAGE 2 of 5 (duplicate row removal).
2024-01-03 21:40:00.003 [INF] PSQL: Dropping temporary index on addresses(tx_vin_vout_row_id).
2024-01-03 21:40:00.170 [INF] PSQL: Finding and removing duplicate table rows before indexing...
2024-01-03 21:40:00.170 [INF] PSQL: Finding and removing duplicate vins entries...
2024-01-03 21:47:00.622 [INF] PSQL: Removed 0 duplicate vins entries.
2024-01-03 21:47:00.623 [INF] PSQL: Finding and removing duplicate vouts entries...
2024-01-03 21:55:25.063 [INF] PSQL: Removed 0 duplicate vouts entries.
2024-01-03 21:55:25.064 [INF] PSQL: Finding and removing duplicate transactions entries...
2024-01-03 21:56:49.243 [INF] PSQL: Removed 0 duplicate transactions entries.
2024-01-03 21:56:49.244 [INF] PSQL: Finding and removing duplicate agendas entries...
2024-01-03 21:56:49.255 [INF] PSQL: Removed 0 duplicate agendas entries.
2024-01-03 21:56:49.255 [INF] PSQL: Finding and removing duplicate agenda_votes entries...
2024-01-03 21:57:08.451 [INF] PSQL: Removed 0 duplicate agenda_votes entries.
2024-01-03 21:57:08.451 [INF] PSQL: Finding and removing duplicate votes entries...
2024-01-03 21:57:35.071 [INF] PSQL: Removed 8 duplicate votes entries.
2024-01-03 21:57:35.161 [INF] PSQL: Beginning SYNC STAGE 3 of 5 (table indexing and analyzing).
  1. Duplicates for misses removed successfully:
2024-01-03 22:42:58.134 [INF] PSQL: Beginning SYNC STAGE 2 of 5 (duplicate row removal).
2024-01-03 22:42:58.134 [INF] PSQL: Dropping temporary index on addresses(tx_vin_vout_row_id).
2024-01-03 22:42:58.287 [INF] PSQL: Finding and removing duplicate table rows before indexing...
2024-01-03 22:42:58.287 [INF] PSQL: Finding and removing duplicate vins entries...
2024-01-03 22:53:28.812 [INF] PSQL: Removed 0 duplicate vins entries.
2024-01-03 22:53:28.812 [INF] PSQL: Finding and removing duplicate vouts entries...
2024-01-03 23:08:03.250 [INF] PSQL: Removed 0 duplicate vouts entries.
2024-01-03 23:08:03.251 [INF] PSQL: Finding and removing duplicate transactions entries...
2024-01-03 23:10:11.626 [INF] PSQL: Removed 0 duplicate transactions entries.
2024-01-03 23:10:11.627 [INF] PSQL: Finding and removing duplicate agendas entries...
2024-01-03 23:10:11.652 [INF] PSQL: Removed 0 duplicate agendas entries.
2024-01-03 23:10:11.652 [INF] PSQL: Finding and removing duplicate agenda_votes entries...
2024-01-03 23:10:40.645 [INF] PSQL: Removed 0 duplicate agenda_votes entries.
2024-01-03 23:10:40.646 [INF] PSQL: Finding and removing duplicate votes entries...
2024-01-03 23:11:17.805 [INF] PSQL: Removed 0 duplicate votes entries.
2024-01-03 23:11:17.806 [INF] PSQL: Finding and removing duplicate misses entries...
2024-01-03 23:11:18.786 [INF] PSQL: Removed 2 duplicate misses entries.
2024-01-03 23:11:18.786 [INF] PSQL: Beginning SYNC STAGE 3 of 5 (table indexing and analyzing).
  1. Duplicates for treasury removed successfully:
2024-01-04 15:07:34.407 [INF] PSQL: Beginning SYNC STAGE 2 of 5 (duplicate row removal).
2024-01-04 15:07:34.407 [INF] PSQL: Dropping temporary index on addresses(tx_vin_vout_row_id).
2024-01-04 15:07:34.718 [INF] PSQL: Finding and removing duplicate table rows before indexing...
2024-01-04 15:07:34.719 [INF] PSQL: Finding and removing duplicate vins entries...
2024-01-04 15:13:54.914 [INF] PSQL: Removed 0 duplicate vins entries.
2024-01-04 15:13:54.915 [INF] PSQL: Finding and removing duplicate vouts entries...
2024-01-04 15:23:43.447 [INF] PSQL: Removed 0 duplicate vouts entries.
2024-01-04 15:23:43.448 [INF] PSQL: Finding and removing duplicate transactions entries...
2024-01-04 15:25:10.908 [INF] PSQL: Removed 0 duplicate transactions entries.
2024-01-04 15:25:10.909 [INF] PSQL: Finding and removing duplicate agendas entries...
2024-01-04 15:25:10.925 [INF] PSQL: Removed 0 duplicate agendas entries.
2024-01-04 15:25:10.925 [INF] PSQL: Finding and removing duplicate agenda_votes entries...
2024-01-04 15:25:29.573 [INF] PSQL: Removed 0 duplicate agenda_votes entries.
2024-01-04 15:25:29.573 [INF] PSQL: Finding and removing duplicate votes entries...
2024-01-04 15:25:52.580 [INF] PSQL: Removed 0 duplicate votes entries.
2024-01-04 15:25:52.580 [INF] PSQL: Finding and removing duplicate misses entries...
2024-01-04 15:25:53.819 [INF] PSQL: Removed 0 duplicate misses entries.
2024-01-04 15:25:53.819 [INF] PSQL: Finding and removing duplicate treasury entries...
2024-01-04 15:25:56.690 [INF] PSQL: Removed 2 duplicate treasury entries.
2024-01-04 15:25:56.690 [INF] PSQL: Beginning SYNC STAGE 3 of 5 (table indexing and analyzing).

@ukane-philemon ukane-philemon changed the title dcrpg:remove duplicates for votes, misses and treasury table dcrpg: Remove duplicates for votes, misses and treasury table Jan 5, 2024
Comment on lines +413 to +421
// DeleteVotesDuplicateRows removes rows that would violate the unique
// index uix_votes_hashes_index. This should be run prior to creating the index.
DeleteVotesDuplicateRows = `DELETE FROM votes
WHERE id IN (SELECT id FROM (
SELECT id,
row_number() OVER (PARTITION BY tx_hash, block_hash ORDER BY id DESC) AS rnum
FROM votes) t
WHERE t.rnum > 1);`

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I've been hitting this more often cuz I've had to drop my testnet dcrdata database and resync for both mainnet and testnet.

Got this log from mainnet dcrdata.

2024-08-22 21:27:40.740 [INF] PSQL: Indexing votes table on candidate block...
2024-08-22 21:28:01.216 [INF] PSQL: Indexing votes table on block hash...
2024-08-22 21:28:12.322 [INF] PSQL: Indexing votes table on block+tx hash...
2024-08-22 21:28:15.537 [ERR] DATD: dcrpg.SyncChainDB failed at height 898277.
2024-08-22 21:28:15.537 [INF] DATD: Shutdown requested. Shutting down...
2024-08-22 21:28:15.538 [DBG] JAPI: Got quit signal. Exiting block connected handler for STATUS monitor.
2024-08-22 21:28:15.539 [WRN] EXPR: ExchangeBot has quit.
2024-08-22 21:28:15.539 [DBG] DATD: Block prefetcher hits = 108, misses = 2.
2024-08-22 21:28:15.539 [INF] DATD: Gracefully shutting down web server...
2024-08-22 21:28:15.541 [INF] PUBS: Stopping websocket hub.
2024-08-22 21:28:15.541 [DBG] PUBS: Unregistered and killed 0 clients.
2024-08-22 21:28:15.541 [INF] EXPR: Stopping websocket hub.
2024-08-22 21:28:15.540 [ERR] XBOT: dcrdex websocket error: read tcp 192.168.1.243:58607->45.77.164.152:7232: use of closed network connection
2024-08-22 21:28:15.677 [DBG] SKDB: badger: Got compaction priority: {level:0 score:1.73 dropPrefixes:[]}
2024-08-22 21:28:16.381 [DBG] SKDB: badger: LOG Compact. Added 898280 keys. Skipped 1 keys. Iteration took: 701.425775ms
2024-08-22 21:28:16.836 [DBG] SKDB: badger: Discard stats: map[]
2024-08-22 21:28:16.859 [INF] SKDB: badger: LOG Compact 0->1, del 2 tables, add 1 tables, took 1.180893312s
2024-08-22 21:28:16.859 [DBG] SKDB: badger: Compaction for level: 0 DONE
2024-08-22 21:28:16.951 [INF] DATD: Closing connection to dcrd.
2024-08-22 21:28:16.952 [INF] DATD: Bye!
2024-08-22 21:28:17.203 [ERR] DATD: IndexAll failed: pq: could not create unique index "uix_votes_hashes_index"

@ukane-philemon
Copy link
Collaborator Author

Turns out we also need to remove duplicates for uix_ticket_hashes_index.

2024-08-23 01:13:57.732 [INF] PSQL: Indexing treasury on tx hash...
2024-08-23 01:13:59.190 [INF] PSQL: Indexing treasury on block height...
2024-08-23 01:13:59.465 [INF] PSQL: Indexing swaps on spend height...
2024-08-23 01:13:59.963 [INF] PSQL: Indexing tickets table on ticket hash...
2024-08-23 01:14:06.736 [ERR] DATD: dcrpg.SyncChainDB failed at height 898324.
2024-08-23 01:14:06.736 [INF] DATD: Shutdown requested. Shutting down...
2024-08-23 01:14:06.736 [DBG] JAPI: Got quit signal. Exiting block connected handler for STATUS monitor.
2024-08-23 01:14:06.737 [INF] DATD: Gracefully shutting down web server...
2024-08-23 01:14:06.737 [DBG] DATD: Block prefetcher hits = 92, misses = 2.
2024-08-23 01:14:06.738 [INF] PUBS: Stopping websocket hub.
2024-08-23 01:14:06.739 [DBG] PUBS: Unregistered and killed 0 clients.
2024-08-23 01:14:06.739 [INF] EXPR: Stopping websocket hub.
2024-08-23 01:14:06.738 [WRN] EXPR: ExchangeBot has quit.
2024-08-23 01:14:06.739 [ERR] XBOT: dcrdex websocket error: read tcp 192.168.1.243:60761->45.77.164.152:7232: use of closed network connection
2024-08-23 01:14:06.901 [DBG] SKDB: badger: Got compaction priority: {level:0 score:1.73 dropPrefixes:[]}
2024-08-23 01:14:07.909 [DBG] SKDB: badger: LOG Compact. Added 898327 keys. Skipped 1 keys. Iteration took: 1.00608323s
2024-08-23 01:14:08.405 [DBG] SKDB: badger: Discard stats: map[]
2024-08-23 01:14:08.415 [INF] SKDB: badger: LOG Compact 0->1, del 2 tables, add 1 tables, took 1.513239135s
2024-08-23 01:14:08.415 [DBG] SKDB: badger: Compaction for level: 0 DONE
2024-08-23 01:14:08.508 [INF] DATD: Closing connection to dcrd.
2024-08-23 01:14:08.509 [INF] DATD: Bye!
2024-08-23 01:14:08.760 [ERR] DATD: IndexTicketsTable failed: pq: could not create unique index "uix_ticket_hashes_index"

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

dcrpg: Remove duplicates for votes, misses and treasury table.
1 participant