Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Flushing issue on small block ranges #20

Closed
0237h opened this issue Oct 29, 2024 · 1 comment · Fixed by #25
Closed

Flushing issue on small block ranges #20

0237h opened this issue Oct 29, 2024 · 1 comment · Fixed by #25
Assignees

Comments

@0237h
Copy link
Contributor

0237h commented Oct 29, 2024

When trying to run the sink on small block ranges, presumably smaller than the default flush intervals, the sink doesn't appear to flush any data to the database.

To reproduce:

  1. Create a ClickHouse database with the following schema.sql
CREATE TABLE IF NOT EXISTS cursors
(
    id        String,
    cursor    String,
    block_num Int64,
    block_id  String
)
    ENGINE = ReplacingMergeTree()
        PRIMARY KEY (id)
        ORDER BY (id);

CREATE TABLE IF NOT EXISTS contract_creation
(
    block_hash              String,
    block_number            UInt64,
    block_timestamp_seconds UInt64,
    contract_address        String,
    creation_bytecode       String,
    creator_address         String,
    creator_tx              String
)
    ENGINE = ReplacingMergeTree()
        PRIMARY KEY (block_hash, contract_address)
        ORDER BY (block_hash, contract_address);
  1. Run the sink with the default parameters on a small block range (e.g. 13042456:13042458)
sink-sql run clickhouse://<user>:<password>@<node-url>:9000/<database> https://github.com/pinax-network/subgraph-contract-creation/releases/download/v0.1.1/evm-contract-creation-tracker-v0.1.1.spkg 13042456:13042458 -e <ethereum-endpoint> --undo-buffer-size 1 --plaintext
  1. See that no data has been flushed to the DB, no messages mentioning flushing with TRACE=true.

If you run it on a bigger range (i.e. bigger than the flush interval), you should see those messages appear:

{"severity":"DEBUG","timestamp":"2024-10-29T15:06:02.486524953Z","logger":"sink-sql","message":"flushing to database","block":"#13041000 (fe1e4293ad23436efad71f00827b7eba2e7d342176546cb99e7686b0de10e9b4)","is_live":false,"block_flush_interval_reached":true,"row_flush_interval_reached":false,"logging.googleapis.com/labels":{}}
{"severity":"DEBUG","timestamp":"2024-10-29T15:06:02.495166368Z","logger":"sink-sql","message":"flushing table entries","table_name":"contract_creation","entry_count":93,"logging.googleapis.com/labels":{}}
@fschoell
Copy link
Contributor

this is fixed in #25

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
2 participants