enhancement: don't allow missing blocks

[lostman]

Description

Closes #1150.

This branch is built on top of #1337 and will be merged after.

Testing steps

We don't have any means to check the indexer's status—whether it is running or terminated with an error code, etc.

The database trigger can be observed in manual testing by adding something like this to executor.rs:

diff --git a/packages/fuel-indexer/src/executor.rs b/packages/fuel-indexer/src/executor.rs
index d771ef79..75dc9bad 100644
--- a/packages/fuel-indexer/src/executor.rs
+++ b/packages/fuel-indexer/src/executor.rs
@@ -142,7 +142,7 @@ pub fn run_executor<T: 'static + Executor + Send + Sync>(
             }
 
             // Fetch the next page of blocks, and the starting cursor for the subsequent page
-            let (block_info, next_cursor, _has_next_page) =
+            let (mut block_info, next_cursor, _has_next_page) =
                 match retrieve_blocks_from_node(
                     &client,
                     node_block_page_size,
@@ -180,6 +180,10 @@ pub fn run_executor<T: 'static + Executor + Send + Sync>(
                 continue;
             }
 
+            if block_info.len() > 10 {
+                block_info.remove(5);
+            }
+
             // The client responded with actual blocks, so attempt to index them.
             let result = executor.handle_events(block_info).await;
 

Then, running the indexer:

cargo run -p fuel-indexer -- run --fuel-node-host beta-4.fuel.network --fuel-node-port 80 --replace-indexer --run-migrations --manifest examples/fuel-explorer/fuel-explorer/fuel_explorer.manifest.yaml --remove-data

Should produce:

2023-09-13T13:00:04.812050Z ERROR fuel_indexer::ffi: 229: Failed to put_object: Error from sqlx: Database(
    PgDatabaseError {
        severity: Error,
        code: "P0001",
        message: "fuellabs_explorer.indexmetadataentity: attempted to insert value with block_height = 7 while last indexed block_height = 5. block_height values must be consecutive.",
        detail: None,
        hint: None,
        position: None,
        where: Some(
            "PL/pgSQL function ensure_block_height_consecutive() line 8 at RAISE",
        ),
        schema: None,
        table: None,
        column: None,
        data_type: None,
        constraint: None,
        file: Some(
            "pl_exec.c",
        ),
        line: Some(
            3907,
        ),
        routine: Some(
            "exec_stmt_raise",
        ),
    },
)
2023-09-13T13:00:04.812915Z ERROR fuel_indexer::executor: 892: Indexer(fuellabs.explorer) WASM execution failed: Some blocks are missing.
2023-09-13T13:00:04.813305Z ERROR fuel_indexer::executor: 199: Indexer(fuellabs.explorer) terminating due to missing blocks.

CI testing

I have added a test case that simulates missing a block by deleting a row from indexmetadataentity. The expected output is:

running 1 test
test indexing::test_no_missing_blocks ... Sep 13 17:57:02.183 ERROR fuel_indexer::ffi: Failed to put_object: Error from sqlx: Database(
    PgDatabaseError {
        severity: Error,
        code: "P0001",
        message: "fuel_indexer_test_index1.indexmetadataentity: attempted to insert value with block_height = 4 while last indexed block_height = 2. block_height values must be consecutive.",
        detail: None,
        hint: None,
        position: None,
        where: Some(
            "PL/pgSQL function ensure_block_height_consecutive() line 8 at RAISE",
        ),
        schema: None,
        table: None,
        column: None,
        data_type: None,
        constraint: None,
        file: Some(
            "pl_exec.c",
        ),
        line: Some(
            3907,
        ),
        routine: Some(
            "exec_stmt_raise",
        ),
    },
)
Sep 13 17:57:02.185 ERROR fuel_indexer::executor: Indexer(fuel_indexer_test.index1) WASM execution failed: Some blocks are missing.
Sep 13 17:57:02.185 ERROR fuel_indexer::executor: Indexer(fuel_indexer_test.index1) terminating due to missing blocks.
ok

test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 42 filtered out; finished in 22.65s

Since missing blocks are not recoverable in the executor loop, it terminates without retries.

Changelog

• add: database trigger to indexmetadataentity table that ensures indexed blocks must be consecutive.
• add: MissingBlocks error code for WASM indexers returned when the database operation fails due to the exception raised by the trigger.
• change: adjust get_start_block to return last + 1 instead of last to avoid processing the same block twice (and thus triggering an exception).
• change: last_block_height_for_an_indexer now returns 0 if the indexer processed no blocks.

[ra0x3]

@lostman

• Looks mostly OK
• Left some small pieces of feedback

[ra0x3]

utACK ✅

• Will defer to @deekerno