Failed blocks cannot be retried after spawner task is finished

[0237h]

Problem

For the async_single_channel and async_multi_channel extractors, the task spawner finishes as soon as no more blocks are available in the block pool.
However, if one of the blocks from the remaining ranges fails to be extracted, even if it's correctly added back to the block pool, it will never be picked up again since the spawner task has already been killed.

Actual

No retries after a block has failed to be processed since spawner task is stopped.

2022-12-08 12:05:41:T+192 [*] Getting JWT token...
2022-12-08 12:05:41:T+196 [*] Got JWT token (cached) [SUCCESS]
2022-12-08 12:05:41:T+209 [*] Getting JWT token...
2022-12-08 12:05:41:T+210 [*] Got JWT token (cached) [SUCCESS]
2022-12-08 12:05:41:T+211 [*] Streaming 1000 blocks on EOS chain...
2022-12-08 12:05:44:T+2840 [!] [Task-03] No more blocks in block_pool["0"], stopping spawner task NOW...
2022-12-08 12:05:45:T+4043 [ERROR] [Task-46] Failed to process block number #274268900: <AioRpcError of RPC that terminated with:
        status = None
        details = "None"
        debug_error_string = "None"
>
2022-12-08 12:05:45:T+4044 [ERROR] Task-46 encountered an exception
2022-12-08 12:05:49:T+8027 [*] [Task-02] Block streaming done !
2022-12-08 12:05:49:T+8038 [*] Finished block processing, parsed 4 rows of data [SUCCESS]
2022-12-08 12:05:49:T+8040 [*] Wrote 4 rows of data to jsonl/EOS_274268400_to_274269400.jsonl [SUCCESS]

real    0m8.119s
user    0m0.611s
sys     0m0.170s

Expected

Using the async_optimized extractor that correctly retries after failure.

2022-12-08 12:00:19:T+176 [*] Getting JWT token...
2022-12-08 12:00:19:T+181 [*] Got JWT token (cached) [SUCCESS]
2022-12-08 12:00:19:T+185 [*] Streaming 1000 blocks on EOS chain (running 25 workers)...
2022-12-08 12:00:19:T+187 [*] Getting JWT token...
2022-12-08 12:00:19:T+188 [*] Got JWT token (cached) [SUCCESS]
2022-12-08 12:00:30:T+11050 [ERROR] [Task-14] Failed to process block number #274268900: <AioRpcError of RPC that terminated with:
        status = None
        details = "None"
        debug_error_string = "None"
>
2022-12-08 12:00:44:T+24906 [!] Could not fetch block #274268900: retrying... (1/2 retries)
2022-12-08 12:00:44:T+25047 [ERROR] [Task-77] Failed to process block number #274268900: <AioRpcError of RPC that terminated with:
        status = None
        details = "None"
        debug_error_string = "None"
>
2022-12-08 12:00:44:T+25047 [!] Could not fetch block #274268900: retrying... (2/2 retries)
2022-12-08 12:00:44:T+25344 [ERROR] [Task-80] Failed to process block number #274268900: <AioRpcError of RPC that terminated with:
        status = None
        details = "None"
        debug_error_string = "None"
>
2022-12-08 12:00:44:T+25344 [ERROR] Could not fetch block #274268900: maximum number of retries reached (2)
2022-12-08 12:00:44:T+25344 [*] Block streaming done !
2022-12-08 12:00:44:T+25369 [*] Finished block processing, parsed 4 rows of data [SUCCESS]
2022-12-08 12:00:44:T+25370 [*] Wrote 4 rows of data to jsonl/EOS_274268400_to_274269400.jsonl [SUCCESS]

real    0m25.436s
user    0m0.501s
sys     0m0.142s

Possible solutions

• Keep the task spawner alive until all blocks have been marked as processed (meaning until all stream_blocks tasks have finished).
• Restart the spawner when adding back blocks to the block pool if it's not already running.