fix raw writer when batch size is too small #285
Conversation
| @@ -444,4 +444,5 @@ function* pack<T>(items: T[], size: number): Iterable<T[]> { | |||
| } | |||
|
|
|||
| items.splice(0, offset) | |||
| yield items | |||
There was a problem hiding this comment.
This breaks the whole idea of committing the data in fixed sized chunks. This is crucial for reproducibility.
There was a problem hiding this comment.
but if we keep this code as it is then some chunks will not be complete.
recently i discovered that solana raw dataset had ~5 blocks gap. 2 dump isntances were running: one from a specific block and without --last-block and another with --last-block. so that one with --last-block instead of writing a final chunk from 5 blocks were finishing its process without writing anything at all.
and it happend only because the dumper restarted before this final chunk otherwise it still would be able to write this final chunk
There was a problem hiding this comment.
The bug is not here. It's just that, blocks left unwritten after the end of ingest loop, should be saved -
belopash
force-pushed
the
master
branch
2 times, most recently
from
cc23118
to
1f6a573
Compare
No description provided.