Periodic missing rows when the system is under load

[s373r]

Describe the bug

When reinforcing the E2E test coverage within the PR (kamu-data/kamu-cli#882), an intermittent bug was detected.

Periodic replay is available when running all tests from the kamu-cli repository:

make test-full

To Reproduce

Sorry, the error is not reproduced every time, so there is only a screenshot:

To reproduce the environment from the test locally, you need:

kamu init

kamu add https://raw.githubusercontent.com/kamu-data/kamu-cli/refs/heads/master/examples/leaderboard/player-scores.yaml https://raw.githubusercontent.com/kamu-data/kamu-cli/refs/heads/master/examples/leaderboard/leaderboard.yaml

{ \
    echo '{"match_time": "2000-01-01", "match_id": 1, "player_id": "Alice", "score": 100}'; \
    echo '{"match_time": "2000-01-01", "match_id": 1, "player_id": "Bob", "score": 80}' \
} \
    | kamu ingest player-scores --stdin

kamu pull leaderboard

kamu tail leaderboard

Expected behavior

Expected:

┌────────┬────┬──────────────────────────┬──────────────────────┬───────┬──────────┬───────────┬───────┐
│ offset │ op │       system_time        │      match_time      │ place │ match_id │ player_id │ score │
├────────┼────┼──────────────────────────┼──────────────────────┼───────┼──────────┼───────────┼───────┤
│      0 │ +A │ 2024-10-15T15:49:41.364Z │ 2000-01-01T00:00:00Z │     1 │        1 │     Alice │   100 │
│      1 │ +A │ 2024-10-15T15:49:41.364Z │ 2000-01-01T00:00:00Z │     2 │        1 │       Bob │    80 │
└────────┴────┴──────────────────────────┴──────────────────────┴───────┴──────────┴───────────┴───────┘

Actual behavior

See screenshot above