[OPIK-410] improve batch mechanism in SDK #648
+206
−29
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Details
This PR improves batching in a way that now SDK sends batched requests with the payload no bigger than 50 MB (unless one batch item already exceeds this limit).
It affects spans, feedback scores and dataset items.
Implementation details:
The batching for spans (since only spans support background batching today) could be implemented differently. We already have a batching mechanism that works inside the
Streamer
instance -BatchManager
. This mechanism creates batches based on time intervals and the amount of accumulated items.Memory checks were NOT added to
BatchManager
because they can be time-consuming and today's implementation of this class requires them to be done in the user's thread. Instead, in this PR they are performed inside the background worker threads. So, whenever the worker receives the batch created byBatchManager
, it can split it additionally into smaller batches based on memory, and then send them sequentially.An alternative approach could be to implement a new
BatchManager
which would work fully in the background thread and have it's own messages queue. We can switch to this approach for spans if we need it later.Testing
Unit tests for splitting function are added.
Manual tests on big batches with heavy payloads were run.