Sliding Sync: Slight optimization when fetching state for the room (get_events_as_list(...)
)
#17718
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Spawning from @kegsay pointing out that the Sliding Sync endpoint doesn't handle a large room with a lot of state well on initial sync (requesting all state via
required_state: [ ["*","*"] ]
) (it just takes forever).After investigating further, the slow part is just
get_events_as_list(...)
fetching all of the current state ID's out for the room (which can be 100k+ events for rooms with a lot of membership). This is just a slow thing in Synapse in general and the same thing happens in Sync v2 or the/state
endpoint.The only idea I had to improve things was to use
batch_iter
to only try fetching a fixed amount at a time instead of working with large maps, lists, and sets. This doesn't seem to have much effect though.There is already a
batch_iter(event_ids, 200)
in_fetch_event_rows(...)
for when we actually have to touch the database and that's inside a queue to deduplicate work.I did notice one slight optimization to use
get_events_as_list(...)
directly instead ofget_events(...)
.get_events(...)
just turns the result fromget_events_as_list(...)
into a dict and since we're just iterating over the events, we don't need the dict/map.Dev notes
Run the new tests
Enable Jaeger tracing in Twisted Trial tests:
Run Jaeger locally with Docker: https://www.jaegertracing.io/docs/1.6/getting-started/#all-in-one-docker-image
Add the following to
tests/unittest.py
insetUp(...)
:Add the following to your tests (in
tests/rest/client/test_asdf.py
for example):Add a sleep to the end of the test you're running so the traces have a chance to flow to Jaeger before the process exits.
If you're not tracing an endpoint/servlet (which will already have a logging context and tracing span), you will need to wrap your function under test in a logging context for the tracing to be happy. Also be sure to add
@trace
decorators to what you want to trace downstream.Run your tests.
Visit http://localhost:16686/ and choose the
test master
service and find the traces you're interesting in.Pull Request Checklist
EventStore
toEventWorkerStore
.".code blocks
.(run the linters)