Element-R: sending messages takes several seconds

[richvdh]

Every time you send a message, we make a /keys/claim request for each device where we don't already have an olm session. This can take several seconds.

If there are devices in the room which have run out of OTKs, we will do this for every single message.

[richvdh]

matrix-org/matrix-rust-sdk#281 will address this.

[ara4n]

moved comment to element-hq/element-meta#2149 at @richvdh's request

[richvdh]

might be improved by #26382; to revisit once that is merged

[richvdh]

might be improved by #26382; to revisit once that is merged

It's still taking me 8 seconds to send each message in #megolm:matrix.org, of which 5 seconds is spent waiting for a /keys/claim request.

[richvdh]

/keys/claim being slow (and repeated) is not the only problem here; see also matrix-org/matrix-rust-sdk#2793

[richvdh]

Also: matrix-org/matrix-rust-sdk#2804

[richvdh]

We need to improve the logging in the message send path. It's generally impossible to tell what is going on and how long it takes.

[richvdh]

Recent work in this area includes:

• Handle missing devices in /keys/claim responses matrix-org/matrix-rust-sdk#2805
• Improve performance of get_missing_sessions matrix-org/matrix-rust-sdk#2845
• Improve performance of share_room_key matrix-org/matrix-rust-sdk#2862
• Logging improvements (not directly improving send time but certainly helping diagnosis):
  • Improve logging of event encryption in RustCrypto matrix-org/matrix-js-sdk#3898
  • Rationalise logging in receive_keys_claim_response matrix-org/matrix-rust-sdk#2860
  • Add instrument annotations to various methods matrix-org/matrix-rust-sdk#2861
  • Reduce logspam during encryption matrix-org/matrix-rust-sdk#2859
  • Be consistent about endpoint names matrix-org/matrix-rust-sdk#2857
  • Implement a custom event formatter for javascript logging matrix-org/matrix-rust-sdk#2856
• General log spam reduction measures:
  • Reduce console log spam matrix-org/matrix-js-sdk#3896
  • Stop logging decryptions as retries matrix-org/matrix-js-sdk#3897
  • Reduce amount of log spam matrix-org/matrix-react-sdk#11898
  • Fix "not attempting encryption" warning matrix-org/matrix-react-sdk#11899

We still need to wait for a number of those rust-sdk PRs to land, and then we can release them into a matrix-sdk-crypto-wasm release, but I think that once that happens we can call this "done" for now. It still takes a few seconds to send messages in #megolm:matrix.org due to the unresponsive servers, but that is not a common case.

[richvdh]

Currently blocked on matrix-org/matrix-rust-sdk-crypto-wasm#58