v1.17: Adjust receive window to make them linear to the count of streams (backport of #33913) #35495
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
![mergify[bot]](https://avatars.githubusercontent.com/in/10562?s=60&v=4){kind=small}
anza-team
reviewed
Apr 5, 2024
There was a problem hiding this comment.
Thank you for the contribution! This repository is no longer in use. Please re-open this pull request in the agave repo: https://github.com/anza-xyz/agave
anza-team
reviewed
Apr 5, 2024
There was a problem hiding this comment.
Thank you for the contribution! This repository is no longer in use. Please re-open this pull request in the agave repo: https://github.com/anza-xyz/agave
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Problem
It has been found in across region testing, transaction sending to remote regions are some time failing with WriteError(Stopped). This is caused that after a stream is opened, the client is blocked from sending due to the following error in Quinn on blocked by connection level flow control. This is due to limited receive window among the multiple streams opened, some stream spent long time in waiting for the limit to be available. The problem is not as obvious when the latency is low as contenders might release the usage fast enough before the time out is reached. But in the case of wide area network with longer latency the problem becomes more severe. Fundamentally, if we allow that many of streams to be opened, we need to ensure they have enough bandwidth to be processed concurrently.
Summary of Changes
Make connection receive window ratio scale to the streams limit.
Without the change, with the following configuration: 4 nodes in the us-west and 1 node in eu-west, running bench-tps tests with thin-client, with duration set to > 300 seconds, we can observe send transaction failures in quic_client with WriteError(Stopped)., with the fix, the problem is no longer observed.
Fixes #
This is an automatic backport of pull request #33913 done by [Mergify](https://mergify.com).