Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Segment Replication] consider to decouple the rateLimiter of segrep and recovery #12939

Closed
Ferrari248 opened this issue Mar 27, 2024 · 3 comments · Fixed by #12959
Closed

[Segment Replication] consider to decouple the rateLimiter of segrep and recovery #12939

Ferrari248 opened this issue Mar 27, 2024 · 3 comments · Fixed by #12959
Labels
enhancement Enhancement or improvement to existing feature or request Indexing:Replication Issues and PRs related to core replication framework eg segrep Storage:Durability Issues and PRs related to the durability framework Storage:Remote v2.14.0

Comments

@Ferrari248
Copy link
Contributor

Ferrari248 commented Mar 27, 2024 •
edited
Loading

Is your feature request related to a problem? Please describe

Hi, I am interested in [Segment Replication] and use this mod a lot. I found that when indexing pressure is a little high, the replica shards using segrep usually get a larger and larger lag time, and then breach the back pressure limit to reject indexing request. That troubles me a lot.
Finally, I found in codes that segrep and peerRecovery share the same setting indices.recovery.max_bytes_per_sec and its corresponding rateLimiter, default to 40mb/s. When I raise this value up, the lag bytes and lag time decreased soon. I don't know why we use the same rateLimiter? Can we use two different settings to decouple these two? Is there anything other to consider?

Describe the solution you'd like

add segrep.max_bytes_per_sec setting for segrep.

Related component

Indexing:Replication

Describe alternatives you've considered

No response

Additional context

No response
@Ferrari248 Ferrari248 added enhancement Enhancement or improvement to existing feature or request untriaged labels Mar 27, 2024
@github-actions github-actions bot added the Indexing:Replication Issues and PRs related to core replication framework eg segrep label Mar 27, 2024
@vikasvb90 vikasvb90 added the Storage:Durability Issues and PRs related to the durability framework label Mar 27, 2024
@vikasvb90
Copy link
Contributor

cc: @Bukhtawar @gbbafna @sachinpkale

@peternied
Copy link
Member

[Triage - attendees 1 2 3 4 5]
@Ferrari248 Thanks for creating this issue, we'd welcome a pull request to address.

@Ferrari248
Copy link
Contributor Author

[Triage - attendees 1 2 3 4 5] @Ferrari248 Thanks for creating this issue, we'd welcome a pull request to address.

Thanks for your reply, I will try to pull request soon.

@Ferrari248 Ferrari248 mentioned this issue Mar 28, 2024
Merged
8 tasks
Ferrari248 added a commit to Ferrari248/OpenSearch that referenced this issue Apr 10, 2024
@Ferrari248
[segment replication] decouple the rateLimiter of segrep and recovery (…
25936fc 
…opensearch-project#12939)

add setting "indices.replication.max_bytes_per_sec" which takes effect when not negative

Signed-off-by: maxliu <[email protected]>
Ferrari248 added a commit to Ferrari248/OpenSearch that referenced this issue Apr 10, 2024
@Ferrari248
[segment replication] decouple the rateLimiter of segrep and recovery (…
9f1e604 
…opensearch-project#12939)

add setting "indices.replication.max_bytes_per_sec" which takes effect when not negative

Signed-off-by: maxliu <[email protected]>
Ferrari248 added a commit to Ferrari248/OpenSearch that referenced this issue Apr 11, 2024
@Ferrari248
[segment replication] decouple the rateLimiter of segrep and recovery (…
8c1a6ce 
…opensearch-project#12939)

add setting "indices.replication.max_bytes_per_sec" which takes effect when not negative

Signed-off-by: maxliu <[email protected]>

Adds change log

Signed-off-by: maxliu <[email protected]>
mch2 pushed a commit that referenced this issue Apr 14, 2024
@Ferrari248
[segment replication] decouple the rateLimiter of segrep and recovery (…
6bc04b4 
…#12959)

* [segment replication] decouple the rateLimiter of segrep and recovery (12939)

add setting "segrep.max_bytes_per_sec"

Signed-off-by: maxliu <[email protected]>

* [segment replication] decouple the rateLimiter of segrep and recovery (12939)

use setting "indices.replication.max_bytes_per_sec" if enable "indices.replication.use_individual_rate_limiter"

Signed-off-by: maxliu <[email protected]>

* [segment replication] decouple the rateLimiter of segrep and recovery (12939)

setting "indices.replication.max_bytes_per_sec" takes effect when not negative

Signed-off-by: maxliu <[email protected]>

* [segment replication] decouple the rateLimiter of segrep and recovery (#12939)

add setting "indices.replication.max_bytes_per_sec" which takes effect when not negative

Signed-off-by: maxliu <[email protected]>

Adds change log

Signed-off-by: maxliu <[email protected]>

---------

Signed-off-by: maxliu <[email protected]>
@github-project-automation github-project-automation bot moved this from 🆕 New to ✅ Done in Storage Project Board Apr 14, 2024
opensearch-trigger-bot bot pushed a commit that referenced this issue Apr 14, 2024
@github-actions
[segment replication] decouple the rateLimiter of segrep and recovery (…
429e144 
…#12959)

* [segment replication] decouple the rateLimiter of segrep and recovery (12939)

add setting "segrep.max_bytes_per_sec"

Signed-off-by: maxliu <[email protected]>

* [segment replication] decouple the rateLimiter of segrep and recovery (12939)

use setting "indices.replication.max_bytes_per_sec" if enable "indices.replication.use_individual_rate_limiter"

Signed-off-by: maxliu <[email protected]>

* [segment replication] decouple the rateLimiter of segrep and recovery (12939)

setting "indices.replication.max_bytes_per_sec" takes effect when not negative

Signed-off-by: maxliu <[email protected]>

* [segment replication] decouple the rateLimiter of segrep and recovery (#12939)

add setting "indices.replication.max_bytes_per_sec" which takes effect when not negative

Signed-off-by: maxliu <[email protected]>

Adds change log

Signed-off-by: maxliu <[email protected]>

---------

Signed-off-by: maxliu <[email protected]>
(cherry picked from commit 6bc04b4)
Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
mch2 pushed a commit that referenced this issue Apr 15, 2024
@github-actions @mch2
[segment replication] decouple the rateLimiter of segrep and recovery (…
b12306c 
…#12959)

* [segment replication] decouple the rateLimiter of segrep and recovery (12939)

add setting "segrep.max_bytes_per_sec"

Signed-off-by: maxliu <[email protected]>

* [segment replication] decouple the rateLimiter of segrep and recovery (12939)

use setting "indices.replication.max_bytes_per_sec" if enable "indices.replication.use_individual_rate_limiter"

Signed-off-by: maxliu <[email protected]>

* [segment replication] decouple the rateLimiter of segrep and recovery (12939)

setting "indices.replication.max_bytes_per_sec" takes effect when not negative

Signed-off-by: maxliu <[email protected]>

* [segment replication] decouple the rateLimiter of segrep and recovery (#12939)

add setting "indices.replication.max_bytes_per_sec" which takes effect when not negative

Signed-off-by: maxliu <[email protected]>

Adds change log

Signed-off-by: maxliu <[email protected]>

---------

Signed-off-by: maxliu <[email protected]>
(cherry picked from commit 6bc04b4)
Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
mch2 added a commit that referenced this issue Apr 15, 2024
@opensearch-trigger-bot @github-actions @mch2
[Backport 2.x] [segment replication] decouple the rateLimiter of segr…
c295d97 
…ep and recovery (#13181)

* [segment replication] decouple the rateLimiter of segrep and recovery (#12959)

* [segment replication] decouple the rateLimiter of segrep and recovery (12939)

add setting "segrep.max_bytes_per_sec"

Signed-off-by: maxliu <[email protected]>

* [segment replication] decouple the rateLimiter of segrep and recovery (12939)

use setting "indices.replication.max_bytes_per_sec" if enable "indices.replication.use_individual_rate_limiter"

Signed-off-by: maxliu <[email protected]>

* [segment replication] decouple the rateLimiter of segrep and recovery (12939)

setting "indices.replication.max_bytes_per_sec" takes effect when not negative

Signed-off-by: maxliu <[email protected]>

* [segment replication] decouple the rateLimiter of segrep and recovery (#12939)

add setting "indices.replication.max_bytes_per_sec" which takes effect when not negative

Signed-off-by: maxliu <[email protected]>

Adds change log

Signed-off-by: maxliu <[email protected]>

---------

Signed-off-by: maxliu <[email protected]>
(cherry picked from commit 6bc04b4)
Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>

* Add back public API in RecoverySettings to 2.x

Signed-off-by: Marc Handalian <[email protected]>

---------

Signed-off-by: maxliu <[email protected]>
Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
Signed-off-by: Marc Handalian <[email protected]>
Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
Co-authored-by: Marc Handalian <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
enhancement Enhancement or improvement to existing feature or request Indexing:Replication Issues and PRs related to core replication framework eg segrep Storage:Durability Issues and PRs related to the durability framework Storage:Remote v2.14.0
Projects
Storage Project Board
Status: ✅ Done
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

[segment replication] decouple the rateLimiter of segrep and recovery Ferrari248/OpenSearch
4 participants
@peternied @anasalkouz @vikasvb90 @Ferrari248