Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[FEATURE] Dynamically adjust rate limit for OpenSearch requests #570

Closed
ykmr1224 opened this issue Aug 16, 2024 · 2 comments
Closed

[FEATURE] Dynamically adjust rate limit for OpenSearch requests #570

ykmr1224 opened this issue Aug 16, 2024 · 2 comments
Labels
enhancement New feature or request

Comments

@ykmr1224
Copy link
Collaborator

Is your feature request related to a problem?
When creating/updating index into OpenSearch, it could cause throttling due to too much request.
Although rate limit is implemented (#567), as it uses Spark parameter, we cannot adjust the limit once the streaming job is started.

What solution would you like?
We want to come up with a way to dynamically adjust rate limit so that user can modify late limit even after they started a job.

What alternatives have you considered?
Alter the index to manual refresh, and change back to auto refresh, which would start new streaming job, and can pick up new rate limit config.

Do you have any additional context?
n/a

@ykmr1224 ykmr1224 added enhancement New feature or request untriaged labels Aug 16, 2024
@dblock dblock removed the untriaged label Sep 9, 2024
@dblock
Copy link
Member

dblock commented Sep 9, 2024

[Catch All Triage - 1, 2, 3, 4, 5]

@ykmr1224
Copy link
Collaborator Author

Replaced with #640

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

No branches or pull requests

2 participants