Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add looped mode to bulk operation #1830

Merged
merged 1 commit into from
Feb 21, 2024

Conversation

gbanasiak
Copy link
Contributor

@gbanasiak gbanasiak commented Feb 20, 2024

Adds looped mode to bulk operation. That is useful in benchmarks where we want Rally to ingest for a specific amount of time even if corpus is not big enough to sustain indexing at a specific rate for this time.

This mode is meant to be used together with time-period or iterations at the task level. Otherwise Rally will never complete the task.

Example use:

{
  "name": "some-bulk-task",
  "operation": {
    "operation-type": "bulk",
    "bulk-size": 5000,
    "looped": true
  },
  "time-period": 300,
  "target-throughput": 2
}

@gbanasiak gbanasiak added the enhancement Improves the status quo label Feb 20, 2024
@gbanasiak gbanasiak requested a review from a team February 20, 2024 11:03
@gbanasiak
Copy link
Contributor Author

Re-running IT test failure of which seems unrelated.

Copy link
Member

@inqueue inqueue left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

# start from the beginning in looped mode, otherwise stop the run
if self.looped:
self.current_bulk = 0
self._init_internal_params()
Copy link
Member

@inqueue inqueue Feb 20, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

With looped mode, we will have the possibility to add an override for bulk and batch sizes in steps if it is something we can use in the future.

Edit: I see bulk size would be relatively easy to adjust in the task definition itself, and not necessarily here directly.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The idea is looped operation should be limited at the task level, and you can have as many tasks with different batch sizes, etc. as you want. Looped mode basically means "keep on generating bulk requests until some external criteria stop the task".

@gbanasiak gbanasiak merged commit 43e56f6 into elastic:master Feb 21, 2024
15 checks passed
@gbanasiak gbanasiak deleted the bulk-looped-mode branch February 21, 2024 12:09
@gareth-ellis gareth-ellis added this to the 2.11 milestone May 23, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement Improves the status quo
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants