Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Update create-track in rally #1836

Closed
wants to merge 0 commits into from

Conversation

gareth-ellis
Copy link
Member

Add option to increase batch size when creating a track to speed up download of data. Update track layout to adhere to best practices

Creating tracks from large corpus can take quite a bit of time, so I have added the option to increase the batch size of scan, so if a user is running with a stable enough network connection and enough hardware resources on the rally instance, they can increase the batch size to speed up download.

I have also updated the track layout to better match how we tend to layout our tracks

@gareth-ellis gareth-ellis added the enhancement Improves the status quo label Mar 1, 2024
@gareth-ellis gareth-ellis added this to the 2.x milestone Mar 1, 2024
@gareth-ellis gareth-ellis requested a review from a team March 1, 2024 21:07
@ebadyano
Copy link
Contributor

ebadyano commented Mar 5, 2024

Thank you so much for the changes! Does it make sense to update command line reference to include the new option to increase the batch size? And update the create -tracks docs?

@gareth-ellis
Copy link
Member Author

Yes, that makes sense. I'll update those too

@ebadyano
Copy link
Contributor

ebadyano commented Mar 5, 2024

tested with regular index and data streams. I think once we have updated docs it's good to go!

One interesting thing for data-streams scenario: we create a regular index in the new track, it's the same as the original behaviour and out of scope for this pr, but I wonder if for data-streams it makes sense if we create a track that index docs into data streams?..

Copy link
Member

@inqueue inqueue left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

++ @ebadyano's comments. The rest LGTM.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement Improves the status quo
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants