Multi-Processing Functionality for create-workload #403
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
IanHoang
reviewed
Nov 1, 2023
| client, index, out_path, start_doc, end_doc, total_docs, progress_message_suffix="" | ||
| ): | ||
| """ | ||
| Extract documents in the range of start_doc and end_doc and write to induvidual files |
There was a problem hiding this comment.
Nit: induvidual files --> individual files
IanHoang
reviewed
Nov 1, 2023
| Extract documents in the range of start_doc and end_doc and write to induvidual files | ||
|
|
||
| :param client: OpenSearch client used to extract data | ||
| :param index: Name of index to dump |
There was a problem hiding this comment.
To help newcomers understand better, we can redefine this as:
name of OpenSearch index to extract documents from
|
Thanks for tackling this @AkshathRaghav! We should still offer users the option to go with the original route and not perform concurrent dumping. Adding a parameter like
Some additional notes:
|
gkamat
requested changes
Nov 2, 2023
IanHoang
reviewed
Nov 2, 2023
|
@AkshathRaghav please also elaborate the title so it clear which component of OSB this is addressing. Thanks |
AkshathRaghav
changed the title
|
Some benchmarks:
All these methods run in under a second, so instead I've been comparing them with respect to extracting+dumping speed which is overall faster in the concurrent runs. @gkamat I am working on making some parameters more configurable right now! |
|
@gkamat I've made the requested changes. Currently working on changing unit tests. |
Signed-off-by: AkshathRaghav <[email protected]>
Signed-off-by: AkshathRaghav <[email protected]>
Signed-off-by: AkshathRaghav <[email protected]>
Signed-off-by: AkshathRaghav <[email protected]>
…e for documents. Needs documentation update, and configurability Signed-off-by: AkshathRaghav <[email protected]>
Signed-off-by: AkshathRaghav <[email protected]>
…ensearch-project#407) Signed-off-by: vivek palakkat <[email protected]> Signed-off-by: AkshathRaghav <[email protected]>
Signed-off-by: AkshathRaghav <[email protected]>
AkshathRaghav
force-pushed
the
AkshathRaghav/multi_process_index_extraction
branch
from
b6ea5ca to
a4bbb2d
Compare
Signed-off-by: AkshathRaghav <[email protected]>
|
@gkamat @IanHoang Hi! I think I've done most of what is required for this PR. Please let me know if everything is ok. I tried adding the commit from the conflicting file, but security check still shows it. Not sure what to do on that front. |
Signed-off-by: AkshathRaghav <[email protected]>
This was referenced Nov 8, 2023
The checks aren't running since there are conflicts in |
IanHoang
reviewed
Nov 14, 2023
| help="Batch size to use for dumping documents (default: false)", | ||
| ) | ||
| create_workload_parser.add_argument( | ||
| "--custom_dump_query", |
There was a problem hiding this comment.
To keep it consistent, --custom_dump_query --> --custom-dump-query
IanHoang
reviewed
Nov 14, 2023
| help="Number of threads for dumping documents from indices (default: false)", | ||
| ) | ||
| create_workload_parser.add_argument( | ||
| "--bsize", |
There was a problem hiding this comment.
Nit: I think it's worth changing this to --batch-size to make it easier to understand on first glance and to keep it consistent with every other parameter that spells out the option. One thing we can do in the future is add an abbreviated option for specific parameters. For example, users can either use --batch-size or -b.
IanHoang
reviewed
Nov 14, 2023
| "--threads", | ||
| type=positive_number, | ||
| default=8, | ||
| help="Number of threads for dumping documents from indices (default: false)", |
There was a problem hiding this comment.
To clarify, we should elaborate the help statement:
Number of threads used to extract documents from indices and dump to workload. --threads parameter should be used with --concurrent flag. By default, --threads is disabled. If no argument is provided to --threads parameter, number of threads defaults to 8.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description
Adding multi-process functionality that works concurrently.
Issues Resolved
#375
Testing
REQUEST FOR HELP:
Below, is the sample output for the create_workloads with multi-processing. The render_progress is not working as expected, I have added a lock() to not overload and print for every thread thats running, but it doesn't seem to be working. Please help if you can!
By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.