Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Documentation updates for DataPrepper 2.8 #7135

Merged
merged 28 commits into from
May 24, 2024

Conversation

kkondaka
Copy link
Contributor

Description

Documentation changes for

  • write-json processor
  • typeof operator
  • event_json input/output codec
  • key value processor new config options
  • local_mode in aggregate processor
  • s3 scan workers config option

Issues Resolved

List any issues this PR will resolve, e.g. Closes [...].

Checklist

  • [X ] By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license and subject to the Developers Certificate of Origin.
    For more information on following Developer Certificate of Origin and signing off your commits, please check here.

@hdhalter
Copy link
Contributor

Thanks, @kkondaka ! Can you please fix the Vale errors?

Signed-off-by: Kondaka <[email protected]>
@kkondaka
Copy link
Contributor Author

@hdhalter fixed the vale errors

@hdhalter hdhalter added 4 - Doc review PR: Doc review in progress labels May 13, 2024
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please name the file write_json.md. Let's try to keep the naming consistent with the processor/source/sink name.

Copy link
Contributor

@vagimeli vagimeli left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Doc review complete. Please see my suggested revisions. Once doc review comments are addressed, I'll move this forward ti editorial review. Thank you.

_data-prepper/pipelines/expression-syntax.md Outdated Show resolved Hide resolved
_data-prepper/pipelines/expression-syntax.md Outdated Show resolved Hide resolved
_data-prepper/pipelines/expression-syntax.md Outdated Show resolved Hide resolved
_data-prepper/pipelines/expression-syntax.md Outdated Show resolved Hide resolved
_data-prepper/pipelines/expression-syntax.md Outdated Show resolved Hide resolved
@vagimeli vagimeli added the 3 - Tech review PR: Tech review in progress label May 17, 2024
@vagimeli
Copy link
Contributor

Doc review complete. Please see my suggested revisions. Once doc review comments are addressed, I'll move this forward ti editorial review. Thank you.

@kkondaka Please see the above. Once you review and accept doc review changes, I can move this into editorial and we can get it merged. Reach out with any questions. Thank you.

kkondaka and others added 7 commits May 23, 2024 16:02
Co-authored-by: Melissa Vagi <[email protected]>
Signed-off-by: Krishna Kondaka <[email protected]>
Co-authored-by: Melissa Vagi <[email protected]>
Signed-off-by: Krishna Kondaka <[email protected]>
Co-authored-by: Melissa Vagi <[email protected]>
Signed-off-by: Krishna Kondaka <[email protected]>
Co-authored-by: Melissa Vagi <[email protected]>
Signed-off-by: Krishna Kondaka <[email protected]>
Copy link
Contributor

@vagimeli vagimeli left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@kkondaka Thank you for addressing the doc review comments. The PR is scheduled for editorial review. Once that review is completed, I'll merge the PR.

Copy link
Collaborator

@natebower natebower left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@kkondaka @vagimeli Please see my comments and changes and let me know if you have any questions. Thanks!


| Option | Description | Example |
| :--- | :--- | :--- |
| source | Mandatory field that specifies the name of the field in the event containing the message or object to be parsed. | If `source` is `"message"`, `{"message": {"key1":"value1", "key2":{"key3":"value3"}}`, the `write_json` processor, generates `{"message": "{\"key1\":\"value`\", \"key2\":"{\"key3\":\"value3\"}"}"`
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm not following the example here. Are the tic marks/commas in the right places, and should there be a "then" in there somewhere?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I revised it.

@@ -104,7 +104,7 @@ Option | Required | Type | Description
`s3_select` | No | [s3_select](#s3_select) | The Amazon S3 Select configuration.
`scan` | No | [scan](#scan) | The S3 scan configuration.
`delete_s3_objects_on_read` | No | Boolean | When `true`, the S3 scan attempts to delete S3 objects after all events from the S3 object are successfully acknowledged by all sinks. `acknowledgments` should be enabled when deleting S3 objects. Default is `false`.
`workers` | No | Integer | Configures the number of worker threads that the source uses to read data from S3. Leaving this value at the default unless your S3 objects are less than 1MB. Performance may decrease for larger S3 objects. This setting only affects SQS-based sources. Default is `1`.
`workers` | No | Integer | Configures the number of worker threads that the source uses to read data from S3. Leaving this value at the default unless your S3 objects are less than 1 MB. Performance may decrease for larger S3 objects. This setting affects SQS-based sources and S3-Scan source. Default is `1`.
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'll assume that we've defined SQS previously.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, Line 1 intro para

@hdhalter hdhalter added 5 - Editorial review PR: Editorial review in progress backport 2.14 PR: Backport label for 2.14 and removed 3 - Tech review PR: Tech review in progress 4 - Doc review PR: Doc review in progress labels May 24, 2024
@vagimeli vagimeli merged commit 2941ecf into opensearch-project:main May 24, 2024
6 checks passed
@vagimeli vagimeli removed the 5 - Editorial review PR: Editorial review in progress label May 24, 2024
opensearch-trigger-bot bot pushed a commit that referenced this pull request May 24, 2024
* Documentation updates for DataPrepper 2.8

Signed-off-by: Kondaka <[email protected]>

* Fixed vale errors

Signed-off-by: Kondaka <[email protected]>

* Addressed review comments

Signed-off-by: Kondaka <[email protected]>

* Update _data-prepper/common-use-cases/codec-processor-combinations.md

Co-authored-by: Melissa Vagi <[email protected]>
Signed-off-by: Krishna Kondaka <[email protected]>

* Update _data-prepper/common-use-cases/codec-processor-combinations.md

Co-authored-by: Melissa Vagi <[email protected]>
Signed-off-by: Krishna Kondaka <[email protected]>

* Update _data-prepper/pipelines/configuration/processors/key-value.md

Co-authored-by: Melissa Vagi <[email protected]>
Signed-off-by: Krishna Kondaka <[email protected]>

* Update _data-prepper/pipelines/configuration/processors/aggregate.md

Co-authored-by: Melissa Vagi <[email protected]>
Signed-off-by: Krishna Kondaka <[email protected]>

* Update _data-prepper/pipelines/configuration/processors/key-value.md

Co-authored-by: Melissa Vagi <[email protected]>
Signed-off-by: Krishna Kondaka <[email protected]>

* Update _data-prepper/pipelines/expression-syntax.md

Co-authored-by: Melissa Vagi <[email protected]>
Signed-off-by: Krishna Kondaka <[email protected]>

* Update _data-prepper/pipelines/configuration/processors/aggregate.md

Co-authored-by: Melissa Vagi <[email protected]>
Signed-off-by: Krishna Kondaka <[email protected]>

* Update _data-prepper/pipelines/configuration/processors/write_json.md

Co-authored-by: Melissa Vagi <[email protected]>
Signed-off-by: Krishna Kondaka <[email protected]>

* Update _data-prepper/pipelines/expression-syntax.md

Co-authored-by: Melissa Vagi <[email protected]>
Signed-off-by: Krishna Kondaka <[email protected]>

* Update _data-prepper/pipelines/expression-syntax.md

Co-authored-by: Melissa Vagi <[email protected]>
Signed-off-by: Krishna Kondaka <[email protected]>

* Update _data-prepper/pipelines/expression-syntax.md

Co-authored-by: Melissa Vagi <[email protected]>
Signed-off-by: Krishna Kondaka <[email protected]>

* Update _data-prepper/pipelines/expression-syntax.md

Co-authored-by: Melissa Vagi <[email protected]>
Signed-off-by: Krishna Kondaka <[email protected]>

* Update _data-prepper/pipelines/configuration/processors/write_json.md

Co-authored-by: Melissa Vagi <[email protected]>
Signed-off-by: Krishna Kondaka <[email protected]>

* Update _data-prepper/pipelines/configuration/processors/write_json.md

Co-authored-by: Melissa Vagi <[email protected]>
Signed-off-by: Krishna Kondaka <[email protected]>

* Update _data-prepper/pipelines/configuration/processors/aggregate.md

Co-authored-by: Nathan Bower <[email protected]>
Signed-off-by: Melissa Vagi <[email protected]>

* Update _data-prepper/pipelines/configuration/processors/key-value.md

Co-authored-by: Nathan Bower <[email protected]>
Signed-off-by: Melissa Vagi <[email protected]>

* Update _data-prepper/pipelines/configuration/processors/write_json.md

Co-authored-by: Nathan Bower <[email protected]>
Signed-off-by: Melissa Vagi <[email protected]>

* Update _data-prepper/pipelines/configuration/processors/write_json.md

Co-authored-by: Nathan Bower <[email protected]>
Signed-off-by: Melissa Vagi <[email protected]>

* Update _data-prepper/pipelines/configuration/processors/aggregate.md

Co-authored-by: Nathan Bower <[email protected]>
Signed-off-by: Melissa Vagi <[email protected]>

* Update _data-prepper/pipelines/configuration/processors/key-value.md

Co-authored-by: Nathan Bower <[email protected]>
Signed-off-by: Melissa Vagi <[email protected]>

* Update _data-prepper/pipelines/configuration/sources/s3.md

Signed-off-by: Melissa Vagi <[email protected]>

* Update _data-prepper/pipelines/configuration/processors/write_json.md

Signed-off-by: Melissa Vagi <[email protected]>

* Update _data-prepper/pipelines/configuration/processors/aggregate.md

Signed-off-by: Melissa Vagi <[email protected]>

---------

Signed-off-by: Kondaka <[email protected]>
Signed-off-by: Krishna Kondaka <[email protected]>
Signed-off-by: Melissa Vagi <[email protected]>
Co-authored-by: Melissa Vagi <[email protected]>
Co-authored-by: Nathan Bower <[email protected]>
(cherry picked from commit 2941ecf)
Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
backport 2.14 PR: Backport label for 2.14 data-prepper 2.8
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants