Created the blog post announcing Data Prepper 2.0 #1066

dlvenable · 2022-10-07T23:18:38Z

Description

We are releasing Data Prepper 2.0.0 on Oct 10. This is our announcement blog post.

This requires the bio for @oeyh as supplied in #1067.

Issues Resolved

N/A

Check List

Commits are signed per the DCO using --signoff
Update authors list after the bio is available from @oeyh.
Merge in Add author info for oeyh #1067

By submitting this pull request, I confirm that my contribution is made under the terms of the BSD-3-Clause License.

Co-authored-by: Hai Yan <[email protected]> Signed-off-by: David Venable <[email protected]>

dlvenable · 2022-10-08T17:59:11Z

I added Hai to the list of authors based on the username he supplied in #1067. We will need to merge that in prior to this PR.

Naarcha-AWS

Added my rewrites for each section in the review. One comment = one section.

Might need to wait to add documentation links until this PR is merged: opensearch-project/documentation-website#1510

Naarcha-AWS · 2022-10-11T14:43:35Z

_posts/2022-10-10-Announcing-Data-Prepper-2.0.0.md

+  - technical-post
+---
+
+Today the maintainers are announcing the release of Data Prepper 2.0. It has been over a year since Data Prepper 1.0 was first introduced


Let's change this paragraph to:

The Data Prepper maintainers are proud to announce the release of Data Prepper 2.0. This release makes Data Prepper easier to use and helps you improve your observability stack based on feedback from our users.

Here are some of the major changes and enhancements made for Data Prepper 2.0.

Or maybe:

The Data Prepper maintainers are proud to announce the release of Data Prepper 2.0. This release makes Data Prepper easier to use and helps you improve your observability stack based on feedback from you, our users.

Here are some of the major changes and enhancements made for Data Prepper 2.0.

@dlvenable: Could we add a line in this intro or somewhere in the blog about OpenSearch compatibility? Data Prepper 2.0 is compatible with all OpenSearch versions, correct?

I added the following:

Data Prepper 2.0 retains compatibility with all current versions of OpenSearch.

_posts/2022-10-10-Announcing-Data-Prepper-2.0.0.md

Naarcha-AWS · 2022-10-11T15:36:25Z

_posts/2022-10-10-Announcing-Data-Prepper-2.0.0.md

+* The HTTP source now supports loading TLS/SSL credentials from either Amazon S3 or Amazon Certificate Manager. The OTel Trace Source supported these options and now pipeline authors can configure them for their log ingestion use-cases as well.
+* Data Prepper now requires Java 11 and the Docker image deploys with JDK 17.
+
+Please see our release notes for a complete list.


Do we have a link to these release notes?

_posts/2022-10-10-Announcing-Data-Prepper-2.0.0.md

Naarcha-AWS · 2022-10-11T15:40:18Z

_posts/2022-10-10-Announcing-Data-Prepper-2.0.0.md

+  - technical-post
+---
+
+Today the maintainers are announcing the release of Data Prepper 2.0. It has been over a year since Data Prepper 1.0 was first introduced


Or maybe:

The Data Prepper maintainers are proud to announce the release of Data Prepper 2.0. This release makes Data Prepper easier to use and helps you improve your observability stack based on feedback from you, our users.

Here are some of the major changes and enhancements made for Data Prepper 2.0.

Signed-off-by: David Venable <[email protected]>

dlvenable · 2022-10-11T17:23:29Z

Thanks @Naarcha-AWS ! I took most of the changes to all sections except the Directory Structure. I want to check with @oeyh on those first.

I did make some tweaks from your suggestions - most of them were to try to be more accurate.

I also wasn't quite sure about some of the paragraphs. Did you intend all those paragraphs? The ones in the examples read too broken up and didn't keep the same train of thought.

Signed-off-by: David Venable <[email protected]>

Naarcha-AWS

A few more minor tweaks before we pass them off to @natebower.

_posts/2022-10-10-Announcing-Data-Prepper-2.0.0.md

Naarcha-AWS · 2022-10-11T17:47:02Z

_posts/2022-10-10-Announcing-Data-Prepper-2.0.0.md

+accepts log data from external sources such as Fluent Bit. 
+
+The pipeline then uses the `grok` processor to split the log line into multiple fields. 
+The `grok` processor adds named `loglevel` to the event. Pipeline authors can use that field in routes. This pipeline has two OpenSearch sinks. The first sink only receives 


Let's break this up a little more:

The pipeline then uses the grok processor to split the log line into multiple fields. The grok processor adds a named loglevel to the event. Pipeline authors can use that field in routes.

This pipeline contains two OpenSearch sinks. The first sink will only receive logs with a log level of WARN or ERROR. Data Prepper will route all events to the second sink.

I took your suggestion and made one clarification by adding "field" which you can see here: "... adds a
field named loglevel ..."

_posts/2022-10-10-Announcing-Data-Prepper-2.0.0.md

oeyh

Two small things:

oeyh · 2022-10-11T17:53:35Z

_posts/2022-10-10-Announcing-Data-Prepper-2.0.0.md

+Simply pick a name appropriate for the domain and a Data Prepper expression.  
+Then for any sink that should only have some data coming through, define one or more routes to apply Data Prepper will evaluate 


I think this supposed to be a space, not a line break; also missing a period in front of Data Prepper will evaluate...:

Suggested change

Simply pick a name appropriate for the domain and a Data Prepper expression.

Then for any sink that should only have some data coming through, define one or more routes to apply Data Prepper will evaluate

Simply pick a name appropriate for the domain and a Data Prepper expression. Then for any sink that should only have some data coming through, define one or more routes to apply. Data Prepper will evaluate

Line breaks should not affect the rendered page.

There was a space and line break which did create a new paragraph in the rendered page. Thanks for noting that!

Signed-off-by: David Venable <[email protected]>

dlvenable · 2022-10-11T19:37:49Z

It took all the suggested changes.

natebower

@dlvenable Please see my changes and comments, and let me know if you have any questions. Thanks!

_posts/2022-10-10-Announcing-Data-Prepper-2.0.0.md

natebower · 2022-10-12T14:49:50Z

_posts/2022-10-10-Announcing-Data-Prepper-2.0.0.md

+
+One common use case for conditional routing is reducing the volume of data going to some clusters.
+When you want info logs that produce large volumes of data to go to a cluster, index with more frequent rollovers, or add deletions to clear out large volumes of data, you can now configure pipelines to route the data with your chosen action.
+deletions to clear out these large volumes of data, you now configure pipelines to route your data.


Suggested change

deletions to clear out these large volumes of data, you now configure pipelines to route your data.

_posts/2022-10-10-Announcing-Data-Prepper-2.0.0.md

natebower · 2022-10-12T14:53:27Z

_posts/2022-10-10-Announcing-Data-Prepper-2.0.0.md

+
+
+Simply pick a name appropriate for the domain and a Data Prepper expression. 
+Then for any sink that should only have some data coming through, define one or more routes to apply. Data Prepper will evaluate 


Second sentence: "to route these events to"?

_posts/2022-10-10-Announcing-Data-Prepper-2.0.0.md

natebower · 2022-10-12T15:47:32Z

_posts/2022-10-10-Announcing-Data-Prepper-2.0.0.md

+For example, when one large object includes a serialized JSON string, you can use the `parse_json` processor to extract 
+the fields from the JSON into your event.
+
+Data Prepper can now import CSV or TSV formatted files from Amazon S3 sources. This is useful for systems like Amazon CloudFront 


Can we remove "formatted"? Otherwise, this would need to be "CSV- or TSV-formatted files".

natebower · 2022-10-12T15:57:51Z

_posts/2022-10-10-Announcing-Data-Prepper-2.0.0.md

+Data Prepper 2.0 includes a number of other improvements. We want to highlight a few of them.
+
+* The OpenSearch sink now supports `create` actions for OpenSearch when writing documents. Pipeline authors can configure their pipelines to only create new documents and not update existing ones.
+* The HTTP source now supports loading TLS/SSL credentials from either Amazon S3 or Amazon Certificate Manager. Pipeline authors can now configure them for their log ingestion use cases. Before Data Prepper 2.0, only the OTel Trace Source supported these options.


Suggested change

* The HTTP source now supports loading TLS/SSL credentials from either Amazon S3 or Amazon Certificate Manager. Pipeline authors can now configure them for their log ingestion use cases. Before Data Prepper 2.0, only the OTel Trace Source supported these options.

* The HTTP source now supports loading SSL/TLS credentials from either Amazon S3 or AWS Certificate Manager (ACM). Pipeline authors can now configure them for their log ingestion use cases. Before Data Prepper 2.0, only the OTel Trace Source supported these options.

I believe either SSL/TLS or TLS/SSL is in use. I intentially chose TLS/SSL because we are using TLS. The SSL part is mostly there for historical reasons.

You can also see that the term TLS/SSL is used in the following Wikipedia article.

https://en.wikipedia.org/wiki/Transport_Layer_Security

natebower · 2022-10-12T15:58:42Z

_posts/2022-10-10-Announcing-Data-Prepper-2.0.0.md

+Data Prepper 2.0 includes a number of other improvements. We want to highlight a few of them.
+
+* The OpenSearch sink now supports `create` actions for OpenSearch when writing documents. Pipeline authors can configure their pipelines to only create new documents and not update existing ones.
+* The HTTP source now supports loading TLS/SSL credentials from either Amazon S3 or Amazon Certificate Manager. Pipeline authors can now configure them for their log ingestion use cases. Before Data Prepper 2.0, only the OTel Trace Source supported these options.


I'm assuming we were referring to AWS Certificate Manager (ACM).

natebower · 2022-10-12T16:13:17Z

_posts/2022-10-10-Announcing-Data-Prepper-2.0.0.md

+* The HTTP source now supports loading TLS/SSL credentials from either Amazon S3 or Amazon Certificate Manager. Pipeline authors can now configure them for their log ingestion use cases. Before Data Prepper 2.0, only the OTel Trace Source supported these options.
+* Data Prepper now requires Java 11 or higher. The Docker image deploys with JDK 17.
+
+Please see our [release notes](https://github.com/opensearch-project/data-prepper/releases/tag/2.0.0) for a complete list.


The only thing we're missing here is a call to action. We need to conclude with a couple sentences telling the reader what we'd like for them to do next or where they can go to learn more. The below is an example from a recent blog post announcing Snapshot Management (SM):

Wrapping it up

SM automates taking snapshots of your cluster and provides useful features like notifications. To learn more about SM, check out the SM documentation section. For more technical details, read the SM meta issue.

If you’re interested in snapshots, consider contributing to the next improvement we’re working on: searchable snapshots.

Signed-off-by: David Venable <[email protected]>

dlvenable

I pushed all the changes except the final call-to-action section. I will push that soon.

dlvenable · 2022-10-12T18:23:32Z

_posts/2022-10-10-Announcing-Data-Prepper-2.0.0.md

+With peer forwarding as a core feature, pipeline authors can perform stateful 
+aggregations on multiple Data Prepper nodes. When performing stateful aggregations, Data Prepper uses a hash ring to determine 
+which nodes are responsible for processing different events based on the values of certain fields. Peer forwarder 
+routes events to the node responsible for processing the event. That node then holds all the state necessary for performing the aggregation.


I'm not sure about the change to "states" here. Using a singular noun for state is quite common.

In information technology and computer science, a system is described as stateful if it is designed to remember preceding events or user interactions; the remembered information is called the state of the system.

https://en.wikipedia.org/wiki/State_(computer_science)

dlvenable · 2022-10-12T18:29:14Z

_posts/2022-10-10-Announcing-Data-Prepper-2.0.0.md

+Data Prepper 2.0 includes a number of other improvements. We want to highlight a few of them.
+
+* The OpenSearch sink now supports `create` actions for OpenSearch when writing documents. Pipeline authors can configure their pipelines to only create new documents and not update existing ones.
+* The HTTP source now supports loading TLS/SSL credentials from either Amazon S3 or Amazon Certificate Manager. Pipeline authors can now configure them for their log ingestion use cases. Before Data Prepper 2.0, only the OTel Trace Source supported these options.


I believe either SSL/TLS or TLS/SSL is in use. I intentially chose TLS/SSL because we are using TLS. The SSL part is mostly there for historical reasons.

You can also see that the term TLS/SSL is used in the following Wikipedia article.

https://en.wikipedia.org/wiki/Transport_Layer_Security

natebower · 2022-10-12T19:09:40Z

@dlvenable I changed to "states" to match @Naarcha-AWS edits and also to avoid "all the state". If you want to use "state", remove "all".

Signed-off-by: David Venable <[email protected]>

dlvenable · 2022-10-12T19:13:44Z

@dlvenable I changed to "states" to match @Naarcha-AWS edits and also to avoid "all the state". If you want to use "state", remove "all".

That makes sense. I've removed "all" from the sentence.

dlvenable · 2022-10-12T19:14:52Z

I have also pushed a short conclusion section.

natebower · 2022-10-12T19:33:06Z

_posts/2022-10-10-Announcing-Data-Prepper-2.0.0.md

+
+## Try Data Prepper 2.0
+
+Data Prepper 2.0 is available for [download](https://opensearch.org/downloads.html#data-prepper) now. The maintainers encourage you to


Because this is a blog, an exclamation point would work after the first sentence. Other than that small nit, LGTM.

natebower

LGTM

Signed-off-by: David Venable <[email protected]>

krisfreedain

looks good!

Created the blog post announcing Data Prepper 2.0.

1b1a91e

Co-authored-by: Hai Yan <[email protected]> Signed-off-by: David Venable <[email protected]>

oeyh mentioned this pull request Oct 8, 2022

Add author info for oeyh #1067

Merged

1 task

Adding oeyh to the author list.

616f362

Co-authored-by: Hai Yan <[email protected]> Signed-off-by: David Venable <[email protected]>

dlvenable marked this pull request as ready for review October 8, 2022 17:58

dlvenable requested a review from a team as a code owner October 8, 2022 17:58

Naarcha-AWS approved these changes Oct 11, 2022

View reviewed changes

Naarcha-AWS suggested changes Oct 11, 2022

View reviewed changes

PR feedback on the blog post.

8e1ec9e

Signed-off-by: David Venable <[email protected]>

PR feedback to the Directory structure section.

39fb094

Signed-off-by: David Venable <[email protected]>

Naarcha-AWS suggested changes Oct 11, 2022

View reviewed changes

oeyh reviewed Oct 11, 2022

View reviewed changes

Applied suggestions from review.

3bc00b6

Signed-off-by: David Venable <[email protected]>

dlvenable force-pushed the data-prepper-2.0.0-blog-post branch from 509bfed to 3bc00b6 Compare October 11, 2022 19:12

Other corrections from the review.

b57dd0b

Signed-off-by: David Venable <[email protected]>

natebower reviewed Oct 12, 2022

View reviewed changes

Applied suggestions from review.

0f160f1

Signed-off-by: David Venable <[email protected]>

dlvenable force-pushed the data-prepper-2.0.0-blog-post branch from 4fd8501 to 0f160f1 Compare October 12, 2022 18:18

Other minor tweaks from review that were not auto-accepted.

faa17e2

Signed-off-by: David Venable <[email protected]>

dlvenable commented Oct 12, 2022

View reviewed changes

dlvenable added 2 commits October 12, 2022 14:12

Added a call to action at the end.

87cb277

Signed-off-by: David Venable <[email protected]>

Removed "all" per recommendation.

64fb607

Signed-off-by: David Venable <[email protected]>

natebower reviewed Oct 12, 2022

View reviewed changes

natebower previously approved these changes Oct 12, 2022

View reviewed changes

Exclamation point!

c65831f

Signed-off-by: David Venable <[email protected]>

dlvenable dismissed natebower’s stale review via c65831f October 12, 2022 21:08

krisfreedain approved these changes Oct 12, 2022

View reviewed changes

krisfreedain merged commit b535e82 into opensearch-project:main Oct 12, 2022

krisfreedain mentioned this pull request Oct 12, 2022

Data Prepper 2.0.0 blog post and new author bio #1075

Merged

dlvenable deleted the data-prepper-2.0.0-blog-post branch July 12, 2023 17:18

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Created the blog post announcing Data Prepper 2.0 #1066

Created the blog post announcing Data Prepper 2.0 #1066

dlvenable commented Oct 7, 2022 •

edited

Loading

dlvenable commented Oct 8, 2022

Naarcha-AWS left a comment

Naarcha-AWS Oct 11, 2022

Naarcha-AWS Oct 11, 2022

Naarcha-AWS Oct 11, 2022

dlvenable Oct 11, 2022

Naarcha-AWS Oct 11, 2022

Naarcha-AWS Oct 11, 2022

dlvenable commented Oct 11, 2022

Naarcha-AWS left a comment

Naarcha-AWS Oct 11, 2022

dlvenable Oct 11, 2022

oeyh left a comment

oeyh Oct 11, 2022 •

edited

Loading

dlvenable Oct 11, 2022

dlvenable Oct 11, 2022

dlvenable commented Oct 11, 2022

natebower left a comment

natebower Oct 12, 2022

natebower Oct 12, 2022

natebower Oct 12, 2022

natebower Oct 12, 2022

dlvenable Oct 12, 2022

natebower Oct 12, 2022

natebower Oct 12, 2022

dlvenable left a comment

dlvenable Oct 12, 2022

dlvenable Oct 12, 2022

natebower commented Oct 12, 2022

dlvenable commented Oct 12, 2022

dlvenable commented Oct 12, 2022

natebower Oct 12, 2022

natebower left a comment

krisfreedain left a comment

		Simply pick a name appropriate for the domain and a Data Prepper expression.
		Then for any sink that should only have some data coming through, define one or more routes to apply Data Prepper will evaluate

	* The HTTP source now supports loading TLS/SSL credentials from either Amazon S3 or Amazon Certificate Manager. Pipeline authors can now configure them for their log ingestion use cases. Before Data Prepper 2.0, only the OTel Trace Source supported these options.
	* The HTTP source now supports loading SSL/TLS credentials from either Amazon S3 or AWS Certificate Manager (ACM). Pipeline authors can now configure them for their log ingestion use cases. Before Data Prepper 2.0, only the OTel Trace Source supported these options.


		## Try Data Prepper 2.0

		Data Prepper 2.0 is available for [download](https://opensearch.org/downloads.html#data-prepper) now. The maintainers encourage you to

Created the blog post announcing Data Prepper 2.0 #1066

Created the blog post announcing Data Prepper 2.0 #1066

Conversation

dlvenable commented Oct 7, 2022 • edited Loading

Description

Issues Resolved

Check List

dlvenable commented Oct 8, 2022

Naarcha-AWS left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

dlvenable commented Oct 11, 2022

Naarcha-AWS left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

oeyh left a comment

Choose a reason for hiding this comment

oeyh Oct 11, 2022 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

dlvenable commented Oct 11, 2022

natebower left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

dlvenable left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

natebower commented Oct 12, 2022

dlvenable commented Oct 12, 2022

dlvenable commented Oct 12, 2022

Choose a reason for hiding this comment

natebower left a comment

Choose a reason for hiding this comment

krisfreedain left a comment

Choose a reason for hiding this comment

dlvenable commented Oct 7, 2022 •

edited

Loading

oeyh Oct 11, 2022 •

edited

Loading