Skip to content

Latest commit

 

History

History
1842 lines (1086 loc) · 66.9 KB

data-prepper.change-log-2.6.0.md

File metadata and controls

1842 lines (1086 loc) · 66.9 KB
  • Release notes for Data Prepper 2.6.0 (#3710) (#3712)

    opensearch-trigger-bot[bot] - Tue, 28 Nov 2023 08:59:32 -0800

    Adds the release notes for Data Prepper 2.6.0. Signed-off-by: David Venable <[email protected]> (cherry picked from commit 2cb172466de9a0a02b43928c82480138f1b02b37) Co-authored-by: David Venable <[email protected]>

  • Fix bug so GLOBAL read-only items do not expire from TTL in ddb source coordination store (#3703) (#3711)

    opensearch-trigger-bot[bot] - Tue, 28 Nov 2023 10:20:50 -0600

    Fix bug so GLOBAL read-only items do not expire from TTL in ddb source coordination store Signed-off-by: Taylor Gray <[email protected]> (cherry picked from commit c8548a0962c2396f111e60a32b0f471a0d424f30) Co-authored-by: Taylor Gray <[email protected]>

  • Generated THIRD-PARTY file for 250e1a0 (#3707)

    opensearch-trigger-bot[bot] - Mon, 27 Nov 2023 15:48:27 -0800

    Signed-off-by: GitHub <[email protected]> Co-authored-by: dlvenable <[email protected]>

  • Gradle parallel max (#3700) (#3706)

    opensearch-trigger-bot[bot] - Mon, 27 Nov 2023 15:47:57 -0800

    Set the maximum workers to 2 when running the GHA build and release tasks. Signed-off-by: David Venable <[email protected]> (cherry picked from commit 6878f56f4add30520448211352d0965744a96812) Co-authored-by: David Venable <[email protected]>

  • Check if failedDeleteCount is positive before logging (#3686) (#3705)

    opensearch-trigger-bot[bot] - Mon, 27 Nov 2023 16:01:30 -0600

    Signed-off-by: Hai Yan <[email protected]> (cherry picked from commit 6dc1d12a4b84ade389d7cc311799363e3ea3114d) Co-authored-by: Hai Yan <[email protected]>

  • Update Data Prepper version to 2.6.0. (#3697)

    David Venable - Mon, 27 Nov 2023 13:25:03 -0800

    Signed-off-by: David Venable <[email protected]>

  • Remove projects which are not ready and not releasing with 2.6.0. (#3702)

    David Venable - Mon, 27 Nov 2023 13:21:21 -0800

    Signed-off-by: David Venable <[email protected]>

  • Generated THIRD-PARTY file for c88c27f (#3701)

    opensearch-trigger-bot[bot] - Mon, 27 Nov 2023 13:11:01 -0800

    Signed-off-by: GitHub <[email protected]> Co-authored-by: dlvenable <[email protected]>

  • Require Apache Avro 1.11.3 to fix CVE-2023-39410. Resolves #3430. (#3695)

    David Venable - Mon, 27 Nov 2023 10:58:50 -0800

    Signed-off-by: David Venable <[email protected]>

  • Updates werkzeug to 3.0.1 which fixes CVE-2023-46136. This required updating to dash 2.14.1 as 2.13 does not support newer versions of werkzeug. Resolves #3552. (#3690)

    David Venable - Mon, 27 Nov 2023 08:52:29 -0800

    efs/heads/data-prepper-2.6.0-thank-you Signed-off-by: David Venable <[email protected]>

  • Fix S3 scan failing tests (#3693)

    Asif Sohail Mohammed - Mon, 27 Nov 2023 08:49:02 -0800

    Signed-off-by: Asif Sohail Mohammed <[email protected]>

  • Updates the opensearch-java client to 2.8.1 and opensearch to 1.3.13. This includes a transitive dependency update to parsson to resolve CVE-2023-4043. (#3689)

    David Venable - Mon, 27 Nov 2023 08:45:36 -0800

    Update required version of org.json library to resolve CVE-2023-5072. Require a Zookeeper version which resolves CVE-2023-44981. Require a transitive Scala library to resolve CVE-2023-46122. Resolves #3588, #3522, #3491, #3547 Signed-off-by: David Venable <[email protected]>

  • TST: validate special data in opensearch sink (#3685)

    Qi Chen - Mon, 27 Nov 2023 10:02:33 -0600

    • TST: validate special data in opensearch sink Signed-off-by: George Chen <[email protected]>
  • Fix crash in Kafka consumer when negative acknowledments are received (#3691)

    kkondaka - Mon, 27 Nov 2023 08:00:42 -0800

    Signed-off-by: Krishna Kondaka <[email protected]> Co-authored-by: Krishna Kondaka <[email protected]>

  • Select require_alias for OS bulk inserts from ISM Policy (#3560)

    Karsten Schnitter - Tue, 21 Nov 2023 14:48:05 -0800

    • Select require_alias for OS bulk inserts from ISM Policy This change requires an alias when writing to an aliased index. This avoids creation of an index without alias, when a previous existing alias and index was deleted. It increases robustness of DataPrepper's trace index against OS user interactions. Signed-off-by: Karsten Schnitter <[email protected]>

    • 3342 Determine Alias Configuration from OS During OS sink initialization it is determined from OS, whether the

    configured index actually is an alias. If so, bulk request will require the index to always be an alias. The response is cached to avoid further requests. This also ensures, that the alias configuration is kept in the initially intended state. After all, this change is about to prevent an automatic index creation for a formerly existing alias. Signed-off-by: Karsten Schnitter <[email protected]>

    • Fix imports for checkstyle Signed-off-by: Karsten Schnitter <[email protected]>

    • Fix integration tests The specific user used in some tests of OpenSerachSinkIT needs get permissions on all aliases to test for their existence. Another bug with determining the alias name is fixed as well. As a final result, the DataPrepper OpenSearch user requires write access to the indices and now additionally read access to the aliases. This can be a change for self-managed indices. Signed-off-by: Karsten Schnitter <[email protected]>

    • Fix Bulk Requests for older OD versions The require_alias parameter for bulk requests was only introduced with ES 7.10. Since DataPrepper needs to be compatible down to 6.8, the parameter should not be used in earlier OD versions. This change will apply the parameter only when OpenSearch is detected as target. Signed-off-by: Karsten Schnitter <[email protected]>

    • Add Permission to get Cluster Info For checking the OS version, the test user needs an additional permission. Signed-off-by: Karsten Schnitter <[email protected]>


    Signed-off-by: Karsten Schnitter <[email protected]>

  • Use Amazon Linux 2023 as the base image for the Data Prepper Docker image. This install Temurin for the Amazon Linux 2 distribution. Resolves #3505. (#3671)

    David Venable - Fri, 17 Nov 2023 12:34:51 -0800

    Signed-off-by: David Venable <[email protected]>

  • E2E: aws secrets tests (#3654)

    Qi Chen - Fri, 17 Nov 2023 10:34:09 -0600

  • ShardId is not passed to ShardConsumer, resulting in logs saying shard is null on shutdown (#3683)

    Taylor Gray - Thu, 16 Nov 2023 17:32:55 -0600

    Signed-off-by: Taylor Gray <[email protected]>

  • Improve logging message for no shards found to indicate that export m… (#3681)

    Taylor Gray - Thu, 16 Nov 2023 16:59:00 -0600

    Improve logging message for no shards found to indicate that export may still be ongoing Signed-off-by: Taylor Gray <[email protected]>

  • Add ddb source fixes/improvements (#3676)

    Taylor Gray - Thu, 16 Nov 2023 10:52:55 -0600

    Add ddb source fixes/improvements Signed-off-by: Taylor Gray <[email protected]>

  • Fix an NPE when the DynamoDB LeaderScheduler does not receive a leader partition. To help test this, I also allowed for a smaller lease interval in a package protected constructor. (#3672)

    David Venable - Thu, 16 Nov 2023 06:37:47 -0800

    Signed-off-by: David Venable <[email protected]>

  • Improves the DefaultPluginFactory class design by creating a new class for providing arguments from the Application Context to plugin parameters. (#3615)

    David Venable - Wed, 15 Nov 2023 11:02:06 -0800

    Signed-off-by: David Venable <[email protected]>

  • Uses export time minus 5 minutes for export document version (#3668)

    Taylor Gray - Wed, 15 Nov 2023 09:06:28 -0800

    Uses export time minus 5 minutes for export document version Signed-off-by: Taylor Gray <[email protected]>

  • ADD: initial AWS testing resources CDK (#3501)

    Qi Chen - Wed, 15 Nov 2023 10:46:14 -0600

  • Correct single quote escape character in DynamoDB [#3664] (#3667)

    David Venable - Wed, 15 Nov 2023 08:02:31 -0800

    Resolves a bug with escaped single quotes in the DynamoDB source by updating the AWS SDK to 2.21.23. Also, skip data that cannot be parsed entirely rather than silently send empty data. Resolves #3664. Signed-off-by: David Venable <[email protected]>

  • Fix leader partition time out issue due to exception (#3666)

    Aiden Dai - Wed, 15 Nov 2023 09:42:09 -0600

    Signed-off-by: Aiden Dai <[email protected]>

  • Update gRPC and HTTP logging (#3658)

    Asif Sohail Mohammed - Tue, 14 Nov 2023 17:26:11 -0800

    Signed-off-by: Asif Sohail Mohammed <[email protected]>

  • Undo rename change done in PR 3656 (#3661)

    kkondaka - Tue, 14 Nov 2023 17:10:33 -0800

    Signed-off-by: Krishna Kondaka <[email protected]> Co-authored-by: Krishna Kondaka <[email protected]>

  • Do not send empty lists to the DLQ when all items share the same retryable failure. Resolves #3644 (#3660)

    David Venable - Tue, 14 Nov 2023 16:58:55 -0800

    Signed-off-by: David Venable <[email protected]>

  • Adds integration tests to KafkaBufferIT to verify that raw bytes are wrapped in the Protobuf Kafka message wrapper. Adds a missing validation when reading data after a writeBytes call and some other minor test improvements. (#3645)

    David Venable - Tue, 14 Nov 2023 16:24:23 -0800

    Signed-off-by: David Venable <[email protected]>

  • Catch failure to change visibility timeout and maintain a counter (#3657)

    kkondaka - Tue, 14 Nov 2023 16:22:43 -0800

    Catch failure to change visibility timeout and maintain a counter Signed-off-by: Krishna Kondaka <[email protected]> Co-authored-by: Krishna Kondaka <[email protected]>

  • Adds a configuration to the random string source to configure the wait delay between writes to the buffer. Resolves #3595. Also uses a single thread for this source to avoid an unnecessary thread pool and increases the code coverage. (#3602)

    David Venable - Tue, 14 Nov 2023 14:53:29 -0800

    Signed-off-by: David Venable <[email protected]>

  • Change latency metric names (#3656)

    kkondaka - Tue, 14 Nov 2023 14:05:31 -0800

    Signed-off-by: Krishna Kondaka <[email protected]> Co-authored-by: Krishna Kondaka <[email protected]>

  • Go back to processing only 1 data node file at a time instead of 3 (#3652)

    Taylor Gray - Tue, 14 Nov 2023 11:04:47 -0600

    Signed-off-by: Taylor Gray <[email protected]>

  • Convert Number types to BigDecimal plainString for consistency between partition and sort keys for export and streams (#3650)

    Taylor Gray - Tue, 14 Nov 2023 09:51:39 -0600

    Signed-off-by: Taylor Gray <[email protected]>

  • Fix no acknowledgments for closed shard issue (#3651)

    Aiden Dai - Tue, 14 Nov 2023 09:51:10 -0600

    Signed-off-by: Aiden Dai <[email protected]>

  • Include the full exception and stack trace when an unexpected error occurs in the LeaderScheduler. (#3648)

    David Venable - Mon, 13 Nov 2023 16:48:11 -0800

    Signed-off-by: David Venable <[email protected]>

  • MAINT: add bytes metrics into opensearch source (#3646)

    Qi Chen - Mon, 13 Nov 2023 16:54:23 -0600

  • MAINT: add bytes metrics into dynamo source (#3647)

    Qi Chen - Mon, 13 Nov 2023 16:54:06 -0600

    Signed-off-by: George Chen <[email protected]>

  • Fix initialization issue in DynamoDB source (#3643)

    Aiden Dai - Mon, 13 Nov 2023 11:29:23 -0800

    Signed-off-by: Aiden Dai <[email protected]>

  • Use a Protobuf buffer message for data in the Kafka buffer (#3635)

    David Venable - Mon, 13 Nov 2023 11:12:32 -0800

    Adds a Protobuf buffer message for the Kafka buffer. Data sent to the topic is wrapped in this and then parsed back into this. Contributes toward #3620. Correct the Kafka buffer tests to test correctly as bytes, adds bytes tests, fixes some serialization issues with the Kafka buffer. Signed-off-by: David Venable <[email protected]>

  • Refactor to use Admin client instead of second set of consumers for empty check (#3637)

    Chase - Sun, 12 Nov 2023 12:20:03 -0600

    • Refactor to use Admin client instead of second set of consumers for empty check Signed-off-by: Chase Engelbrecht <[email protected]>

    • Remove debug log Signed-off-by: Chase Engelbrecht <[email protected]>


    Signed-off-by: Chase Engelbrecht <[email protected]>

  • Pass custom metric prefix if present to AbstractBuffer when using KafkaBuffer (#3638)

    Dinu John - Sun, 12 Nov 2023 11:31:16 -0600

    Signed-off-by: Dinu John <[email protected]>

  • Improvements to prevent data loss in DynamoDB source (#3614)

    Aiden Dai - Sun, 12 Nov 2023 11:20:58 -0600

    Signed-off-by: Aiden Dai <[email protected]>

  • Remove verbose debug log from JacksonEvent (#3639)

    Taylor Gray - Sun, 12 Nov 2023 11:14:04 -0600

    Signed-off-by: Taylor Gray <[email protected]>

  • Add distribution_version flag to opensearch source (#3636)

    Taylor Gray - Sun, 12 Nov 2023 10:42:27 -0600

    Signed-off-by: Taylor Gray <[email protected]>

  • Add normalize_index flag to normalize invalid dynamic indices (#3634)

    Taylor Gray - Sat, 11 Nov 2023 15:54:29 -0600

    Signed-off-by: Taylor Gray <[email protected]>

  • Bump com.fasterxml.jackson.datatype:jackson-datatype-jdk8 (#3570)

    dependabot[bot] - Fri, 10 Nov 2023 16:43:31 -0800

    Bumps com.fasterxml.jackson.datatype:jackson-datatype-jdk8 from 2.15.2 to 2.15.3.


    updated-dependencies:

    • dependency-name: com.fasterxml.jackson.datatype:jackson-datatype-jdk8 dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
  • Bump net.bytebuddy:byte-buddy in /data-prepper-plugins/opensearch (#3571)

    dependabot[bot] - Fri, 10 Nov 2023 16:42:56 -0800

    Bumps net.bytebuddy:byte-buddy from 1.14.8 to 1.14.9.

    Commits


    updated-dependencies:

    • dependency-name: net.bytebuddy:byte-buddy dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
  • Create network policy for aoss source. (#3613)

    Adi Suresh - Fri, 10 Nov 2023 14:15:49 -0800

    Signed-off-by: Adi Suresh <[email protected]>

  • Add support OTEL traces and logs with Kafka buffer (#3625)

    kkondaka - Fri, 10 Nov 2023 14:13:26 -0800


    Signed-off-by: Krishna Kondaka <[email protected]> Co-authored-by: Krishna Kondaka <[email protected]>

  • Retry on dynamic index creation when an OpenSearchException is thrown (#3541)

    Chase - Fri, 10 Nov 2023 15:53:30 -0600

    • Retry on dynamic index creation when an OpenSearchException is thrown Signed-off-by: Chase Engelbrecht <[email protected]>

    • Use correct exception type in unit tests Signed-off-by: Chase Engelbrecht <[email protected]>

    • Remove older cache imports Signed-off-by: Chase Engelbrecht <[email protected]>


    Signed-off-by: Chase Engelbrecht <[email protected]> Signed-off-by: Chase <[email protected]>

  • Start unit tests for the OpenSearch sink testing the document_version error cases (#3599)

    Taylor Gray - Fri, 10 Nov 2023 13:27:29 -0600

    Signed-off-by: Taylor Gray <[email protected]>

  • Attempting to fix the flaky tests in AwsCloudMapPeerListProviderTest. This uses a higher wait for changes, and refreshes at sub-second intervals for testing. (#3628)

    David Venable - Fri, 10 Nov 2023 11:15:41 -0800

    Signed-off-by: David Venable <[email protected]>

  • Add metric for shards actively being processed, lower ownership timeo… (#3629)

    Taylor Gray - Fri, 10 Nov 2023 12:56:00 -0600

    Add metric for shards actively being processed, lower ownership timeout from 10 minutes to 5 minutes for ddb source Signed-off-by: Taylor Gray <[email protected]>

  • Updated the Router_ThreeRoutesIT test with a longer timeout. Also, use Awaitility's during() method to verify that certain data never reaches a sink that it never should reach. (#3624)

    David Venable - Fri, 10 Nov 2023 09:57:00 -0800

    Signed-off-by: David Venable <[email protected]>

  • Disable the circuit breaker for buffers that write data off-heap only… (#3619)

    David Venable - Fri, 10 Nov 2023 08:45:05 -0800

    Disable the circuit breaker for buffers that write data off-heap only. This is currently only the Kafka buffer. Resolves #3616 Signed-off-by: David Venable <[email protected]>

  • Add splitExportTraceServiceRequest API to OTelProtoDecoder (#3600)

    kkondaka - Thu, 9 Nov 2023 21:09:45 -0800


    Signed-off-by: Krishna Kondaka <[email protected]> Co-authored-by: Krishna Kondaka <[email protected]>

  • Add support for OTEL metrics source to use Kafka buffer (#3539)

    kkondaka - Thu, 9 Nov 2023 15:32:41 -0800

    • Add support for OTEL metrics source to use Kafka buffer Signed-off-by: Krishna Kondaka <[email protected]>

    • Added tests and fixed test failures Signed-off-by: Krishna Kondaka <[email protected]>


    Signed-off-by: Krishna Kondaka <[email protected]> Co-authored-by: Krishna Kondaka <[email protected]>

  • Add Java 11/17/21 matrix for build, test and performance test checks (#3622)

    Andriy Redko - Thu, 9 Nov 2023 15:21:03 -0800

    Signed-off-by: Andriy Redko <[email protected]>

  • Fix race condition in DefaultEventHandle (#3618)

    kkondaka - Thu, 9 Nov 2023 10:09:20 -0800

    Signed-off-by: Krishna Kondaka <[email protected]> Co-authored-by: Krishna Kondaka <[email protected]>

  • Add Kafka Producer metrics for send record failures (#3611)

    Dinu John - Wed, 8 Nov 2023 12:46:35 -0800

    Signed-off-by: Dinu John <[email protected]>

  • Fix issue where we start from checkpoint for PIT with acks to instead start from beginning (#3610)

    Taylor Gray - Wed, 8 Nov 2023 12:08:42 -0800

    Signed-off-by: Taylor Gray <[email protected]>

  • Pause reading from Kafka in the Kafka buffer when the circuit breaker is open (#3595)

    David Venable - Wed, 8 Nov 2023 11:21:20 -0800

    Use the CircuitBreaker in the Kafka buffer to stop reading data from the Kafka topic and putting it into the in-memory buffer. Moves the CircuitBreaker class into data-prepper-api. Adds a DelegatingBuffer class to data-prepper-api. Resolves #3578. Signed-off-by: David Venable <[email protected]>

  • Move opensearch source into same module as opensearch sink. (#3607)

    Adi Suresh - Wed, 8 Nov 2023 10:07:27 -0800

    This change is required to share code between the source and sink plugins. Signed-off-by: Adi Suresh <[email protected]>

  • Updated visibility protection timeout (#3608)

    Asif Sohail Mohammed - Wed, 8 Nov 2023 09:40:42 -0800

    Signed-off-by: Asif Sohail Mohammed <[email protected]>

  • Check topic for emptiness during KafkaBuffer shutdown (#3545)

    Chase - Wed, 8 Nov 2023 11:13:45 -0600

    • Add shutdown method to buffer API Signed-off-by: Chase Engelbrecht <[email protected]>

    • Add unit tests Signed-off-by: Chase Engelbrecht <[email protected]>

    • Only allow single thread to check emptiness Signed-off-by: Chase Engelbrecht <[email protected]>

    • Fix logic and add 1 minute wait before requerying kafka Signed-off-by: Chase Engelbrecht <[email protected]>

    • Add unit tests for thread safety logic Signed-off-by: Chase Engelbrecht <[email protected]>

    • Refactor metadata related to emptiness into own class Signed-off-by: Chase Engelbrecht <[email protected]>

    • Single instance per topic rather than per worker Signed-off-by: Chase Engelbrecht <[email protected]>

    • Add unit tests for topic emptiness class Signed-off-by: Chase Engelbrecht <[email protected]>

    • Resolve rebase conflicts Signed-off-by: Chase Engelbrecht <[email protected]>


    Signed-off-by: Chase Engelbrecht <[email protected]>

  • Some changes to the configurations for the new visibility timeout extension feature. Increase the maximum default to 2 hours, increase the maximum configurable extension to 24 hours, and rename to use the work "maximum" to remain consistent (e.g. maximum_messages). (#3604)

    David Venable - Wed, 8 Nov 2023 08:45:39 -0800

    Signed-off-by: David Venable <[email protected]>

  • Change Latency Metric names (#3603)

    kkondaka - Wed, 8 Nov 2023 09:51:55 -0600

    Signed-off-by: Krishna Kondaka <[email protected]> Co-authored-by: Krishna Kondaka <[email protected]>

  • Support ISM correctly when using composable index templates (#3590)

    David Venable - Wed, 8 Nov 2023 06:38:49 -0800

    Correctly support custom properties in composable index templates in the OpenSearch sink. This resolves #3506. Signed-off-by: David Venable <[email protected]>

  • Add Kafka Producer Metrics and Kafka Buffer Metrics (#3598)

    Dinu John - Tue, 7 Nov 2023 22:45:19 -0600

    Signed-off-by: Dinu John <[email protected]>

  • Add dynamodb_item_version metadata that is derived from timestamp for… (#3596)

    Taylor Gray - Tue, 7 Nov 2023 19:16:16 -0600

    Add dynamodb_item_version metadata that is derived from timestamp for stream events Signed-off-by: Taylor Gray <[email protected]>

  • Add document_version and document_version_type parameters to the open… (#3591)

    Taylor Gray - Tue, 7 Nov 2023 18:38:46 -0600

    Add document_version and document_version_type parameters to the opensearch sink for conditional indexing of documents Signed-off-by: Taylor Gray <[email protected]>

  • Optimize idle item between GetRecords call (#3593)

    Aiden Dai - Tue, 7 Nov 2023 17:55:18 -0600

    Signed-off-by: Aiden Dai <[email protected]>

  • Add CMK encryption support to DynamoDB export (#3592)

    Aiden Dai - Tue, 7 Nov 2023 16:26:24 -0600

    Signed-off-by: Aiden Dai <[email protected]>

  • Javadoc fixes (#3594)

    kkondaka - Tue, 7 Nov 2023 12:18:43 -0800

    Signed-off-by: Krishna Kondaka <[email protected]> Co-authored-by: Krishna Kondaka <[email protected]>

  • Add Internal and external latency to OpenSearch and S3 sinks. (#3583)

    kkondaka - Tue, 7 Nov 2023 09:42:58 -0800

    Add Internal and external latency to OpenSearch and S3 sinks Signed-off-by: Krishna Kondaka <[email protected]> Co-authored-by: Krishna Kondaka <[email protected]>

  • Enable acknowledgements for KafkaBuffer plugin (#3584)

    Hardeep Singh - Mon, 6 Nov 2023 09:57:04 -0800

    Signed-off-by: Hardeep <[email protected]>

  • Add dynamodb_event_name metadata attribute, change mapping for Ddb INSERT and MODIFY to be index bulk action (#3585)

    Taylor Gray - Sun, 5 Nov 2023 10:49:05 -0600

    Signed-off-by: Taylor Gray <[email protected]>

  • Change s3 scan and opensearch to only save state every 5 minutes, fix… (#3581)

    Taylor Gray - Sun, 5 Nov 2023 10:31:01 -0600

    Change s3 scan and opensearch to only save state every 5 minutes, fix bug where any action was valid in OpenSearch sink Signed-off-by: Taylor Gray <[email protected]>

  • Replace Guava Cache with Caffeine for OpenSearch integration (#3586)

    Roman Kvasnytskyi - Sat, 4 Nov 2023 13:24:12 -0700

    Signed-off-by: Roman Kvasnytskyi <[email protected]>

  • Add ProgressCheck callbacks to end-to-end acknowledgements (#3565)

    kkondaka - Sat, 4 Nov 2023 08:04:22 -0700

    Add ProgressCheck callbacks to end-to-end acknowledgements Signed-off-by: Krishna Kondaka <[email protected]> Co-authored-by: Krishna Kondaka <[email protected]>

  • Update end-to-end tests to use the release Docker image or a custom image with a specific Java version. Resolves #3566 (#3576)

    David Venable - Fri, 3 Nov 2023 11:26:53 -0700

    Signed-off-by: David Venable <[email protected]>

  • Add acknowledgments for the ddb source (#3575)

    Taylor Gray - Thu, 2 Nov 2023 11:14:41 -0500

    Signed-off-by: Taylor Gray <[email protected]>

  • Modify EventHandle to be created for every event and support internal and external origination times (#3546)

    kkondaka - Wed, 1 Nov 2023 13:52:58 -0700


    Signed-off-by: Krishna Kondaka <[email protected]> Co-authored-by: Krishna Kondaka <[email protected]>

  • Bump org.apache.logging.log4j:log4j-bom in /data-prepper-expression (#3569)

    dependabot[bot] - Wed, 1 Nov 2023 11:46:48 -0700

    Bumps org.apache.logging.log4j:log4j-bom from 2.20.0 to 2.21.1.

    Commits


    updated-dependencies:

    • dependency-name: org.apache.logging.log4j:log4j-bom dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
  • Bump org.apache.logging.log4j:log4j-jpl in /data-prepper-core (#3574)

    dependabot[bot] - Wed, 1 Nov 2023 10:02:27 -0700

    Bumps org.apache.logging.log4j:log4j-jpl from 2.20.0 to 2.21.1.


    updated-dependencies:

    • dependency-name: org.apache.logging.log4j:log4j-jpl dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
  • Bump org.apache.logging.log4j:log4j-bom in /data-prepper-core (#3573)

    dependabot[bot] - Wed, 1 Nov 2023 10:01:53 -0700

    Bumps org.apache.logging.log4j:log4j-bom from 2.20.0 to 2.21.1.

    Commits


    updated-dependencies:

    • dependency-name: org.apache.logging.log4j:log4j-bom dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
  • OpenSearch Sink: Add log messages when there is no exception (#3532)

    kkondaka - Tue, 31 Oct 2023 13:23:55 -0700

    Add log messages when there is exception Signed-off-by: Kondaka <[email protected]>

  • Add bug fixes and improvements to DDB source (#3559)

    Aiden Dai - Tue, 31 Oct 2023 14:48:32 -0500

    Signed-off-by: Aiden Dai <[email protected]>

  • Updates Kafka configurations such that plugin has its own topic configurations (#3551)

    David Venable - Tue, 31 Oct 2023 10:20:09 -0700

    Updates Kafka buffer configurations to disallow configurations which are not needed - e.g. schema and the serde_format for the topic configuration. As part of this change, I also split the TopicConfig into three distinct interfaces and classes. This allows each plugin to either accept a configuration or provide a value of the plugin's own choosing. Also adds copyright headers to all files modified as part of this commit. Renamed is_topic_create to create_topic. Also made this a boolean internally instead of Boolean since it will have a value. Adds a zeroBytes() static method to ByteCount as a convenience. Signed-off-by: David Venable <[email protected]>

  • Fix for Kafka buffer encryption with bytes serde_format by returning null for null input in EncryptionSerializer. (#3556)

    David Venable - Fri, 27 Oct 2023 14:39:02 -0700

    Signed-off-by: David Venable <[email protected]>

  • Catch exceptions and backoff and retry ddb source threads instead of shutting down on exception (#3554)

    Taylor Gray - Fri, 27 Oct 2023 10:37:39 -0500

    Signed-off-by: Taylor Gray <[email protected]>

  • Fix CVE error for Jetty version 11.0.12 (#3548)

    wanghd89 - Thu, 26 Oct 2023 13:12:39 -0500


    Signed-off-by: Haidong <[email protected]> Co-authored-by: Haidong <[email protected]>

  • Implement writeBytes and isByteBuffer in the CircuitBreakingBuffer. Also update the Buffer to throw UnsupportedOperationException when writeBytes is called. (#3553)

    David Venable - Thu, 26 Oct 2023 09:50:44 -0700

    Signed-off-by: David Venable <[email protected]>

  • Add tests to InMemoryBufferTest and LocalFileBufferTest (#3550)

    kkondaka - Wed, 25 Oct 2023 14:34:18 -0700

    Signed-off-by: Krishna Kondaka <[email protected]> Co-authored-by: Krishna Kondaka <[email protected]>

  • Add obfuscate_when parameter and tags_on_match failure to obfuscate processor (#3544)

    Taylor Gray - Wed, 25 Oct 2023 07:52:13 -0700

    Add obfuscate_when parameter to obfuscate processor Signed-off-by: Taylor Gray <[email protected]>

  • HttpSink plugin functionality for #874. (#3065)

    mallikagogoi7 - Tue, 24 Oct 2023 17:10:29 -0700

    HttpSink plugin functionality for #874. Signed-off-by: mallikagogoi7 <[email protected]>

  • Add bug fixes and improvements to DDB source (#3534)

    Aiden Dai - Mon, 23 Oct 2023 18:08:21 -0500

    Signed-off-by: Aiden Dai <[email protected]>

  • Removed the deprecated annotation from Record and RecordMetadata as these are currently still very necessary. Resolves #3536. (#3540)

    David Venable - Mon, 23 Oct 2023 14:52:26 -0700

    Signed-off-by: David Venable <[email protected]>

  • Test against newer versions of OpenSearch and use odd versions in the 2.x series to avoid testing against too many different versions. Updated to the latest 1.3 version. (#3512)

    David Venable - Mon, 23 Oct 2023 14:51:48 -0700

    Signed-off-by: David Venable <[email protected]>

  • Perform a full upgrade on the base Docker image when building the Data Prepper Docker image to get latest patches. (#3497)

    David Venable - Mon, 23 Oct 2023 14:51:22 -0700

    Signed-off-by: David Venable <[email protected]>

  • Run the end-to-end tests on Java 21 in the GitHub Actions. (#3523)

    David Venable - Mon, 23 Oct 2023 14:50:56 -0700

    Signed-off-by: David Venable <[email protected]>

  • Bump @babel/traverse in /release/staging-resources-cdk (#3521)

    dependabot[bot] - Mon, 23 Oct 2023 12:13:44 -0700

    Bumps @babel/traverse from 7.22.5 to 7.23.2.

    Commits


    updated-dependencies:

    • dependency-name: "@babel/traverse" dependency-type: indirect ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
  • Add logic to create or update serverless network policy. (#3510)

    Adi Suresh - Mon, 23 Oct 2023 12:12:59 -0700

    Signed-off-by: Adi Suresh <[email protected]>

  • Bump net.bytebuddy:byte-buddy-agent in /data-prepper-plugins/opensearch (#3527)

    dependabot[bot] - Mon, 23 Oct 2023 06:14:52 -0700

    Bumps net.bytebuddy:byte-buddy-agent from 1.14.8 to 1.14.9.

    Commits


    updated-dependencies:

    • dependency-name: net.bytebuddy:byte-buddy-agent dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
  • Adding support for storing raw bytes in Kafka Buffer (#3519)

    kkondaka - Fri, 20 Oct 2023 16:34:54 -0700

    • Adding support for storing raw bytes in Kafka Buffer Signed-off-by: Krishna Kondaka <[email protected]>

    • Modified to wait for the send() to finish before returning Signed-off-by: Krishna Kondaka <[email protected]>

    • Addressed review comments Signed-off-by: Krishna Kondaka <[email protected]>

    • Removed unused imports Signed-off-by: Krishna Kondaka <[email protected]>

    • Fixed Kafka integration test Signed-off-by: Krishna Kondaka <[email protected]>

    • Fixed json processor check style errors Signed-off-by: Krishna Kondaka <[email protected]>

    • Addressed review comments and added a new test case Signed-off-by: Krishna Kondaka <[email protected]>

    • Addressed review comments and added a new tests Signed-off-by: Krishna Kondaka <[email protected]>


    Signed-off-by: Krishna Kondaka <[email protected]> Co-authored-by: Krishna Kondaka <[email protected]>

  • Reorder formatString expression check for JacksonEvent (#3533)

    Taylor Gray - Fri, 20 Oct 2023 15:50:17 -0500

    Reorder formatString expression check for JacksonEvent Signed-off-by: Taylor Gray <[email protected]>

  • Fix broken DefaultKafkaClusterConfigSupplier get API (#3529)

    kkondaka - Wed, 18 Oct 2023 21:04:08 -0500

    Signed-off-by: Kondaka <[email protected]>

  • Bump net.bytebuddy:byte-buddy in /data-prepper-plugins/opensearch (#3413)

    dependabot[bot] - Wed, 18 Oct 2023 13:49:34 -0700

    Bumps net.bytebuddy:byte-buddy from 1.14.7 to 1.14.8.

    Commits


    updated-dependencies:

    • dependency-name: net.bytebuddy:byte-buddy dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
  • Rebased to latest (#3476)

    kkondaka - Wed, 18 Oct 2023 13:08:45 -0700

    Signed-off-by: Kondaka <[email protected]>

  • Adds new AWS configurations for the KMS encryption. Resolves #3516. (#3517)

    David Venable - Wed, 18 Oct 2023 08:26:20 -0700

    Signed-off-by: David Venable <[email protected]>

  • Bump urllib3 in /examples/trace-analytics-sample-app/sample-app (#3518)

    dependabot[bot] - Wed, 18 Oct 2023 06:17:48 -0700

    Bumps urllib3 from 2.0.6 to 2.0.7.


    updated-dependencies:

    • dependency-name: urllib3 dependency-type: direct:production ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
  • Bump urllib3 in /release/smoke-tests/otel-span-exporter (#3520)

    dependabot[bot] - Wed, 18 Oct 2023 06:17:13 -0700

    Bumps urllib3 from 1.26.17 to 1.26.18.


    updated-dependencies:

    • dependency-name: urllib3 dependency-type: direct:production ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
  • Gradle 8.4 (#3492)

    David Venable - Wed, 18 Oct 2023 05:48:26 -0700

    Signed-off-by: David Venable <[email protected]>

  • Rename/add metrics for ddb source (#3498)

    Taylor Gray - Mon, 16 Oct 2023 14:47:20 -0500

    Signed-off-by: Taylor Gray <[email protected]>

  • Add more verbose logging for the DynamoDb source (#3500)

    Aiden Dai - Mon, 16 Oct 2023 10:41:26 -0500

    Add more verbose logging to the DynamoDB source Signed-off-by: Aiden Dai <[email protected]>

  • Miscellaneous fixes/improvements to the DynamoDb source (#3489)

    Taylor Gray - Fri, 13 Oct 2023 09:50:12 -0500

    Signed-off-by: Taylor Gray <[email protected]>

  • Adds KMS encryption_context for KMS encryption in the Kafka buffer. Moves the kms_key_id into a new kms section along with encryption_context. Resolves #3484 (#3486)

    David Venable - Thu, 12 Oct 2023 11:06:49 -0700

    Signed-off-by: David Venable <[email protected]>

  • Data Prepper 2.5.0 change log (#3488)

    David Venable - Thu, 12 Oct 2023 10:22:07 -0700

    Signed-off-by: David Venable <[email protected]>

  • Integrate CDC data from MySQL/Postgres/MongoDb data source (#3313)

    wanghd89 - Wed, 11 Oct 2023 22:10:36 -0500

    Signed-off-by: Haidong <[email protected]>

  • Move ddb source coordinator config to the data-prepper-config.yaml (#3466)

    Taylor Gray - Wed, 11 Oct 2023 14:40:16 -0500

    Signed-off-by: Taylor Gray <[email protected]>

  • Fix e2e acks test (#3471)

    kkondaka - Wed, 11 Oct 2023 12:21:47 -0700


    Signed-off-by: Kondaka <[email protected]>

  • remove WIP plugins from build (#3480)

    Jonah Calvo - Wed, 11 Oct 2023 13:41:37 -0500

    Signed-off-by: Jonah Calvo <[email protected]>

  • Some updates to the 2.5.0 release notes. (#3479)

    David Venable - Wed, 11 Oct 2023 08:30:39 -0700

    Some updates to the 2.5.0 release notes. Signed-off-by: David Venable <[email protected]>

  • Fix #3459 Unit tests fail on Windows machine (#3461)

    GongYi - Tue, 10 Oct 2023 16:38:59 -0700

    1. SinkModelTest: Use system System.lineSeparator() instead of hardcode '\n'

    2. DataPrepperArgsTest: Covert file path separators to local system.

    DateProcessorTests: Covert time to same timezone before comparing. 4. InMemorySourceCoordinationStoreTest: Use greaterThanOrEqualTo to compare time since they may be same. 5. QueuedPartitionsItemTest: Use sleep to get two different time instances. 6. RSSSourceTest: Use mocker server to avoid internet connecting. 7. ParquetOutputCodecTest: Close all outputStream objects in the tests. 8. org.opensearch.dataprepper.plugins.sink.s3.accumulator.InMemoryBufferTest#getDuration_provides_duration_within_expected_range: No solution to fix. Disable it. Please see my comments in the test file. Signed-off-by: Gong Yi <[email protected]>

  • Resolve Netty to 4.1.100.Final, require Jetty 11.0.17 in Data Prepper. Use Tomcat 10.1.14 in the example project. These changes fix CVE-2023-44487 to protect against HTTP/2 reset floods. Resolves #3474. (#3475)

    David Venable - Tue, 10 Oct 2023 16:33:25 -0700

    Signed-off-by: David Venable <[email protected]>

  • Shutdown kafka buffer (#3464)

    Chase - Tue, 10 Oct 2023 15:49:46 -0500


    Signed-off-by: Chase Engelbrecht <[email protected]>

  • Encrypted and decrypt data in the Kafka buffer (#3468)

    David Venable - Tue, 10 Oct 2023 13:18:03 -0700

    Encrypt and decrypt data in the Kafka buffer when the user configures. Use a KMS key to decrypt the data encryption key, if one is provided. Resolves #3422 Signed-off-by: David Venable <[email protected]>

  • Fix broken build and clean up KafkaSource class. (#3469)

    David Venable - Tue, 10 Oct 2023 07:56:36 -0700

    Signed-off-by: David Venable <[email protected]>

  • Fix MSK integration test fix (#3465)

    kkondaka - Mon, 9 Oct 2023 15:01:35 -0700

    Signed-off-by: Kondaka <[email protected]> Co-authored-by: Kondaka <[email protected]>

  • Refactors the Kafka buffer (and Kafka sink) code related to defining the serialization and deserialization classes. This migrates from using Kafka properties to providing concrete instances into the Kafka consumer/producer. We will make use of this when encrypting or decrypting data. (#3463)

    David Venable - Mon, 9 Oct 2023 14:04:23 -0700

    Signed-off-by: David Venable <[email protected]>

  • Kafka drain timeout (#3454)

    Chase - Mon, 9 Oct 2023 13:40:02 -0500

    • Add getDrainTimeout method to buffer interface. Add as configurable value for kafka buffer Signed-off-by: Chase Engelbrecht <[email protected]>

    • Add unit tests Signed-off-by: Chase Engelbrecht <[email protected]>

    • Move getDrainTimeout to default method in the interface, add test for it, disable SNS sink Signed-off-by: Chase Engelbrecht <[email protected]>

    • Remove verification from non-mock Signed-off-by: Chase Engelbrecht <[email protected]>


    Signed-off-by: Chase Engelbrecht <[email protected]>

  • ENH: support index template for serverless (#3071)

    Qi Chen - Mon, 9 Oct 2023 09:02:19 -0700

    • ENH: support index template for serverless Signed-off-by: George Chen <[email protected]>
  • ENH: support index template for serverless (#3071)

    Qi Chen - Mon, 9 Oct 2023 09:01:47 -0700

    • ENH: support index template for serverless Signed-off-by: George Chen <[email protected]>
  • ENH: opensearch source secrets refreshment suppport (#3437)

    Qi Chen - Mon, 9 Oct 2023 08:19:50 -0700

    ENH: opensearch source secrets refreshment suppport (#3437) Signed-off-by: George Chen <[email protected]>

  • Add release-notes for 2.5.0 (#3449)

    Asif Sohail Mohammed - Fri, 6 Oct 2023 14:13:01 -0700


    Signed-off-by: Asif Sohail Mohammed <[email protected]>

  • Fix CVE-2023-39410 (#3450)

    Asif Sohail Mohammed - Sat, 7 Oct 2023 00:17:50 +0530

    Signed-off-by: Asif Sohail Mohammed <[email protected]>

  • Fixes some issues with the Kafka buffer and sink. Adds an integration test for the Kafka buffer and run that in the GitHub Actions. (#3451)

    David Venable - Fri, 6 Oct 2023 10:55:51 -0700

    Signed-off-by: David Venable <[email protected]>

  • Use Awaitility in the KafkaSourceJsonTypeIT to avoid sleeps. Also consolidates logic for creating and deleting topics. (#3447)

    David Venable - Fri, 6 Oct 2023 09:20:37 -0700

    Signed-off-by: David Venable <[email protected]>

  • Add inline template_content support to the opensearch sink (#3431)

    Taylor Gray - Fri, 6 Oct 2023 11:17:22 -0500

    Signed-off-by: Taylor Gray <[email protected]>

  • Run the Kafka integration tests as a GitHub Action. Created Docker compose files for starting Kafka easily and updated the README.md instructions. (#3445)

    David Venable - Fri, 6 Oct 2023 09:08:41 -0700

    Signed-off-by: David Venable <[email protected]>

  • Removes the GeoIP processor from the build. It doesn't work and the tests are failing consistently now. (#3448)

    David Venable - Fri, 6 Oct 2023 08:19:54 -0700

    Signed-off-by: David Venable <[email protected]>

  • Add support for Update/Upsert/Delete operations in OpenSearch Sink (#3424)

    kkondaka - Thu, 5 Oct 2023 17:54:37 -0700

    • Add support for Update/Upsert/Delete operations in OpenSearch Sink Signed-off-by: Krishna Kondaka <[email protected]>

    • Fixed tests and removed unused imports Signed-off-by: Krishna Kondaka <[email protected]>

    • Updated documentation Signed-off-by: Krishna Kondaka <[email protected]>

    • Added test cases to improve code coverage Signed-off-by: Krishna Kondaka <[email protected]>

    • Addressed review comments Signed-off-by: Krishna Kondaka <[email protected]>

    • Fixed check style errors Signed-off-by: Krishna Kondaka <[email protected]>

    • Added another test for upsert action without prior create action Signed-off-by: Krishna Kondaka <[email protected]>

    • Added check for valid action strings at config time Signed-off-by: Krishna Kondaka <[email protected]>


    Signed-off-by: Krishna Kondaka <[email protected]> Co-authored-by: Krishna Kondaka <[email protected]>

  • Refactor Kafka Source and Sink, implement basic Kafka buffer (#3354)

    Jonah Calvo - Thu, 5 Oct 2023 17:30:58 -0700


    Signed-off-by: Jonah Calvo <[email protected]>

  • Remove support for Enum and Duration values from secrets manager (#3433)

    Asif Sohail Mohammed - Thu, 5 Oct 2023 22:35:58 +0530

    • Remove support for Enum and Duration values from secrets manager Signed-off-by: Asif Sohail Mohammed <[email protected]>

    • Added unit tests Signed-off-by: Asif Sohail Mohammed <[email protected]>


    Signed-off-by: Asif Sohail Mohammed <[email protected]>

  • Bump org.xerial.snappy:snappy-java in /data-prepper-plugins/common (#3411)

    dependabot[bot] - Thu, 5 Oct 2023 09:50:41 -0700

    Bumps org.xerial.snappy:snappy-java from 1.1.10.3 to 1.1.10.5.

    Commits


    updated-dependencies:

    • dependency-name: org.xerial.snappy:snappy-java dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
  • Bump urllib3 in /examples/trace-analytics-sample-app/sample-app (#3425)

    dependabot[bot] - Thu, 5 Oct 2023 09:48:34 -0700

    Bumps urllib3 from 2.0.4 to 2.0.6.


    updated-dependencies:

    • dependency-name: urllib3 dependency-type: direct:production ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
  • ENH: data-prepper-core support for secrets refreshment (#3415)

    Qi Chen - Thu, 5 Oct 2023 09:46:44 -0700


    Signed-off-by: George Chen <[email protected]>

  • Set main version to 2.6 (#3439)

    Asif Sohail Mohammed - Thu, 5 Oct 2023 08:58:35 -0700

    Signed-off-by: Asif Sohail Mohammed <[email protected]>

  • Bump urllib3 in /release/smoke-tests/otel-span-exporter (#3427)

    dependabot[bot] - Tue, 3 Oct 2023 09:55:27 -0700

    Bumps urllib3 from 1.26.7 to 1.26.17.


    updated-dependencies:

    • dependency-name: urllib3 dependency-type: direct:production ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
  • -download task support for geoip (#3373)

    rajeshLovesToCode - Tue, 3 Oct 2023 09:39:32 -0700

  • Add DynamoDB source plugin (#3349)

    Aiden Dai - Tue, 3 Oct 2023 11:31:22 -0500

    Signed-off-by: Aiden Dai <[email protected]>

  • Update release notes and change log for 2.4.1 (#3416)

    Asif Sohail Mohammed - Mon, 2 Oct 2023 09:58:20 -0700

    Signed-off-by: Asif Sohail Mohammed <[email protected]>

  • Fix CVE-2022-45688, CVE-2023-43642 (#3404)

    Asif Sohail Mohammed - Fri, 29 Sep 2023 19:14:59 +0530


    Signed-off-by: Asif Sohail Mohammed <[email protected]>

  • Updated release notes file name (#3403)

    Asif Sohail Mohammed - Fri, 29 Sep 2023 16:20:20 +0530

    Signed-off-by: Asif Sohail Mohammed <[email protected]>

  • Add retry to Kafka Consumer Create in source (#3399)

    Jonah Calvo - Fri, 29 Sep 2023 16:19:57 +0530

    Signed-off-by: Jonah Calvo <[email protected]>