Skip to content

Latest commit

 

History

History
2568 lines (1585 loc) · 93.1 KB

data-prepper.change-log-2.1.0.md

File metadata and controls

2568 lines (1585 loc) · 93.1 KB
Raw

  • Add release notes for 2.1.0 (#2354)

    Hai Yan - Thu, 2 Mar 2023 13:52:16 -0600

    EAD -> refs/heads/change-log-2.1, refs/remotes/upstream/main, refs/remotes/origin/main, refs/remotes/origin/HEAD, refs/heads/main

  • Added backoff for SQS to reduce logging (#2326)

    Asif Sohail Mohammed - Thu, 2 Mar 2023 12:14:20 -0600

  • Removed default service endpoint for otel sources (#2346)

    Asif Sohail Mohammed - Thu, 2 Mar 2023 11:38:54 -0600

    • Removed default service endpoint for otel sources Signed-off-by: Asif Sohail Mohammed <[email protected]>

  • Replace the java.util.* allowed pattern in the ObjectInputFilter with specific classes which are commonly used in Data Prepper. (#2351)

    David Venable - Thu, 2 Mar 2023 09:53:00 -0600

    Signed-off-by: David Venable <[email protected]>

  • Explicitly set the GitHub Actions thumbprint to resolve #2343. Updated the AWS CDK as well. (#2345)

    David Venable - Wed, 1 Mar 2023 15:40:05 -0600

    Signed-off-by: David Venable <[email protected]>

  • Updated version to 2.2 on main (#2342)

    Asif Sohail Mohammed - Wed, 1 Mar 2023 13:27:15 -0600

    Signed-off-by: Asif Sohail Mohammed <[email protected]>

  • Generated THIRD-PARTY file for 062ae95 (#2344)

    opensearch-trigger-bot[bot] - Wed, 1 Mar 2023 13:26:20 -0600

    Signed-off-by: GitHub <[email protected]> Co-authored-by: asifsmohammed <[email protected]>

  • Use an ObjectInputFilter to serialize allow deserialization of only certain objects in peer-to-peer connections. Additionally, it refactors some application configurations to improve integration testing. Fixes #2310. (#2311)

    David Venable - Wed, 1 Mar 2023 10:06:52 -0600

    efs/heads/changelog-2.1 Signed-off-by: David Venable <[email protected]>

  • Bump org.apache.logging.log4j:log4j-bom in /data-prepper-core (#2333)

    dependabot[bot] - Wed, 1 Mar 2023 09:47:42 -0600

    Bumps org.apache.logging.log4j:log4j-bom from 2.19.0 to 2.20.0.

    updated-dependencies:

    • dependency-name: org.apache.logging.log4j:log4j-bom dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

  • Bump org.assertj:assertj-core from 3.21.0 to 3.24.2 (#2331)

    dependabot[bot] - Wed, 1 Mar 2023 09:47:06 -0600

    Bumps org.assertj:assertj-core from 3.21.0 to 3.24.2.

    updated-dependencies:

    • dependency-name: org.assertj:assertj-core dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

  • Bump org.junit.jupiter:junit-jupiter-api from 5.9.0 to 5.9.2 (#2332)

    dependabot[bot] - Wed, 1 Mar 2023 09:46:26 -0600

    Bumps org.junit.jupiter:junit-jupiter-api from 5.9.0 to 5.9.2.

    updated-dependencies:

    • dependency-name: org.junit.jupiter:junit-jupiter-api dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

  • Support for path in OTel sources (#2297)

    Asif Sohail Mohammed - Tue, 28 Feb 2023 19:57:38 -0600

    Initial commit for OTel trace path changes Signed-off-by: Asif Sohail Mohammed <[email protected]> Signed-off-by: Asif Sohail Mohammed <[email protected]>

  • Fix grok processor to not create a new record (#2325)

    kkondaka - Tue, 28 Feb 2023 19:56:04 -0600

    • Fix grok processor to not create a new record Signed-off-by: Krishna Kondaka <[email protected]>

    • Fixed checkStyleMain failure Signed-off-by: Krishna Kondaka <[email protected]>

    Signed-off-by: Krishna Kondaka <[email protected]> Co-authored-by: Krishna Kondaka <[email protected]>

  • Bump org.springframework:spring-context in /data-prepper-expression (#2223)

    dependabot[bot] - Mon, 27 Feb 2023 18:27:04 -0600

    efs/heads/otel-paths Bumps org.springframework:spring-context from 5.3.23 to 5.3.25.

    Commits

    updated-dependencies:

    • dependency-name: org.springframework:spring-context dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

  • Bump org.springframework:spring-context in /data-prepper-core (#2214)

    dependabot[bot] - Mon, 27 Feb 2023 16:52:20 -0600

    Bumps org.springframework:spring-context from 5.3.23 to 5.3.25.

    Commits

    updated-dependencies:

    • dependency-name: org.springframework:spring-context dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

  • Bump io.spring.dependency-management from 1.0.11.RELEASE to 1.1.0 (#2106)

    dependabot[bot] - Mon, 27 Feb 2023 16:48:25 -0600

    Bumps io.spring.dependency-management from 1.0.11.RELEASE to 1.1.0.

    updated-dependencies:

    • dependency-name: io.spring.dependency-management dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

  • Bump com.fasterxml.jackson.datatype:jackson-datatype-jdk8 (#2217)

    dependabot[bot] - Mon, 27 Feb 2023 16:47:36 -0600

    Bumps com.fasterxml.jackson.datatype:jackson-datatype-jdk8 from 2.14.1 to 2.14.2.

    updated-dependencies:

    • dependency-name: com.fasterxml.jackson.datatype:jackson-datatype-jdk8 dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

  • Added path support for HTTP source (#2277)

    Asif Sohail Mohammed - Mon, 27 Feb 2023 15:26:58 -0600

    • Added path support for HTTP source Signed-off-by: Asif Sohail Mohammed <[email protected]>

  • Updated Data Prepper base image for e2e tests (#2269)

    Asif Sohail Mohammed - Mon, 27 Feb 2023 15:08:13 -0600

    Signed-off-by: Asif Sohail Mohammed <[email protected]>

  • Loop until interrupted in SqsWorker (#2306)

    Chase - Fri, 24 Feb 2023 00:10:26 -0600

    Signed-off-by: Chase Engelbrecht <[email protected]>

  • Performance Improvement - Avoid event copying for the first sub-pipeline (#2290)

    kkondaka - Thu, 23 Feb 2023 22:08:47 -0600

    Avoid event copying for the first sub-pipeline Signed-off-by: Krishna Kondaka <[email protected]>

  • Bump slf4j-simple from 1.7.36 to 2.0.6 (#2112)

    dependabot[bot] - Wed, 22 Feb 2023 10:39:21 -0600

    Bumps slf4j-simple from 1.7.36 to 2.0.6.

    updated-dependencies:

    • dependency-name: org.slf4j:slf4j-simple dependency-type: direct:production update-type: version-update:semver-major ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

  • Upgrades opentelemetry dependencies (#2288)

    Shivani Shukla - Tue, 21 Feb 2023 21:16:58 -0600

    Signed-off-by: Shivani Shukla <[email protected]>

  • Fix error message printed when OpenSearch Sink fails to initialize (#2296)

    kkondaka - Tue, 21 Feb 2023 14:38:36 -0600

    Fix error message printed when OpenSearch Sink fails to initialize Signed-off-by: Krishna Kondaka <[email protected]>

    Signed-off-by: Krishna Kondaka <[email protected]> Co-authored-by: Krishna Kondaka <[email protected]>

  • adding backwards compatibility support in versioning check (#2295)

    Christopher Manning - Tue, 21 Feb 2023 14:13:50 -0600

    Signed-off-by: Christopher Manning <[email protected]>

  • [2263] adding versioning property to pipeline yaml configuration (#2292)

    Christopher Manning - Tue, 21 Feb 2023 09:08:39 -0600

    • [2263] adding versioning property to pipeline yaml configuration Signed-off-by: Christopher Manning <[email protected]>

    • adding hashCode support and providing documentation for updating the version Signed-off-by: Christopher Manning <[email protected]>

    Signed-off-by: Christopher Manning <[email protected]>

  • Updates to opensearch-java 2.2.0 (#2287)

    David Venable - Mon, 20 Feb 2023 15:18:55 -0600

    Signed-off-by: David Venable <[email protected]>

  • Replace an unnecessary builder pattern in trace raw processor with a factory method. This is a little simpler code and it reduces an object creation. (#2271)

    David Venable - Fri, 17 Feb 2023 14:34:21 -0600

    Signed-off-by: David Venable <[email protected]>

  • Create DataPrepper server after pipeline initialization (#2284)

    Chase - Fri, 17 Feb 2023 11:12:08 -0600

    • Create DataPrepper HttpServer after pipeline initialization Signed-off-by: Chase Engelbrecht <[email protected]>

    • Remove injection provider implementation Signed-off-by: Chase Engelbrecht <[email protected]>

    Signed-off-by: Chase Engelbrecht <[email protected]>

  • Refactor PeerForwarderReceiveBuffer to extend AbstractBuffer to pick up metric logic (#2286)

    Chase - Fri, 17 Feb 2023 10:48:29 -0600

    Signed-off-by: Chase Engelbrecht <[email protected]>

  • Add more metrics to OTEL metrics source (#2283)

    Chase - Fri, 17 Feb 2023 10:24:30 -0600

    Signed-off-by: Chase Engelbrecht <[email protected]>

  • Make batching queue depth configurable, fill in some missing documentation around batching (#2278)

    Chase - Fri, 17 Feb 2023 10:24:06 -0600

    • Make batching queue depth configurable, fill in some missing documentation around batching Signed-off-by: Chase Engelbrecht <[email protected]>

    • Fix validation message Signed-off-by: Chase Engelbrecht <[email protected]>

    • Actually fix validation message Signed-off-by: Chase Engelbrecht <[email protected]>

    Signed-off-by: Chase Engelbrecht <[email protected]>

  • Update to opensearch 1.3.8. Resolves #2192, #2193 (#2285)

    David Venable - Thu, 16 Feb 2023 15:56:36 -0600

    Signed-off-by: David Venable <[email protected]>

  • OTel Trace Source: Logging and integration tests (#2262)

    David Venable - Thu, 16 Feb 2023 11:12:26 -0600

    Modified some of the logging for the OTel Trace source. Added integration tests which actually verify that gRPC requests operate as expected. Print out SLF4J logs from tests to stdout to help with debugging. Signed-off-by: David Venable <[email protected]>

  • Created untriaged issue workflow. (#2276)

    Daniel (dB.) Doubrovkine - Wed, 15 Feb 2023 13:15:53 -0600

    Signed-off-by: dblock <[email protected]>

  • Long int fix (#2265)

    kkondaka - Tue, 14 Feb 2023 16:13:20 -0600

    • OpenSearchSink should close open files before retrying initialization Signed-off-by: Krishna Kondaka <[email protected]>

    • Minor fixes to data prepper plugins documentation Signed-off-by: Krishna Kondaka <[email protected]>

    • Fix Long Integer comparisons Signed-off-by: Krishna Kondaka <[email protected]>

    • Added more tests for long integer comparisons Signed-off-by: Krishna Kondaka <[email protected]>

    Signed-off-by: Krishna Kondaka <[email protected]> Co-authored-by: Krishna Kondaka <[email protected]>

  • CVE: updated guava version (#2254)

    Asif Sohail Mohammed - Tue, 14 Feb 2023 15:08:18 -0600

    Signed-off-by: Asif Sohail Mohammed <[email protected]> Signed-off-by: Asif Sohail Mohammed <[email protected]>

  • Update to Armeria 1.22.1 which fixes #2206. (#2274)

    David Venable - Tue, 14 Feb 2023 13:36:21 -0600

    Signed-off-by: David Venable <[email protected]>

  • Fix: e2e tests and added documentation (#2267)

    Asif Sohail Mohammed - Mon, 13 Feb 2023 14:40:27 -0600

    Signed-off-by: Asif Sohail Mohammed <[email protected]>

  • Removes some unnecessary methods from ServiceMapRelationship and adds additional tests to verify behavior. (#2200)

    David Venable - Mon, 13 Feb 2023 13:45:31 -0600

    Signed-off-by: David Venable <[email protected]>

  • Created a metric for the overall JVM memory usage - both heap and non-heap. (#2266)

    David Venable - Mon, 13 Feb 2023 13:24:48 -0600

    Signed-off-by: David Venable <[email protected]>

  • Fixes index_type convention (#2261)

    Shivani Shukla - Fri, 10 Feb 2023 14:47:20 -0600

    Signed-off-by: Shivani Shukla <[email protected]>

  • Minor fixes to data prepper plugins documentation (#2260)

    kkondaka - Fri, 10 Feb 2023 13:39:12 -0600

    • OpenSearchSink should close open files before retrying initialization Signed-off-by: Krishna Kondaka <[email protected]>

    • Minor fixes to data prepper plugins documentation Signed-off-by: Krishna Kondaka <[email protected]>

    Signed-off-by: Krishna Kondaka <[email protected]> Co-authored-by: Krishna Kondaka <[email protected]>

  • ENH: peer forwarding codec and model (#2256)

    Qi Chen - Fri, 10 Feb 2023 09:19:51 -0600

    Signed-off-by: George Chen <[email protected]>

  • Use the Gradle version catalog for common software versions to the extent that it can replace the versionMap. (#2253)

    David Venable - Thu, 9 Feb 2023 13:49:11 -0600

    Signed-off-by: David Venable <[email protected]>

  • OpenSearchSink should close open files before retrying initialization (#2255)

    kkondaka - Thu, 9 Feb 2023 11:15:19 -0600

    Signed-off-by: Krishna Kondaka <[email protected]> Co-authored-by: Krishna Kondaka <[email protected]>

  • Added trace peer forwarder doc (#2245)

    Asif Sohail Mohammed - Wed, 8 Feb 2023 18:10:26 -0600

  • Refactor to remove stream in RemotePeerForwarder as micro optimization (#2250)

    Chase - Wed, 8 Feb 2023 18:09:17 -0600

    • Refactor to remove stream in RemotePeerForwarder as micro optimization Signed-off-by: Chase Engelbrecht <[email protected]>

    • Remove unused import Signed-off-by: Chase Engelbrecht <[email protected]>

    Signed-off-by: Chase Engelbrecht <[email protected]>

  • Bump org.assertj:assertj-core (#2221)

    dependabot[bot] - Wed, 8 Feb 2023 16:01:58 -0600

    Bumps org.assertj:assertj-core from 3.23.1 to 3.24.2.

    updated-dependencies:

    • dependency-name: org.assertj:assertj-core dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

  • Bump certifi in /release/smoke-tests/otel-span-exporter (#2063)

    dependabot[bot] - Wed, 8 Feb 2023 15:51:47 -0600

    Bumps certifi from 2021.10.8 to 2022.12.7.

    Commits

    updated-dependencies:

    • dependency-name: certifi dependency-type: direct:production ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

  • Bump com.google.guava:guava in /data-prepper-plugins/aggregate-processor (#2219)

    dependabot[bot] - Wed, 8 Feb 2023 15:50:29 -0600

    Bumps com.google.guava:guava from 10.0.1 to 23.0.

    updated-dependencies:

    • dependency-name: com.google.guava:guava dependency-type: direct:production update-type: version-update:semver-major ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

  • Bump org.assertj:assertj-core (#2226)

    dependabot[bot] - Wed, 8 Feb 2023 15:42:11 -0600

    Bumps org.assertj:assertj-core from 3.23.1 to 3.24.2.

    updated-dependencies:

    • dependency-name: org.assertj:assertj-core dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

  • Bump org.springframework:spring-test in /data-prepper-expression (#2224)

    dependabot[bot] - Wed, 8 Feb 2023 14:53:37 -0600

    Bumps org.springframework:spring-test from 5.3.23 to 5.3.25.

    Commits

    updated-dependencies:

    • dependency-name: org.springframework:spring-test dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

  • Moved Random Cut Forest libraries to the Anomaly Detection processor. (#2252)

    David Venable - Wed, 8 Feb 2023 11:44:14 -0600

    Signed-off-by: David Venable <[email protected]>

  • Sink doInitialize() should be able throw exception to stop execution (#2249)

    kkondaka - Wed, 8 Feb 2023 09:53:30 -0600

    • Sink doInitialize() should be able throw exception to stop execution Signed-off-by: Krishna Kondaka <[email protected]>

    • Sink doInitialize() should be able throw exception to stop execution - fixed code coverage failures Signed-off-by: Krishna Kondaka <[email protected]>

    Signed-off-by: Krishna Kondaka <[email protected]> Co-authored-by: Krishna Kondaka <[email protected]>

  • Consistent AWS Pipeline Configurations - #2184 (#2248)

    kkondaka - Tue, 7 Feb 2023 15:58:59 -0600

    • Consistent AWS Pipeline Configurations - #2184 Signed-off-by: Krishna Kondaka <[email protected]>

    • Consistent AWS Pipeline Configurations - updated documentation Signed-off-by: Krishna Kondaka <[email protected]>

    • Consistent AWS Pipeline Configurations - fixed documentation Signed-off-by: Krishna Kondaka <[email protected]>

    Signed-off-by: Krishna Kondaka <[email protected]> Co-authored-by: Krishna Kondaka <[email protected]>

  • Made some changes to the :release:archives:linux:linuxTar task to reduce dependencies and get a better build time when building multiple times. (#2240)

    David Venable - Tue, 7 Feb 2023 14:30:30 -0600

    Signed-off-by: David Venable <[email protected]>

  • Added SDK client metrics and metric filters (#2232)

    Asif Sohail Mohammed - Tue, 7 Feb 2023 14:04:10 -0600

    • Added SDK client metrics and metric filters Signed-off-by: Asif Sohail Mohammed <[email protected]>

  • Improved how the PipelineConnector copies events by using a new static method to perform the copy. For JacksonEvent, this performs a deepCopy() which appears to be more efficient than the old process. (#2241)

    David Venable - Tue, 7 Feb 2023 10:06:18 -0600

    Signed-off-by: David Venable <[email protected]>

  • Use snake case for all configurations #2203 (#2243)

    kkondaka - Mon, 6 Feb 2023 17:30:14 -0600

    • Use snake case for all configurations #2203 Signed-off-by: Krishna Kondaka <[email protected]>

  • Add back LinkedBlockingQueue::poll to avoid busy wait (#2246)

    Chase - Mon, 6 Feb 2023 14:12:32 -0600

    • Add back LinkedBlockingQueue::poll to avoid busy wait Signed-off-by: Chase Engelbrecht <[email protected]>

    • Drop poll timeout when delay=0 to 5 millis Signed-off-by: Chase Engelbrecht <[email protected]>

    Signed-off-by: Chase Engelbrecht <[email protected]>

  • Use Armeria's BlockingTaskExecutor to configure thread names (#2235)

    David Venable - Mon, 6 Feb 2023 09:32:47 -0600

    Use Armeria's BlockingTaskExecutor to configure thread name Signed-off-by: David Venable <[email protected]>

  • Update default values in buffer documentation (#2233)

    JannikBrand - Sat, 4 Feb 2023 14:04:56 -0600

  • The DefaultEventMetadata almost always creates an empty attributes. This uses the static ImmutableMap.of() which uses a shared instance underneath so that new objects are not created each time. (#2239)

    David Venable - Fri, 3 Feb 2023 13:21:31 -0600

    Signed-off-by: David Venable <[email protected]>

  • The RecordMetadata default metadata is used by most records and it creates a new instance each time. This is an immutable class, so just return the same metadata instance for all. (#2238)

    David Venable - Fri, 3 Feb 2023 12:47:25 -0600

    Signed-off-by: David Venable <[email protected]>

  • Abstract sink should not create a new retry thread everytime initialization fails (#2231)

    kkondaka - Thu, 2 Feb 2023 10:25:41 -0600

    Signed-off-by: Krishna Kondaka <[email protected]> Co-authored-by: Krishna Kondaka <[email protected]>

  • Add Support for retry when Sink fails to initialize (#2198)

    kkondaka - Wed, 1 Feb 2023 16:39:50 -0600

    • Remove opensearch availability dependence - Issue #936 Signed-off-by: Krishna Kondaka <[email protected]>

    • Add support for retry when Sink fails to initialize - issue #936 Signed-off-by: Krishna Kondaka <[email protected]>

    • Add support for retry when Sink fails to initialize - added number of retries to dataprepper execute Signed-off-by: Krishna Kondaka <[email protected]>

    • Add support for retry when Sink fails to initialize - Addressed review comments. Fixed test failures Signed-off-by: Krishna Kondaka <[email protected]>

    • Add support for retry when Sink fails to initialize - Addressed review comments Signed-off-by: Krishna Kondaka <[email protected]>

    • Add support for retry when Sink fails to initialize - Fixes for failing tests

    Signed-off-by: Krishna Kondaka <[email protected]>

    • Add support for retry when Sink fails to initialize - Fixes for failing tests

    Signed-off-by: Krishna Kondaka <[email protected]>

    • Add support for retry when Sink fails to initialize - Fixes for code coverage failures Signed-off-by: Krishna Kondaka <[email protected]>

    • Add support for retry when Sink fails to initialize - Modified to check for retryable exception Signed-off-by: Krishna Kondaka <[email protected]>

    • Add support for retry when Sink fails to initialize - addressed review comments Signed-off-by: Krishna Kondaka <[email protected]>

    Signed-off-by: Krishna Kondaka <[email protected]> Co-authored-by: Krishna Kondaka <[email protected]>

  • DynamicIndexTemplate can cause NPE that shuts down pipeline - Issue #2210 (#2211)

    kkondaka - Wed, 1 Feb 2023 15:15:01 -0600

    • DynamicIndexTemplate can cause NPE that shuts down pipeline - Issue #2210 Signed-off-by: Krishna Kondaka <[email protected]>

    • DynamicIndexTemplate can cause NPE that shuts down pipeline - addressed review comments Signed-off-by: Krishna Kondaka <[email protected]>

    • DynamicIndexTemplate can cause NPE that shuts down pipeline - addressed review comments Signed-off-by: Krishna Kondaka <[email protected]>

    • DynamicIndexTemplate can cause NPE that shuts down pipeline - fixed build failure Signed-off-by: Krishna Kondaka <[email protected]>

    Signed-off-by: Krishna Kondaka <[email protected]> Co-authored-by: Krishna Kondaka <[email protected]>

  • Add missing metrics for Opensearch Sink #2168 (#2205)

    kkondaka - Wed, 1 Feb 2023 15:10:21 -0600

    • Add missing metrics for Opensearch Sink #2168 Signed-off-by: Krishna Kondaka <[email protected]>

    • Add missing metrics for Opensearch Sink -- updated the documentation Signed-off-by: Krishna Kondaka <[email protected]>

    • Add missing metrics for Opensearch Sink -- addressed review comments Signed-off-by: Krishna Kondaka <[email protected]>

    • Add missing metrics for Opensearch Sink -- addressed review comments Signed-off-by: Krishna Kondaka <[email protected]>

    • Add missing metrics for Opensearch Sink -- modified documentation Signed-off-by: Krishna Kondaka <[email protected]>

    • Add missing metrics for Opensearch Sink -- addressed review comments Signed-off-by: Krishna Kondaka <[email protected]>

    • Add missing metrics for Opensearch Sink -- fixed build failure Signed-off-by: Krishna Kondaka <[email protected]>

    Signed-off-by: Krishna Kondaka <[email protected]> Co-authored-by: Krishna Kondaka <[email protected]>

  • Perform serialization sequentially, block for each batch of forwards (#2228)

    Chase - Wed, 1 Feb 2023 11:04:43 -0600

    • Perform serialization sequentially, block for each batch of forwards Signed-off-by: Chase Engelbrecht <[email protected]>

    • Add test for mixed future results Signed-off-by: Chase Engelbrecht <[email protected]>

    • Rename forwardRecords to forwardBatchedRecords to avoid naming duplication Signed-off-by: Chase Engelbrecht <[email protected]>

    • Add missing log statement Signed-off-by: Chase Engelbrecht <[email protected]>

    Signed-off-by: Chase Engelbrecht <[email protected]>

  • Removed getObjectAttributes API call to check object size (#2179)

    Asif Sohail Mohammed - Wed, 1 Feb 2023 10:16:49 -0600

    • Removed getObjectAttributes API call to check object size Signed-off-by: Asif Sohail Mohammed <[email protected]>

  • Bump org.springframework:spring-core in /data-prepper-expression (#2222)

    dependabot[bot] - Tue, 31 Jan 2023 21:00:33 -0600

    Bumps org.springframework:spring-core from 5.3.23 to 5.3.25.

    Commits

    updated-dependencies:

    • dependency-name: org.springframework:spring-core dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

  • Bump org.assertj:assertj-core (#2225)

    dependabot[bot] - Tue, 31 Jan 2023 20:46:44 -0600

    Bumps org.assertj:assertj-core from 3.23.1 to 3.24.2.

    updated-dependencies:

    • dependency-name: org.assertj:assertj-core dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

  • Bump org.assertj:assertj-core in /data-prepper-plugins/http-source (#2227)

    dependabot[bot] - Tue, 31 Jan 2023 20:46:18 -0600

    Bumps org.assertj:assertj-core from 3.23.1 to 3.24.2.

    updated-dependencies:

    • dependency-name: org.assertj:assertj-core dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

  • Bump org.assertj:assertj-core in /data-prepper-plugins/otel-trace-source (#2212)

    dependabot[bot] - Tue, 31 Jan 2023 20:45:50 -0600

    Bumps org.assertj:assertj-core from 3.23.1 to 3.24.2.

    updated-dependencies:

    • dependency-name: org.assertj:assertj-core dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

  • Bump org.springframework:spring-test in /data-prepper-core (#2215)

    dependabot[bot] - Tue, 31 Jan 2023 20:41:19 -0600

    Bumps org.springframework:spring-test from 5.3.23 to 5.3.25.

    Commits

    updated-dependencies:

    • dependency-name: org.springframework:spring-test dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

  • Bump org.springframework:spring-core in /data-prepper-core (#2216)

    dependabot[bot] - Tue, 31 Jan 2023 20:40:24 -0600

    Bumps org.springframework:spring-core from 5.3.23 to 5.3.25.

    Commits

    updated-dependencies:

    • dependency-name: org.springframework:spring-core dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

  • Implement batching for peer forwarder request documents (#2197)

    Chase - Tue, 31 Jan 2023 16:42:25 -0600

    • Implement batching for peer forwarder request documents Signed-off-by: Chase Engelbrecht <[email protected]>

    • Add configurable forwarding_batch_timeout for low-traffic scenarios Signed-off-by: Chase Engelbrecht <[email protected]>

    • Slight refactors and add unit tests for batching Signed-off-by: Chase Engelbrecht <[email protected]>

    • Increase YAML deserialization size Signed-off-by: Chase Engelbrecht <[email protected]>

    • Refactor for clarity and flush all available batches on each iteration Signed-off-by: Chase Engelbrecht <[email protected]>

    • Fix typo in FORWARDING Signed-off-by: Chase Engelbrecht <[email protected]>

    • Use getOrDefault when checking last flushed time Signed-off-by: Chase Engelbrecht <[email protected]>

    Signed-off-by: Chase Engelbrecht <[email protected]>

  • Integration test for a pipeline with multiple process workers (#2017)

    David Venable - Tue, 31 Jan 2023 14:29:38 -0600

    Created an integration test for a pipeline with multiple process workers. Signed-off-by: David Venable <[email protected]>

  • Add clientTimeout to peer forwarder configuration, optimize CPF seria… (#2190)

    Chase - Tue, 31 Jan 2023 11:53:15 -0600

    • Add clientTimeout to peer forwarder configuration, optimize CPF serialization

    Signed-off-by: Chase Engelbrecht <[email protected]>

    • Fix miss in rename and use lower write timeout rather than higher request timeout Signed-off-by: Chase Engelbrecht <[email protected]>

    • Add missing private Signed-off-by: Chase Engelbrecht <[email protected]>

    Signed-off-by: Chase Engelbrecht <[email protected]>

  • Bump hibernate-validator in /data-prepper-core (#1854)

    dependabot[bot] - Mon, 30 Jan 2023 10:29:58 -0600

    Bumps hibernate-validator from 7.0.5.Final to 8.0.0.Final.

    Changelog

    Commits

    updated-dependencies:

    • dependency-name: org.hibernate.validator:hibernate-validator dependency-type: direct:production update-type: version-update:semver-major ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

  • Bump jackson-datatype-jdk8 from 2.13.3 to 2.14.1 in /data-prepper-api (#2101)

    dependabot[bot] - Mon, 30 Jan 2023 10:25:40 -0600

    Bumps jackson-datatype-jdk8 from 2.13.3 to 2.14.1.

    updated-dependencies:

    • dependency-name: com.fasterxml.jackson.datatype:jackson-datatype-jdk8 dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

  • Support OpenTelementry Logs (#1372)

    kmssap - Sat, 28 Jan 2023 09:21:54 -0600

    • Move mapping from processor to source
    • Move mapping logic to OtelProtoCodec
    • Enhance metrics of LogsGrpcService Signed-off-by: Kai Sternad <[email protected]>

    Signed-off-by: Kai Sternad <[email protected]> Signed-off-by: Kai Sternad <[email protected]> Co-authored-by: Kai Sternad <[email protected]>

    Co-authored-by: Tomas Longo <[email protected]> Co-authored-by: Karsten Schnitter <[email protected]> Co-authored-by: Kai Sternad <[email protected]>

  • Optimize buffer reads, allow delay to be 0 (#2189)

    Chase - Fri, 27 Jan 2023 17:11:49 -0600

    Signed-off-by: Chase Engelbrecht <[email protected]>

  • ENH: implement custom LogEventPatternConverter for desensitization on Data Prepper logging (#2188)

    Qi Chen - Fri, 27 Jan 2023 16:27:09 -0600

  • Generated THIRD-PARTY file for bd60dcc (#2201)

    opensearch-trigger-bot[bot] - Fri, 27 Jan 2023 14:30:58 -0600

    Signed-off-by: GitHub <[email protected]> Co-authored-by: asifsmohammed <[email protected]>

  • Updated OpenSearch version to 1.3.7 (#2191)

    Asif Sohail Mohammed - Fri, 27 Jan 2023 09:24:13 -0600

    Signed-off-by: Asif Sohail Mohammed <[email protected]> Signed-off-by: Asif Sohail Mohammed <[email protected]>

  • Fix Data Prepper to not terminate on invalid open telemetry metric/trace data (#2176)

    kkondaka - Thu, 26 Jan 2023 19:03:00 -0600

    • Rebased to latest and removed unnecessary System.out.println Signed-off-by: Krishna Kondaka <[email protected]>

    • Modified the test to send one valid record and one invalid record, as per the review comments Signed-off-by: Krishna Kondaka <[email protected]> Signed-off-by: Krishna Kondaka <[email protected]> Co-authored-by: Krishna Kondaka <[email protected]>

  • Add configurations and metrics to OTelTraceRawProcessor (#2164)

    David Venable - Thu, 26 Jan 2023 19:01:47 -0600

    Updated the OTelTraceRawProcessor to use the new plugin configuration model. Added two plugin metrics to tracking the usage of two of the collections used in this processor. Signed-off-by: David Venable <[email protected]>

  • Don't block on forwarding request response, populate records that failed to forward in local CPF buffer (#2175)

    Chase - Thu, 26 Jan 2023 17:26:14 -0600

    • Handle peer forwarding in the background Signed-off-by: Chase Engelbrecht <[email protected]>

    • Rework tests to pass with new implementation Signed-off-by: Chase Engelbrecht <[email protected]>

    • Small tweaks to fix the diff Signed-off-by: Chase Engelbrecht <[email protected]>

    • Refactor to use Consumer<AggregateHttpResponse> and ExecutorService for background forwarding work Signed-off-by: Chase Engelbrecht <[email protected]>

    • Add unit tests for processFailedRequestsLocally Signed-off-by: Chase Engelbrecht <[email protected]>

    • Add more unit tests Signed-off-by: Chase Engelbrecht <[email protected]>

    • Move requestsReceivedFromPeers to HttpService to avoid gauge usage Signed-off-by: Chase Engelbrecht <[email protected]>

    • Coming full circle + an ExecutorService Signed-off-by: Chase Engelbrecht <[email protected]>

    • Add missing private modifier Signed-off-by: Chase Engelbrecht <[email protected]>

    • Reuse existing clientThreadCount parameter, cleanup unused code Signed-off-by: Chase Engelbrecht <[email protected]>

    • Fix log message Signed-off-by: Chase Engelbrecht <[email protected]> Signed-off-by: Chase Engelbrecht <[email protected]>

  • Add support to convert metrics to json strings without flattening attributes - issue #2146 (#2163)

    kkondaka - Thu, 26 Jan 2023 15:45:02 -0600

    Add support to convert metrics to json strings without flattening attributes field - issue #2146 Signed-off-by: Krishna Kondaka <[email protected]>

  • ENH: add and populate markers (#2180)

    Qi Chen - Thu, 26 Jan 2023 11:31:25 -0600

  • Fixes links to use org/opensearch from com/amazon now that this is the correct the package name. (#2186)

    David Venable - Wed, 25 Jan 2023 08:29:54 -0600

    Signed-off-by: David Venable <[email protected]>

  • Added AWS STS header override configurations for OpenSearch sink/S3 source (#1898)

    David Venable - Tue, 24 Jan 2023 17:17:13 -0600

    Added AWS STS header override configurations for the OpenSearch sink and the Amazon S3 source. Resolves #1888. Signed-off-by: David Venable <[email protected]>

  • ENH: add buffer records overflow metrics (#2170)

    Qi Chen - Tue, 24 Jan 2023 09:51:27 -0600

    • ENH: add buffer records overflow metrics Signed-off-by: George Chen <[email protected]>

  • Bump decode-uri-component in /release/staging-resources-cdk (#2064)

    dependabot[bot] - Mon, 23 Jan 2023 19:49:14 -0600

    Bumps decode-uri-component from 0.2.0 to 0.2.2.

    Commits

    updated-dependencies:

    • dependency-name: decode-uri-component dependency-type: indirect ... Signed-off-by: dependabot[bot] <[email protected]> Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

  • New aggregate action - percent sampler (#2096)

    kkondaka - Mon, 23 Jan 2023 17:36:31 -0600

    Percent Sampler aggregate action Signed-off-by: Krishna Kondaka <[email protected]>

  • Implemented a heap-based circuit breaker (#2155)

    David Venable - Mon, 23 Jan 2023 17:11:16 -0600

    Implemented a heap-based circuit breaker. This circuit breaker will prevent entry buffers from accepting events after the heap usage reaches a specified value. This checks for heap usage in a background thread and updates the state, which the buffer will then use to determine if the circuit breaker is open or closed. This also signals to the JVM to start a GC when the threshold is reached. Resolves #2150. Signed-off-by: David Venable <[email protected]>

  • Test against multiple OTel version - Issue #1963 (#2154)

    kkondaka - Mon, 23 Jan 2023 15:08:18 -0600

  • Bump byte-buddy in /data-prepper-plugins/opensearch (#2161)

    dependabot[bot] - Sat, 21 Jan 2023 14:48:03 -0600

    Bumps byte-buddy from 1.12.18 to 1.12.22.

    Commits

    updated-dependencies:

    • dependency-name: net.bytebuddy:byte-buddy dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <[email protected]> Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

  • Bump byte-buddy-agent in /data-prepper-plugins/opensearch (#2160)

    dependabot[bot] - Fri, 20 Jan 2023 11:11:23 -0600

    Bumps byte-buddy-agent from 1.12.20 to 1.12.22.

    Commits

    updated-dependencies:

    • dependency-name: net.bytebuddy:byte-buddy-agent dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <[email protected]> Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

  • New Aggregate Action - Event Rate Limiter (#2090)

    kkondaka - Thu, 19 Jan 2023 17:28:12 -0600

    Signed-off-by: Krishna Kondaka <[email protected]> Co-authored-by: Krishna Kondaka <[email protected]>

  • Bump byte-buddy-agent in /data-prepper-plugins/opensearch (#2100)

    dependabot[bot] - Thu, 19 Jan 2023 16:22:01 -0600

    Bumps byte-buddy-agent from 1.12.18 to 1.12.20.

    Commits

    updated-dependencies:

    • dependency-name: net.bytebuddy:byte-buddy-agent dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <[email protected]> Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

  • Added s3 support in Opensearch sink (#2121)

    Asif Sohail Mohammed - Thu, 19 Jan 2023 16:21:32 -0600

    • Added implementation of s3 support in Opensearch sink Signed-off-by: Asif Sohail Mohammed <[email protected]>

  • Added batch delay to CPF configuration (#2159)

    Asif Sohail Mohammed - Thu, 19 Jan 2023 13:05:26 -0600

    Signed-off-by: Asif Sohail Mohammed <[email protected]> Signed-off-by: Asif Sohail Mohammed <[email protected]>

  • Fix for null pointer exception in remote peer forwarding (fix for issue 2123) (#2124)

    kkondaka - Wed, 18 Jan 2023 20:56:57 -0600

    • Fix for null pointer exception in remote peer forwarding (fix for issue #2123) Signed-off-by: Krishna Kondaka <[email protected]>

    • Addressed review comments to add a counter and not skip when an identification key is missing Signed-off-by: Krishna Kondaka <[email protected]>

    • Addressed review comments. Modified to increment the counter only when all identification keys are missing Signed-off-by: Krishna Kondaka <[email protected]>

    • Added 'final' to the local variable Signed-off-by: Krishna Kondaka <[email protected]>

    • Addressed review comments Signed-off-by: Krishna Kondaka <[email protected]>

    • Added a test with all missing keys Signed-off-by: Krishna Kondaka <[email protected]> Signed-off-by: Krishna Kondaka <[email protected]> Co-authored-by: Krishna Kondaka <[email protected]>

  • Adds ScheduledExecutorService for Polling the RSS feed (#2140)

    Shivani Shukla - Wed, 18 Jan 2023 15:10:08 -0600

    • Adds ScheduledExecutor Service and runnable task Signed-off-by: Shivani Shukla <[email protected]>

  • Combined two integration tests for conditional routes into one. Also fixed a bug in the tests where the data was not sent to the correct sink. (#2061)

    David Venable - Tue, 17 Jan 2023 21:04:00 -0600

    Signed-off-by: David Venable <[email protected]>

  • Add e2etest for testing log metrics (#2127)

    kkondaka - Fri, 13 Jan 2023 15:19:16 -0600

  • Adds count metrics to the service_map_stateful processor (#2130)

    David Venable - Thu, 12 Jan 2023 15:17:03 -0600

    Adds new metrics to the service_map_stateful processor. These count the number of items in the collections used by the service map. These do not have byte sizes, but use object counts. Signed-off-by: David Venable <[email protected]>

  • Bump guava from 10.0.1 to 23.0 in /data-prepper-plugins/opensearch (#2050)

    dependabot[bot] - Wed, 11 Jan 2023 12:07:17 -0600

    Bumps guava from 10.0.1 to 23.0.

    updated-dependencies:

    • dependency-name: com.google.guava:guava dependency-type: direct:production update-type: version-update:semver-major ... Signed-off-by: dependabot[bot] <[email protected]> Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

  • Bump json5 from 2.2.0 to 2.2.3 in /release/staging-resources-cdk (#2119)

    dependabot[bot] - Wed, 11 Jan 2023 12:06:05 -0600

    Bumps json5 from 2.2.0 to 2.2.3.

    updated-dependencies:

    • dependency-name: json5 dependency-type: indirect ... Signed-off-by: dependabot[bot] <[email protected]> Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

  • Updated MAINTAINERS.md to match recommended opensearch-project format. (#2117)

    Daniel (dB.) Doubrovkine - Thu, 5 Jan 2023 14:58:26 -0600

    Signed-off-by: dblock <[email protected]> Signed-off-by: dblock <[email protected]>

  • Update OTelProtoCodec for InstrumentationLibrary to InstrumentationScope rename (#2114)

    Chase - Wed, 4 Jan 2023 09:40:06 -0600

  • Update slf4j-simple; update log4j-slf4j-impl to log4j-slf4j2-impl (#2113)

    Hai Yan - Tue, 3 Jan 2023 17:16:03 -0600

    Signed-off-by: Hai Yan <[email protected]>

  • MOD: fix typos (#2084)

    Shanelle Marasigan - Tue, 3 Jan 2023 15:51:49 -0600

    Signed-off-by: Russianhielle Marasigan <[email protected]>

  • Created a GitHub Action to generate the Third Party report. (#2033)

    David Venable - Thu, 22 Dec 2022 15:01:37 -0800

    Signed-off-by: David Venable <[email protected]>

  • updating S3 source documentation to include all codecs (#2091)

    Christopher Manning - Thu, 22 Dec 2022 16:04:08 -0600

    Signed-off-by: Christopher Manning <[email protected]>

  • Provide a type conversion / cast processor #2010 (#2020)

    kkondaka - Thu, 22 Dec 2022 15:50:01 -0600

  • Histogram Aggregate Action - Added duration and fixed end time in the aggregated output (#2085)

    kkondaka - Thu, 22 Dec 2022 12:20:08 -0600

    Signed-off-by: Krishna Kondaka <[email protected]>

  • Fix CVE-2022-41881, CVE-2021-21290 and CVE-2022-41915 (#2093)

    Asif Sohail Mohammed - Wed, 21 Dec 2022 16:36:04 -0800

    Signed-off-by: Asif Sohail Mohammed <[email protected]>

  • Fix: CVE-2022-3509, CVE-2022-3510 (#2079)

    Asif Sohail Mohammed - Mon, 19 Dec 2022 10:14:22 -0800

    Signed-off-by: Asif Sohail Mohammed <[email protected]>

  • Add robust retry strategy to AcmClients (#2082)

    Chase - Fri, 16 Dec 2022 19:29:07 -0600

    Signed-off-by: Chase Engelbrecht <[email protected]>

  • Updated info logs to debug level (#2083)

    Asif Sohail Mohammed - Fri, 16 Dec 2022 16:35:28 -0800

    Signed-off-by: Asif Sohail Mohammed <[email protected]>

  • Fix: CVE-2022-36944 (#2080)

    Asif Sohail Mohammed - Fri, 16 Dec 2022 15:59:07 -0800

    Signed-off-by: Asif Sohail Mohammed <[email protected]>

  • Histogram aggregate action (#2078)

    kkondaka - Fri, 16 Dec 2022 11:22:23 -0800

  • Updated jackson bom dependency (#2068)

    Asif Sohail Mohammed - Thu, 15 Dec 2022 14:44:59 -0800

    Signed-off-by: Asif Sohail Mohammed <[email protected]>

  • Anomaly detector (#2058)

    kkondaka - Thu, 15 Dec 2022 16:28:28 -0600

    • Add support for anomaly detection in the pipeline with new anomaly detector processor Signed-off-by: Krishna Kondaka <[email protected]> Co-authored-by: Krishna Kondaka <[email protected]>

  • Add retry strategy to StsClient used for sigv4 auth against OpenSearch sinks (#2069)

    Chase - Thu, 15 Dec 2022 11:26:50 -0600

    Add retry strategy to STS client used for sigv4 auth Signed-off-by: Chase Engelbrecht <[email protected]>

  • Parse RSS feed URL items and convert Item to Event (#2073)

    Shivani Shukla - Wed, 14 Dec 2022 13:16:46 -0600

    Signed-off-by: Shivani Shukla <[email protected]>

    • Adds Jackson Jdk8Module to enable usage of Optional in Rss Item model

    Signed-off-by: Shivani Shukla <[email protected]>

  • Fix: Updated parse json processor documentation (#2071)

    Asif Sohail Mohammed - Wed, 14 Dec 2022 08:47:12 -0800

    Signed-off-by: Asif Sohail Mohammed <[email protected]>

  • Count Aggregate Action - fix aggregation temporality (#2067)

    kkondaka - Thu, 8 Dec 2022 17:21:41 -0600

    Signed-off-by: Krishna Kondaka <[email protected]>

  • Add support for count aggregate action (#2034)

    kkondaka - Wed, 7 Dec 2022 12:51:11 -0600

    Add support for count aggregate action with raw and otel_metrics output format Signed-off-by: Krishna Kondaka <[email protected]>

  • Setup boilerplate for RSS Source Plugin (#2062)

    Shivani Shukla - Wed, 7 Dec 2022 12:44:37 -0600

    • Adds boilerplate config and code for rss source Signed-off-by: Shivani Shukla <[email protected]>

    • Adds RSS Source plugin class Signed-off-by: Shivani Shukla <[email protected]>

    • Adds Document Event Type Signed-off-by: Shivani Shukla <[email protected]>

    • Adds Document interface and JacksonDocument class Signed-off-by: Shivani Shukla <[email protected]>

    • Adds a simple unit test checking default for pollingFrequency Signed-off-by: Shivani Shukla <[email protected]>

  • Bump slf4j-simple in /data-prepper-logstash-configuration (#2053)

    dependabot[bot] - Fri, 2 Dec 2022 14:04:27 -0600

    Bumps slf4j-simple from 1.7.36 to 2.0.5.

    updated-dependencies:

    • dependency-name: org.slf4j:slf4j-simple dependency-type: direct:production update-type: version-update:semver-major ... Signed-off-by: dependabot[bot] <[email protected]> Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

  • Bump slf4j-api from 1.7.36 to 2.0.5 (#2057)

    dependabot[bot] - Fri, 2 Dec 2022 14:00:34 -0600

    Bumps slf4j-api from 1.7.36 to 2.0.5.

    updated-dependencies:

    • dependency-name: org.slf4j:slf4j-api dependency-type: direct:production update-type: version-update:semver-major ... Signed-off-by: dependabot[bot] <[email protected]> Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

  • Bump jsonassert from 1.5.0 to 1.5.1 in /data-prepper-api (#2043)

    dependabot[bot] - Thu, 1 Dec 2022 17:51:03 -0600

    Bumps jsonassert from 1.5.0 to 1.5.1.

    Changelog

    Commits

    updated-dependencies:

    • dependency-name: org.skyscreamer:jsonassert dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <[email protected]> Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

  • Bump protobuf-java-util in /data-prepper-plugins/otel-trace-raw-prepper (#2042)

    dependabot[bot] - Thu, 1 Dec 2022 17:50:08 -0600

    Bumps protobuf-java-util from 3.21.9 to 3.21.10.

    updated-dependencies:

    • dependency-name: com.google.protobuf:protobuf-java-util dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <[email protected]> Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

  • Router integration tests to verify that conditional pipeline routes work as expected. (#1988)

    David Venable - Thu, 1 Dec 2022 15:51:52 -0600

    Signed-off-by: David Venable <[email protected]>

  • Updated the OpenSearch libraries to 1.3.6 and run the integration tests against 1.3.6 instead of 1.3.5. Fixes #2022 (#2030)

    David Venable - Wed, 30 Nov 2022 12:57:39 -0600

    Signed-off-by: David Venable <[email protected]>

  • Removes double brace initialization and includes this as a checkstyle rule to prevent future use. (#2035)

    David Venable - Tue, 22 Nov 2022 16:16:14 -0600

    Signed-off-by: David Venable <[email protected]>

  • Implemented additional metrics for the S3 source. (#2028)

    David Venable - Tue, 22 Nov 2022 14:25:20 -0600

    Implemented additional metrics for the S3 source. Resolves #2024 Signed-off-by: David Venable <[email protected]>

  • Use the same compression engines for GZIP and NONE in the AUTOMATIC compression engine. Fixes #2026. (#2027)

    David Venable - Mon, 21 Nov 2022 11:15:05 -0600

    Signed-off-by: David Venable <[email protected]>

  • Log the number of messages received from the SQS queue, including a count of the number of messages that will need to be processed. Also, include logging of deletes at the debug level. (#2011)

    David Venable - Sat, 19 Nov 2022 13:06:23 -0600

    Signed-off-by: David Venable <[email protected]>

  • Add PluginMetrics in the Auth Plugin for Http, OTel and Metrics Source (#2023)

    Dinu John - Fri, 18 Nov 2022 21:27:15 -0600

    Signed-off-by: Dinu John <[email protected]>

  • Add when condition to aggregate processor (#2018)

    kkondaka - Fri, 18 Nov 2022 15:31:54 -0600

  • Dynamic Index Name in OpenSearch sink - Resolves #1459 (#1999)

    kkondaka - Thu, 17 Nov 2022 20:36:47 -0600

    • Dynamic Index Name in OpenSearch sink #1459 Signed-off-by: Krishna Kondaka <[email protected]>

    • Dynamic Index Name in OpenSearch sink #1459 -- updated documentation Signed-off-by: Krishna Kondaka <[email protected]>

    • Dynamic Index Name in OpenSearch sink #1459 -- fixed a bug and increased test coverage Signed-off-by: Krishna Kondaka <[email protected]>

    • Addressed review comments to add more tests and re-design index manager to accommodate dynamic indexes Signed-off-by: Krishna Kondaka <[email protected]>

    • Added tests for DynamicIndexManager Signed-off-by: Krishna Kondaka <[email protected]>

    • Dynamic Index Name in OpenSearch sink #1459 -- changed cache weigher to have constant value Signed-off-by: Krishna Kondaka <[email protected]>

    • Dynamic Index Name in OpenSearch sink #1459 -- addressed review comments Signed-off-by: Krishna Kondaka <[email protected]>

    • Dynamic Index Name in OpenSearch sink - Resolves Issue #1459 -- addressed review comments Signed-off-by: Krishna Kondaka <[email protected]>

    • Dynamic Index Name in OpenSearch sink - Fixed checkSytleMain issues in opensearch Signed-off-by: Krishna Kondaka <[email protected]>

    • Dynamic Index Name in OpenSearch sink - Fixed spotlessJavaCheck issues in opensearch Signed-off-by: Krishna Kondaka <[email protected]> Signed-off-by: Krishna Kondaka <[email protected]> Co-authored-by: Krishna Kondaka <[email protected]>

  • updated samples for dataperpper 2 (#2019)

    Arunachalam Lakshmanan - Thu, 17 Nov 2022 10:02:31 -0600

  • Add IntelliJ's .iml file extension to the .gitignore. (#2016)

    David Venable - Wed, 16 Nov 2022 14:15:00 -0600

    Signed-off-by: David Venable <[email protected]>

  • Bump protobuf-java-util in /data-prepper-plugins/otel-trace-raw-prepper (#2001)

    dependabot[bot] - Fri, 11 Nov 2022 09:58:40 -0600

    Bumps protobuf-java-util from 3.19.4 to 3.21.9.

    updated-dependencies:

    • dependency-name: com.google.protobuf:protobuf-java-util dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <[email protected]> Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

  • Data Prepper Core integration tests (#1949)

    David Venable - Thu, 10 Nov 2022 09:50:03 -0600

    Creates an integration test source set for data-prepper-core and adds a small framework for running integration tests on data-prepper-core functionality. Signed-off-by: David Venable <[email protected]>

  • Adding Chase Engelbrecht to the MAINTAINERS.md. (#2005)

    David Venable - Wed, 9 Nov 2022 09:55:59 -0600

    Signed-off-by: David Venable <[email protected]>

  • Adding Hai Yan to the maintainers. (#2004)

    David Venable - Tue, 8 Nov 2022 16:13:31 -0600

    Signed-off-by: David Venable <[email protected]>

  • Bump opensearch-java from 1.0.0 to 2.1.0 (#1733)

    dependabot[bot] - Tue, 8 Nov 2022 08:17:20 -0600

    Bumps opensearch-java from 1.0.0 to 2.1.0.

    Commits

    updated-dependencies:

    • dependency-name: org.opensearch.client:opensearch-java dependency-type: direct:production update-type: version-update:semver-major ... Signed-off-by: dependabot[bot] <[email protected]> Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

  • Bump assertj-core in /data-prepper-plugins/otel-trace-raw-prepper (#2003)

    dependabot[bot] - Tue, 8 Nov 2022 08:15:44 -0600

    Bumps assertj-core from 3.22.0 to 3.23.1.

    Commits

    updated-dependencies:

    • dependency-name: org.assertj:assertj-core dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <[email protected]> Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

  • Bump awaitility in /data-prepper-plugins/otel-trace-raw-prepper (#2002)

    dependabot[bot] - Tue, 8 Nov 2022 08:15:12 -0600

    Bumps awaitility from 4.1.1 to 4.2.0.

    Changelog

    Commits

    updated-dependencies:

    • dependency-name: org.awaitility:awaitility dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <[email protected]> Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

  • Support remaining OpenTelemetry Metrics proto spec features (#1335)

    kmssap - Fri, 4 Nov 2022 10:58:14 -0500

    • Address review comments
    • Remove unused import, breaking Checkstyle
    • Change Exponential Histogram filter
    • Add lenient to some Mockito calls
    • Clarify metrics processor documentation Signed-off-by: Kai Sternad <[email protected]> Co-authored-by: Tomas Longo <[email protected]> Co-authored-by: Karsten Schnitter <[email protected]>

    Co-authored-by: Karsten Schnitter <[email protected]>

  • Bump byte-buddy in /data-prepper-plugins/opensearch (#1993)

    dependabot[bot] - Wed, 2 Nov 2022 20:22:12 -0500

    Bumps byte-buddy from 1.12.17 to 1.12.18.

    Commits

    updated-dependencies:

    • dependency-name: net.bytebuddy:byte-buddy dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <[email protected]> Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

  • Bump byte-buddy-agent in /data-prepper-plugins/opensearch (#1992)

    dependabot[bot] - Wed, 2 Nov 2022 19:20:41 -0500

    Bumps byte-buddy-agent from 1.12.17 to 1.12.18.

    Commits

    updated-dependencies:

    • dependency-name: net.bytebuddy:byte-buddy-agent dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <[email protected]> Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

  • Fixes for issues #1456 and #1458 - support for complex document ID and routing ID (#1966)

    kkondaka - Wed, 2 Nov 2022 18:57:46 -0500

    • Fixes for issues #1456 and #1458 - support for complex document ID and routing ID Signed-off-by: Krishna Kondaka <[email protected]>

    • Fixes for issues #1456 and #1458 - updated README.md and added unit tests Signed-off-by: Krishna Kondaka <[email protected]>

    • Fixes for issues #1456 and #1458 - addressed review comments Signed-off-by: Krishna Kondaka <[email protected]>

    • Fixes for issues #1456 and #1458 - fixed check style build failure Signed-off-by: Krishna Kondaka <[email protected]>

    • Add OpenSearch e2e test to github workflows Signed-off-by: Krishna Kondaka <[email protected]>

    • Addressed review comments. Deleted e2e test for OpenSearch DocumentId/RoutingField testing Signed-off-by: Krishna Kondaka <[email protected]>

    • Remove e2e test entry for removed e2e test Signed-off-by: Krishna Kondaka <[email protected]>

    • Addressed review comments and added tests for fromStringAndOptionals method Signed-off-by: Krishna Kondaka <[email protected]>

    • Fixed checkstyle failure Signed-off-by: Krishna Kondaka <[email protected]> Signed-off-by: Krishna Kondaka <[email protected]> Co-authored-by: Krishna Kondaka <[email protected]>

  • Update a few logs that were dumping the actual customer logs and traces into the DataPrepper logs on failure. Logging the request sizes instead, to avoid customer data being logged (#1989)

    Deep Datta - Mon, 31 Oct 2022 14:30:12 -0500

    Signed-off-by: Deep Datta <[email protected]>

  • Updated release notes and change log with spring change (#1981)

    Asif Sohail Mohammed - Thu, 27 Oct 2022 17:02:02 -0500

    Signed-off-by: Asif Sohail Mohammed <[email protected]>

  • Created release notes for Data Prepper 2.0.1. (#1969)

    David Venable - Thu, 27 Oct 2022 13:28:47 -0500

    Signed-off-by: David Venable <[email protected]>

  • Added release notes for 1.5.2 (#1976)

    Asif Sohail Mohammed - Thu, 27 Oct 2022 12:44:57 -0500

    Signed-off-by: Asif Sohail Mohammed <[email protected]>

  • Added change log for 1.5.2 (#1977)

    Asif Sohail Mohammed - Thu, 27 Oct 2022 12:28:10 -0500

    Signed-off-by: Asif Sohail Mohammed <[email protected]>

  • Added the change log for 2.0.1. (#1968)

    David Venable - Thu, 27 Oct 2022 10:38:17 -0500

    Signed-off-by: David Venable <[email protected]>

  • Expressions with null (#1946)

    kkondaka - Tue, 25 Oct 2022 15:00:55 -0500

    • fix for issue #1136 Add null support to DataPrepperExpressions Signed-off-by: Krishna Kondaka <[email protected]>

    • fix for issue #1136 Add null support to DataPrepperExpressions - added more tests Signed-off-by: Krishna Kondaka <[email protected]>

    • fix for issue #1136 Add null support to DataPrepperExpressions - updated docs

    Signed-off-by: Krishna Kondaka <[email protected]>

  • Reduce smoke test timeout to 8 minutes from 30 minutes. These tests tend to pass within 3 minutes in my personal GitHub branch. So this leaves quite a bit of buffer time. It helps speed up retrying failures from flaky tests. (#1956)

    David Venable - Fri, 21 Oct 2022 09:48:44 -0500

    Signed-off-by: David Venable <[email protected]>

  • Attempt to reduce flakiness in RandomStringSourceTests by using awaitility. Split tests into two. JUnit 5. (#1921)

    David Venable - Fri, 21 Oct 2022 09:48:29 -0500

    Signed-off-by: David Venable <[email protected]>

  • Run smoke tests against OpenSearch 1.3.6. (#1955)

    David Venable - Fri, 21 Oct 2022 06:47:01 -0500

    Signed-off-by: David Venable <[email protected]>

  • Use Python grpcio 1.50.0 in smoke tests to reduce time to run. (#1954)

    David Venable - Fri, 21 Oct 2022 06:46:43 -0500

    Signed-off-by: David Venable <[email protected]>

  • Delete s3:TestEvent objects and log them when they are found in the SQS queue. Resolves #1924. (#1939)

    David Venable - Thu, 20 Oct 2022 12:56:56 -0500

    Signed-off-by: David Venable <[email protected]>

  • Add ExecutorService to DataPrepperServer (#1948)

    Chase - Thu, 20 Oct 2022 11:18:27 -0500

  • Updated k8s manifest to suit Data Prepper 2.0 (#1928)

    Rafael Gumiero - Wed, 19 Oct 2022 17:04:42 -0500

  • Require protobuf-java-util 3.21.7 to fix #1891 (#1938)

    David Venable - Wed, 19 Oct 2022 12:35:33 -0500

    Signed-off-by: David Venable <[email protected]>

  • Bug Fix: S3 source key (#1926)

    Asif Sohail Mohammed - Wed, 19 Oct 2022 11:39:56 -0500

  • Jackson 2.13.4.2 (#1925)

    David Venable - Tue, 18 Oct 2022 16:52:21 -0500

    Signed-off-by: David Venable <[email protected]>

  • Refactors the Data Prepper CLI argument parsing into data-prepper-main. Added an interface for the parts of DataPrepperArgs that client classes really need. (#1920)

    David Venable - Fri, 14 Oct 2022 10:03:06 -0500

    Signed-off-by: David Venable <[email protected]>

  • Fix PipelineConnector to duplicate the events (#1897)

    kkondaka - Wed, 12 Oct 2022 16:32:00 -0500

    • Fix string mutate processors to duplicate the events Signed-off-by: Krishna Kondaka <[email protected]>

    • Fix string mutate processors to duplicate the events - made changes as per David's suggestions Signed-off-by: Krishna Kondaka <[email protected]>

    • Removed unnecessary changes leftover from 1st commit Signed-off-by: Krishna Kondaka <[email protected]>

    • Modified PipelineConnector to duplicate JacksonSpan type events too. Added testcases in PipelineConnectorTest Signed-off-by: Krishna Kondaka <[email protected]>

    • Addressed review comments Signed-off-by: Krishna Kondaka <[email protected]>

    • Addressed review comment and added a new testcase for JacksonSpan withData() Signed-off-by: Krishna Kondaka <[email protected]>

    • Addressed review comment and added parallel pipeline test to github/workflows

    Signed-off-by: Krishna Kondaka <[email protected]>

  • Simple duration regex did not allow for 0s or 0ms (#1910)

    Taylor Gray - Tue, 11 Oct 2022 10:58:40 -0500

    Signed-off-by: Taylor Gray <[email protected]> Signed-off-by: Taylor Gray <[email protected]>

  • Updated the release notes for 2.0.0 (#1911)

    David Venable - Mon, 10 Oct 2022 17:46:26 -0500

    Updated the release notes for 2.0.0 Signed-off-by: David Venable <[email protected]>

  • Updated the change log for 2.0.0 with most recent changes. (#1909)

    David Venable - Mon, 10 Oct 2022 17:33:31 -0500

    Signed-off-by: David Venable <[email protected]>

  • Update dev version to 2.1.0-SNAPSHOT (#1904)

    Hai Yan - Mon, 10 Oct 2022 13:34:46 -0500

    Signed-off-by: Hai Yan <[email protected]>

  • Increase the default buffer configurations by 25. Capacity to 12,800 and batch size to 200. (#1906)

    David Venable - Mon, 10 Oct 2022 11:59:38 -0500

    Signed-off-by: David Venable <[email protected]>

  • Conditional routing documentation (#1894)

    David Venable - Mon, 10 Oct 2022 10:35:15 -0500

    Add documentation for conditional routing. Resolves #1890 Signed-off-by: David Venable <[email protected]>

  • Added change log (#1901)

    Asif Sohail Mohammed - Fri, 7 Oct 2022 20:52:38 -0500

    Signed-off-by: Asif Sohail Mohammed <[email protected]>

  • Adds a stack-trace to failures from OpenSearch to help with debugging issues. (#1899)

    David Venable - Fri, 7 Oct 2022 18:32:26 -0500

    Signed-off-by: David Venable <[email protected]>