Skip to content

Data Prepper is a data ingestion component that pre-processes documents before indexing them in OpenSearch.

License

Notifications You must be signed in to change notification settings

asifsmohammed/data-prepper

This branch is 1 commit ahead of, 616 commits behind opensearch-project/data-prepper:main.

Folders and files

NameName
Last commit message
Last commit date
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 

Repository files navigation

Data Prepper

codecov

Data Prepper

We envision Data Prepper as an open source data collector for observability data (trace, logs, metrics) that can filter, enrich, transform, normalize, and aggregate data for downstream analysis and visualization. It will support stateful processing across multiple instances of data pipelines for observability use cases such as distributed tracing and multi-line log events (e.g. stack traces, aggregations, and log-to-metric transformations). Currently Data Prepper supports processing of distributed trace data and log ingestion with plans to support metric data in the future.

Please read the Overview to understand what Data Prepper is and how it works.

Getting Started

Our Getting Started guide is the best starting point for anybody who wants to run Data Prepper.

Please read the Trace Analytics guide or Log Analytics to get started with using Data Prepper for trace or log analytics use cases.

Project Resources

Contribute

We invite developers from the larger OpenSearch community to contribute and help improve test coverage and give us feedback on where improvements can be made in design, code and documentation. You can look at contribution guide for more information on how to contribute.

If you are looking to contribute code, or just to build from source, please see our Developer Guide.

Code of Conduct

This project has adopted an Open Source Code of Conduct.

Security Issue Notifications

If you discover a potential security issue in this project, please refer to the security policy.

License

This library is licensed under the Apache 2.0 License

Copyright

Copyright OpenSearch Contributors. See NOTICE for details.

About

Data Prepper is a data ingestion component that pre-processes documents before indexing them in OpenSearch.

Resources

License

Code of conduct

Stars

Watchers

Forks

Packages

No packages published

Languages

  • Java 99.5%
  • Shell 0.2%
  • TypeScript 0.2%
  • ANTLR 0.1%
  • JavaScript 0.0%
  • Dockerfile 0.0%