Skip to content

RamanDamayeu/odd-collector-aws

 
 

Repository files navigation

forthebadge forthebadge

odd-collector-aws

ODD Collector is a lightweight service which gathers metadata from all your data sources.

To learn more about collector types and ODD Platform's architecture, read the documentation.

Preview

Implemented adapters

Service Config example
Athena config
DynamoDB config
Glue config
Kinesis config
Quicksight config
S3 config
Sagemaker config
SQS config
SagemakerFeaturestore config

Building

docker build .

Docker compose example

Due to the Plugin is inherited from pydantic.BaseSetting, each field missed in collector-config.yaml can be taken from env variables.

Custom .env file for docker-compose.yaml

AWS_REGION=
AWS_ACCESS_KEY_ID=
AWS_SECRET_ACCESS_KEY=
PLATFORM_HOST_URL=http://odd-platform:8080

Custom collector-config.yaml

platform_host_url: "http://localhost:8080"
default_pulling_interval: 10
token: ""
plugins:
  - type: s3
    name: test_s3_adapter
    datasets:
      - bucket: bucket_name
        path: some_data

docker-compose.yaml

version: "3.8"
services:
  # --- ODD Platform ---
  database:
    ...

  odd-platform:
    ...
  
  odd-collector-aws:
    image: 'ghcr.io/opendatadiscovery/odd-collector-aws:latest'
    restart: always
    volumes:
      - collector_config.yaml:/app/collector_config.yaml
    environment:
      - AWS_REGION=${AWS_REGION}
      - AWS_ACCESS_KEY_ID=${AWS_ACCESS_KEY_ID}
      - AWS_SECRET_ACCESS_KEY=${AWS_SECRET_ACCESS_KEY}
      - PLATFORM_HOST_URL=${PLATFORM_HOST_URL}
      - LOGLEVEL='DEBUG'
    depends_on:
      - odd-platform

About

No description, website, or topics provided.

Resources

License

Stars

Watchers

Forks

Packages

No packages published

Languages

  • Python 99.3%
  • Other 0.7%