ODD Collector is a lightweight service which gathers metadata from all your data sources.
To learn more about collector types and ODD Platform's architecture, read the documentation.
Service | Config example |
---|---|
Athena | config |
DynamoDB | config |
Glue | config |
Kinesis | config |
Quicksight | config |
S3 | config |
Sagemaker | config |
SQS | config |
SagemakerFeaturestore | config |
docker build .
Due to the Plugin is inherited from pydantic.BaseSetting
, each field missed in collector-config.yaml
can be taken from env variables.
Custom .env
file for docker-compose.yaml
AWS_REGION=
AWS_ACCESS_KEY_ID=
AWS_SECRET_ACCESS_KEY=
PLATFORM_HOST_URL=http://odd-platform:8080
Custom collector-config.yaml
platform_host_url: "http://localhost:8080"
default_pulling_interval: 10
token: ""
plugins:
- type: s3
name: test_s3_adapter
datasets:
- bucket: bucket_name
path: some_data
docker-compose.yaml
version: "3.8"
services:
# --- ODD Platform ---
database:
...
odd-platform:
...
odd-collector-aws:
image: 'ghcr.io/opendatadiscovery/odd-collector-aws:latest'
restart: always
volumes:
- collector_config.yaml:/app/collector_config.yaml
environment:
- AWS_REGION=${AWS_REGION}
- AWS_ACCESS_KEY_ID=${AWS_ACCESS_KEY_ID}
- AWS_SECRET_ACCESS_KEY=${AWS_SECRET_ACCESS_KEY}
- PLATFORM_HOST_URL=${PLATFORM_HOST_URL}
- LOGLEVEL='DEBUG'
depends_on:
- odd-platform