README.md

odd-collector-azure

ODD Collector is a lightweight service which gathers metadata from all your data sources.

To learn more about collector types and ODD Platform's architecture, read the documentation.

Preview

• Implemented adapters
• How to build
• Azure Data Factory authentication
• Docker compose example

Implemented adapters

Service	Config example
PowerBi	config
AzureSQL	config
BlobStorage	config
AzureDataFactory	config

Azure Data Factory authentication

ADF adapter is using DefaultAzureCredential class for authentication.
By default, it requires following environmental variables: AZURE_TENANT_ID, AZURE_CLIENT_ID, AZURE_CLIENT_SECRET.
Docs regarding class and variables: DefaultAzureCredential, EnvironmentCredential

Building

docker build .

Docker compose example

Due to the Plugin is inherited from pydantic.BaseSetting, each field missed in collector-config.yaml can be taken from env variables.

Custom .env file for docker-compose.yaml

CLIENT_ID=
CLIENT_SECRET=
USERNAME=
PASSWORD=
DOMAIN=
PLATFORM_HOST_URL=http://localhost:8080

Custom collector-config.yaml

platform_host_url: "http://localhost:8080"
default_pulling_interval: 10
token: ""
plugins:
  - type: powerbi
    name: powerbi_adapter
    client_id: <client_id of registered in AD app>
    client_secret: <client secret of registered in AD app>
    username: <email>
    password: password
    domain: yourmomain.com

docker-compose.yaml

version: "3.8"
services:
  # --- ODD Platform ---
  database:
    ...

  odd-platform:
    ...
  
  odd-collector-azure:
    image: 'ghcr.io/opendatadiscovery/odd-collector-azure:latest'
    restart: always
    volumes:
      - collector_config.yaml:/app/collector_config.yaml
    environment:
      - CLIENT_ID=${CLIENT_ID}
      - CLIENT_SECRET=${CLIENT_SECRET}
      - USERNAME=${USERNAME}
      - PASSWORD=${PASSWORD}
      - DOMAIN=${DOMAIN}
      - PLATFORM_HOST_URL=${PLATFORM_HOST_URL}
      - LOGLEVEL='DEBUG'
    depends_on:
      - odd-platform