Skip to content
This repository has been archived by the owner on Sep 23, 2024. It is now read-only.

Latest commit

 

History

History
213 lines (167 loc) · 10.4 KB

README.md

File metadata and controls

213 lines (167 loc) · 10.4 KB
Raw

Notice

To better serve Wise business and customer needs, the PipelineWise codebase needs to shrink. We have made the difficult decision that, going forward many components of PipelineWise will be removed or incorporated in the main repo. The last version before this decision is v0.64.1

We thank all in the open-source community, that over the past 6 years, have helped to make PipelineWise a robust product for heterogeneous replication of many many Terabytes, daily

pipelinewise-tap-mixpanel

PyPI version PyPI - Python Version License: AGPL

Singer tap that extracts data from a Mixpanel API and produces JSON-formatted data following the Singer spec.

This is a PipelineWise compatible tap connector.

This tap:

  • Pulls raw data from the Mixpanel Event Export API and the Mixpanel Query API.
  • Extracts the following resources:
    • Export (Events)
    • Engage (People/Users)
    • Funnels
    • Annotations
    • Cohorts
    • Cohort Members
    • Revenue
  • Outputs the schema for each resource
  • Incrementally pulls data based on the input state
  • Uses date-windowing to chunk/loop through export, revenue, funnels.
  • Incorporates attribution window for latency look-back to accommodate delays in data reconciliation.

Streams

export

  • Endpoint: https://data.mixpanel.com/api/2.0/export
  • Primary key fields: event, time, distinct_id
  • Replication strategy: INCREMENTAL (query filtered)
    • Bookmark: time
    • Bookmark query field: from_date, to_date
  • Transformations: De-nest properties to root-level, re-name properties with leading $... to mp_reserved_..., convert datetimes from project timezone to UTC.
  • Optional parameters
    • export_events to export only certain events

engage

  • Endpoint: https://mixpanel.com/api/2.0/engage
  • Primary key fields: distinct_id
  • Replication strategy: FULL_TABLE (all records, every load)
  • Transformations: De-nest $properties to root-level, re-name properties with leading $... to mp_reserved_....

funnels

  • Endpoint 1 (name, id): https://data.mixpanel.com/api/2.0/export
  • Endpoint 2 (date, measures): https://mixpanel.com/api/2.0/funnels
  • Primary key fields: funnel_id, date
  • Parameters:
    • funnel_id: {funnel_id} (from Endpoint 1)
    • unit: day
  • Replication strategy: INCREMENTAL (query filtered)
    • Bookmark: date
    • Bookmark query field: from_date, to_date
  • Transformations: Combine Endpoint 1 & 2 results, convert date keys to list to results list-array.

revenue

  • Endpoint: https://mixpanel.com/api/2.0/engage/revenue
  • Primary key fields: date
  • Parameters:
    • unit: day
  • Replication strategy: INCREMENTAL (query filtered)
    • Bookmark: date
    • Bookmark query field: from_date, to_date
  • Transformations: Convert date keys to list to results list-array.

annotations

cohorts

cohort_members (engage)

  • Endpoint: https://mixpanel.com/api/2.0/cohorts/list
  • Primary key fields: distinct_id, cohort_id
  • Parameters:
    • filter_by_cohort: {cohort_id} (from cohorts endpoint)
  • Replication strategy: FULL_TABLE
  • Transformations: For each cohort_id in cohorts endpoint, query engage endpoint with filter_by_cohort parameter to create list of distinct_id for each cohort_id.

Authentication

The Mixpanel API uses Basic Authorization with the api_secret from the tap config in base-64 encoded format. It is slightly different than normal Basic Authorization with username/password. All requests should include this header with the api_secret as the username, with no password:

  • Authorization: Basic <base-64 encoded api_secret>

More details may be found in the Mixpanel API Authentication instructions.

Quick Start

  1. Install

    make venv

  2. Create your tap's config.json file. The tap config file for this tap should include these entries:

    • start_date - the default value to use if no bookmark exists for an endpoint (rfc3339 date string)
    • user_agent (string, optional): Process and email for API logging purposes. Example: tap-mixpanel <api_user_email@your_company.com>
    • api_secret (string, ABCdef123): an API secret for each project in Mixpanel. This can be found in the Mixpanel Console, upper-right Settings (gear icon), Organization Settings > Projects and in the Access Keys section. For this tap, only the api_secret is needed (the api_key is legacy and the token is used only for uploading data). Each Mixpanel project has a different api_secret; therefore each Singer tap pipeline instance is for a single project.
    • date_window_size (integer, 30): Number of days for date window looping through transactional endpoints with from_date and to_date. Default date_window_size is 30 days. Clients with large volumes of events may want to decrease this to 14, 7, or even down to 1-2 days.
    • attribution_window (integer, 5): Latency minimum number of days to look-back to account for delays in attributing accurate results. Default attribution window is 5 days.
    • project_timezone (string like US/Pacific): Time zone in which integer date times are stored. The project timezone may be found in the project settings in the Mixpanel console. More info about timezones.
    • select_properties_by_default (true or false): Mixpanel properties are not fixed and depend on the date being uploaded. During Discovery mode and catalog.json setup, all current/existing properties will be captured. Setting this config parameter to true ensures that new properties on events and engage records are captured. Otherwise new properties will be ignored.
    • denest_properties (true or false): To denest large and nested JSON Mixpanel responses in the extract and engage streams. To avoid very wide schema you can disable the denesting feature and the original JSON response will be sent in the RECORD message as plain object. Default denest_properties is true.
    {
    "api_secret": "YOUR_API_SECRET",
    "date_window_size": "30",
    "attribution_window": "5",
    "project_timezone": "US/Pacific",
    "select_properties_by_default": "true",
    "denest_properties": "true",
    "start_date": "2019-01-01T00:00:00Z",
    "user_agent": "tap-mixpanel <api_user_email@your_company.com>"
}

    If you want to export only certain events from the Raw export API then add export_events option to the config.json and list the required event names:

    "export_events": ["event_one", "event_two"]

    Optionally, also create a state.json file. currently_syncing is an optional attribute used for identifying the last object to be synced in case the job is interrupted mid-stream. The next run would begin where the last job left off.

    {
    "currently_syncing": "engage",
    "bookmarks": {
        "export": "2019-09-27T22:34:39.000000Z",
        "funnels": "2019-09-28T15:30:26.000000Z",
        "revenue": "2019-09-28T18:23:53Z"
    }
}

  3. Run the Tap in Discovery Mode This creates a catalog.json for selecting objects/fields to integrate:

    tap-mixpanel --config config.json --discover > catalog.json

    See the Singer docs on discovery mode here.

  4. Run the Tap in Sync Mode (with catalog) and write out to state file

    For Sync mode:

    > tap-mixpanel --config tap_config.json --catalog catalog.json

    Messages are written to standard output following the Singer specification. The resultant stream of JSON data can be consumed by a Singer target. To load to json files to verify outputs:

    > tap-mixpanel --config tap_config.json --catalog catalog.json | target-json > state.json
> tail -1 state.json > state.json.tmp && mv state.json.tmp state.json

    To pseudo-load to Stitch Import API with dry run:

    > tap-mixpanel --config tap_config.json --catalog catalog.json | target-stitch --config target_config.json --dry-run > state.json
> tail -1 state.json > state.json.tmp && mv state.json.tmp state.json

Test

  1. Install python test dependencies in a virtual env

    make venv

  2. Run unit tests

    make unit_test

Linting

  1. Install python test dependencies in a virtual env

    make venv

  2. Run linter

    make pylint

Licence

GNU AFFERO GENERAL PUBLIC LICENSE