feat(messaging): providers & broadcasts

[mariusandra]

Problem

We can't send e-mails and other messages to users directly from PostHog

Changes

• basic scaffolding
• add type to hog functions
• split the lists for "email", "destination" and "broadcast"
  • ingestion only uses "destination" (or null)
  • frontend lists show different results
  • separate new/edit pages
  • remove filters and inputs for email functions
  • add person filters for brodcast, only show one email input
  • keep source for broadcast? --> by default it's just (send_email(inputs.email); posthogCapture('sent'))
  • show matching persons
  • logs and metrics pages
  • refactor "destination" logic and components
• call send_email from a destination or a broadcast
  • do we have access to the functions every time, e.g also in callbacks?
  • get bytecode chunks with a callback
  • get_team_function(name) in hog executor, sync returns bytecode + callable/globals for 'send_email'
  • get globals working
  • get args working
• big broadcast button
  • make testing possible
  • makes a query to fetch all persons
  • runs the broadcast's bytecode on each of them
    • via the cdp api system
    • via kafka
  • move to a crashproof background job

Next steps

Going to split it up into these chunks:

• PR 1 - migration and backend split
  • Migration to add type to the HogFunction model. Type will be set to destination for all existing functions.
  • The plugin server only runs destination functions on events, but also loads email functions (shared provider code) into memory.
  • The backend returns hog functions based on type, and the frontend only asks for destination
  • Tests for all of the above
• PR 2 - hog work
  • Add a system to synchronously import bytecode/files/code/functions in Hog
  • Pass loaded code around in the vmState
  • Tests
  • Backport to Python HogVM
  • Release new version
• PR 3 - frontend work
  • Create a "messaging" menu item with "broadcasts" and "providers"
  • Refactor "destinationsLogic" and "DestinationsTable" into something more abstract and use it for all 3
  • Separate view/edit pages for the new function types
  • Show matching persons
  • Custom testing data
  • Logs and metrics pages
• PR 4 - cdp work & broadcasting
  • Hardcode just one import source, provider/email, which exports sendEmail() (runs the team's email provider's bytecode)
  • Can invoke/tests both providers and broadcasts
  • Make a query to fetch all persons
  • Execute the broadcast for each person
  • Tests for all of the above

Future work not yet in this PR:

• Broadcast in a background job (will add something in PR 5)
• Cancellation/retry insurance for broadcasts
• Capture/log events when sending mails (will add in PR 3)
• Layout templates and saved defaults for e-mails
• Metrics about sent/delivered/bounced
• Actual export fun sendEmail(){} syntax instead of the current return { 'sendEmail': sendEmail }

The work above sets the groundwork for workflows (totally out of scope right now):

• Each node in a workflow can be its own Hog function.
• The workflow itself will be another Hog function.
• All of these can be imported in the plugin server at will (import('fun/{id}'))
• When we import some code, we cache it in the VM state which gets passed around (mostly to make sure when we return from fetch we're running the right thing)
• This means that in the code sleep(60sec); importAndRun1(), what we import can change within the 60 seconds, and we'll still take the latest version when running importAndRun1(), however importAndRun1(); sleep(60); importAndRun1() will only import the code once, and reuse it a second time.
• We can be super flexible here and if we want each node to be "always latest", we can just discard the bytecode or reimport as needed (e.g. when it's out of the call stack and a new version is available)

[posthog-bot]

📸 UI snapshots have been updated

4 snapshot changes in total. 0 added, 4 modified, 0 deleted:

• chromium: 0 added, 4 modified, 0 deleted (diff for shard 1)
• webkit: 0 added, 0 modified, 0 deleted

Triggered by this commit.

👉 Review this PR's diff of snapshots.

[github-actions]

Size Change: +372 B (+0.03%)

Total Size: 1.15 MB

ℹ️ View Unchanged

Filename	Size	Change
frontend/dist/toolbar.js	1.15 MB	+372 B (+0.03%)

_{compressed-size-action}

[posthog-bot]

📸 UI snapshots have been updated

2 snapshot changes in total. 0 added, 2 modified, 0 deleted:

• chromium: 0 added, 2 modified, 0 deleted (diff for shard 1)
• webkit: 0 added, 0 modified, 0 deleted

Triggered by this commit.

👉 Review this PR's diff of snapshots.

[posthog-bot]

📸 UI snapshots have been updated

2 snapshot changes in total. 0 added, 2 modified, 0 deleted:

• chromium: 0 added, 2 modified, 0 deleted (diff for shard 1)
• webkit: 0 added, 0 modified, 0 deleted

Triggered by this commit.

👉 Review this PR's diff of snapshots.

[posthog-bot]

📸 UI snapshots have been updated

2 snapshot changes in total. 0 added, 2 modified, 0 deleted:

• chromium: 0 added, 2 modified, 0 deleted (diff for shard 1)
• webkit: 0 added, 0 modified, 0 deleted

Triggered by this commit.

👉 Review this PR's diff of snapshots.

[posthog-bot]

📸 UI snapshots have been updated

2 snapshot changes in total. 0 added, 2 modified, 0 deleted:

• chromium: 0 added, 2 modified, 0 deleted (diff for shard 1)
• webkit: 0 added, 0 modified, 0 deleted

Triggered by this commit.

👉 Review this PR's diff of snapshots.

[posthog-bot]

📸 UI snapshots have been updated

2 snapshot changes in total. 0 added, 2 modified, 0 deleted:

• chromium: 0 added, 2 modified, 0 deleted (diff for shard 1)
• webkit: 0 added, 0 modified, 0 deleted

Triggered by this commit.

👉 Review this PR's diff of snapshots.

[posthog-bot]

📸 UI snapshots have been updated

2 snapshot changes in total. 0 added, 2 modified, 0 deleted:

• chromium: 0 added, 2 modified, 0 deleted (diff for shard 1)
• webkit: 0 added, 0 modified, 0 deleted

Triggered by this commit.

👉 Review this PR's diff of snapshots.

[posthog-bot]

📸 UI snapshots have been updated

2 snapshot changes in total. 0 added, 2 modified, 0 deleted:

• chromium: 0 added, 2 modified, 0 deleted (diff for shard 1)
• webkit: 0 added, 0 modified, 0 deleted

Triggered by this commit.

👉 Review this PR's diff of snapshots.

[mariusandra]

All code here has been moved into 4 smaller PRs. Three of them have been merged. I'll close this and move the other notes into a new issue/epic.