Sprint November 27 - December 8

[mariusandra]

Global Sprint Planning

3 things that might take us down

1. Clickhouse EU/cloud -> spinning up our own cluster
2. Clickhouse visibility - had an incident, and then we had to sudo command in -> educating everyone on housewatch to get logs
3. Gunicorn -> NGINX is working, hoping to roll out tomorrow in EU

Team sprint planning

For your team sprint planning copy this template into a comment below for each team.

# Team ___

**Support hero:** ___

## Retro

<!-- Grab the high and low priority items from last time and add whether that item was completed or not -->

- 

## Hang over items from previous sprint

<!-- For each item, decide to re-prioritise (and add below) or deprioritise -->

- Item 1. prioritised/deprioritise

## OKR

1. OKR, status (red/yellow/green) and action points if yellow/red


### High priority

-

### Low priority / side quests

-

[mariusandra]

Project Query Engine

Support hero: Marius 1st week, Michael 2nd week.

Retro

• Async slow lane
  • 🟢 In testing, can be unflagged for select customers. Will add metrics and roll out.
• Launch trends & lifecycle
  • 🟡 On track to release both for our team by end of week.
• Start building BI features
  • 🔴 Didn't start, will roll over.

Hang over items from previous sprint

Insights and BI.

OKR

Project "Finish the darned Query Engine" (Tom, Julian, Marius)

• Finish insight HogQL conversion.
  • 🟡 We're on track this year to convert everything except funnels
  • Trends - 90% (~5 failing tests?)
  • Lifecycle - 98% (missing PoE v2 in HogQL to verify data integrity)
  • Retention - 10% (started)
  • Funnels, paths, stickiness - 0% (not started)
• Support non-event data sources on HogQL insights (BI tools for data warehouse).
  • 🟡 Focus for next sprint. Will update OKR after.
• Build tools to measure HogQL query performance, and catch regressions before they hit users.
  • 🔴 We haven't been able to focus on this. Fixing regressions one off as they're reported.

Next sprint

High priority

• HogQL BI features @Gilbert09
  • Trends graphs from any HogQL query
  • Eventual goal: replicating the go/revenue dashboard in PostHog
• Convert retention insight @webjunkie and launch trends/lifecycle @Gilbert09 @mariusandra

Low priority / side quests

• Performance work as it comes in.

[benjackwhite]

Team No-im-the-ceo! Replay

Support hero: @pauldambra

Retro

• @pauldambra can't wait till 3000 is finished to have full-stack team mates again
• @pauldambra easy to break things e.g. cache busting posthog-js assets
• @benjackwhite back to a Replay team focus would be great
• @benjackwhite wondering what the future of NoteForce3000 is... 🤔
• @daibhin agrees with all the above
• @daibhin some of the support stuff is really complex - especially with replay issues
• @benjackwhite - at some point, we need to get a handle on our own rrweb deployment / local dev tools for debugging
• @marandaneto building a mobile session replay based on RRWeb not knowing much of RRWeb is tricky, Ben/Paul are always sharing their expertise with stuff that is not documented/publicly known about RRWeb internals, learning bit by bit slowly.
• @marandaneto trying to reuse as much as possible from RRWeb on the ingestion/player side makes it a bit trickier since the concepts don't match 1:1, but we don't want to reinvent the wheel either.
• @marandaneto learned how to debug missing events with Tiina's help, that was cool.

Hang over items from previous sprint

• 🟢 Investigate whether canvas as a target makes sense for iOS (PoC) @marandaneto
  • Nope, it's not possible to intercept and delegate drawing cmds as on Android, it's low-level C-APIs.
• 🟢 Figure out Canvas player - either within RRweb or our own player @pauldambra
  • No it doesn't, started our own player spec
  • Still need to figure out our usage in posthog (EE or otherwise)

OKR

• 📱 Mobile Replay 🟡
  • We're doing okay but the solution we need is clearly more effort so realistically Android would already be a stretch

High priority

• Collect and playback wireframe recordings in PostHog player @marandaneto @pauldambra

Low priority / side quests

• Network response payloads release @pauldambra

[EDsCODE]

Team Data Warehouse

OKR Q4

Objective

PostHog users can effortlessly bring their business data to PostHog and analyze it 🟢

• Key Results:
  • Be able to import data into data warehouse in one click from most popular SaaS tools (X, Y, Z connectors built)
  • 5 happy users/teams who have imported external data
  • Data Warehouse queries are compatible with insights hogQL

PostHog is moved over to whatever CH infrastructure we will be using for the next 1 year (at least) 🟡

• Key Results:
  • We are either on new infrastructure for EU that we host or on ClickHouse Cloud
  • Queries for large customers are performing faster
  • We can scale more easily (without James bespoke effort)

Retro

Product

• get billing deployed
• have a logging and monitoring dashboard setup
• stream selector interface

Infra

• Complete required performance improvements with ClickHouse.cloud
• Start back on migrations to new CH world

High priority

Product

• Deploy either airbyte system or inhouse system (RFC here

Infra

• Begin TF + Ansible configs for EU on prem cluster
• Continue performance improvements with CH.cloud's help
• More investigation and experiments with Projections

[benjackwhite]

Project Noteforce3000

Hang over items from previous sprint

High priority

• Roll out internally and follow up on any feedback / issues
  • #posthog-3000 slack channel (and then we make sure it is in the list on PostHog 3000 Revival Megaissue #18089)
• Get docs and screenshots up to date
• Roll experiment for everybody (end of sprint)

The remaining details...

see #18089

[robbie-c]

Team Web Analytics

Support hero: @robbie-c

Hang over items from previous sprint

• Searchable pathnames, better pathname widget - deprioritize the searchable pathnames part

OKR

🟢 Build the dashboards to match 80% of functionality - work underway and making good progress
⚪ Start working towards the architecture to scale to a general release - not started

High priority

• Continue working through Web Analytics Public Beta To-Do List #18547
• e.g. improve the pathname widget
• merge feat: Change default storage to localStorage+cookie, store client session props posthog-js#906 in a few days, after we are confident that changing the default on posthog.com didn't break anything.

[neilkakkar]

Team Feature Success

Support hero: @jurajmajerik

Days off:
Juraj: 0 days
Neil: 7 days
Li: 0 days

Retro

• Generate cohort of all current users for whom flag is enabled: easy access to all exposed users (@neilkakkar ) -> side quest became the main quest, & main quest is deferred
• Scheduled flag changes - @jurajmajerik - we didn't consider copying cohorts in flag environments, which extended it quite a bit into the current sprint, and scheduled rollout will do a scheduled rollover.
• Survey fixes & user interviews - @liyiy
  • reorder survey questions once created
  • Next/submit buttons per survey
  • open ended choices
  • Thank you timer is optional
  • inactive surveys disable flags

Hang over items from previous sprint

---

OKRs

1. Get 5 reference customers for surveys
2. Get reference customers for feature flags and experiments

High priority

• Survey permanent feedback widget - @liyiy
• Survey quota limiting - @liyiy
• Scheduled flag changes (and potentially everything else): Automated/scheduled rollouts of feature flags #13842 - @jurajmajerik
• Make progress on SDK updates for relative date targeting & targeting by distinct id - @neilkakkar

Low priority / side quests

[ellie]

Team Infra 🚀

Support hero: @ellie

High priority

• roll out tracing in prod-eu, and use opentelemetry collector for performance @ellie
• first Python tracing integration setup @ellie
• Work on kafka switch document (events + sessions) @danielxnj
• Finalize eu bigint type change @danielxnj
• Nginx unit rollout @frankh
• Separate ingress endpoint that skips cloudfront @frankh

Low priority / side quests

[xrdt]

Team Growth

Support hero: Raquel, then Bianca

Retro

• Make sure unsub survey response is only sent if there is a response @xrdt fix: Only send unsubscribe survey if response is non-empty #18431
• Roll out new onboarding to everyone @raquelmsmith
  • add the project switcher to the /products screen
  • make mobile friendly
  • make sure people who have existing projects aren't suddenly redirected
    • use snipped_onboarding_completed
• Launch data warehouse pricing https://github.com/PostHog/billing/issues/378
  • Automatically set $500 limit on subscribe to data_warehouse @xrdt
• Make sure people aren't receiving billing limit emails every day @xrdt
• Deal with the incident @raquelmsmith
• Change MRR per product reporting to be calculated based on usage, not invoices
• Store an email for every billing customer, not just the subscribed ones
  • PRs open, will finish shortly
  • https://github.com/PostHog/billing/pull/405
  • feat: Update the organization customer email upon signup #18724
• Website pricing page update Improved pricing UI for both website and product posthog.com#7056 @xrdt https://github.com/PostHog/billing/pull/398

OKR

• 🟡 Objective 1: Better understand what leads to revenue
  • Reasoning: We're still a bit fuzzy about what metrics are leading indicators for revenue.
  • KR: A company scorecard with actionable insights that help us understand where to invest
  • KR: An understanding (doc or dashboard) of how people use multiple products
• 🟡 Objective 2: Continue experimenting with onboarding and activation metrics
  • KR: Product-specific onboarding performs better than the old onboarding flow ✅
  • KR: 2 experiments for Product Analytics & Session Replay activation metrics each
  • KR: figure out how to track activation
    • Pin first step in funnel

This sprint

High priority

• Launch data warehouse pricing https://github.com/PostHog/billing/issues/378 ...delayed by airbyte
• Alert people in UI when an initial billing limit has been sent @xrdt feat: show alert when we automatically apply billing limit upon subscribe #18668
• Look into pinning first step in funnel @xrdt. The product analytics team is working a major insights refactor to make them HogQL backed so we are delaying any feature implementation. Neil gave a number of suggestions on how we might get around this with the current PostHog tooling. There are a number of pitfalls to his suggestions so we settled on a HogQL based solution for now. Tracking activation is really quite a non-trivial task.
• Reinstate recovery billing feature https://github.com/PostHog/billing/pull/409 @raquelmsmith
• Finish invoice split for upcoming invoices @raquelmsmith
• Change MRR per product reporting to be calculated based on usage, not invoice line items @raquelmsmith Discussing here
• Onboarding config step for product analytics and session replay @raquelmsmith feat(onboarding): add product config screen #18720
• Review 3000 onboarding and signup pages fix: Some issues with onboarding for 3000 #18746 @raquelmsmith
• org level info on users feat: add membership level to user / person data #18332 @raquelmsmith

Low priority / side quests

• @xrdt Figure out how to test unsubscribe survey events in cypress feat: Test that 'survey sent' Event is Triggered when Submitting Non-Empty Unsubscribe Survey #18805
• Think about how to better enforce API contracts between posthog FE and billing service @xrdt ?? Should we use graphql?
• Getting logged out when subscribing locally only @xrdt. IDK why I'm being shown as anonymousUser but setting the AUTO_LOGIN env var resolves the surface level issue of having to manually log in every time I try to upgrade.
• It's not quite that simple to ensure that emails are case-insensitive...
• Unset billing limit upon unsubscribe @xrdt. Decided that this is probably a :sparks_joy: moment and can be left as is.

[davemurphysf]

Team Pipeline

Support hero: @tkaemming then @xvello

Retro

• 🟡 Capture-rs @xvello

  • Rolled out to 36k events/min yesterday with good performance, but rolled back for now while we investigate some edge cases

• ✅ Batch Exports @tomasfarias

  • Finished all destination migrations 🎉
  • Added tracing & metrics to batch exports (w/ @bretthoerner)
  • All destinations are now running Python async to prevent blocking on IO.
  • Kill exportEvents & createBuffer @tiina303

• 🔴 PoE

  • Was originally planning on using new tables, now accommodating that marketing wants to keep using the existing implementation while we move to the new async overrides approach
  • @tkaemming helped out with the incident with UA plugin disabled

Hang over items from previous sprint

OKR

1. OKR, status (red/yellow/green) and action points if yellow/red

• 🟢 keep the lights on @Team-pipeline
  • 🟢 Redis split done on both regions, we improve our failure isolation and can increase our Redis use without fear
  • 🟡 Person processing is still the bottleneck for ingestion (MX traffic spike leading to 50M events lag on overflow). We need to continue investing on it
• ✅ Batch exports migrations complete 🎉 - @tomasfarias
• Webhooks v2 @tiina303/@davemurphysf
  • 🟡 New delivery system
    • kickoff this week with @ellie, @tomasfarias & @davemurphysf
  • 🟢 New interface for webhooks and onevent apps
    • ingestion handles composeWebhooks (i.e. backend integration with new delivery system is ready, though with very limited testing coverage and all onEvent plugins still need to be updated to composeWebhooks) @tiina303
• 🟡 PoE @tkaemming
  • plan is to get write path person_overrides ready EOY (will likely not finish squash nor backfill)

High priority

• Capture-rs @xvello
  • 100% posthog-js rollout
  • instrument all the special-cases and kludges in capture-py to check whether they need porting
• new pipeline UI ready to roll out to users @tiina303
• elements_chain 100% rollout to all new posthog-js version @davemurphysf
• Book venue/lodging for Jan 22-26 offsite @davemurphysf
• PoE write path person_overrides for our team @tkaemming
• Webhooks delivery system @davemurphysf: scaffolding @ellie & database schema @tomasfarias

Low priority / side quests

• capture-rs: end2end probe revamp @xvello
• Make progress on CDP pricing RFC @davemurphysf
• Batch exports autoscaling @bretthoerner / @tomasfarias