feat(product-assistant): a human in the loop for the taxonomy agent #26767

skoob13 · 2024-12-09T17:41:10Z

Problem

If the request is ambiguous or the taxonomy doesn't provide a clear picture, we should ask the user to clarify their request.

Demo

Changes

Add a checkpointer implementation on Django for LangGraph.
Threads are now stored in the database (+ checkpoints, checkpoint writes, and checkpoint blobs).
Implement a human in the loop for the taxonomy agent.
Refactor how we restore a conversation since human messages can now appear in any order.
Move the assistant to its endpoint.
Use Pydantic for the state to simplify further state serialization.
Add more tests.

Further work:

Improve serialization of checkpoints. I've decided to split these PRs because this appears to be more complicated than I expected.
Improve frontend UI, so users can easily search and add specific events to a prompt.
Add eval tests.

Does this work well for both Cloud and self-hosted?

No

How did you test this code?

Manual testing, Storybook, more unit tests.

…uman-in-the-loop

…thog into feat/agent-human-in-the-loop

skoob13 · 2024-12-12T18:58:24Z

ee/models/assistant.py

+
+class Conversation(UUIDModel):
+    user = models.ForeignKey(User, on_delete=models.CASCADE)
+    team = models.ForeignKey(Team, on_delete=models.CASCADE)


Now, I think that there should be a ref to the project because property values can differ. @Twixes what do you think?

We can't really have a meaningful analysis shared between environments because of data differing, so team-level (aka environment-level) is the right way here

posthog-bot · 2024-12-12T18:59:02Z

📸 UI snapshots have been updated

10 snapshot changes in total. 0 added, 10 modified, 0 deleted:

chromium: 0 added, 10 modified, 0 deleted (diff for shard 1)
webkit: 0 added, 0 modified, 0 deleted

Triggered by this commit.

👉 Review this PR's diff of snapshots.

…thog into feat/agent-human-in-the-loop

…uman-in-the-loop

skoob13 · 2024-12-16T13:45:00Z

The CI fails because there is a bug in migrations tests, which I've fixed in #26928.

…uman-in-the-loop

Twixes

A mammoth PR so I might have missed something, but overall looks good, with a few comments!

Twixes · 2024-12-17T00:59:47Z

ee/models/assistant.py

+
+class Conversation(UUIDModel):
+    user = models.ForeignKey(User, on_delete=models.CASCADE)
+    team = models.ForeignKey(Team, on_delete=models.CASCADE)


We can't really have a meaningful analysis shared between environments because of data differing, so team-level (aka environment-level) is the right way here

frontend/src/types.ts

frontend/__snapshots__/scenes-app-max-ai--welcome-loading-suggestions--dark.png

ee/urls.py

ee/hogai/utils/types.py

…thog into feat/agent-human-in-the-loop

…uman-in-the-loop

posthog-bot · 2024-12-17T11:19:59Z

📸 UI snapshots have been updated

2 snapshot changes in total. 0 added, 2 modified, 0 deleted:

chromium: 0 added, 2 modified, 0 deleted (diff for shard 1)
webkit: 0 added, 0 modified, 0 deleted

Triggered by this commit.

👉 Review this PR's diff of snapshots.

Twixes

Let's get this in

Twixes · 2024-12-17T13:01:39Z

ee/hogai/django_checkpoint/checkpointer.py

Not important to shipping, but could be useful in terms of maintainability: Would it make sense to split this out into its own PyPI package, langgraph_checkpoint_django? This part seems pretty much abstract, independent from PostHog specifics, could be useful for others building on Django too

Twixes

Let's get this in

Twixes · 2024-12-17T13:30:13Z

dashboard.cy.ts has failed 4 times, so something seems to be up 🤔 @skoob13

sentry-io · 2024-12-19T18:28:08Z

Suspect Issues

This pull request was deployed and Sentry observed the following issues:

‼️ IntegrityError: duplicate key value violates unique constraint "unique_checkpoint_write" /api/environments/{parent_lookup_team_id}/conve... View Issue
‼️ IntegrityError: duplicate key value violates unique constraint "unique_checkpoint_write" ee.hogai.django_checkpoint.checkpointer in put_... View Issue
‼️ IntegrityError: null value in column "checkpoint_ns" of relation "ee_conversationcheckpointblob" violates not-nul... ee.hogai.django_checkpoint.checkpointer in put View Issue

_{Did you find this useful? React with a 👍 or 👎}

skoob13 added 30 commits December 4, 2024 16:31

feat: in-memory human in the loop

dcb1bca

feat: force the agent to ask for clarification

7ef5260

chore: bump deps

d26ad87

feat: checkpointer v1

6c686f5

fix: tests

ac9491b

test: more tests

80d39e5

feat: thread retrieval

1b80601

fix: debug

80222a7

fix: lock avoiding race conditions

a38229a

fix: lock order

c2a3c62

fix: refactor locking

88614f9

test: test interruptions and time travelling

bad9d66

refactor: move some methods on the model level

288fea1

test: make sure a checkpoint is copied

207b832

Merge branch 'master' of github.com:PostHog/posthog into feat/agent-h…

cb45539

…uman-in-the-loop

chore: migrations

8ebab16

fix: mypy issues

6c40f48

refactor: restoring state

c21d4e8

feat: split human in the loop prompt

8f2c583

test: writes

f6a60dd

refactor: message history reconstruction

221308a

fix: message restoration for taxonomy agent

273c0c9

feat: track initiators

8bf9c19

Merge branch 'master' of github.com:PostHog/posthog into feat/agent-h…

1dafce5

…uman-in-the-loop

fix: rename filter conversation

1b5695f

fix: test failures

32b783d

fix: utils test

a5d018d

fix: assistant tests

ce766d4

fix: tests

6ee0a5c

feat: assign id to human messages

55a1244

skoob13 and others added 2 commits December 12, 2024 19:41

Merge branch 'feat/agent-human-in-the-loop' of github.com:PostHog/pos…

1e055d1

…thog into feat/agent-human-in-the-loop

Update UI snapshots for chromium (1)

582f175

skoob13 commented Dec 12, 2024

View reviewed changes

skoob13 requested a review from Twixes December 12, 2024 18:59

skoob13 marked this pull request as ready for review December 12, 2024 18:59

github-actions bot and others added 5 commits December 12, 2024 19:00

Update query snapshots

1ef8667

fix: eval tests

d5c7bbd

Merge branch 'feat/agent-human-in-the-loop' of github.com:PostHog/pos…

e52dd3c

…thog into feat/agent-human-in-the-loop

Merge branch 'master' of github.com:PostHog/posthog into feat/agent-h…

ed78866

…uman-in-the-loop

fix: unify tests

0e3a01a

skoob13 mentioned this pull request Dec 16, 2024

fix(tests): restore the latest migration after migration tests #26928

Merged

skoob13 and others added 2 commits December 16, 2024 16:46

Merge branch 'master' of github.com:PostHog/posthog into feat/agent-h…

7995131

…uman-in-the-loop

Merge branch 'master' into feat/agent-human-in-the-loop

c75a857

Twixes reviewed Dec 17, 2024

View reviewed changes

skoob13 and others added 5 commits December 17, 2024 11:06

Merge branch 'feat/agent-human-in-the-loop' of github.com:PostHog/pos…

29818b1

…thog into feat/agent-human-in-the-loop

fix: review comments

adea361

fix: broke a story accidentally

723a123

Merge branch 'master' of github.com:PostHog/posthog into feat/agent-h…

eb29ef4

…uman-in-the-loop

Update UI snapshots for chromium (1)

76e80bb

skoob13 requested a review from Twixes December 17, 2024 11:50

Twixes approved these changes Dec 17, 2024

View reviewed changes

This comment was marked as duplicate.

Sign in to view

skoob13 merged commit 738fa46 into master Dec 17, 2024
96 checks passed

skoob13 deleted the feat/agent-human-in-the-loop branch December 17, 2024 16:28

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat(product-assistant): a human in the loop for the taxonomy agent #26767

feat(product-assistant): a human in the loop for the taxonomy agent #26767

skoob13 commented Dec 9, 2024 •

edited

Loading

skoob13 Dec 12, 2024

Twixes Dec 17, 2024

posthog-bot commented Dec 12, 2024

skoob13 commented Dec 16, 2024

Twixes left a comment

Twixes Dec 17, 2024

posthog-bot commented Dec 17, 2024

Twixes left a comment

Twixes Dec 17, 2024 •

edited

Loading

Twixes left a comment

This comment was marked as duplicate.

Twixes commented Dec 17, 2024

sentry-io bot commented Dec 19, 2024 •

edited

Loading

feat(product-assistant): a human in the loop for the taxonomy agent #26767

feat(product-assistant): a human in the loop for the taxonomy agent #26767

Conversation

skoob13 commented Dec 9, 2024 • edited Loading

Problem

Changes

Does this work well for both Cloud and self-hosted?

How did you test this code?

skoob13 Dec 12, 2024

Choose a reason for hiding this comment

Twixes Dec 17, 2024

Choose a reason for hiding this comment

posthog-bot commented Dec 12, 2024

📸 UI snapshots have been updated

skoob13 commented Dec 16, 2024

Twixes left a comment

Choose a reason for hiding this comment

Twixes Dec 17, 2024

Choose a reason for hiding this comment

posthog-bot commented Dec 17, 2024

📸 UI snapshots have been updated

Twixes left a comment

Choose a reason for hiding this comment

Twixes Dec 17, 2024 • edited Loading

Choose a reason for hiding this comment

Twixes left a comment

Choose a reason for hiding this comment

This comment was marked as duplicate.

Twixes commented Dec 17, 2024

sentry-io bot commented Dec 19, 2024 • edited Loading

Suspect Issues

skoob13 commented Dec 9, 2024 •

edited

Loading

Twixes Dec 17, 2024 •

edited

Loading

sentry-io bot commented Dec 19, 2024 •

edited

Loading