Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat(max): Make summary more actionable and self-aware #27005

Merged
merged 9 commits into from
Dec 20, 2024

Conversation

Twixes
Copy link
Member

@Twixes Twixes commented Dec 18, 2024

Problem

Agent doesn't always recognize it hasn't really answered the question.

Changes

This tweaks the summarizer to be more self-aware and self-critical.

How did you test this code?

TODO!

Copy link
Member

@rafaeelaudibert rafaeelaudibert left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Wondering if we've ever want to try something that helps us version prompts? :)
Or maybe use a Posthog tm experiment?

Copy link
Contributor

Size Change: 0 B

Total Size: 1.11 MB

ℹ️ View Unchanged
Filename Size
frontend/dist/toolbar.js 1.11 MB

compressed-size-action

@Twixes Twixes force-pushed the assistant-self-aware branch from 02a7d0c to 6ab4d88 Compare December 20, 2024 12:18
@Twixes Twixes requested a review from skoob13 December 20, 2024 12:25
Comment on lines 163 to 164
ports:
- 3000:3000
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This was supposed to quickly solve a lot of pointless errors in logs (this is what I asked Frank to review here), but apparently there's already a PR (#27008) and a major discussion around proper routing of capture has ensued (Slack) – so reverted here

@@ -1,17 +1,24 @@
SUMMARIZER_SYSTEM_PROMPT = """
Act as an expert product manager. Your task is to summarize query results in a a concise way.
Act as an expert product manager. Your task is to help the user build a successful product and business. Remember, you're a hedeghog.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why a hedgehog role?😆 Do you want to force it to answer as a hedgehog?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It's more setting the ground for the pure conversation skill, but we definitely will want to be aware it's Max the Hedgehog rather than "expert product manager, an OpenAI large language model"

@@ -100,5 +104,6 @@ def _construct_messages(self, state: AssistantState) -> list[tuple[str, str]]:
elif isinstance(message, AssistantMessage):
conversation.append(("assistant", message.content))

conversation.append(("human", SUMMARIZER_INSTRUCTION_PROMPT))
conversation.append(("system", SUMMARIZER_RESULTS_PROMPT))
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Two system prompts? It's better to keep a single system prompt.

Copy link
Member Author

@Twixes Twixes Dec 20, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Was playing with a different approach, but actually better to consolidate. Pushed


If there are interesting trends or anomalies, succintly point them out. I can see the chart, so don't just describe all of it.
The answer needs to be high-impact, no more than a few sentences. Bullets can improve clarity of action points.
Use Silicon Valley lingo. Be informal, but without fluff. NEVER USE TITLE CASE, even in headings. Our style is sentence case always.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why didn't you include the language style instructions in the first system prompt? Or you can concatenate the second system prompt and the user prompt.

@Twixes Twixes requested a review from skoob13 December 20, 2024 17:11
Copy link
Contributor

@skoob13 skoob13 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@Twixes Twixes enabled auto-merge (squash) December 20, 2024 17:39
@Twixes Twixes merged commit c10a386 into master Dec 20, 2024
91 checks passed
@Twixes Twixes deleted the assistant-self-aware branch December 20, 2024 18:03
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants