feat(product-assistant): better failover for the ReAct agent

[skoob13]

Problem

Since the ReAct agent doesn't use the structured output, we have to parse manually the response. There are three potential pitfalls:

• The agent can completely omit the action block.
• The agent can output a malformed JSON, e.g., missing action or action_input fields.
• The agent can hallucinate and output non-existing tools or use a wrong schema for the tool input.

For the first two cases, LangChain provides a parser. However, the parser is very basic and doesn't guide the agent on what it should do when the output's format is incorrect. We frequently are in situations when the agent does 5 to 10 iterations until it corrects itself.

The improved guidelines should help to recover after the first unsuccessful attempt.

Example conversation where the agent generated a final answer on the first run without formatting

Improve the previously generated plan based on the feedback: filter out the us from all events

Thought:
To filter out the US from all events, I need to apply the same property filter to each event in the series.

Events:

• $pageview
  • math operation: total count
  • property filter:
    • entity: event
    • property name: $geoip_country_code
    • property type: String
    • operator: doesn't equal
    • property value: US
• signed_up
  • math operation: total count
  • property filter:
    • entity: event
    • property name: $geoip_country_code
    • property type: String
    • operator: doesn't equal
    • property value: US
• $identify
  • math operation: total count
  • property filter:
    • entity: event
    • property name: $geoip_country_code
    • property type: String
    • operator: doesn't equal
    • property value: US

Formula:
A/(B+C), where A is the $pageview event, B is the signed_up event, and C is the $identify event.
Action: I didn't output the Action: block.

Observation:
Your previous answer didn't output the Action: block. You must always follow the format described in the system prompt.

Thought:
Action:

{
  "action": "final_answer",
  "action_input": "Events:\n- $pageview\n    - math operation: total count\n    - property filter:\n        - entity: event\n        - property name: $geoip_country_code\n        - property type: String\n        - operator: doesn't equal\n        - property value: US\n- signed_up\n    - math operation: total count\n    - property filter:\n        - entity: event\n        - property name: $geoip_country_code\n        - property type: String\n        - operator: doesn't equal\n        - property value: US\n- $identify\n    - math operation: total count\n    - property filter:\n        - entity: event\n        - property name: $geoip_country_code\n        - property type: String\n        - operator: doesn't equal\n        - property value: US\n\nFormula:\n`A/(B+C)`, where `A` is the $pageview event, `B` is the signed_up event, and `C` is the $identify event."
}

Changes

Added detailed observations based on the improved parsing of the output:

• When the agent omits the Action: block, we now append the block to the prompt and output an observation that the output had a wrong format.
• When the agent outputs a malformed JSON, add the observation to remind that it must follow the format instruction.
• When the agent outputs a non-existing tool or tool's schema that doesn't match the input, return the descriptive observation describing the Pydantic exception.

Does this work well for both Cloud and self-hosted?

N/A

How did you test this code?

Unit tests, manual testing, Langfuse traces.