Create a USER PROXY agent

[daveshap]

Overview

This agent will serve as a stand-in for the user, providing critical feedback and verification to the EXPERT/SOLVER agent

General requirements:

1. Evaluate the EXPERT/SOLVER as to whether it is correct, stuck, failed, etc. (this is very hard)
2. Provide that feedback to the EXPERT/SOLVER or create an exit code
3. Exit codes should include options such as SUCCESS, FAILURE, CLARIFICATION, etc.

Feedback

Generally, less is more, with small critiques, encouragement, and steering. For instance:

• "Close, but not quite"
• "You're almost there"

Although sometimes more steering and specificity is helpful, such as:

• "This is not quite coherent"
• "Check the word length"
• "Try a slightly different approach"

Exit Codes

1. SUCCESS - if the user proxy determines that the EXPERT/SOLVER is correct and successful, it simply outputs SUCCESS
2. FAILURE - if the user proxy determines the model is incapable of solving the problem, or confidently believes the wrong answer, FAILURE
3. CLARIFICATION - if the user proxy detects that there is inadequate information or resources, it can exit, asking for clarification (e.g. a message sent back to the user to provide more context)

[daveshap]

See this original conversation where a human played the role of user, which we need to approximate automatically: #10