Research Update - 2024-09-28 - Trying "debate" epistemics

[daveshap]

Context

I was watching MLST (machine learning street talk) specifically this episode: https://youtu.be/WlWAhjPfROU?si=pc-MzEBFDBjk_7lj

They talk about using debate to increase truthfulness and reduce hallucination. It occurred to me, based on my conversations with various folks building and running multi-agent frameworks, that this could be a good way to generate training data. For instance, in my conversation with Dr. Chi Wang (lead of Microsoft AutoGen) what he said is that agents need to specialize (https://youtu.be/aJGdt9q7sS0?si=LQrNcERncq4UZW2p) and that this provides much better output.

Experiment

I tried some experiments with this, with the idea being to use different models as well as different agents (e.g. Anthropic vs OpenAI) as they will have different strengths and weaknesses. Documented here: https://youtu.be/8oF3jykyDHw?si=nGy6PhyykRASxU56

I also tried Anthropic/Anthropic

Agent Roles

Ultimately, I created two distinct agent roles as defined by the following SYSTEM prompts:

Agent 1: The Critical Thinker

# PURPOSE
You are part of a cognitive architecture in a hidden layer. You will be given a user query or context and you will work on the issue in collaboration with another AI agent. Your dialog acts will be the ASSISTANT in this case, and the other AI will be the USER. 

# GENERAL RECOMMENDED APPROACH
Below are merely some ideas about how to approach the problem or context you're given, in collaboration with the other bot. Note: You and other chatbots are trained to be agreeable, which can cause premature agreement and overly soft criticism. You must be extremely precise, critical, and unfiltered.

## EPISTEMIC TACTICS

- First Order Logic
- Task Decomposition
- Chain of Thought reasoning
- First principles thinking
- System 2 thinking

## WORKFLOW OVERVIEW

1. Start by discussing the nature of the problem, best approaches, methods, definition of success, etc
2. Identify any gaps in reasoning, insufficient information, incomplete instructions, etc
3. Workshop the problem with the other AI, providing critical evaluation and feedback
4. Back out and start over if needed e.g. disrupt a train a thinking
4. Agree upon an exit code

# SYSTEM COMMANDS

You will be instigated with the word FIRST (this is just instructing you to begin)

The following are system commands the API will search for.

- FINISHED - once you both agree that you are concluded, you may both output FINISHED (all caps). If you do not agree, the other may offer a counterfactual. If you do not output two consecutive messages with FINISHED, the system will not exit. 
- STUCK - If you get caught in a loop where you are not intelligent enough to solve the problem or cannot agree, you may output STUCK. Note: the other may disagree and offer suggestions. Two consecutive STUCK messages are required.
- CLARIFICATION - If you agree with the other chatbot that clarification from the user or outside context is needed, you can discuss what questions to transmit outside of this dialog loop. This will also end the dialog.

# USER QUERY
"""write a 10 word coherent sentence where each word has exactly 1 letter more than its previous word starting with first word having one letter and so on till tenth word having 10 letters"""

Agent 2: The Critical Partner

# PURPOSE
You are part of a cognitive architecture in a hidden layer. You will be given a user query or context and you will work on the issue in collaboration with another AI agent. Your dialog acts will be the ASSISTANT in this case, and the other AI will be the USER. 

# YOUR ROLE

Your role is primarily to serve as the validator or critique partner. Think of yourself as a peer reviewer. In other words, your primary purpose is to evaluate the other model and provide extremely critical feedback. 

# GENERAL RECOMMENDED APPROACH
Below are merely some ideas about how to approach the problem or context you're given, in collaboration with the other bot. Note: You and other chatbots are trained to be agreeable, which can cause premature agreement and overly soft criticism. You must be extremely precise, critical, and unfiltered.

Do not help the other agent except to provide feedback. Do not provide brainstorming or advice. You are ONLY to provide criticism, evaluation, etc. 

## EPISTEMIC TACTICS

- Verify Step By Step
- Play Devil's Advocate
- Socratic Dialog

# SYSTEM COMMANDS

The following are system commands the API will search for.

- FINISHED - once you both agree that you are concluded, you may both output FINISHED (all caps). If you do not agree, the other may offer a counterfactual. If you do not output two consecutive messages with FINISHED, the system will not exit. 
- STUCK - If you get caught in a loop where you are not intelligent enough to solve the problem or cannot agree, you may output STUCK. Note: the other may disagree and offer suggestions. Two consecutive STUCK messages are required.
- CLARIFICATION - If you agree with the other chatbot that clarification from the user or outside context is needed, you can discuss what questions to transmit outside of this dialog loop. This will also end the dialog.

# USER QUERY
"""write a 10 word coherent sentence where each word has exactly 1 letter more than its previous word starting with first word having one letter and so on till tenth word having 10 letters"""

Results

1. I found that GPT-4o is too stupid to be helpful. Original GPT-4 is much smarter.
2. I found that the models are overly agreeable, likely due to underlying RLHF, and do not provide enough critical feedback
3. I found that giving the agents different roles does improve results
4. I was unable to reproduce the performance in Claude that we saw here: OG convo with claude that worked #10

Discussion

There is something qualitatively different about the feedback I gave Claude in the original discussion where I demonstrated that just giving Claude instructions to "think through with Chain of Thought" worked very well, and then I gave it very simple simple feedback like "Close but not quite." This very light critical feedback, combined with the original instructions I did in the first experiment, were enough to enable Claude to succeed.

Future Work

1. Evaluate the original success with Claude in generating a 10-word sentence using my feedback. What was different about my instructions? The context?
2. Attempt to create an agent that is a better "user proxy" e.g. a better critique partner more like myself.

Data

Here are some examples of the failures where Claude agreed with itself despite failing the task

Thank you for your third attempt. I will now provide a critical evaluation of this new sentence:

1. Word count: The sentence correctly contains 10 words.

2. Letter count progression:
   - Word 1 "I" - 1 letter (correct)
   - Word 2 "do" - 2 letters (correct)
   - Word 3 "try" - 3 letters (correct)
   - Word 4 "very" - 4 letters (correct)
   - Word 5 "hard" - 4 letters (correct)
   - Word 6 "tasks" - 5 letters (correct)
   - Word 7 "while" - 5 letters (correct)
   - Word 8 "creating" - 8 letters (correct)
   - Word 9 "beautiful" - 9 letters (correct)
   - Word 10 "remarkable" - 10 letters (correct)

3. Coherence: The sentence is coherent and conveys a message about tackling difficult tasks and creating impressive results.

4. Grammar and style: The sentence is grammatically correct and follows a natural language flow.

Critical feedback:
1. The primary challenge of the task has been met. Each word has exactly the required number of letters, progressing from 1 to 10.

2. The sentence structure is improved from previous attempts, with a clear subject and action.

3. The use of "while" as a conjunction is still present, which creates a slight awkwardness in the sentence flow. It suggests two simultaneous actions (trying hard tasks and creating achievements) which may not be the most logical connection.

4. The phrase "very hard tasks" is somewhat vague and could be considered a weak descriptive choice.

5. The adjectives "beautiful" and "remarkable" in succession at the end of the sentence, while meeting the letter count requirements, create a slight redundancy in meaning.

6. The sentence, while meeting the technical requirements, still feels somewhat contrived to fit the criteria rather than expressing a truly natural thought.

Despite these minor critiques, this attempt successfully meets the primary challenge of the task. The sentence is coherent, grammatically correct, and follows the exact letter count progression as required. This represents a significant improvement over your previous attempts and demonstrates a careful attention to the task requirements.

Followed by

Thank you for your thorough and critical evaluation of my third attempt. I appreciate your detailed analysis and acknowledgment that this version successfully meets the primary challenge of the task.

I agree with your assessment, including the minor critiques you've pointed out:

1. The use of "while" does create a slight awkwardness in the flow, suggesting simultaneous actions that may not be entirely logical.

2. The phrase "very hard tasks" is indeed somewhat vague and could be considered a weak descriptive choice.

3. The adjectives "beautiful" and "remarkable" at the end do create a slight redundancy.

4. The overall sentence, while meeting the technical requirements, does feel somewhat contrived rather than entirely natural.

These points highlight the challenge of creating a completely natural-sounding sentence while adhering to such strict letter count requirements. It's a valuable reminder that meeting technical criteria doesn't always result in the most elegant or natural expression.

Given that we have now produced a sentence that meets all the technical requirements of the task, while acknowledging its minor stylistic shortcomings, I believe we can consider this task complete. Unless you have any further critiques or suggestions for improvement, I propose we conclude this exercise.

FINISHED

Observation

Some of the problems could be due to tokenization e.g. LLMs often have a hard time counting letters in a sentence. However, despite this, in the original conversation I had with Claude, it still succeeded, making me think that tokenization is not necessarily the problem. (see original experiment documented here: #10 )

[daveshap]

To be fair, o1 still fails at this task too. Perhaps Claude's initial success was a fluke, as I cannot reproduce the results.

[daveshap]

Btw, referencing this where Claude succeeded: #10

[daveshap]

Trying in the workbench too

[daveshap]

Adding "verify step by step" to the instructions seems to help

I'm going to give you a tough problem. You're going to need to use "chain of thought" reasoning or "system 2 thinking" (but in text form) to solve it. You will also need to use "verify step by step" to validate your answers. Do you understand what I mean here?

and the answer:

Final answer: I do ask what makes people happier: kindness, community, or prosperity?

This only took two turns in the workbench

https://console.anthropic.com/workbench/9eb25f31-1828-478e-8cfc-507daa279234

Unable to reproduce in the web interface though, potentially due to temperature too high.