[Agent, LLM] Make sure codeact agent produce message in u/a/u/a order (…

…#3193) * make sure codeact agent produce message in u/a/u/a order * integration tests * sync message changes to codeact swe * fix integration tests --------- Co-authored-by: Engel Nyst <[email protected]>
All-Hands-AI · Aug 1, 2024 · 2e60d25 · 2e60d25
1 parent 0627af8
commit 2e60d25
Show file tree

Hide file tree

Showing 34 changed files with 43 additions and 95 deletions.
diff --git a/agenthub/codeact_agent/codeact_agent.py b/agenthub/codeact_agent/codeact_agent.py
@@ -222,7 +222,13 @@ def _get_messages(self, state: State) -> list[dict[str, str]]:
 
             # add regular message
             if message:
-                messages.append(message)
+                # handle error if the message is the SAME role as the previous message
+                # litellm.exceptions.BadRequestError: litellm.BadRequestError: OpenAIException - Error code: 400 - {'detail': 'Only supports u/a/u/a/u...'}
+                # there should not have two consecutive messages from the same role
+                if messages and messages[-1]['role'] == message['role']:
+                    messages[-1]['content'] += '\n\n' + message['content']
+                else:
+                    messages.append(message)
 
         # the latest user message is important:
         # we want to remind the agent of the environment constraints

diff --git a/agenthub/codeact_swe_agent/codeact_swe_agent.py b/agenthub/codeact_swe_agent/codeact_swe_agent.py
@@ -173,7 +173,13 @@ def _get_messages(self, state: State) -> list[dict[str, str]]:
 
             # add regular message
             if message:
-                messages.append(message)
+                # handle error if the message is the SAME role as the previous message
+                # litellm.exceptions.BadRequestError: litellm.BadRequestError: OpenAIException - Error code: 400 - {'detail': 'Only supports u/a/u/a/u...'}
+                # there should not have two consecutive messages from the same role
+                if messages and messages[-1]['role'] == message['role']:
+                    messages[-1]['content'] += '\n\n' + message['content']
+                else:
+                    messages.append(message)
 
         # the latest user message is important:
         # we want to remind the agent of the environment constraints

diff --git a/tests/integration/mock/CodeActAgent/test_browse_internet/prompt_001.log b/tests/integration/mock/CodeActAgent/test_browse_internet/prompt_001.log
@@ -396,8 +396,6 @@ The server is running on port 5000 with PID 126. You can access the list of numb
 
 NOW, LET'S START!
 
-----------
-
 Browse localhost:8000, and tell me the ultimate answer to life. Do not ask me for confirmation at any point.
 
 ENVIRONMENT REMINDER: You have 14 turns left to complete the task. When finished reply with <finish></finish>
diff --git a/tests/integration/mock/CodeActAgent/test_browse_internet/prompt_005.log b/tests/integration/mock/CodeActAgent/test_browse_internet/prompt_005.log
@@ -396,8 +396,6 @@ The server is running on port 5000 with PID 126. You can access the list of numb
 
 NOW, LET'S START!
 
-----------
-
 Browse localhost:8000, and tell me the ultimate answer to life. Do not ask me for confirmation at any point.
 
 ----------

diff --git a/tests/integration/mock/CodeActAgent/test_edits/prompt_001.log b/tests/integration/mock/CodeActAgent/test_edits/prompt_001.log
@@ -396,8 +396,6 @@ The server is running on port 5000 with PID 126. You can access the list of numb
 
 NOW, LET'S START!
 
-----------
-
 Fix typos in bad.txt. Do not ask me for confirmation at any point.
 
 ENVIRONMENT REMINDER: You have 14 turns left to complete the task. When finished reply with <finish></finish>
diff --git a/tests/integration/mock/CodeActAgent/test_edits/prompt_002.log b/tests/integration/mock/CodeActAgent/test_edits/prompt_002.log
@@ -396,8 +396,6 @@ The server is running on port 5000 with PID 126. You can access the list of numb
 
 NOW, LET'S START!
 
-----------
-
 Fix typos in bad.txt. Do not ask me for confirmation at any point.
 
 ----------

diff --git a/tests/integration/mock/CodeActAgent/test_edits/prompt_003.log b/tests/integration/mock/CodeActAgent/test_edits/prompt_003.log
@@ -396,8 +396,6 @@ The server is running on port 5000 with PID 126. You can access the list of numb
 
 NOW, LET'S START!
 
-----------
-
 Fix typos in bad.txt. Do not ask me for confirmation at any point.
 
 ----------

diff --git a/tests/integration/mock/CodeActAgent/test_edits/prompt_004.log b/tests/integration/mock/CodeActAgent/test_edits/prompt_004.log
@@ -396,8 +396,6 @@ The server is running on port 5000 with PID 126. You can access the list of numb
 
 NOW, LET'S START!
 
-----------
-
 Fix typos in bad.txt. Do not ask me for confirmation at any point.
 
 ----------

diff --git a/tests/integration/mock/CodeActAgent/test_edits/prompt_005.log b/tests/integration/mock/CodeActAgent/test_edits/prompt_005.log
@@ -396,8 +396,6 @@ The server is running on port 5000 with PID 126. You can access the list of numb
 
 NOW, LET'S START!
 
-----------
-
 Fix typos in bad.txt. Do not ask me for confirmation at any point.
 
 ----------

diff --git a/tests/integration/mock/CodeActAgent/test_ipython/prompt_001.log b/tests/integration/mock/CodeActAgent/test_ipython/prompt_001.log
@@ -396,8 +396,6 @@ The server is running on port 5000 with PID 126. You can access the list of numb
 
 NOW, LET'S START!
 
-----------
-
 Use Jupyter IPython to write a text file containing 'hello world' to '/workspace/test.txt'. Do not ask me for confirmation at any point.
 
 ENVIRONMENT REMINDER: You have 14 turns left to complete the task. When finished reply with <finish></finish>
diff --git a/tests/integration/mock/CodeActAgent/test_ipython/prompt_002.log b/tests/integration/mock/CodeActAgent/test_ipython/prompt_002.log
@@ -396,8 +396,6 @@ The server is running on port 5000 with PID 126. You can access the list of numb
 
 NOW, LET'S START!
 
-----------
-
 Use Jupyter IPython to write a text file containing 'hello world' to '/workspace/test.txt'. Do not ask me for confirmation at any point.
 
 ----------

diff --git a/tests/integration/mock/CodeActAgent/test_ipython/prompt_003.log b/tests/integration/mock/CodeActAgent/test_ipython/prompt_003.log
@@ -396,8 +396,6 @@ The server is running on port 5000 with PID 126. You can access the list of numb
 
 NOW, LET'S START!
 
-----------
-
 Use Jupyter IPython to write a text file containing 'hello world' to '/workspace/test.txt'. Do not ask me for confirmation at any point.
 
 ----------

diff --git a/tests/integration/mock/CodeActAgent/test_ipython_module/prompt_001.log b/tests/integration/mock/CodeActAgent/test_ipython_module/prompt_001.log
@@ -396,8 +396,6 @@ The server is running on port 5000 with PID 126. You can access the list of numb
 
 NOW, LET'S START!
 
-----------
-
 Install and import pymsgbox==1.0.9 and print it's version in /workspace/test.txt. Do not ask me for confirmation at any point.
 
 ENVIRONMENT REMINDER: You have 14 turns left to complete the task. When finished reply with <finish></finish>
diff --git a/tests/integration/mock/CodeActAgent/test_ipython_module/prompt_002.log b/tests/integration/mock/CodeActAgent/test_ipython_module/prompt_002.log
@@ -396,8 +396,6 @@ The server is running on port 5000 with PID 126. You can access the list of numb
 
 NOW, LET'S START!
 
-----------
-
 Install and import pymsgbox==1.0.9 and print it's version in /workspace/test.txt. Do not ask me for confirmation at any point.
 
 ----------

diff --git a/tests/integration/mock/CodeActAgent/test_ipython_module/prompt_003.log b/tests/integration/mock/CodeActAgent/test_ipython_module/prompt_003.log
@@ -396,8 +396,6 @@ The server is running on port 5000 with PID 126. You can access the list of numb
 
 NOW, LET'S START!
 
-----------
-
 Install and import pymsgbox==1.0.9 and print it's version in /workspace/test.txt. Do not ask me for confirmation at any point.
 
 ----------

diff --git a/tests/integration/mock/CodeActAgent/test_ipython_module/prompt_004.log b/tests/integration/mock/CodeActAgent/test_ipython_module/prompt_004.log
@@ -396,8 +396,6 @@ The server is running on port 5000 with PID 126. You can access the list of numb
 
 NOW, LET'S START!
 
-----------
-
 Install and import pymsgbox==1.0.9 and print it's version in /workspace/test.txt. Do not ask me for confirmation at any point.
 
 ----------

diff --git a/tests/integration/mock/CodeActAgent/test_write_simple_script/prompt_001.log b/tests/integration/mock/CodeActAgent/test_write_simple_script/prompt_001.log
@@ -396,8 +396,6 @@ The server is running on port 5000 with PID 126. You can access the list of numb
 
 NOW, LET'S START!
 
-----------
-
 Write a shell script 'hello.sh' that prints 'hello'. Do not ask me for confirmation at any point.
 
 ENVIRONMENT REMINDER: You have 14 turns left to complete the task. When finished reply with <finish></finish>
diff --git a/tests/integration/mock/CodeActAgent/test_write_simple_script/prompt_002.log b/tests/integration/mock/CodeActAgent/test_write_simple_script/prompt_002.log
@@ -396,8 +396,6 @@ The server is running on port 5000 with PID 126. You can access the list of numb
 
 NOW, LET'S START!
 
-----------
-
 Write a shell script 'hello.sh' that prints 'hello'. Do not ask me for confirmation at any point.
 
 ----------

diff --git a/tests/integration/mock/CodeActAgent/test_write_simple_script/prompt_003.log b/tests/integration/mock/CodeActAgent/test_write_simple_script/prompt_003.log
@@ -396,8 +396,6 @@ The server is running on port 5000 with PID 126. You can access the list of numb
 
 NOW, LET'S START!
 
-----------
-
 Write a shell script 'hello.sh' that prints 'hello'. Do not ask me for confirmation at any point.
 
 ----------

diff --git a/tests/integration/mock/CodeActAgent/test_write_simple_script/prompt_004.log b/tests/integration/mock/CodeActAgent/test_write_simple_script/prompt_004.log
@@ -396,8 +396,6 @@ The server is running on port 5000 with PID 126. You can access the list of numb
 
 NOW, LET'S START!
 
-----------
-
 Write a shell script 'hello.sh' that prints 'hello'. Do not ask me for confirmation at any point.
 
 ----------

diff --git a/tests/integration/mock/CodeActSWEAgent/test_edits/prompt_001.log b/tests/integration/mock/CodeActSWEAgent/test_edits/prompt_001.log
@@ -587,8 +587,6 @@ exit
 
 NOW, LET'S START!
 
-----------
-
 Fix typos in bad.txt. Do not ask me for confirmation at any point.
 
 ENVIRONMENT REMINDER: You have 14 turns left to complete the task.
diff --git a/tests/integration/mock/CodeActSWEAgent/test_edits/prompt_002.log b/tests/integration/mock/CodeActSWEAgent/test_edits/prompt_002.log
@@ -587,8 +587,6 @@ exit
 
 NOW, LET'S START!
 
-----------
-
 Fix typos in bad.txt. Do not ask me for confirmation at any point.
 
 ----------

diff --git a/tests/integration/mock/CodeActSWEAgent/test_edits/prompt_003.log b/tests/integration/mock/CodeActSWEAgent/test_edits/prompt_003.log
@@ -587,8 +587,6 @@ exit
 
 NOW, LET'S START!
 
-----------
-
 Fix typos in bad.txt. Do not ask me for confirmation at any point.
 
 ----------

diff --git a/tests/integration/mock/CodeActSWEAgent/test_edits/prompt_004.log b/tests/integration/mock/CodeActSWEAgent/test_edits/prompt_004.log
@@ -587,8 +587,6 @@ exit
 
 NOW, LET'S START!
 
-----------
-
 Fix typos in bad.txt. Do not ask me for confirmation at any point.
 
 ----------

diff --git a/tests/integration/mock/CodeActSWEAgent/test_edits/prompt_005.log b/tests/integration/mock/CodeActSWEAgent/test_edits/prompt_005.log
@@ -587,8 +587,6 @@ exit
 
 NOW, LET'S START!
 
-----------
-
 Fix typos in bad.txt. Do not ask me for confirmation at any point.
 
 ----------
@@ -601,13 +599,13 @@ open_file("bad.txt")
 ----------
 
 OBSERVATION:
-[File: /workspace/bad.txt (4 lines total)]
-(this is the beginning of the file)
-1|This is a stupid typoo.
-2|Really?
-3|No mor typos!
-4|Enjoy!
-(this is the end of the file)
+[File: /workspace/bad.txt (4 lines total)]
+(this is the beginning of the file)
+1|This is a stupid typoo.
+2|Really?
+3|No mor typos!
+4|Enjoy!
+(this is the end of the file)
 
 
 ----------
@@ -629,12 +627,12 @@ edit_file_by_replace(
 ----------
 
 OBSERVATION:
-[No exact match found in bad.txt for
-```
-This is a stupid typoo.
-No mor typos!
-```
-]
+[No exact match found in bad.txt for
+```
+This is a stupid typoo.
+No mor typos!
+```
+]
 
 
 ----------
@@ -656,14 +654,14 @@ edit_file_by_replace(
 ----------
 
 OBSERVATION:
-[File: /workspace/bad.txt (4 lines total after edit)]
-(this is the beginning of the file)
-1|This is a stupid typo.
-2|Really?
-3|No mor typos!
-4|Enjoy!
-(this is the end of the file)
-[File updated (edited at line 1). Please review the changes and make sure they are correct (correct indentation, no duplicate lines, etc). Edit the file again if necessary.]
+[File: /workspace/bad.txt (4 lines total after edit)]
+(this is the beginning of the file)
+1|This is a stupid typo.
+2|Really?
+3|No mor typos!
+4|Enjoy!
+(this is the end of the file)
+[File updated (edited at line 1). Please review the changes and make sure they are correct (correct indentation, no duplicate lines, etc). Edit the file again if necessary.]
 
 
 ----------
@@ -680,14 +678,14 @@ edit_file_by_replace(
 ----------
 
 OBSERVATION:
-[File: /workspace/bad.txt (4 lines total after edit)]
-(this is the beginning of the file)
-1|This is a stupid typo.
-2|Really?
-3|No more typos!
-4|Enjoy!
-(this is the end of the file)
-[File updated (edited at line 3). Please review the changes and make sure they are correct (correct indentation, no duplicate lines, etc). Edit the file again if necessary.]
+[File: /workspace/bad.txt (4 lines total after edit)]
+(this is the beginning of the file)
+1|This is a stupid typo.
+2|Really?
+3|No more typos!
+4|Enjoy!
+(this is the end of the file)
+[File updated (edited at line 3). Please review the changes and make sure they are correct (correct indentation, no duplicate lines, etc). Edit the file again if necessary.]
 
 
 ENVIRONMENT REMINDER: You have 10 turns left to complete the task.
diff --git a/tests/integration/mock/CodeActSWEAgent/test_ipython/prompt_001.log b/tests/integration/mock/CodeActSWEAgent/test_ipython/prompt_001.log
@@ -587,8 +587,6 @@ exit
 
 NOW, LET'S START!
 
-----------
-
 Use Jupyter IPython to write a text file containing 'hello world' to '/workspace/test.txt'. Do not ask me for confirmation at any point.
 
 ENVIRONMENT REMINDER: You have 14 turns left to complete the task.
diff --git a/tests/integration/mock/CodeActSWEAgent/test_ipython/prompt_002.log b/tests/integration/mock/CodeActSWEAgent/test_ipython/prompt_002.log
@@ -587,8 +587,6 @@ exit
 
 NOW, LET'S START!
 
-----------
-
 Use Jupyter IPython to write a text file containing 'hello world' to '/workspace/test.txt'. Do not ask me for confirmation at any point.
 
 ----------

diff --git a/tests/integration/mock/CodeActSWEAgent/test_ipython_module/prompt_001.log b/tests/integration/mock/CodeActSWEAgent/test_ipython_module/prompt_001.log
@@ -587,8 +587,6 @@ exit
 
 NOW, LET'S START!
 
-----------
-
 Install and import pymsgbox==1.0.9 and print it's version in /workspace/test.txt. Do not ask me for confirmation at any point.
 
 ENVIRONMENT REMINDER: You have 14 turns left to complete the task.
diff --git a/tests/integration/mock/CodeActSWEAgent/test_ipython_module/prompt_002.log b/tests/integration/mock/CodeActSWEAgent/test_ipython_module/prompt_002.log
@@ -587,8 +587,6 @@ exit
 
 NOW, LET'S START!
 
-----------
-
 Install and import pymsgbox==1.0.9 and print it's version in /workspace/test.txt. Do not ask me for confirmation at any point.
 
 ----------

diff --git a/tests/integration/mock/CodeActSWEAgent/test_ipython_module/prompt_003.log b/tests/integration/mock/CodeActSWEAgent/test_ipython_module/prompt_003.log
@@ -587,8 +587,6 @@ exit
 
 NOW, LET'S START!
 
-----------
-
 Install and import pymsgbox==1.0.9 and print it's version in /workspace/test.txt. Do not ask me for confirmation at any point.
 
 ----------

diff --git a/tests/integration/mock/CodeActSWEAgent/test_write_simple_script/prompt_001.log b/tests/integration/mock/CodeActSWEAgent/test_write_simple_script/prompt_001.log
@@ -587,8 +587,6 @@ exit
 
 NOW, LET'S START!
 
-----------
-
 Write a shell script 'hello.sh' that prints 'hello'. Do not ask me for confirmation at any point.
 
 ENVIRONMENT REMINDER: You have 14 turns left to complete the task.
diff --git a/tests/integration/mock/CodeActSWEAgent/test_write_simple_script/prompt_002.log b/tests/integration/mock/CodeActSWEAgent/test_write_simple_script/prompt_002.log
@@ -587,8 +587,6 @@ exit
 
 NOW, LET'S START!
 
-----------
-
 Write a shell script 'hello.sh' that prints 'hello'. Do not ask me for confirmation at any point.
 
 ----------

diff --git a/tests/integration/mock/CodeActSWEAgent/test_write_simple_script/prompt_003.log b/tests/integration/mock/CodeActSWEAgent/test_write_simple_script/prompt_003.log
@@ -587,8 +587,6 @@ exit
 
 NOW, LET'S START!
 
-----------
-
 Write a shell script 'hello.sh' that prints 'hello'. Do not ask me for confirmation at any point.
 
 ----------