Chat Engine

[d3buggerdan]

@dosu
How can i make this with the best or context mode and have chat history.

Initialize custom chat history

custom_chat_history = [
]

Create the chat engine using the low-level API

chat_engine = CondenseQuestionChatEngine.from_defaults(
query_engine=query_engine,
#condense_question_prompt=custom_prompt,
chat_history=custom_chat_history,
verbose=True,)

[dosubot[bot]]

To create a chat engine with the best context mode and include chat history, you can use the following code:

# Assuming `index` is already created and configured
chat_engine = index.as_chat_engine(chat_mode="best", verbose=True)

# To include chat history, you can initialize the conversation with some existing history
from llama_index.core.llms import ChatMessage, MessageRole

custom_chat_history = [
    ChatMessage(
        role=MessageRole.USER,
        content="Hello assistant, we are having an insightful discussion about Paul Graham today.",
    ),
    ChatMessage(role=MessageRole.ASSISTANT, content="Okay, sounds good."),
]

# Assuming `query_engine` is already created and configured
chat_engine = CondenseQuestionChatEngine.from_defaults(
    query_engine=query_engine,
    chat_history=custom_chat_history,
    verbose=True,
)

# Now you can have a conversation with your data
response = chat_engine.chat("Tell me a joke.")
print(response)

This code sets up a chat engine with the "best" context mode and includes some initial chat history. For more detailed information, you can refer to the LlamaIndex Chat Engines documentation ^[1][2].

To continue talking to Dosu, mention @dosu.

[d3buggerdan]

@dosu
custom_chat_history = [
]

Create the chat engine using the low-level API

chat_engine = CondenseQuestionChatEngine.from_defaults(
query_engine=query_engine,
condense_question_prompt=custom_prompt,
chat_history=custom_chat_history,
verbose=True,)

def analyze_query(query_str):
# Use LLM to determine if the query is about contract details
analysis_prompt = f"Is the following query asking about specific contract details (requires the contract text to give a answer) or just general information? Please answer with '1' if yes or '0' if no. Query: {query_str}"
analysis_result = llm.complete(prompt=analysis_prompt)

return analysis_result

def handle_chat(query_str):
# Get the response from the chat engine
response = chat_engine.chat(query_str)

# Append the query and response to the chat history
custom_chat_history.append({"user": query_str, "assistant": response})

# Print the current chat history
print("\nChat History:")
for idx, chat in enumerate(custom_chat_history):
    print(f"{idx + 1}. User: {chat['user']}")
    print(f"   Assistant: {chat['assistant']}")

# Return the response
return response

Querying with: hello

Chat History:

1. User: hello
   Assistant: Hello! How can I assist you today?
   127.0.0.1 - - [15/Aug/2024 22:36:13] "POST /api/query HTTP/1.1" 200 -
   127.0.0.1 - - [15/Aug/2024 22:36:22] "OPTIONS /api/query HTTP/1.1" 200 -
   Querying with: In which country is New York located?

Chat History:

1. User: hello
   Assistant: Hello! How can I assist you today?
2. User: what country is the new york in
   Assistant: No Context

[dosubot[bot]]

To improve your chat engine and ensure the chat history is properly maintained, you can make a few adjustments:

1. Use the Correct Chat History Format:
   Ensure that the chat history is in the correct format expected by the CondenseQuestionChatEngine. The ChatMessage class should be used for each message in the history.

2. Append Chat History Correctly:
   When appending to the chat history, use the ChatMessage class to maintain consistency.

3. Analyze and Handle Queries:
   Use the analyze_query function to determine the nature of the query and handle it accordingly.

Here is the updated code:

from llama_index.core.llms import ChatMessage, MessageRole

# Initialize custom chat history
custom_chat_history = [
    ChatMessage(role=MessageRole.USER, content="Hello! How can I assist you today?"),
    ChatMessage(role=MessageRole.ASSISTANT, content="Hello! How can I assist you today?"),
]

# Create the chat engine using the low-level API
chat_engine = CondenseQuestionChatEngine.from_defaults(
    query_engine=query_engine,
    condense_question_prompt=custom_prompt,
    chat_history=custom_chat_history,
    verbose=True,
)

def analyze_query(query_str):
    # Use LLM to determine if the query is about contract details
    analysis_prompt = f"Is the following query asking about specific contract details (requires the contract text to give an answer) or just general information? Please answer with '1' if yes or '0' if no. Query: {query_str}"
    analysis_result = llm.complete(prompt=analysis_prompt)
    
    return analysis_result

def handle_chat(query_str):
    # Get the response from the chat engine
    response = chat_engine.chat(query_str)
    
    # Append the query and response to the chat history
    custom_chat_history.append(ChatMessage(role=MessageRole.USER, content=query_str))
    custom_chat_history.append(ChatMessage(role=MessageRole.ASSISTANT, content=response))

    # Print the current chat history
    print("\nChat History:")
    for idx, chat in enumerate(custom_chat_history):
        print(f"{idx + 1}. User: {chat.content if chat.role == MessageRole.USER else ''}")
        print(f"   Assistant: {chat.content if chat.role == MessageRole.ASSISTANT else ''}")

    # Return the response
    return response

# Example usage
query_str = "In which country is New York located?"
response = handle_chat(query_str)
print(response)

This code ensures that the chat history is properly maintained and formatted, and it uses the ChatMessage class for consistency. The analyze_query function helps determine the nature of the query, and the handle_chat function manages the chat interaction and updates the chat history accordingly.

To continue talking to Dosu, mention @dosu.

[d3buggerdan]

@dosu this doesnt make sense why doesnt it have context

[dosubot[bot]]

Your chat engine likely doesn't have context because it is not properly retrieving or setting the context in the system prompt. To fix this, ensure that the context is being generated and included in the prefix messages correctly.

Here is a key part of the ContextChatEngine class that handles context generation and inclusion:

1. Context Generation:

   def _generate_context(self, message: str) -> Tuple[str, List[NodeWithScore]]:
       """Generate context information from a message."""
       nodes = self._retriever.retrieve(message)
       for postprocessor in self._node_postprocessors:
           nodes = postprocessor.postprocess_nodes(
               nodes, query_bundle=QueryBundle(message)
           )
   
       context_str = "\n\n".join(
           [n.node.get_content(metadata_mode=MetadataMode.LLM).strip() for n in nodes]
       )
   
       return self._context_template.format(context_str=context_str), nodes

2. Including Context in Prefix Messages:

   def _get_prefix_messages_with_context(self, context_str: str) -> List[ChatMessage]:
       """Get the prefix messages with context."""
       system_prompt = ""
       prefix_messages = self._prefix_messages
       if (
           len(self._prefix_messages) != 0
           and self._prefix_messages[0].role == MessageRole.SYSTEM
       ):
           system_prompt = str(self._prefix_messages[0].content)
           prefix_messages = self._prefix_messages[1:]
   
       context_str_w_sys_prompt = system_prompt.strip() + "\n" + context_str
       return [
           ChatMessage(
               content=context_str_w_sys_prompt, role=self._llm.metadata.system_role
           ),
           *prefix_messages,
       ]

3. Using the Context in Chat:

   @trace_method("chat")
   def chat(
       self, message: str, chat_history: Optional[List[ChatMessage]] = None
   ) -> AgentChatResponse:
       if chat_history is not None:
           self._memory.set(chat_history)
       self._memory.put(ChatMessage(content=message, role="user"))
   
       context_str_template, nodes = self._generate_context(message)
       prefix_messages = self._get_prefix_messages_with_context(context_str_template)
       prefix_messages_token_count = len(
           self._memory.tokenizer_fn(
               " ".join([(m.content or "") for m in prefix_messages])
           )
       )
       all_messages = prefix_messages + self._memory.get(
           initial_token_count=prefix_messages_token_count
       )
       chat_response = self._llm.chat(all_messages)
       ai_message = chat_response.message
       self._memory.put(ai_message)
   
       return AgentChatResponse(
           response=str(chat_response.message.content),
           sources=[
               ToolOutput(
                   tool_name="retriever",
                   content=str(prefix_messages[0]),
                   raw_input={"message": message},
                   raw_output=prefix_messages[0],
               )
           ],
           source_nodes=nodes,
       )

Ensure that your retriever is correctly set up and that the context is being generated and included in the prefix messages as shown above. Additionally, make sure that the ContextChatEngine class is properly initialized with the necessary components such as the retriever, LLM, memory, and prefix messages ^[1][2].

To continue talking to Dosu, mention @dosu.