streamText with Tool Calling returns empty message

[DR7777]

Hello everyone!

I am building a RAG application which basically works like any other RAG.

1. User uploads doc, it is vectorised and saved
2. User asks question
3. Chatbot answers.

I want the LLM to choose between 3 tools:

1. search vectorstore
2. If no sufficient answer, load whole doc into context
3. search Bing if necessary

I am using:
"ai": 3.4.33
"next": "14.1.3"

For some reason, whenever the LLM calls a tool, it appends an empty message to the messages Array, which leads to my UI rendering one or more empty chat bubbles:

This is my Chat component:

'use client';
import { useEffect, useRef, useState, MutableRefObject } from 'react';
import { useChat } from 'ai/react';
// more imports


export function Chat({
//props
}: ChatProps) {

  const { messages, input, handleInputChange, handleSubmit, isLoading } =
    useChat({
      api: '/api/chat',
      maxSteps: 5,
      body: {
        chatId,
        pagesText
      }
    });

  return (
    <div className="flex flex-col w-full h-full">
      <div className="flex-1 bg-white border rounded overflow-auto">
        <div ref={interactionsRef} className="p-4">
          {messages.map((m, index) => (
            <Alert
              key={m.id || index}
              className={
                m.role === 'assistant' ? 'm-2' : 'bg-primary-500 text-white m-2'
              }
            >
              {m.role === 'assistant' ? (
                <QQIcon className="h-4 w-4" />
              ) : (
                <UserIcon className="h-4 w-4 stroke-white" />
              )}
              <AlertDescription>
                <CustomMarkdown>{m.content}</CustomMarkdown>
              </AlertDescription>
            </Alert>
          ))}
        </div>
      </div>

      <div className="flex justify-center items-center">
        <form
          onSubmit={handleSubmit}
          className="relative w-full sm:pt-4 pt-2 flex justify-start items-center flex-row"
        >
          <Input
            disabled={isLoading}
            type="text"
            placeholder="Stelle eine beliebige Frage... (max. 1000 Zeichen)"
            onChange={handleInputChange}
            value={input}
            maxLength={1000}
            className="resize-none p-3 pr-10 rounded-md border border-gray-300 bg-white text-black focus:outline-gray-400 w-full"
          />
          <Button type="submit" disabled={isLoading} className="ml-2">
            {isLoading ? (
              <Loader2 className="h-4 w-4 animate-spin" />
            ) : (
              'Send'
            )}
          </Button>
        </form>
      </div>
    </div>
  );
}

This is my route:

import { openai } from '@ai-sdk/openai';
import { streamText, generateText, StreamData } from 'ai';
import { z } from 'zod';
import { formatDocumentsAsString } from 'langchain/util/document';
// other imports

export const maxDuration = 300;


export async function POST(req: Request) {
  const { messages, chatId, pagesText } = await req.json();
  const question = messages[messages.length - 1].content;


  const result = await streamText({
    model: openai('gpt-4o-mini'),
    system: `You are a helpful assistant who answers questions from a document.
    Use tools on every request.

    Follow these steps:
    1. Always use the searchDocument tool first to find relevant information with RAG technique and try to answer the question.
    2. If the results from searchDocument are not enough to answer the question, use the useWholeDocument tool.
    3. If neither tool provides relevant information, respond with "Sorry, I don't have enough information to answer that question."

    Always answer in the same language as the question, unless otherwise specified.`,
    messages,
    maxSteps: 3,
    temperature: 0.5,

    tools: {
      searchDocument: {
        description: `Use this tool to search the document for relevant information to answer the question. Use this tool on every request.`,
        parameters: z.object({
          question: z.string().describe('The users question')
        }),

        execute: async ({ question }) => {
          console.log('calling searchDocuments');

       // create standalone Question

          const vectorStore = await SupabaseVectorStore.fromExistingIndex(
            embeddingModel,
            {
              tableName: 'documents',
              queryName: 'match_documents',
              client: supabaseClient
            }
          );

          const retriever = vectorStore.asRetriever({
            filter: {
              chatId: chatId
            }
          });
          const searchResults = await retriever.getRelevantDocuments(
            standaloneQuestion.text
          );
          const context = formatDocumentsAsString(searchResults);
          return context;
        }
      },
      useWholeDocument: {
        description: `Use this tool if you cannot answer the question using searchDocument.`,
        parameters: z.object({}),
        execute: async () => {
          console.log('calling getFullDocument');
          return pagesText.map(page => page.textContent).join('\n\n');
        }
       // Bing Chat Tool
      }
    }
       // save to firebase with onFinish
  });
  console.log('sending result');
  return result.toDataStreamResponse();
}

Is this expected behaviour and I need to adjust my UI and filter out those emtpy messges or am I doing something wrong?

Thank you!

[janvennemann]

I'm also running into this behavior. Another side effect of this is that the toolCalls and toolResults in the onFinish callback are always empty. For example, the docs recommend the following piece of code to implement a persistent chat history:

import { openai } from '@ai-sdk/openai';
import { streamText } from 'ai';

// Allow streaming responses up to 30 seconds
export const maxDuration = 30;

export async function POST(req: Request) {
  const { messages } = await req.json();

  const result = streamText({
    model: openai('gpt-4-turbo'),
    messages,
    async onFinish({ text, toolCalls, toolResults, usage, finishReason, responseMessages }) {
      // implement your own storage logic:
      // Note: toolCalls and toolResults are always empty even when tools were called
      await saveChat({ text, toolCalls, toolResults });
    },
  });

  return result.toDataStreamResponse();
}

The responseMessages from the onFinish callback look like this which explains the additional empty assistant message:

[
  {
    role: 'assistant',
    content: [
      { type: 'text', text: '' },
      {
        type: 'tool-call',
        toolCallId: 'call_9bchLFghtVPs1ub4JXgBFjHu',
        toolName: 'getWeatherInformation',
        args: { city: 'Kapstadt' }
      }
    ]
  },
  {
    role: 'tool',
    content: [
      {
        type: 'tool-result',
        toolCallId: 'call_9bchLFghtVPs1ub4JXgBFjHu',
        toolName: 'getWeatherInformation',
        result: 'windy'
      }
    ]
  },
  {
    role: 'assistant',
    content: [
      {
        type: 'text',
        text: 'Das Wetter in Kapstadt ist derzeit windig.'
      }
    ]
  }
]

The vercel/ai-chatbot currently uses the responseMessages for this reason to save messages i assume.

Would be great to know whether this is a bug in the way messages are structured, or expected behavior and the onFinish callback is wrong.

[lgrammel]

please filter out the empty messages or combine multiple consecutive assistant messages into one before rendering. we are aware of the problem and are working on a solution, but it'll require larger changes

[janvennemann]

Sure, no problem, thank's for the quick reply. Can you say what the expected behavior should be, so i can design my workaround accordingly? Should everything be combined into a single assistant message that includes the text and tool invocations?

[lgrammel]

most likely it'll be a new helper in addition to messages that has differently structure objects and ensures that there is a clear user-assistant-user-assistant conversation