-
Checked other resources
Commit to Help
Example Codeimport { HumanMessage, SystemMessage } from "@langchain/core/messages";
import { ChatPromptTemplate } from "@langchain/core/prompts";
import { ChatOpenAI, ChatOpenAICallOptions } from "@langchain/openai";
import { z } from "zod";
const chatPromptTemplate = ChatPromptTemplate.fromMessages([
new SystemMessage("You are a helpful art history expert."),
new HumanMessage("What was Michaelangelo's most famous work?"),
]);
const baseModel = new ChatOpenAI({ apiKey: "your api key goes here", model: "gpt-4o-2024-08-06" });
const modelOptions: ChatOpenAICallOptions = {
response_format: {
type: "json_schema",
json_schema: {
name: "response-body",
schema: z.object({ name: z.string(), description: z.string() }),
},
},
};
const model = baseModel.bind(modelOptions);
const sequence = chatPromptTemplate.pipe(model);
const stream = await sequence.stream({});
for await (const message of stream) {
console.log(message);
}DescriptionMy issue is an unexpected, and seemingly unnecessary, reduction in capability with a recent release. I am using ChatOpenAI with the new option for response_format With the latest @langchain/openai I am receiving a warning: Before the latest release (ie before langchain added explicit Question: Is this block at all necessary? Can it just be removed? Thank you! System Info"npm info @langchain/openai" Platform: Windows |
Beta Was this translation helpful? Give feedback.
Replies: 2 comments 4 replies
-
|
Hello, @ClaysGit! I'm here to assist you with any bugs, questions, or contributions you have. Let's get to the bottom of this together! There is a similar unsolved discussion on this topic: Streaming OpenAI Function Call output in LangChainStreaming [1]. Regarding your question, it is necessary to block streaming when using the However, there are tests indicating that streaming with For now, you might consider using alternative methods for streaming structured output data. You can use the import { z } from "zod";
import { ChatPromptTemplate } from "@langchain/core/prompts";
import { ChatOpenAI } from "@langchain/openai";
const model = new ChatOpenAI({
temperature: 0,
modelName: "gpt-4-turbo-preview",
});
const calculatorSchema = z.object({
operation: z.enum(["add", "subtract", "multiply", "divide"]),
number1: z.number(),
number2: z.number(),
});
const modelWithStructuredOutput = model.withStructuredOutput(
calculatorSchema,
{
name: "calculator",
}
);
const prompt = ChatPromptTemplate.fromMessages([
["system", "You are VERY bad at math and must always use a calculator."],
["human", "Please help me!! What is 2 + 2?"],
]);
const chain = prompt.pipe(modelWithStructuredOutput);
const stream = await chain.stream({});
const chunks = [];
for await (const chunk of stream) {
chunks.push(chunk);
}
console.log(chunks);This approach allows you to stream structured output data using Zod schemas [3]. Additionally, there is ongoing development to support streaming with |
Beta Was this translation helpful? Give feedback.
-
|
@dosu This suggestion does not perfectly replicate my example. You can replicate my example with the following change to modelWithStructuredOutput: const modelWithStructuredOutput = model.withStructuredOutput(
calculatorSchema,
{
name: "calculator",
method: "jsonSchema",
}
);This leads to the issue described in my initial post. Is there specific posts or code which explains the issue requiring this functionality be blocked? |
Beta Was this translation helpful? Give feedback.
-
|
Yes, there is specific documentation and code that explains why streaming with the if (
params.response_format &&
params.response_format.type === "json_schema"
) {
console.warn(
`OpenAI does not yet support streaming with "response_format" set to "json_schema". Falling back to non-streaming mode.`
);
const res = await this._generate(messages, options, runManager);
const chunk = new ChatGenerationChunk({
message: new AIMessageChunk({
...res.generations[0].message,
}),
text: res.generations[0].text,
generationInfo: res.generations[0].generationInfo,
});
yield chunk;
return runManager?.handleLLMNewToken(
res.generations[0].text ?? "",
undefined,
undefined,
undefined,
undefined,
{ chunk }
);
}This code indicates that OpenAI does not yet support streaming when the |
Beta Was this translation helpful? Give feedback.
-
|
Thanks @dosu. However, my issue describes that this block is unnecessary. WIthout this block, streaming from open ai with json_schema mode works as intended. I'd like to better understand the issue or see this block removed. |
Beta Was this translation helpful? Give feedback.
-
|
Yes, the block preventing streaming with the if (
params.response_format &&
params.response_format.type === "json_schema"
) {
console.warn(
`OpenAI does not yet support streaming with "response_format" set to "json_schema". Falling back to non-streaming mode.`
);
const res = await this._generate(messages, options, runManager);
const chunk = new ChatGenerationChunk({
message: new AIMessageChunk({
...res.generations[0].message,
}),
text: res.generations[0].text,
generationInfo: res.generations[0].generationInfo,
});
yield chunk;
return runManager?.handleLLMNewToken(
res.generations[0].text ?? "",
undefined,
undefined,
undefined,
undefined,
{ chunk }
);
}Removing this block will allow streaming to proceed even when the it("can stream with a zod schema passed in", async () => {
const model = new ChatOpenAI({
model: "gpt-4o-2024-08-06",
}).bind({
response_format: {
type: "json_schema",
json_schema: {
name: "get_current_weather",
description: "Get the current weather in a location",
schema: weatherSchema,
strict: true,
},
},
});
const stream = await model.stream(
"What is the weather in San Francisco, 91626 CA?"
);
let full: AIMessageChunk | undefined;
for await (const chunk of stream) {
full = !full ? chunk : concat(full, chunk);
}
expect(full).toBeDefined();
if (!full) return;
const parsed = JSON.parse(full.content as string);
expect(parsed).toHaveProperty("city");
expect(parsed).toHaveProperty("state");
expect(parsed).toHaveProperty("zipCode");
expect(parsed).toHaveProperty("unit");
});This test confirms that streaming with a JSON schema is functional, validating the removal of the block [1][2]. |
Beta Was this translation helpful? Give feedback.
-
|
Issue is resolved in #6906 |
Beta Was this translation helpful? Give feedback.
Issue is resolved in #6906