Skip to content

Commit

Permalink
OpenAI: Ignore chunks with an empty choices list (#192951)
Browse files Browse the repository at this point in the history
## Summary

OpenAI added usage stats to their APIs recently:
https://community.openai.com/t/usage-stats-now-available-when-using-streaming-with-the-chat-completions-api-or-completions-api/738156

However, it's somewhat non BWC:

> Note that this usage-specific chunk will have choices: [], so if you
turn this feature on, you may need to update any code that accesses

(Héhéhé)

Leveraging this feature is for another day. This PR is just about
avoiding the whole thing to explode.
  • Loading branch information
pgayvallet authored Sep 15, 2024
1 parent 7b3fa3a commit 8bffd61
Show file tree
Hide file tree
Showing 3 changed files with 5 additions and 3 deletions.
Original file line number Diff line number Diff line change
Expand Up @@ -90,7 +90,9 @@ export async function getTokenCountFromOpenAIStream({
delta: { content?: string; function_call?: { name?: string; arguments: string } };
}>;
} => {
return 'object' in line && line.object === 'chat.completion.chunk';
return (
'object' in line && line.object === 'chat.completion.chunk' && line.choices.length > 0
);
}
)
.reduce(
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -74,7 +74,7 @@ export const openAIAdapter: InferenceConnectorAdapter = {
}),
filter(
(line): line is OpenAI.ChatCompletionChunk =>
'object' in line && line.object === 'chat.completion.chunk'
'object' in line && line.object === 'chat.completion.chunk' && line.choices.length > 0
),
map((chunk): ChatCompletionChunkEvent => {
const delta = chunk.choices[0].delta;
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -98,7 +98,7 @@ export function processOpenAiStream({
}),
filter(
(line): line is CreateChatCompletionResponseChunk =>
'object' in line && line.object === 'chat.completion.chunk'
'object' in line && line.object === 'chat.completion.chunk' && line.choices.length > 0
),
map((chunk): ChatCompletionChunkEvent => {
const delta = chunk.choices[0].delta;
Expand Down

0 comments on commit 8bffd61

Please sign in to comment.