Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

feat (provider/xai): add groq-vision-beta support #3771

Merged
merged 1 commit into from
Nov 19, 2024
Merged

feat (provider/xai): add groq-vision-beta support #3771

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 5 additions & 0 deletions .changeset/brave-poets-sing.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,5 @@
---
'@ai-sdk/xai': patch
---

feat (provider/xai): add groq-vision-beta support
1 change: 1 addition & 0 deletions content/docs/02-foundations/02-providers-and-models.mdx
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -77,6 +77,7 @@ Here are the capabilities of popular models:
| [Google Vertex](/providers/ai-sdk-providers/google-vertex) | `gemini-1.5-flash` | <Check size={18} /> | <Check size={18} /> | <Check size={18} /> | <Check size={18} /> |
| [Google Vertex](/providers/ai-sdk-providers/google-vertex) | `gemini-1.5-pro` | <Check size={18} /> | <Check size={18} /> | <Check size={18} /> | <Check size={18} /> |
| [xAI Grok](/providers/ai-sdk-providers/xai) | `grok-beta` | <Cross size={18} /> | <Check size={18} /> | <Check size={18} /> | <Check size={18} /> |
| [xAI Grok](/providers/ai-sdk-providers/xai) | `grok-vision-beta` | <Check size={18} /> | <Check size={18} /> | <Check size={18} /> | <Check size={18} /> |
| [Groq](/providers/ai-sdk-providers/groq) | `llama-3.1-405b-reasoning` | <Cross size={18} /> | <Check size={18} /> | <Check size={18} /> | <Check size={18} /> |
| [Groq](/providers/ai-sdk-providers/groq) | `llama-3.1-70b-versatile` | <Cross size={18} /> | <Check size={18} /> | <Check size={18} /> | <Check size={18} /> |
| [Groq](/providers/ai-sdk-providers/groq) | `llama-3.1-8b-instant` | <Cross size={18} /> | <Check size={18} /> | <Check size={18} /> | <Check size={18} /> |
Expand Down
7 changes: 4 additions & 3 deletions content/providers/01-ai-sdk-providers/22-xai.mdx
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -116,9 +116,10 @@ The following optional settings are available for xAI chat models:
xAI available models are listed in [the xAI console](https://console.x.ai/)
under "View models", including:

| Model | Image Input | Object Generation | Tool Usage | Tool Streaming |
| ----------- | ------------------- | ------------------- | ------------------- | ------------------- |
| `grok-beta` | <Cross size={18} /> | <Check size={18} /> | <Check size={18} /> | <Check size={18} /> |
| Model | Image Input | Object Generation | Tool Usage | Tool Streaming |
| ------------------ | ------------------- | ------------------- | ------------------- | ------------------- |
| `grok-beta` | <Cross size={18} /> | <Check size={18} /> | <Check size={18} /> | <Check size={18} /> |
| `grok-vision-beta` | <Check size={18} /> | <Check size={18} /> | <Check size={18} /> | <Check size={18} /> |

<Note>
You can also pass any available provider model ID as a string if needed.
Expand Down
1 change: 1 addition & 0 deletions content/providers/01-ai-sdk-providers/index.mdx
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -37,6 +37,7 @@ Not all providers support all AI SDK features. Here's a quick comparison of the
| [Google Vertex](/providers/ai-sdk-providers/google-vertex) | `gemini-1.5-flash` | <Check size={18} /> | <Check size={18} /> | <Check size={18} /> | <Check size={18} /> |
| [Google Vertex](/providers/ai-sdk-providers/google-vertex) | `gemini-1.5-pro` | <Check size={18} /> | <Check size={18} /> | <Check size={18} /> | <Check size={18} /> |
| [xAI Grok](/providers/ai-sdk-providers/xai) | `grok-beta` | <Cross size={18} /> | <Check size={18} /> | <Check size={18} /> | <Check size={18} /> |
| [xAI Grok](/providers/ai-sdk-providers/xai) | `grok-vision-beta` | <Check size={18} /> | <Check size={18} /> | <Check size={18} /> | <Check size={18} /> |
| [Groq](/providers/ai-sdk-providers/groq) | `llama-3.1-405b-reasoning` | <Cross size={18} /> | <Check size={18} /> | <Check size={18} /> | <Check size={18} /> |
| [Groq](/providers/ai-sdk-providers/groq) | `llama-3.1-70b-versatile` | <Cross size={18} /> | <Check size={18} /> | <Check size={18} /> | <Check size={18} /> |
| [Groq](/providers/ai-sdk-providers/groq) | `llama-3.1-8b-instant` | <Cross size={18} /> | <Check size={18} /> | <Check size={18} /> | <Check size={18} /> |
Expand Down
25 changes: 25 additions & 0 deletions examples/ai-core/src/stream-text/xai-image.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,25 @@
import { xai } from '@ai-sdk/xai';
import { streamText } from 'ai';
import 'dotenv/config';
import fs from 'node:fs';

async function main() {
const result = streamText({
model: xai('grok-vision-beta'),
messages: [
{
role: 'user',
content: [
{ type: 'text', text: 'Describe the image in detail.' },
{ type: 'image', image: fs.readFileSync('./data/comic-cat.png') },
],
},
],
});

for await (const textPart of result.textStream) {
process.stdout.write(textPart);
}
}

main().catch(console.error);
29 changes: 29 additions & 0 deletions packages/xai/src/convert-to-xai-chat-messages.test.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,6 +1,35 @@
import { convertToXaiChatMessages } from './convert-to-xai-chat-messages';

describe('user messages', () => {
it('should convert messages with image parts', async () => {
const result = convertToXaiChatMessages([
{
role: 'user',
content: [
{ type: 'text', text: 'Hello' },
{
type: 'image',
image: new Uint8Array([0, 1, 2, 3]),
mimeType: 'image/png',
},
],
},
]);

expect(result).toEqual([
{
role: 'user',
content: [
{ type: 'text', text: 'Hello' },
{
type: 'image_url',
image_url: { url: 'data:image/png;base64,AAECAw==' },
},
],
},
]);
});

it('should convert messages with only a text part to a string content', async () => {
const result = convertToXaiChatMessages([
{
Expand Down
14 changes: 11 additions & 3 deletions packages/xai/src/convert-to-xai-chat-messages.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -31,9 +31,17 @@ export function convertToXaiChatMessages(
return { type: 'text', text: part.text };
}
case 'image': {
throw new UnsupportedFunctionalityError({
functionality: 'Image content parts in user messages',
});
return {
type: 'image_url',
image_url: {
url:
part.image instanceof URL
? part.image.toString()
: `data:${
part.mimeType ?? 'image/jpeg'
};base64,${convertUint8ArrayToBase64(part.image)}`,
},
};
}
case 'file': {
throw new UnsupportedFunctionalityError({
Expand Down
2 changes: 1 addition & 1 deletion packages/xai/src/xai-chat-settings.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
// https://console.x.ai and see "View models"
export type XaiChatModelId = 'grok-beta' | (string & {});
export type XaiChatModelId = 'grok-beta' | 'grok-vision-beta' | (string & {});

export interface XaiChatSettings {
/**
Expand Down
Loading