Add chat completion method

[radames]

Supersede #581. Thanks to @Wauplin, I can import the types from "@huggingface/tasks"
I've followed the pattern for textGeneration and textGenerationStream.

[radames]

One question @coyotte508 , do we want to abstract the /v1/chat/completions endpoint if the user uses chatCompletion with our hosted models, let's say mistralai/Mistral-7B-Instruct-v0.2, shall this also work?

const stream = hf.chatCompletionStream({
				model: "mistralai/Mistral-7B-Instruct-v0.2",
				messages: [{ role: "user", content: "Complete the equation 1+1= ,just the answer" }],
				max_tokens: 500,
				return_full_text: false,
				temperature: 0.1,
				seed: 0,
			});

[radames]

The only issue is now all models are served/compatible with /v1/chat/completions we might ended up on the same @Wauplin huggingface/huggingface_hub#2094

[Wauplin]

The only issue is now all models are served/compatible with /v1/chat/completions we might ended up on the same

@radames not quite exactly. TGI-served models expose a /v1/chat/completions route which is fully compatible with the chat completion API. transformers-served models on the other hand don't have a /v1/chat/completions route but they still accept to receive a list of messages as input in their text-generation pipeline. In this case messages are rendered server-side which makes figure from huggingface/huggingface_hub#2094 (comment) obsolete. I made the change in huggingface/huggingface_hub#2258 if you are interested to look at it. In particular you'll see that the transformers response is simply a text string since it is still the text-generation pipeline. To return a proper ChatCompletionOutput object, you'll have to populate fields manually (in my case a lot of fields are set to "dummy"). Hope this will help, please let me know if you have some remaining questions.

[Wauplin]

Oh and btw, TGI-served models have a /info route exposed (see https://api-inference.huggingface.co/models/meta-llama/Meta-Llama-3-70B-Instruct/info). With this endpoint you can be 100% sure whether a model is TGI or transformers backed. In the current huggingface_hub implementation we rely on errors to "guess/infer" if a model is not TGI-served but don't do this 😬 I'll open a PR to fix this on hfh-side as well.

[radames]

The only issue is now all models are served/compatible with /v1/chat/completions we might ended up on the same

@radames not quite exactly. TGI-served models expose a /v1/chat/completions route which is fully compatible with the chat completion API. transformers-served models on the other hand don't have a /v1/chat/completions route but they

thanks @Wauplin, yes I miswrote my sentence, and I wanted to say that not all models are served with mixedbread-ai/mxbai-embed-large-v1 exactly your point!

and I think we should copy this logic to complete the model url, in case ones provide only a model id

           model_url = self._resolve_url(model)
            if not model_url.endswith("/chat/completions"):
                model_url += "/v1/chat/completions"

[radames]

I've add chatCompletion hint, feel free to change if there is a better way to send that to the request call

[coyotte508]

Can you update the tests in HfInference.spec.ts to use the chatCompletionStream method instead of streamingRequest?

(also check my README changes)

It should be good afterwards

[radames]

Can you update the tests in HfInference.spec.ts to use the chatCompletionStream method instead of streamingRequest?

(also check my README changes)

It should be good afterwards

@coyotte508 for the external providers, it has to be streamingRequest since they don't support receiving extra arguments that comes with the chatCompletionStream , which sends options params, that are compatible with our serveless API and TGI

Need to see exactly what options is sent to other endpoints and if they can be removed

[radames]

@coyotte508 here is the issue, when we use predefined tasks, such as textGeneration, which is compatible with our serveless API, https://huggingface.co/docs/api-inference/en/detailed_parameters, option parameter is valid. That's why on my example I use the custom request, since it won't send anything options

huggingface.js/packages/inference/src/lib/makeRequestOptions.ts

Lines 114 to 117 in b66fcf3

	: JSON.stringify({
	...(otherArgs.model && isUrl(otherArgs.model) ? omit(otherArgs, "model") : otherArgs),
	...(otherOptions && !isObjectEmpty(otherOptions) && { options: otherOptions }),
	}),

[coyotte508]

@coyotte508 here is the issue, when we use predefined tasks, such as textGeneration, which is compatible with our serveless API, huggingface.co/docs/api-inference/en/detailed_parameters, option parameter is valid. That's why on my example I use the custom request, since it won't send anything options

huggingface.js/packages/inference/src/lib/makeRequestOptions.ts

Lines 114 to 117 in b66fcf3

	: JSON.stringify({
	...(otherArgs.model && isUrl(otherArgs.model) ? omit(otherArgs, "model") : otherArgs),
	...(otherOptions && !isObjectEmpty(otherOptions) && { options: otherOptions }),
	}),

Yes I just want to know what the value of otherOptions is - what options we send and if they're necessary or can be sent via headers instead.

We could also detect mistral/openai in the domain name and remove the options in that case

[radames]

Great point. We might not need otherOptions. It seems like we can do it via headers, and we're already doing it.

[coyotte508]

Great point. We might not need otherOptions. It seems like we can do it via headers, and we're already doing it.

@radames can you run tests again with VCR_MODE=cache to record the new tapes.json?

Hopefully it should be good afterwards

[radames]

thanks @coyotte508 done! I ran the tests only for the new api VCR_MODE=cache pnpm exec vitest run -t "OpenAI Specs"

[julien-c]

yay! this is close to being merged? i want to play with it:)

[radames]

I think is just the final ok from @coyotte508 it's ready to be merged and released. I'm also waiting for this, to build a couple of things.

[coyotte508]

Thanks @radames !