Support for tool/function calling

[christianliebel]

I would like to request the addition of tool/function calling functionality to the Prompt API. This feature is available in some models and allows the model to invoke specific actions using a well-defined contract, typically in JSON format. This functionality is beneficial for various use cases that require outputs of a specific structure.

Examples:

• https://docs.retellai.com/retell-llm/tool-calling
• https://docs.mistral.ai/capabilities/function_calling/
• https://platform.openai.com/docs/guides/function-calling

[KenjiBaheux]

Thank you for the suggestion to add tool/function calling functionality.

To assess the feasibility of this feature, we would appreciate it if folks could provide more details on the typical context size required for defining the functions needed in actual use cases. This information will help us understand the potential impact on performance and resource requirements, especially for an on-device context.

[CakeCrusher]

Function calling at this size of model is not practical. Although constrained generation would be a practical intermediate solution for this.
You can delegate constrained generation to the user by giving them access to output logits.

This is the most practical approach IMO

[ChristianWeyer]

Function Calling may indeed be too heavy for those models.
However, models fine-tuned for JSON data extraction would be really helpful. Then we can use approved patterns that are e.g. implemented in Instructor (https://js.useinstructor.com/)

[schlessera]

Function calling might be viable for specially tuned models even at lower sizes. Having a flexible API in place to allow for function execution would allow experimentation for that. There might be alternative ways to make this work with limited resources, like having dedicated helper logic in the browser to structure input and output, or extracting arguments and formatting them. And with advances in training the smaller models, they could still drastically improve in that area. There could also be a path way where two smaller models with separate responsibilities could collaborate, one for understanding the generic language and one for reasoning about solving the problem at hand outside of the boundaries of human languages.

If the model can, within a certain threshold of reliability, assess whether a function might be adapted to solve an identified task, it can state so and pass arguments to back to the consumer code. The consumer code could then opt to either run that function directly within the browser thread, or forward it to a service worker. It might even make sense to default to service workers as the default execution model, so that the in-browser API knows about the functions and the service worker to execute them in, and the entire flow can be executed without requiring intermediate assistance by the main thread.

This could even allow for browser extensions to provide a set of standard functions to be called to augment the capabilities of an LLM model in an easy way for end users (provided the security implications are correctly handled).