v0.2.5: Chat Turns, LLM Scorers

[ianarawjo]

We're excited to release two new nodes: Chat Turns and LLM Scorers. These nodes came from feedback during user sessions:

• Some users wanted to first tell chat models 'how to act', and then wanted to put their real prompt in the second turn.
• Some users wanted a quicker, cheaper way to 'evaluate' responses and visualize results.

We describe these new nodes below, as well as a few quality-of-life improvements.

🗣️ Chat Turn nodes

Chat models are all the rage (in fact, they are so important that OpenAI announced it would no longer support plain-old text generation models going forward.) Yet strikingly, very few prompt engineering tools let you evaluate LLM outputs beyond a prompt.

Now with Chat Turn nodes, you can continue conversations beyond a single prompt. In fact, you can:

Continue multiple conversations simultaneously across multiple LLMs

Just connect the Chat Turn to your initial Prompt Node, and voilà:

Here, I've first prompted four chat models: GPT3.5, GPT-4, Claude-2, and PaLM with the question: "What was the first {game} game?". Then I ask a follow-up question, "What was the second?" By default, Chat Turns continue the conversation with all LLMs that were used before, allowing you to follow-up on LLM responses in parallel. (You can also toggle that off, if you want to query different models --more details below).

Template chat messages, just like prompts

You can do everything you can with Chat Turns that you could with Prompt Nodes, including prompt templating and adding input variables. For instance, here's a prompt template as a follow-up message:

Note
In fact, Chat Turns are merely modified Prompt Nodes, and use the underlying PromptNode class.

Start a conversation with one LLM, and continue it with a different LLM

Chat Turns include a toggle of whether you'd like to continue chatting with the same LLMs, or query different ones, passing chat context to the new models. With this, you can start a conversation with one LLM and continue it with another (or several):

Supported chat models

Simple in concept, chat turns were the result of 2 weeks' work, revising many parts of the ChainForge backend to store and carry chat context. Chat history is automatically translated to the appropriate format for a number of providers:

• OpenAI chat models
• Anthropic models (Claude)
• Google PaLM2 chat
• HuggingFace (you need to set 'Model Type' in Settings to 'chat', and choose a Conversation model or custom endpoint. Currently there's only one chat model listed in ChainForge dropdown: microsoft/DialoGPT. Go to the HuggingFace site to find more!)

Warning
If you use a non-chat, text completions model like GPT-2, chat turns will still function, but the chat context won't be passed into the text completions model.

Let us know what you think!

🤖 LLM Scorer nodes

More commonly called "LLM evaluators", LLM scorer nodes allow you to use an LLM to 'grade'/score outputs of other LLMs:

Although ChainForge supported this functionality before via prompt chaining, it was not straightforward and required an additional chain to a code evaluator node for postprocessing. You can now connect the output of the scorer directly to a Vis Node to plot outputs. For instance, here's GPT-4 scoring whether different LLM responses apologized for a mistake:

Note that LLM scores are finicky --if one score isn't in the right format (true/false), visualization nodes won't work properly, because they'll think the outputs are notof boolean type but categorical. We'll work on improving this, but, for now, enjoy LLM scorers!

❗ Why we're not calling LLM scorers 'LLM evaluators'

We thought long and hard about what to call LLMs that score outputs of other LLMs. Ultimately, using LLMs to score outputs is helpful, and can save time when it's hard to write code to achieve the same effect. However, LLMs are imperfect. Although the AI community currently uses the term 'LLM evaluator,' we ultimately decided not to use that term, for a few reasons:

1. LLM scores should not be blindly trusted. They are helpful if you already have a sense of what you're looking for, and want to grade hundreds of responses and don't care about picture-perfect accuracy. This is especially true after playing with LLM scorer nodes for a while and finding that small tweaks to the scoring prompt can result in vast differences in results.
2. Evaluators, like 'graders' or 'annotations,' is a term that has connotations with humans (i.e. human evaluator). We want to avoid anthropomorphizing LLMs, which contributes to peoples' over-trust in them. 'Scorers' still has human connotations, but arguably less so, and less authoritative ones than 'evaluator'.
3. Evaluators is a term in ChainForge that refers to programs that score responses. Calling LLM scorers 'evaluators' loosely equates them with programmatic evaluators, suggesting they carry the same authority. Although code can be wrong or incorrect, the scoring process for code is inspectable and auditable --not so with LLMs.

Fundamentally, then, we disagree with the positions taken by projects like LangChain, which tend to emphasize LLM scorers as the go-to solution for evaluation. We believe this is a massive mistake that tends to mislead people and causing them to over-trust AI outputs, including ML researchers at MIT. In choosing the term Scorers, we aim to --at the very least --distance ourselves from such positions.

Other changes

• Inspecting true/false scored responses (in Evaluators or LLM scorers) will now show false in red, to easily eyeball failure cases:

• In Response Inspectors, the term "Hierarchy" has been replaced with "Grouped List". Grouped Lists are again the default.
• In table view of the response inspector, you can now choose what variable to use for columns. With this method you can compare across prompt templates or indeed anything of interest:

Future Work

Chat Turns opened up a whole new can of worms, both for the UI, and for evaluation. Some open questions are:

• How can we display Chat History in response inspectors? Right now, you'll only see the latest response from the LLM. There's more design work to do such that you can view the chat context of specific responses.
• Should there be a Chat History node so you can predefine/preset chat histories to test on, without needing to query an LLM?

We hope to prioritize such features based on user feedback. If you use Chat Turns or LLM Scorers, let us know what you think --open an Issue or start a Discussion! 👍

---

This discussion was created from the release v0.2.5: Chat Turns, LLM Scorers.