-
Notifications
You must be signed in to change notification settings - Fork 383
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
Add example of using
generate_events_async
with streaming.
- Loading branch information
Showing
1 changed file
with
195 additions
and
0 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,195 @@ | ||
{ | ||
"cells": [ | ||
{ | ||
"cell_type": "markdown", | ||
"source": [ | ||
"# Using `generate_events_async` and Streaming\n", | ||
"\n", | ||
"This guide will teach you how to use the `LLMRails.generate_events_async` method and stream the bot response. This is useful when your rails need to process other events, in addition to user and bot messages. \n", | ||
"\n", | ||
"**Important**: the streaming option does not work with the synchronous methods `LLMRails.generate_events`.\n", | ||
"\n", | ||
"**Note**: this guide assumes you have successfully installed NeMo Guardrails and the OpenAI package. If not, please refer to the [Hello World](../../docs/getting_started/1_hello_world) guide." | ||
], | ||
"metadata": { | ||
"collapsed": false | ||
}, | ||
"id": "53e0d6f2f984979d" | ||
}, | ||
{ | ||
"cell_type": "code", | ||
"execution_count": 1, | ||
"outputs": [], | ||
"source": [ | ||
"import os\n", | ||
"\n", | ||
"# Setting the TOKENIZERS_PARALLELISM to get rid of the forking warning\n", | ||
"os.environ[\"TOKENIZERS_PARALLELISM\"] = \"false\"" | ||
], | ||
"metadata": { | ||
"collapsed": false, | ||
"ExecuteTime": { | ||
"end_time": "2023-11-23T08:47:33.941631Z", | ||
"start_time": "2023-11-23T08:47:33.939231Z" | ||
} | ||
}, | ||
"id": "4b18190855adfe3a" | ||
}, | ||
{ | ||
"cell_type": "markdown", | ||
"source": [ | ||
"## Step 1: create a config \n", | ||
"\n", | ||
"Let's create a simple config:" | ||
], | ||
"metadata": { | ||
"collapsed": false | ||
}, | ||
"id": "35fb674a4026ec51" | ||
}, | ||
{ | ||
"cell_type": "code", | ||
"execution_count": 2, | ||
"outputs": [], | ||
"source": [ | ||
"from nemoguardrails import RailsConfig, LLMRails\n", | ||
"\n", | ||
"YAML_CONFIG = \"\"\"\n", | ||
"models:\n", | ||
" - type: main\n", | ||
" engine: openai\n", | ||
" model: gpt-4\n", | ||
"\n", | ||
"streaming: True\n", | ||
"\"\"\"\n", | ||
"\n", | ||
"config = RailsConfig.from_content(yaml_content=YAML_CONFIG)\n", | ||
"app = LLMRails(config)" | ||
], | ||
"metadata": { | ||
"collapsed": false, | ||
"ExecuteTime": { | ||
"end_time": "2023-11-23T08:47:39.137061Z", | ||
"start_time": "2023-11-23T08:47:33.942980Z" | ||
} | ||
}, | ||
"id": "d9bac50b3383915e" | ||
}, | ||
{ | ||
"cell_type": "markdown", | ||
"source": [ | ||
"Next, we need to create a streaming handler and register it in the current async context by setting the value of the AsyncIO context variable `streaming_handler_var`, create a demo task that prints the tokens and make the `generate_events_async` call:" | ||
], | ||
"metadata": { | ||
"collapsed": false | ||
}, | ||
"id": "9036d5ce62e352f0" | ||
}, | ||
{ | ||
"cell_type": "code", | ||
"execution_count": 3, | ||
"outputs": [ | ||
{ | ||
"name": "stdout", | ||
"output_type": "stream", | ||
"text": [ | ||
"CHUNK: Hello\n", | ||
"CHUNK: !\n", | ||
"CHUNK: As\n", | ||
"CHUNK: an\n", | ||
"CHUNK: AI\n", | ||
"CHUNK: ,\n", | ||
"CHUNK: I\n", | ||
"CHUNK: don\n", | ||
"CHUNK: 't\n", | ||
"CHUNK: have\n", | ||
"CHUNK: feelings\n", | ||
"CHUNK: ,\n", | ||
"CHUNK: but\n", | ||
"CHUNK: I\n", | ||
"CHUNK: 'm\n", | ||
"CHUNK: here\n", | ||
"CHUNK: and\n", | ||
"CHUNK: ready\n", | ||
"CHUNK: to\n", | ||
"CHUNK: assist\n", | ||
"CHUNK: you\n", | ||
"CHUNK: .\n", | ||
"CHUNK: How\n", | ||
"CHUNK: can\n", | ||
"CHUNK: I\n", | ||
"CHUNK: help\n", | ||
"CHUNK: you\n", | ||
"CHUNK: today\n", | ||
"CHUNK: ?\n", | ||
"There were 12 new events.\n" | ||
] | ||
} | ||
], | ||
"source": [ | ||
"import asyncio \n", | ||
"from nemoguardrails.streaming import StreamingHandler\n", | ||
"from nemoguardrails.context import streaming_handler_var\n", | ||
"\n", | ||
"# Create the streaming handler and register it\n", | ||
"streaming_handler = StreamingHandler()\n", | ||
"streaming_handler_var.set(streaming_handler)\n", | ||
"\n", | ||
"# For demo purposes, create a task that prints the tokens.\n", | ||
"async def process_tokens():\n", | ||
" async for chunk in streaming_handler:\n", | ||
" print(f\"CHUNK: {chunk}\")\n", | ||
"\n", | ||
"asyncio.create_task(process_tokens())\n", | ||
"\n", | ||
"# Call the events-based API\n", | ||
"events = [{\n", | ||
" \"type\": \"UtteranceUserActionFinished\",\n", | ||
" \"final_transcript\": \"Hello! How are you?\"\n", | ||
"}]\n", | ||
"\n", | ||
"new_events = await app.generate_events_async(events)\n", | ||
"print(f\"There were {len(new_events)} new events.\")" | ||
], | ||
"metadata": { | ||
"collapsed": false, | ||
"ExecuteTime": { | ||
"end_time": "2023-11-23T08:47:42.846315Z", | ||
"start_time": "2023-11-23T08:47:39.143972Z" | ||
} | ||
}, | ||
"id": "60fa80f584cce58c" | ||
}, | ||
{ | ||
"cell_type": "markdown", | ||
"source": [ | ||
"As expected, the tokens were printed as they were generated, and at the end we get the complete list of events that were generated. For more details on the structure of the events, check out the [Events-based API Guide](../../docs/user_guides/advanced/event-based-api.md)." | ||
], | ||
"metadata": { | ||
"collapsed": false | ||
}, | ||
"id": "29f1381b93da53b4" | ||
} | ||
], | ||
"metadata": { | ||
"kernelspec": { | ||
"display_name": "Python 3", | ||
"language": "python", | ||
"name": "python3" | ||
}, | ||
"language_info": { | ||
"codemirror_mode": { | ||
"name": "ipython", | ||
"version": 2 | ||
}, | ||
"file_extension": ".py", | ||
"mimetype": "text/x-python", | ||
"name": "python", | ||
"nbconvert_exporter": "python", | ||
"pygments_lexer": "ipython2", | ||
"version": "2.7.6" | ||
} | ||
}, | ||
"nbformat": 4, | ||
"nbformat_minor": 5 | ||
} |