Skip to content

Commit

Permalink
Expand ReplicateLlama2 example with temperature and evaluation
Browse files Browse the repository at this point in the history
  • Loading branch information
@NivekT
NivekT committed Sep 4, 2023
1 parent 2f13c8f commit 1cb0618
Showing 1 changed file with 160 additions and 19 deletions.
179 changes: 160 additions & 19 deletions examples/notebooks/ReplicateLlama2.ipynb
View file
Original file line number Diff line number Diff line change
Expand Up @@ -98,8 +98,9 @@
"sd1 = \"replicate/llama-2-70b-chat:2796ee9483c3fd7aa2e171d38f4ca12251a30609463dcfd4cd76703f22e96cdf\"\n",
"models = [sd1] # You can specify multiple models here\n",
"input_kwargs = {\"prompt\": [\"Tell me a story about a 19th century wombat gentleman. Keep it short.\",\n",
" \"Tell me a story about a 22nd century wombat gentleman. Keep it short.\"]}\n",
"model_specific_kwargs = {sd1: {}}\n",
" \"Tell me a story about a 22nd century wombat gentleman. Keep it short.\"],\n",
" \"temperature\": [0.1, 1.5]}\n",
"model_specific_kwargs = {sd1: {}} # Optional, if you have any argument specifc to a model, you can pass it in there.\n",
"\n",
"experiment = ReplicateExperiment(models, input_kwargs, model_specific_kwargs)"
]
Expand All @@ -114,7 +115,7 @@
},
{
"cell_type": "code",
"execution_count": 5,
"execution_count": 4,
"id": "53b89b7f",
"metadata": {},
"outputs": [
Expand All @@ -139,40 +140,69 @@
" <thead>\n",
" <tr style=\"text-align: right;\">\n",
" <th></th>\n",
" <th>input</th>\n",
" <th>prompt</th>\n",
" <th>temperature</th>\n",
" <th>response</th>\n",
" <th>latency</th>\n",
" </tr>\n",
" </thead>\n",
" <tbody>\n",
" <tr>\n",
" <th>0</th>\n",
" <td>{'prompt': 'Tell me a story about a 19th century wombat gentleman. Keep it short.'}</td>\n",
" <td>Once upon a time, in the late 19th century, there was a wombat named Winston who lived in the rolling hills of the English countryside. Winston was a gentleman of refined taste and manners, known for his impeccable dress sense and love of fine tea. Every afternoon, he would don his finest top hat and tails and venture into the nearby village for a spot of tea and a chat with the locals. Despite his somewhat unconventional appearance, Winston was well-liked by all who knew him, and his wit</td>\n",
" <td>0.978888</td>\n",
" <td>Tell me a story about a 19th century wombat gentleman. Keep it short.</td>\n",
" <td>0.1</td>\n",
" <td>Once upon a time, in the late 19th century, there was a wombat named Winston who lived in the rolling hills of the English countryside. Winston was a gentleman of refined taste and culture, known for his impeccable manners and love of fine tea. Every afternoon, he would don his finest top hat and take a stroll through the village, exchanging pleasantries with the locals and offering them scones and jam from his personal collection. Despite his dignified demeanor, Winston had a mischievous streak</td>\n",
" <td>0.538249</td>\n",
" </tr>\n",
" <tr>\n",
" <th>1</th>\n",
" <td>{'prompt': 'Tell me a story about a 22nd century wombat gentleman. Keep it short.'}</td>\n",
" <td>Certainly! Here's a short tale about a 22nd century wombat gentleman:\\n\\nIn the year 2105, a wombat named Winston lived in a sprawling underground burrow complex beneath the city of New Sydney. Winston was a gentleman of leisure, spending his days lounging on plush cushions and sipping tea served by his loyal butler-bot. He was known throughout the city for his impeccable taste in fine hats and his ability to recite Shakespearean sonnets with ease.</td>\n",
" <td>0.355229</td>\n",
" <td>Tell me a story about a 19th century wombat gentleman. Keep it short.</td>\n",
" <td>1.5</td>\n",
" <td>Once upon a time, in the late 19th century, there was a wombat named Mr. Whiskers who lived in the rolling hills of the English countryside. Mr. Whiskers was a gentleman of great distinction and refinement, known for his impeccable manners and love of fine tea. He spent his days lounging in the sun, enjoying picnics with his friends, and engaging in spirited debates about the finer points of Victorian literature. Despite his leisurely lifestyle, Mr. Whiskers was a</td>\n",
" <td>0.267989</td>\n",
" </tr>\n",
" <tr>\n",
" <th>2</th>\n",
" <td>Tell me a story about a 22nd century wombat gentleman. Keep it short.</td>\n",
" <td>0.1</td>\n",
" <td>Certainly! Here's a short tale about a 22nd century wombat gentleman:\\n\\nIn the year 2105, a wombat named Winston lived in a sprawling underground burrow complex beneath the city of New Sydney. Winston was a gentleman of leisure, spending his days lounging on plush cushions and sipping tea served by his loyal butler-bot. He was known throughout the city for his impeccable taste in fine hats and his extensive collection of rare, vintage cigars.\\n</td>\n",
" <td>0.279196</td>\n",
" </tr>\n",
" <tr>\n",
" <th>3</th>\n",
" <td>Tell me a story about a 22nd century wombat gentleman. Keep it short.</td>\n",
" <td>1.5</td>\n",
" <td>Certainly! Here's a short tale about a 22nd century wombat gentleman:\\n\\nIn the year 2105, a wombat named Winston lived in a futuristic underground burrow. He was a gentleman of leisure, spending his days lounging on expensive virtual reality furniture and sipping champagne.\\n\\nOne day, Winston received an invitation to a high-society gathering hosted by a wealthy group of aliens. Eager to make a good impression, he donned his finest tux</td>\n",
" <td>0.270601</td>\n",
" </tr>\n",
" </tbody>\n",
"</table>\n",
"</div>"
],
"text/plain": [
" input \\\n",
"0 {'prompt': 'Tell me a story about a 19th century wombat gentleman. Keep it short.'} \n",
"1 {'prompt': 'Tell me a story about a 22nd century wombat gentleman. Keep it short.'} \n",
" prompt \\\n",
"0 Tell me a story about a 19th century wombat gentleman. Keep it short. \n",
"1 Tell me a story about a 19th century wombat gentleman. Keep it short. \n",
"2 Tell me a story about a 22nd century wombat gentleman. Keep it short. \n",
"3 Tell me a story about a 22nd century wombat gentleman. Keep it short. \n",
"\n",
" temperature \\\n",
"0 0.1 \n",
"1 1.5 \n",
"2 0.1 \n",
"3 1.5 \n",
"\n",
" response \\\n",
"0 Once upon a time, in the late 19th century, there was a wombat named Winston who lived in the rolling hills of the English countryside. Winston was a gentleman of refined taste and manners, known for his impeccable dress sense and love of fine tea. Every afternoon, he would don his finest top hat and tails and venture into the nearby village for a spot of tea and a chat with the locals. Despite his somewhat unconventional appearance, Winston was well-liked by all who knew him, and his wit \n",
"1 Certainly! Here's a short tale about a 22nd century wombat gentleman:\\n\\nIn the year 2105, a wombat named Winston lived in a sprawling underground burrow complex beneath the city of New Sydney. Winston was a gentleman of leisure, spending his days lounging on plush cushions and sipping tea served by his loyal butler-bot. He was known throughout the city for his impeccable taste in fine hats and his ability to recite Shakespearean sonnets with ease. \n",
" response \\\n",
"0 Once upon a time, in the late 19th century, there was a wombat named Winston who lived in the rolling hills of the English countryside. Winston was a gentleman of refined taste and culture, known for his impeccable manners and love of fine tea. Every afternoon, he would don his finest top hat and take a stroll through the village, exchanging pleasantries with the locals and offering them scones and jam from his personal collection. Despite his dignified demeanor, Winston had a mischievous streak \n",
"1 Once upon a time, in the late 19th century, there was a wombat named Mr. Whiskers who lived in the rolling hills of the English countryside. Mr. Whiskers was a gentleman of great distinction and refinement, known for his impeccable manners and love of fine tea. He spent his days lounging in the sun, enjoying picnics with his friends, and engaging in spirited debates about the finer points of Victorian literature. Despite his leisurely lifestyle, Mr. Whiskers was a \n",
"2 Certainly! Here's a short tale about a 22nd century wombat gentleman:\\n\\nIn the year 2105, a wombat named Winston lived in a sprawling underground burrow complex beneath the city of New Sydney. Winston was a gentleman of leisure, spending his days lounging on plush cushions and sipping tea served by his loyal butler-bot. He was known throughout the city for his impeccable taste in fine hats and his extensive collection of rare, vintage cigars.\\n \n",
"3 Certainly! Here's a short tale about a 22nd century wombat gentleman:\\n\\nIn the year 2105, a wombat named Winston lived in a futuristic underground burrow. He was a gentleman of leisure, spending his days lounging on expensive virtual reality furniture and sipping champagne.\\n\\nOne day, Winston received an invitation to a high-society gathering hosted by a wealthy group of aliens. Eager to make a good impression, he donned his finest tux \n",
"\n",
" latency \n",
"0 0.978888 \n",
"1 0.355229 "
"0 0.538249 \n",
"1 0.267989 \n",
"2 0.279196 \n",
"3 0.270601 "
]
},
"metadata": {},
Expand All @@ -197,7 +227,118 @@
"id": "bebb8023",
"metadata": {},
"source": [
"This notebook is meant to showcase how you can use Replicate's API for Llama2. Please reference other notebook for evaluating the model's response. The examples from the notebook examples of OpenAI and Anthropic provide good references on how to evaluate LLM responses."
"This notebook is meant to showcase how you can use Replicate's API for Llama2. In this case, we will use a simple evalution - compute semantic distance to check if the model's response is similar to our expected output.\n",
"\n",
"Please reference other notebooks for various methods to evaluate a model's response. The examples include structual evaluation (e.g. for Python, JSON) and auto-evaluations (using another LLM)."
]
},
{
"cell_type": "code",
"execution_count": 11,
"id": "bdc76ecc",
"metadata": {},
"outputs": [
{
"data": {
"text/html": [
"<div>\n",
"<style scoped>\n",
" .dataframe tbody tr th:only-of-type {\n",
" vertical-align: middle;\n",
" }\n",
"\n",
" .dataframe tbody tr th {\n",
" vertical-align: top;\n",
" }\n",
"\n",
" .dataframe thead th {\n",
" text-align: right;\n",
" }\n",
"</style>\n",
"<table border=\"1\" class=\"dataframe\">\n",
" <thead>\n",
" <tr style=\"text-align: right;\">\n",
" <th></th>\n",
" <th>prompt</th>\n",
" <th>temperature</th>\n",
" <th>response</th>\n",
" <th>latency</th>\n",
" <th>similar_to_expected</th>\n",
" </tr>\n",
" </thead>\n",
" <tbody>\n",
" <tr>\n",
" <th>0</th>\n",
" <td>Tell me a story about a 19th century wombat gentleman. Keep it short.</td>\n",
" <td>0.1</td>\n",
" <td>Once upon a time, in the late 19th century, there was a wombat named Winston who lived in the rolling hills of the English countryside. Winston was a gentleman of refined taste and culture, known for his impeccable manners and love of fine tea. Every afternoon, he would don his finest top hat and take a stroll through the village, exchanging pleasantries with the locals and offering them scones and jam from his personal collection. Despite his dignified demeanor, Winston had a mischievous streak</td>\n",
" <td>0.538249</td>\n",
" <td>0.566713</td>\n",
" </tr>\n",
" <tr>\n",
" <th>1</th>\n",
" <td>Tell me a story about a 19th century wombat gentleman. Keep it short.</td>\n",
" <td>1.5</td>\n",
" <td>Once upon a time, in the late 19th century, there was a wombat named Mr. Whiskers who lived in the rolling hills of the English countryside. Mr. Whiskers was a gentleman of great distinction and refinement, known for his impeccable manners and love of fine tea. He spent his days lounging in the sun, enjoying picnics with his friends, and engaging in spirited debates about the finer points of Victorian literature. Despite his leisurely lifestyle, Mr. Whiskers was a</td>\n",
" <td>0.267989</td>\n",
" <td>0.601912</td>\n",
" </tr>\n",
" <tr>\n",
" <th>2</th>\n",
" <td>Tell me a story about a 22nd century wombat gentleman. Keep it short.</td>\n",
" <td>0.1</td>\n",
" <td>Certainly! Here's a short tale about a 22nd century wombat gentleman:\\n\\nIn the year 2105, a wombat named Winston lived in a sprawling underground burrow complex beneath the city of New Sydney. Winston was a gentleman of leisure, spending his days lounging on plush cushions and sipping tea served by his loyal butler-bot. He was known throughout the city for his impeccable taste in fine hats and his extensive collection of rare, vintage cigars.\\n</td>\n",
" <td>0.279196</td>\n",
" <td>0.688690</td>\n",
" </tr>\n",
" <tr>\n",
" <th>3</th>\n",
" <td>Tell me a story about a 22nd century wombat gentleman. Keep it short.</td>\n",
" <td>1.5</td>\n",
" <td>Certainly! Here's a short tale about a 22nd century wombat gentleman:\\n\\nIn the year 2105, a wombat named Winston lived in a futuristic underground burrow. He was a gentleman of leisure, spending his days lounging on expensive virtual reality furniture and sipping champagne.\\n\\nOne day, Winston received an invitation to a high-society gathering hosted by a wealthy group of aliens. Eager to make a good impression, he donned his finest tux</td>\n",
" <td>0.270601</td>\n",
" <td>0.639222</td>\n",
" </tr>\n",
" </tbody>\n",
"</table>\n",
"</div>"
],
"text/plain": [
" prompt \\\n",
"0 Tell me a story about a 19th century wombat gentleman. Keep it short. \n",
"1 Tell me a story about a 19th century wombat gentleman. Keep it short. \n",
"2 Tell me a story about a 22nd century wombat gentleman. Keep it short. \n",
"3 Tell me a story about a 22nd century wombat gentleman. Keep it short. \n",
"\n",
" temperature \\\n",
"0 0.1 \n",
"1 1.5 \n",
"2 0.1 \n",
"3 1.5 \n",
"\n",
" response \\\n",
"0 Once upon a time, in the late 19th century, there was a wombat named Winston who lived in the rolling hills of the English countryside. Winston was a gentleman of refined taste and culture, known for his impeccable manners and love of fine tea. Every afternoon, he would don his finest top hat and take a stroll through the village, exchanging pleasantries with the locals and offering them scones and jam from his personal collection. Despite his dignified demeanor, Winston had a mischievous streak \n",
"1 Once upon a time, in the late 19th century, there was a wombat named Mr. Whiskers who lived in the rolling hills of the English countryside. Mr. Whiskers was a gentleman of great distinction and refinement, known for his impeccable manners and love of fine tea. He spent his days lounging in the sun, enjoying picnics with his friends, and engaging in spirited debates about the finer points of Victorian literature. Despite his leisurely lifestyle, Mr. Whiskers was a \n",
"2 Certainly! Here's a short tale about a 22nd century wombat gentleman:\\n\\nIn the year 2105, a wombat named Winston lived in a sprawling underground burrow complex beneath the city of New Sydney. Winston was a gentleman of leisure, spending his days lounging on plush cushions and sipping tea served by his loyal butler-bot. He was known throughout the city for his impeccable taste in fine hats and his extensive collection of rare, vintage cigars.\\n \n",
"3 Certainly! Here's a short tale about a 22nd century wombat gentleman:\\n\\nIn the year 2105, a wombat named Winston lived in a futuristic underground burrow. He was a gentleman of leisure, spending his days lounging on expensive virtual reality furniture and sipping champagne.\\n\\nOne day, Winston received an invitation to a high-society gathering hosted by a wealthy group of aliens. Eager to make a good impression, he donned his finest tux \n",
"\n",
" latency similar_to_expected \n",
"0 0.538249 0.566713 \n",
"1 0.267989 0.601912 \n",
"2 0.279196 0.688690 \n",
"3 0.270601 0.639222 "
]
},
"metadata": {},
"output_type": "display_data"
}
],
"source": [
"from prompttools.utils import semantic_similarity\n",
"\n",
"expected_responses = [\"wombat gentleman in 19th century\"] * 2 + [\"wombat gentleman in 22nd century\"] * 2\n",
"experiment.evaluate(\"similar_to_expected\", semantic_similarity, expected=expected_responses)\n",
"experiment.visualize()"
]
}
],
Expand Down

0 comments on commit 1cb0618

Please sign in to comment.