\n",
" \n",
" | 0 | \n",
- " {'prompt': 'Tell me a story about a 19th century wombat gentleman. Keep it short.'} | \n",
- " Once upon a time, in the late 19th century, there was a wombat named Winston who lived in the rolling hills of the English countryside. Winston was a gentleman of refined taste and manners, known for his impeccable dress sense and love of fine tea. Every afternoon, he would don his finest top hat and tails and venture into the nearby village for a spot of tea and a chat with the locals. Despite his somewhat unconventional appearance, Winston was well-liked by all who knew him, and his wit | \n",
- " 0.978888 | \n",
+ " Tell me a story about a 19th century wombat gentleman. Keep it short. | \n",
+ " 0.1 | \n",
+ " Once upon a time, in the late 19th century, there was a wombat named Winston who lived in the rolling hills of the English countryside. Winston was a gentleman of refined taste and culture, known for his impeccable manners and love of fine tea. Every afternoon, he would don his finest top hat and take a stroll through the village, exchanging pleasantries with the locals and offering them scones and jam from his personal collection. Despite his dignified demeanor, Winston had a mischievous streak | \n",
+ " 0.538249 | \n",
"
\n",
" \n",
" | 1 | \n",
- " {'prompt': 'Tell me a story about a 22nd century wombat gentleman. Keep it short.'} | \n",
- " Certainly! Here's a short tale about a 22nd century wombat gentleman:\\n\\nIn the year 2105, a wombat named Winston lived in a sprawling underground burrow complex beneath the city of New Sydney. Winston was a gentleman of leisure, spending his days lounging on plush cushions and sipping tea served by his loyal butler-bot. He was known throughout the city for his impeccable taste in fine hats and his ability to recite Shakespearean sonnets with ease. | \n",
- " 0.355229 | \n",
+ " Tell me a story about a 19th century wombat gentleman. Keep it short. | \n",
+ " 1.5 | \n",
+ " Once upon a time, in the late 19th century, there was a wombat named Mr. Whiskers who lived in the rolling hills of the English countryside. Mr. Whiskers was a gentleman of great distinction and refinement, known for his impeccable manners and love of fine tea. He spent his days lounging in the sun, enjoying picnics with his friends, and engaging in spirited debates about the finer points of Victorian literature. Despite his leisurely lifestyle, Mr. Whiskers was a | \n",
+ " 0.267989 | \n",
+ "
\n",
+ " \n",
+ " | 2 | \n",
+ " Tell me a story about a 22nd century wombat gentleman. Keep it short. | \n",
+ " 0.1 | \n",
+ " Certainly! Here's a short tale about a 22nd century wombat gentleman:\\n\\nIn the year 2105, a wombat named Winston lived in a sprawling underground burrow complex beneath the city of New Sydney. Winston was a gentleman of leisure, spending his days lounging on plush cushions and sipping tea served by his loyal butler-bot. He was known throughout the city for his impeccable taste in fine hats and his extensive collection of rare, vintage cigars.\\n | \n",
+ " 0.279196 | \n",
+ "
\n",
+ " \n",
+ " | 3 | \n",
+ " Tell me a story about a 22nd century wombat gentleman. Keep it short. | \n",
+ " 1.5 | \n",
+ " Certainly! Here's a short tale about a 22nd century wombat gentleman:\\n\\nIn the year 2105, a wombat named Winston lived in a futuristic underground burrow. He was a gentleman of leisure, spending his days lounging on expensive virtual reality furniture and sipping champagne.\\n\\nOne day, Winston received an invitation to a high-society gathering hosted by a wealthy group of aliens. Eager to make a good impression, he donned his finest tux | \n",
+ " 0.270601 | \n",
"
\n",
" \n",
"\n",
""
],
"text/plain": [
- " input \\\n",
- "0 {'prompt': 'Tell me a story about a 19th century wombat gentleman. Keep it short.'} \n",
- "1 {'prompt': 'Tell me a story about a 22nd century wombat gentleman. Keep it short.'} \n",
+ " prompt \\\n",
+ "0 Tell me a story about a 19th century wombat gentleman. Keep it short. \n",
+ "1 Tell me a story about a 19th century wombat gentleman. Keep it short. \n",
+ "2 Tell me a story about a 22nd century wombat gentleman. Keep it short. \n",
+ "3 Tell me a story about a 22nd century wombat gentleman. Keep it short. \n",
+ "\n",
+ " temperature \\\n",
+ "0 0.1 \n",
+ "1 1.5 \n",
+ "2 0.1 \n",
+ "3 1.5 \n",
"\n",
- " response \\\n",
- "0 Once upon a time, in the late 19th century, there was a wombat named Winston who lived in the rolling hills of the English countryside. Winston was a gentleman of refined taste and manners, known for his impeccable dress sense and love of fine tea. Every afternoon, he would don his finest top hat and tails and venture into the nearby village for a spot of tea and a chat with the locals. Despite his somewhat unconventional appearance, Winston was well-liked by all who knew him, and his wit \n",
- "1 Certainly! Here's a short tale about a 22nd century wombat gentleman:\\n\\nIn the year 2105, a wombat named Winston lived in a sprawling underground burrow complex beneath the city of New Sydney. Winston was a gentleman of leisure, spending his days lounging on plush cushions and sipping tea served by his loyal butler-bot. He was known throughout the city for his impeccable taste in fine hats and his ability to recite Shakespearean sonnets with ease. \n",
+ " response \\\n",
+ "0 Once upon a time, in the late 19th century, there was a wombat named Winston who lived in the rolling hills of the English countryside. Winston was a gentleman of refined taste and culture, known for his impeccable manners and love of fine tea. Every afternoon, he would don his finest top hat and take a stroll through the village, exchanging pleasantries with the locals and offering them scones and jam from his personal collection. Despite his dignified demeanor, Winston had a mischievous streak \n",
+ "1 Once upon a time, in the late 19th century, there was a wombat named Mr. Whiskers who lived in the rolling hills of the English countryside. Mr. Whiskers was a gentleman of great distinction and refinement, known for his impeccable manners and love of fine tea. He spent his days lounging in the sun, enjoying picnics with his friends, and engaging in spirited debates about the finer points of Victorian literature. Despite his leisurely lifestyle, Mr. Whiskers was a \n",
+ "2 Certainly! Here's a short tale about a 22nd century wombat gentleman:\\n\\nIn the year 2105, a wombat named Winston lived in a sprawling underground burrow complex beneath the city of New Sydney. Winston was a gentleman of leisure, spending his days lounging on plush cushions and sipping tea served by his loyal butler-bot. He was known throughout the city for his impeccable taste in fine hats and his extensive collection of rare, vintage cigars.\\n \n",
+ "3 Certainly! Here's a short tale about a 22nd century wombat gentleman:\\n\\nIn the year 2105, a wombat named Winston lived in a futuristic underground burrow. He was a gentleman of leisure, spending his days lounging on expensive virtual reality furniture and sipping champagne.\\n\\nOne day, Winston received an invitation to a high-society gathering hosted by a wealthy group of aliens. Eager to make a good impression, he donned his finest tux \n",
"\n",
" latency \n",
- "0 0.978888 \n",
- "1 0.355229 "
+ "0 0.538249 \n",
+ "1 0.267989 \n",
+ "2 0.279196 \n",
+ "3 0.270601 "
]
},
"metadata": {},
@@ -197,7 +227,118 @@
"id": "bebb8023",
"metadata": {},
"source": [
- "This notebook is meant to showcase how you can use Replicate's API for Llama2. Please reference other notebook for evaluating the model's response. The examples from the notebook examples of OpenAI and Anthropic provide good references on how to evaluate LLM responses."
+ "This notebook is meant to showcase how you can use Replicate's API for Llama2. In this case, we will use a simple evalution - compute semantic distance to check if the model's response is similar to our expected output.\n",
+ "\n",
+ "Please reference other notebooks for various methods to evaluate a model's response. The examples include structual evaluation (e.g. for Python, JSON) and auto-evaluations (using another LLM)."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 11,
+ "id": "bdc76ecc",
+ "metadata": {},
+ "outputs": [
+ {
+ "data": {
+ "text/html": [
+ "