Initial replicate experiment example

examples/notebooks/ReplicateExperiment.ipynb

@@ -0,0 +1,242 @@
+{
+ "cells": [
+ {
+ "cell_type": "markdown",
+ "id": "0a13ddc8",
+ "metadata": {},
+ "source": [
+ "# Replicate Experiment Example"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "623f0cfe",
+ "metadata": {},
+ "source": [
+ "## Installations"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 1,
+ "id": "885dabeb",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "# !pip install --quiet --force-reinstall prompttools"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "2eac35f8",
+ "metadata": {},
+ "source": [
+ "## Setup imports and API keys"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "5edba05a",
+ "metadata": {},
+ "source": [
+ "First, we'll need to set our API keys."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 2,
+ "id": "ed4e635e",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "import os\n",
+ "\n",
+ "os.environ[\"DEBUG\"] = \"1\" # Set this to \"\" to call Replicate's API, otherwise a mock function is used\n",
+ "os.environ[\"REPLICATE_API_TOKEN\"] = \"\""
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "842f1e47",
+ "metadata": {},
+ "source": [
+ "Then we'll import the relevant `prompttools` modules to setup our experiment."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 3,
+ "id": "beaa70a1",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "from prompttools.experiment import ReplicateExperiment"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "622dea9a",
+ "metadata": {},
+ "source": [
+ "## Run an experiment"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "3babfe5a",
+ "metadata": {},
+ "source": [
+ "Next, we create our test inputs. We can iterate over models, inputs, and configurations like temperature."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 4,
+ "id": "347590cf",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "sd1 = \"stability-ai/stable-diffusion:27b93a2413e7f36cd83da926f3656280b2931564ff050bf9575f1fdf9bcd7478\"\n",
+ "models = [sd1] # You can specify multiple models here\n",
+ "input_kwargs = {\"prompt\": [\"a 19th century portrait of a wombat gentleman\", \"a 22nd century portrait of a wombat gentleman\"]}\n",
+ "model_specific_kwargs = {sd1: {}}\n",
+ "\n",
+ "experiment = ReplicateExperiment(models, input_kwargs, model_specific_kwargs)"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "f3fa5450",
+ "metadata": {},
+ "source": [
+ "We can then run the experiment to get results."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 5,
+ "id": "1a261583",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "experiment.run()"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 6,
+ "id": "fc4c4fd2",
+ "metadata": {},
+ "outputs": [
+ {
+ "data": {
+ "text/html": [
+ "<div>\n",
+ "<style scoped>\n",
+ " .dataframe tbody tr th:only-of-type {\n",
+ " vertical-align: middle;\n",
+ " }\n",
+ "\n",
+ " .dataframe tbody tr th {\n",
+ " vertical-align: top;\n",
+ " }\n",
+ "\n",
+ " .dataframe thead th {\n",
+ " text-align: right;\n",
+ " }\n",
+ "</style>\n",
+ "<table border=\"1\" class=\"dataframe\">\n",
+ " <thead>\n",
+ " <tr style=\"text-align: right;\">\n",
+ " <th></th>\n",
+ " <th>input</th>\n",
+ " <th>response</th>\n",
+ " <th>latency</th>\n",
+ " </tr>\n",
+ " </thead>\n",
+ " <tbody>\n",
+ " <tr>\n",
+ " <th>0</th>\n",
+ " <td>{'prompt': 'a 19th century portrait of a wombat gentleman'}</td>\n",
+ " <td>[https://mock.com/1.jpeg, https://mock.com/2.jpeg, https://mock.com/3.jpeg]</td>\n",
+ " <td>0.000008</td>\n",
+ " </tr>\n",
+ " <tr>\n",
+ " <th>1</th>\n",
+ " <td>{'prompt': 'a 22nd century portrait of a wombat gentleman'}</td>\n",
+ " <td>[https://mock.com/1.jpeg, https://mock.com/2.jpeg, https://mock.com/3.jpeg]</td>\n",
+ " <td>0.000003</td>\n",
+ " </tr>\n",
+ " </tbody>\n",
+ "</table>\n",
+ "</div>"
+ ],
+ "text/plain": [
+ " input \\\n",
+ "0 {'prompt': 'a 19th century portrait of a wombat gentleman'} \n",
+ "1 {'prompt': 'a 22nd century portrait of a wombat gentleman'} \n",
+ "\n",
+ " response \\\n",
+ "0 [https://mock.com/1.jpeg, https://mock.com/2.jpeg, https://mock.com/3.jpeg] \n",
+ "1 [https://mock.com/1.jpeg, https://mock.com/2.jpeg, https://mock.com/3.jpeg] \n",
+ "\n",
+ " latency \n",
+ "0 0.000008 \n",
+ "1 0.000003 "
+ ]
+ },
+ "metadata": {},
+ "output_type": "display_data"
+ }
+ ],
+ "source": [
+ "experiment.visualize()"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "266c13eb",
+ "metadata": {},
+ "source": [
+ "## Evaluate the model response"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "bebb8023",
+ "metadata": {},
+ "source": [
+ "This will be updated once we add image evaluation functions. Please don't hesitate to open an issue if you have any suggestions. Stay tuned."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "78c0c9f6",
+ "metadata": {},
+ "outputs": [],
+ "source": []
+ }
+ ],
+ "metadata": {
+ "kernelspec": {
+ "display_name": "Python 3 (ipykernel)",
+ "language": "python",
+ "name": "python3"
+ },
+ "language_info": {
+ "codemirror_mode": {
+ "name": "ipython",
+ "version": 3
+ },
+ "file_extension": ".py",
+ "mimetype": "text/x-python",
+ "name": "python",
+ "nbconvert_exporter": "python",
+ "pygments_lexer": "ipython3",
+ "version": "3.11.4"
+ }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 5
+}