notebook_examples/feature_store_embeddings_openai.ipynb

{
 "cells": [
  {
   "cell_type": "raw",
   "id": "b1708a81",
   "metadata": {},
   "source": [
    "qweews@notebook{feature_store-querying.ipynb,\n",
    "    title: Using feature store for feature querying using pandas like interface for query and join,\n",
    "    summary: Feature store quickstart guide to perform feature querying using pandas like interface for query and join.,\n",
    "    developed_on: fspyspark32_p38_cpu_v3,\n",
    "    keywords: feature store, querying,\n",
    "    license: Universal Permissive License v 1.0\n",
    "}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "dd069038",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2023-05-24T08:26:08.572567Z",
     "start_time": "2023-05-24T08:26:08.328013Z"
    }
   },
   "outputs": [],
   "source": [
    "!odsc conda install -s fspyspark32_p38_cpu_v3"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "4a7a9e44",
   "metadata": {},
   "outputs": [],
   "source": [
    "!pip install openai==0.28 python-dotenv spacy tiktoken"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0ef82618",
   "metadata": {},
   "source": [
    "Oracle Data Science service sample notebook.\n",
    "\n",
    "Copyright (c) 2022 Oracle, Inc. All rights reserved. Licensed under the [Universal Permissive License v 1.0](https://oss.oracle.com/licenses/upl).\n",
    "\n",
    "***\n",
    "\n",
    "# <font color=\"red\">OpenAI embeddings in feature store</font>\n",
    "<p style=\"margin-left:10%; margin-right:10%;\">by the <font color=\"teal\">Oracle Cloud Infrastructure Data Science Service.</font></p>\n",
    "\n",
    "---\n",
    "# Overview:\n",
    "---\n",
    "\n",
    "Compatible conda pack: [PySpark 3.2 and Feature store](https://docs.oracle.com/iaas/data-science/using/conda-pyspark-fam.htm) for CPU on Python 3.8\n",
    "\n",
    "## Contents:\n",
    "\n",
    "- <a href=\"#concepts\">1. Introduction</a>\n",
    "- <a href='#pre-requisites'>1. Pre-requisites</a>\n",
    "    - <a href='#policies'>2.1 Policies</a>\n",
    "    - <a href='#prerequisites_authentication'>2.2 Authentication</a>\n",
    "    - <a href='#prerequisites_variables'>2.3 Variables</a>\n",
    "- <a href='#featurestore_querying'>3. Feature store querying</a>\n",
    "    - <a href='#data_exploration'>3.1. Exploration of data in feature store</a>\n",
    "    - <a href='#load_featuregroup'>3.2. Load feature groups</a>\n",
    "    - <a href='#explore_featuregroup'>3.3. Explore feature groups</a>\n",
    "    - <a href='#select_subset_featuregroup'>3.4. Select subset of features</a>\n",
    "    - <a href='#filter_featuregroup'>3.5. Filter feature groups</a>\n",
    "    - <a href='#join_featuregroup'>3.6. Apply joins on feature group</a>\n",
    "    - <a href='#create_dataset'>3.7. Create dataset from multiple or one feature group</a>\n",
    "    - <a href='#sql_query'>3.8 Free form sql query</a>\n",
    "- <a href='#ref'>4. References</a>\n",
    "\n",
    "---\n",
    "\n",
    "**Important:**\n",
    "\n",
    "Placeholder text for required values are surrounded by angle brackets that must be removed when adding the indicated content. For example, when adding a database name to `database_name = \"<database_name>\"` would become `database_name = \"production\"`.\n",
    "\n",
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e6732e83",
   "metadata": {},
   "source": [
    "<a id=\"concepts\"></a>\n",
    "# 1. Introduction\n",
    "\n",
    "Oracle feature store is a stack based solution that is deployed in the customer enclave using OCI resource manager. Customer can stand up the service with infrastructure in their own tenancy. The service consists of API which are deployed in customer tenancy using resource manager.\n",
    "\n",
    "The following are some key terms that will help you understand OCI Data Science Feature Store:\n",
    "\n",
    "\n",
    "* **Feature Vector**: Set of feature values for any one primary/identifier key. Eg. All/subset of features of customer id ‘2536’ can be called as one feature vector.\n",
    "\n",
    "* **Feature**: A feature is an individual measurable property or characteristic of a phenomenon being observed.\n",
    "\n",
    "* **Entity**: An entity is a group of semantically related features. The first step a consumer of features would typically do when accessing the feature store service is to list the entities and the entities associated features. Another way to look at it is that an entity is an object or concept that is described by its features. Examples of entities could be customer, product, transaction, review, image, document, etc.\n",
    "\n",
    "* **Feature Group**: A feature group in a feature store is a collection of related features that are often used together in ml models. It serves as an organizational unit within the feature store for users to manage, version and share features across different ml projects. By organizing features into groups, data scientists and ml engineers can efficiently discover, reuse and collaborate on features reducing the redundant work and ensuring consistency in feature engineering.\n",
    "\n",
    "* **Feature Group Job**: Feature group job is the execution instance of a feature group. Each feature group job will include validation results and statistics results.\n",
    "\n",
    "* **Dataset**: A dataset is a collection of feature that are used together to either train a model or perform model inference.\n",
    "\n",
    "* **Dataset Job**: Dataset job is the execution instance of a dataset. Each dataset job will include validation results and statistics results."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a9d1682b",
   "metadata": {},
   "source": [
    "<a id='pre-requisites'></a>\n",
    "# 2. Pre-requisites\n",
    "\n",
    "Data Flow Sessions are accessible through the following conda environment:\n",
    "\n",
    "* **PySpark 3.2, Feature store 1.0 and Data Flow 1.0 (fs_pyspark32_p38_cpu_v1)**\n",
    "\n",
    "The [Data Catalog Hive Metastore](https://docs.oracle.com/en-us/iaas/data-catalog/using/metastore.htm) provides schema definitions for objects in structured and unstructured data assets. The Metastore is the central metadata repository to understand tables backed by files on object storage. You can customize `fs_pyspark32_p38_cpu_v1`, publish it, and use it as a runtime environment for a Data Flow session cluster. The metastore id of hive metastore is tied to feature store construct of feature store service.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "fa22b7ff",
   "metadata": {},
   "source": [
    "<a id='setup_spark-defaults'></a>\n",
    "### `spark-defaults.conf`\n",
    "\n",
    "The `spark-defaults.conf` file is used to define the properties that are used by Spark. A templated version is installed when you install a Data Science conda environment that supports PySpark. However, you must update the template so that the Data Catalog metastore can be accessed. You can do this manually. However, the `odsc data-catalog config` commandline tool is ideal for setting up the file because it gathers information about your environment, and uses that to build the file.\n",
    "\n",
    "The `odsc data-catalog config` command line tool needs the `--metastore` option to define the Data Catalog metastore OCID. No other command line option is needed because settings have default values, or they take values from your notebook session environment. Following are common parameters that you may need to override.\n",
    "\n",
    "The `--authentication` option sets the authentication mode. It supports resource principal and API keys. The preferred method for authentication is resource principal, which is sent with `--authentication resource_principal`. If you want to use API keys, then use the `--authentication api_key` option. If the `--authentication` isn't specified, API keys are used. When API keys are used, information from the OCI configuration file is used to create the `spark-defaults.conf` file.\n",
    "\n",
    "Object Storage and Data Catalog are regional services. By default, the region is set to the region your notebook session is running in. This information is taken from the environment variable, `NB_REGION`. Use the `--region` option to override this behavior.\n",
    "\n",
    "The default location of the `spark-defaults.conf` file is `/home/datascience/spark_conf_dir` as defined in the `SPARK_CONF_DIR` environment variable. Use the `--output` option to define the directory where to write the file.\n",
    "\n",
    "You need to determine what settings are appropriate for your configuration. However, the following works for most configurations and is run in a terminal window.\n",
    "\n",
    "```bash\n",
    "odsc data-catalog config --authentication resource_principal --metastore <metastore_id>\n",
    "```\n",
    "For more assistance, use the following command in a terminal window:\n",
    "\n",
    "```bash\n",
    "odsc data-catalog config --help\n",
    "```\n",
    "\n",
    "<a id='setup_session'></a>\n",
    "### Session Setup\n",
    "\n",
    "The notebook makes connections to the Data Catalog metastore and Object Storage. In the next cell, specify the bucket URI to act as the data warehouse. Use the `warehouse_uri` variable with the `oci://<bucket_name>@<namespace_name>/<key>` format. Update the variable `metastore_id` with the OCID of the Data Catalog metastore."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b9029967",
   "metadata": {},
   "source": [
    "python -m pip install --pre oracle-ads<a id='policies'></a>\n",
    "### 2.1. Policies\n",
    "This section covers the creation of dynamic groups and policies needed to use the service.\n",
    "\n",
    "* [Data Flow Policies](https://docs.oracle.com/iaas/data-flow/using/policies.htm/)\n",
    "* [Data Catalog Metastore Required Policies](https://docs.oracle.com/en-us/iaas/data-catalog/using/metastore.htm)\n",
    "* [Getting Started with Data Flow](https://docs.oracle.com/iaas/data-flow/using/dfs_getting_started.htm)\n",
    "* [About Data Science Policies](https://docs.oracle.com/iaas/data-science/using/policies.htm)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "499eeda1",
   "metadata": {},
   "source": [
    "<a id=\"prerequisites_authentication\"></a>\n",
    "### 2.2. Authentication\n",
    "The [Oracle Accelerated Data Science SDK (ADS)](https://docs.oracle.com/iaas/tools/ads-sdk/latest/index.html) controls the authentication mechanism with the notebook cluster.<br>\n",
    "To setup authentication use the ```ads.set_auth(\"resource_principal\")``` or ```ads.set_auth(\"api_key\")```."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "acd4ea44",
   "metadata": {
    "ExecuteTime": {
     "start_time": "2023-05-24T08:26:08.577504Z"
    },
    "is_executing": true,
    "pycharm": {
     "is_executing": true
    }
   },
   "outputs": [],
   "source": [
    "import ads\n",
    "ads.set_auth(auth=\"resource_principal\", client_kwargs={\"fs_service_endpoint\": \"https://{api_gateway}/20230101\"})"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0dc3e1bb",
   "metadata": {},
   "source": [
    "<a id=\"prerequisites_variables\"></a>\n",
    "### 2.3. Variables\n",
    "To run this notebook, you must provide some information about your tenancy configuration. To create and run a feature store, you must specify a `<compartment_id>` and bucket `<metastore_id>` for offline feature store."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "f5548d8d",
   "metadata": {
    "pycharm": {
     "is_executing": true
    }
   },
   "outputs": [],
   "source": [
    "import os\n",
    "\n",
    "compartment_id = os.environ.get(\"NB_SESSION_COMPARTMENT_OCID\")\n",
    "metastore_id = \"<metastore_id>\"\n",
    "os.environ[\"OPENAI_API_KEY\"] = \"<OPENAI_API_KEY>\""
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cff77fa6",
   "metadata": {},
   "source": [
    "<a id=\"featurestore_querying\"></a>\n",
    "# 3. Feature group use cases\n",
    "By default the **PySpark 3.2, Feature store and Data Flow** conda environment includes pre-installed [great-expectations](https://legacy.docs.greatexpectations.io/en/latest/reference/core_concepts/validation.html) and [deeque](https://github.com/awslabs/deequ) libraries. The joining functionality is heavily inspired by the APIs used by Pandas to merge, join or filter DataFrames. The APIs allow you to specify which features to select from which feature group, how to join them and which features to use in join conditions."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "24a7b05b",
   "metadata": {
    "pycharm": {
     "is_executing": true
    }
   },
   "outputs": [],
   "source": [
    "import warnings\n",
    "warnings.filterwarnings(\"ignore\", message=\"iteritems is deprecated\")\n",
    "warnings.filterwarnings(\"ignore\", category=DeprecationWarning)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "5f27910a",
   "metadata": {
    "pycharm": {
     "is_executing": true
    }
   },
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "from ads.feature_store.feature_store import FeatureStore\n",
    "from ads.feature_store.feature_group import FeatureGroup\n",
    "from ads.feature_store.model_details import ModelDetails\n",
    "from ads.feature_store.dataset import Dataset\n",
    "from ads.feature_store.common.enums import DatasetIngestionMode\n",
    "\n",
    "\n",
    "from ads.feature_store.feature_group_expectation import ExpectationType\n",
    "from great_expectations.core import ExpectationSuite, ExpectationConfiguration\n",
    "from ads.feature_store.feature_store_registrar import FeatureStoreRegistrar"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d1571ed6",
   "metadata": {},
   "source": [
    "<a id=\"data_exploration\"></a>\n",
    "### 3.1. Exploration of data in feature store"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "5f9f0ff9",
   "metadata": {},
   "outputs": [],
   "source": [
    "# load & inspect dataset\n",
    "import pandas as pd\n",
    "input_datapath = \"https://raw.githubusercontent.com/oracle-samples/oci-data-science-ai-samples/main/notebook_examples/data/fine_food_reviews_1k.csv\"\n",
    "df = pd.read_csv(input_datapath, index_col=0)\n",
    "df = df[[\"Time\", \"ProductId\", \"UserId\", \"Score\", \"Summary\", \"Text\"]]\n",
    "df = df.dropna()\n",
    "df[\"combined\"] = (\n",
    "    \"Title: \" + df.Summary.str.strip() + \"; Content: \" + df.Text.str.strip()\n",
    ")\n",
    "df.head(2)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cc23f854",
   "metadata": {},
   "source": [
    "<a id=\"load_featuregroup\"></a>\n",
    "### 3.2. Create feature store logical entities"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c1a93cfe",
   "metadata": {},
   "source": [
    "#### 3.2.1 Feature Store\n",
    "Feature store is the top level entity for feature store service"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "25e53f21",
   "metadata": {
    "pycharm": {
     "is_executing": true
    }
   },
   "outputs": [],
   "source": [
    "feature_store_resource = (\n",
    "    FeatureStore().\n",
    "    with_description(\"Feature Store embeddings\").\n",
    "    with_compartment_id(compartment_id).\n",
    "    with_name(\"Feature Store embeddings\").\n",
    "    with_offline_config(metastore_id=metastore_id)\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "73af27b7",
   "metadata": {},
   "source": [
    "<a id=\"create_feature_store\"></a>\n",
    "##### Create Feature Store\n",
    "\n",
    "Call the ```.create()``` method of the Feature store instance to create a feature store."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "a297171b",
   "metadata": {
    "pycharm": {
     "is_executing": true
    }
   },
   "outputs": [],
   "source": [
    "feature_store = feature_store_resource.create()\n",
    "feature_store"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "663126f5",
   "metadata": {},
   "source": [
    "#### 3.2.2 Entity\n",
    "An entity is a logical segregation of feature store entities"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "3c76c2ab",
   "metadata": {},
   "outputs": [],
   "source": [
    "entity = feature_store.create_entity(\n",
    "    name=\"Feature Store embeddings\",\n",
    "    description=\"Feature Store embeddings\"\n",
    ")\n",
    "entity"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1164d506",
   "metadata": {},
   "source": [
    "#### 3.2.3 Transformation"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "6e72f2c6",
   "metadata": {},
   "outputs": [],
   "source": [
    "def openai_generate_embedding_data(df):\n",
    "    import pandas as pd\n",
    "    import tiktoken\n",
    "    import openai\n",
    "    import dotenv\n",
    "    import os\n",
    "    from ast import literal_eval\n",
    "    dotenv.load_dotenv()\n",
    "    openai.api_key=os.getenv(\"OPENAI_API_KEY\")\n",
    "    \n",
    "    def get_embedding(text: str, engine=\"text-similarity-davinci-001\", **kwargs):\n",
    "        text = text.replace(\"\\n\", \" \")\n",
    "        return openai.Embedding.create(input=[text], engine=engine, **kwargs)[\"data\"][0][\"embedding\"]\n",
    "    \n",
    "    \n",
    "    # embedding model parameters\n",
    "    embedding_model = \"text-embedding-ada-002\"\n",
    "    embedding_encoding = \"cl100k_base\"  # this the encoding for text-embedding-ada-002\n",
    "    max_tokens = 8000  # the maximum for text-embedding-ada-002 is 8191\n",
    "    # subsample to 1k most recent reviews and remove samples that are too long\n",
    "    top_n = 1000\n",
    "    df = df.sort_values(\"Time\").tail(top_n * 2)  # first cut to first 2k entries, assuming less than half will be filtered out\n",
    "    df.drop(\"Time\", axis=1, inplace=True)\n",
    "    encoding = tiktoken.get_encoding(embedding_encoding)\n",
    "    df[\"n_tokens\"] = df.combined.apply(lambda x: len(encoding.encode(x)))\n",
    "    df = df[df.n_tokens <= max_tokens].tail(top_n)\n",
    "    df[\"embedding\"] = df.combined.apply(lambda x: get_embedding(x, engine=embedding_model))\n",
    "    return df"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "dd7b5fa8",
   "metadata": {},
   "outputs": [],
   "source": [
    "from ads.feature_store.transformation import TransformationMode\n",
    "\n",
    "embedding_transformation = feature_store.create_transformation(\n",
    "    transformation_mode=TransformationMode.PANDAS,\n",
    "    source_code_func=openai_generate_embedding_data,\n",
    "    name=\"openai_generate_embedding_data\",\n",
    ")\n",
    "\n",
    "embedding_transformation"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8b86f76e",
   "metadata": {},
   "source": [
    "#### 3.2.3 Feature group\n",
    "A feature group is an object that represents a logical group of time-series feature data as it is found in a datasource."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8038c430",
   "metadata": {},
   "source": [
    "<a id=\"create_feature_group_flights\"></a>\n",
    "##### Open AI feature Group\n",
    "\n",
    "Create feature group for Open AI Feature Group\n",
    "\n",
    "<div>\n",
    "    <img src=\"https://objectstorage.us-ashburn-1.oraclecloud.com/p/hh2NOgFJbVSg4amcLM3G3hkTuHyBD-8aE_iCsuZKEvIav1Wlld-3zfCawG4ycQGN/n/ociodscdev/b/oci-feature-store/o/beta/_images/feature_group_flights.gif\" width=\"700\" height=\"350\" />\n",
    "</div>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "112bfa93",
   "metadata": {},
   "outputs": [],
   "source": [
    "embedding_feature_group = (\n",
    "    FeatureGroup()\n",
    "    .with_feature_store_id(feature_store.id)\n",
    "    .with_primary_keys([])\n",
    "    .with_name(\"embedding_fg\")\n",
    "    .with_entity_id(entity.id)\n",
    "    .with_compartment_id(compartment_id)\n",
    "    .with_schema_details_from_dataframe(df)\n",
    "    .with_transformation_id(embedding_transformation.id)\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "63aca6b1",
   "metadata": {},
   "outputs": [],
   "source": [
    "embedding_feature_group.create()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "3e383d98",
   "metadata": {},
   "outputs": [],
   "source": [
    "embedding_feature_group.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c3478afc",
   "metadata": {},
   "outputs": [],
   "source": [
    "embedding_feature_group.materialise(df)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0f51b3b5",
   "metadata": {},
   "source": [
    "<a id=\"explore_featuregroup\"></a>\n",
    "### 3.3. Explore feature groups"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "27fb8aac",
   "metadata": {},
   "outputs": [],
   "source": [
    "embedding_feature_group.get_statistics().to_viz()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "391ad804",
   "metadata": {},
   "outputs": [],
   "source": [
    "embedding_feature_group.history().show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "ac7bff41",
   "metadata": {},
   "outputs": [],
   "source": [
    "embedding_feature_group.as_of(version_number=0).show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c9150752",
   "metadata": {},
   "outputs": [],
   "source": [
    "embedding_feature_group.get_features_df()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "8804aa6e",
   "metadata": {},
   "outputs": [],
   "source": [
    "embedding_feature_group.get_validation_output().to_pandas()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "3936064c",
   "metadata": {},
   "outputs": [],
   "source": [
    "embedding_feature_group.select().show()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "16bfb2a0",
   "metadata": {},
   "source": [
    "<a id=\"select_subset_featuregroup\"></a>\n",
    "### 3.4. Select subset of features"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "5d774862",
   "metadata": {},
   "outputs": [],
   "source": [
    "embedding_feature_group.select(['UserId', 'Summary']).show()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "835c222f",
   "metadata": {},
   "source": [
    "<a id=\"filter_featuregroup\"></a>\n",
    "### 3.5. Filter feature groups"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "83bee499",
   "metadata": {},
   "outputs": [],
   "source": [
    "embedding_feature_group.filter(embedding_feature_group.UserId == \"A3R7JR3FMEBXQB\").show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "2a12a5a1",
   "metadata": {},
   "outputs": [],
   "source": [
    "tranining_df = embedding_feature_group.select().read().toPandas()\n",
    "tranining_df.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e9ca754e",
   "metadata": {},
   "source": [
    "<a id=\"visualise_embeddings\"></a>\n",
    "### 3.5. Visualise embeddings"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "e9465820",
   "metadata": {},
   "outputs": [],
   "source": [
    "tranining_df['embedding'] = tranining_df['embedding'].apply(lambda x: [float(val) for val in x])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "5a92868f",
   "metadata": {},
   "outputs": [],
   "source": [
    "tranining_df.to_csv(\"fine_food_reviews_with_embeddings_1k.csv\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "6b0f7de6",
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "from sklearn.manifold import TSNE\n",
    "import numpy as np\n",
    "from ast import literal_eval\n",
    "\n",
    "datafile_path = \"fine_food_reviews_with_embeddings_1k.csv\"\n",
    "df = pd.read_csv(datafile_path)\n",
    "\n",
    "# Convert to a list of lists of floats\n",
    "matrix = np.array(df.embedding.apply(literal_eval).to_list())\n",
    "\n",
    "# Create a t-SNE model and transform the data\n",
    "tsne = TSNE(n_components=2, perplexity=15, random_state=42, init='random', learning_rate=200)\n",
    "vis_dims = tsne.fit_transform(matrix)\n",
    "vis_dims.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "fdfe9f68",
   "metadata": {},
   "outputs": [],
   "source": [
    "import matplotlib.pyplot as plt\n",
    "import matplotlib\n",
    "import numpy as np\n",
    "\n",
    "colors = [\"red\", \"darkorange\", \"gold\", \"turquoise\", \"darkgreen\"]\n",
    "x = [x for x,y in vis_dims]\n",
    "y = [y for x,y in vis_dims]\n",
    "color_indices = df.Score.values - 1\n",
    "\n",
    "colormap = matplotlib.colors.ListedColormap(colors)\n",
    "plt.scatter(x, y, c=color_indices, cmap=colormap, alpha=0.3)\n",
    "for score in [0,1,2,3,4]:\n",
    "    avg_x = np.array(x)[df.Score-1==score].mean()\n",
    "    avg_y = np.array(y)[df.Score-1==score].mean()\n",
    "    color = colors[score]\n",
    "    plt.scatter(avg_x, avg_y, marker='x', color=color, s=100)\n",
    "\n",
    "plt.title(\"Amazon ratings visualized in language using t-SNE\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d89e7c38",
   "metadata": {},
   "source": [
    "In this text classification task, we predict the score of a food review (1 to 5) based on the embedding of the review's text. We split the dataset into a training and a testing set for all the following tasks, so we can realistically evaluate performance on unseen data. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "6871348b",
   "metadata": {},
   "outputs": [],
   "source": [
    "# imports\n",
    "import pandas as pd\n",
    "import numpy as np\n",
    "from ast import literal_eval\n",
    "\n",
    "from sklearn.ensemble import RandomForestClassifier\n",
    "from sklearn.model_selection import train_test_split\n",
    "from sklearn.metrics import classification_report, accuracy_score\n",
    "\n",
    "# load data\n",
    "datafile_path = \"fine_food_reviews_with_embeddings_1k.csv\"\n",
    "\n",
    "df = pd.read_csv(datafile_path)\n",
    "df[\"embedding\"] = df.embedding.apply(literal_eval).apply(np.array)  # convert string to array\n",
    "\n",
    "# split data into train and test\n",
    "X_train, X_test, y_train, y_test = train_test_split(\n",
    "    list(df.embedding.values), df.Score, test_size=0.2, random_state=42\n",
    ")\n",
    "\n",
    "# train random forest classifier\n",
    "clf = RandomForestClassifier(n_estimators=100)\n",
    "clf.fit(X_train, y_train)\n",
    "preds = clf.predict(X_test)\n",
    "probas = clf.predict_proba(X_test)\n",
    "\n",
    "report = classification_report(y_test, preds)\n",
    "print(report)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "d918a0db",
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.metrics import average_precision_score, precision_recall_curve\n",
    "def plot_multiclass_precision_recall(\n",
    "    y_score, y_true_untransformed, class_list, classifier_name\n",
    "):\n",
    "    \"\"\"\n",
    "    Precision-Recall plotting for a multiclass problem. It plots average precision-recall, per class precision recall and reference f1 contours.\n",
    "\n",
    "    Code slightly modified, but heavily based on https://scikit-learn.org/stable/auto_examples/model_selection/plot_precision_recall.html\n",
    "    \"\"\"\n",
    "    n_classes = len(class_list)\n",
    "    y_true = pd.concat(\n",
    "        [(y_true_untransformed == class_list[i]) for i in range(n_classes)], axis=1\n",
    "    ).values\n",
    "\n",
    "    # For each class\n",
    "    precision = dict()\n",
    "    recall = dict()\n",
    "    average_precision = dict()\n",
    "    for i in range(n_classes):\n",
    "        precision[i], recall[i], _ = precision_recall_curve(y_true[:, i], y_score[:, i])\n",
    "        average_precision[i] = average_precision_score(y_true[:, i], y_score[:, i])\n",
    "\n",
    "    # A \"micro-average\": quantifying score on all classes jointly\n",
    "    precision_micro, recall_micro, _ = precision_recall_curve(\n",
    "        y_true.ravel(), y_score.ravel()\n",
    "    )\n",
    "    average_precision_micro = average_precision_score(y_true, y_score, average=\"micro\")\n",
    "    print(\n",
    "        str(classifier_name)\n",
    "        + \" - Average precision score over all classes: {0:0.2f}\".format(\n",
    "            average_precision_micro\n",
    "        )\n",
    "    )\n",
    "\n",
    "    # setup plot details\n",
    "    plt.figure(figsize=(9, 10))\n",
    "    f_scores = np.linspace(0.2, 0.8, num=4)\n",
    "    lines = []\n",
    "    labels = []\n",
    "    for f_score in f_scores:\n",
    "        x = np.linspace(0.01, 1)\n",
    "        y = f_score * x / (2 * x - f_score)\n",
    "        (l,) = plt.plot(x[y >= 0], y[y >= 0], color=\"gray\", alpha=0.2)\n",
    "        plt.annotate(\"f1={0:0.1f}\".format(f_score), xy=(0.9, y[45] + 0.02))\n",
    "\n",
    "    lines.append(l)\n",
    "    labels.append(\"iso-f1 curves\")\n",
    "    (l,) = plt.plot(recall_micro, precision_micro, color=\"gold\", lw=2)\n",
    "    lines.append(l)\n",
    "    labels.append(\n",
    "        \"average Precision-recall (auprc = {0:0.2f})\" \"\".format(average_precision_micro)\n",
    "    )\n",
    "\n",
    "    for i in range(n_classes):\n",
    "        (l,) = plt.plot(recall[i], precision[i], lw=2)\n",
    "        lines.append(l)\n",
    "        labels.append(\n",
    "            \"Precision-recall for class `{0}` (auprc = {1:0.2f})\"\n",
    "            \"\".format(class_list[i], average_precision[i])\n",
    "        )\n",
    "\n",
    "    fig = plt.gcf()\n",
    "    fig.subplots_adjust(bottom=0.25)\n",
    "    plt.xlim([0.0, 1.0])\n",
    "    plt.ylim([0.0, 1.05])\n",
    "    plt.xlabel(\"Recall\")\n",
    "    plt.ylabel(\"Precision\")\n",
    "    plt.title(f\"{classifier_name}: Precision-Recall curve for each class\")\n",
    "    plt.legend(lines, labels)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c62655a3",
   "metadata": {},
   "outputs": [],
   "source": [
    "plot_multiclass_precision_recall(probas, y_test, [1, 2, 3, 4, 5], clf)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "eef21e88",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "id": "ebc23c06",
   "metadata": {},
   "source": [
    "<a id='ref'></a>\n",
    "# References\n",
    "\n",
    "- [ADS Library Documentation](https://accelerated-data-science.readthedocs.io/en/latest/index.html)\n",
    "- [Data Science YouTube Videos](https://www.youtube.com/playlist?list=PLKCk3OyNwIzv6CWMhvqSB_8MLJIZdO80L)\n",
    "- [OCI Data Science Documentation](https://docs.cloud.oracle.com/en-us/iaas/data-science/using/data-science.htm)\n",
    "- [Oracle Data & AI Blog](https://blogs.oracle.com/datascience/)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "77384ce0",
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.4"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}