tutorials/Metrics Tutorial.ipynb

{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "84298738",
   "metadata": {},
   "outputs": [],
   "source": [
    "%load_ext autoreload\n",
    "%autoreload 2\n",
    "\n",
    "import os\n",
    "\n",
    "import warnings\n",
    "import functools\n",
    "import numpy as np\n",
    "import matplotlib.pyplot as plt\n",
    "\n",
    "from tensorflow import keras\n",
    "\n",
    "warnings.filterwarnings('ignore')\n",
    "%matplotlib inline"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "52655fa9",
   "metadata": {},
   "outputs": [],
   "source": [
    "import tsgm"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c8fb6ef8",
   "metadata": {},
   "source": [
    "Let's generate a real `d_real` and a synthetic `d_syn` dataset."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "3ea014d9",
   "metadata": {},
   "outputs": [],
   "source": [
    "eps = 1e-5\n",
    "\n",
    "Xr, yr = tsgm.utils.gen_sine_vs_const_dataset(10, 100, 20, max_value=2, const=1)\n",
    "Xs, ys = Xr + eps, yr\n",
    "\n",
    "d_real = tsgm.dataset.Dataset(Xr, yr)\n",
    "d_syn = tsgm.dataset.Dataset(Xs, ys)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a121d403",
   "metadata": {},
   "source": [
    "## Distance metric\n",
    "\n",
    "First, we define a list of summary statistics that reflect the distance between the datasets. Module `tss.metrics.statistics` defines a set of handy statistics."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c167b660",
   "metadata": {},
   "outputs": [],
   "source": [
    "statistics = [functools.partial(tsgm.metrics.statistics.axis_max_s, axis=None),\n",
    "              functools.partial(tsgm.metrics.statistics.axis_min_s, axis=None),\n",
    "              functools.partial(tsgm.metrics.statistics.axis_max_s, axis=1),\n",
    "              functools.partial(tsgm.metrics.statistics.axis_min_s, axis=1)]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "910d217f",
   "metadata": {},
   "source": [
    "Next, we define a discrepancy function. In our case, it is simply Euclidean norm."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "578f5c81",
   "metadata": {},
   "outputs": [],
   "source": [
    "discrepancy_func = lambda x, y: np.linalg.norm(x - y)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b2a72a7c",
   "metadata": {},
   "source": [
    "Finally, we are putting all together using `tss.metrics.DistanceMetric` object."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "e6a1eed6",
   "metadata": {},
   "outputs": [],
   "source": [
    "sim_metric = tsgm.metrics.DistanceMetric(\n",
    "    statistics=statistics, discrepancy=discrepancy_func\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c29f9d6a",
   "metadata": {},
   "outputs": [],
   "source": [
    "sim_metric(d_real, d_syn)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c08fac9f",
   "metadata": {},
   "source": [
    "## Consistency Metric\n",
    "\n",
    "The consistency metric measures whether a family of models show consistent performance on real and synthetic datasets. First, we define an evaluator that returns the predictive performance on a downstream task."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "92d5f21a",
   "metadata": {},
   "outputs": [],
   "source": [
    "import sklearn\n",
    "\n",
    "class EvaluatorConvLSTM():\n",
    "    '''\n",
    "    NB an oversimplified classifier, for educational purposes only.\n",
    "    '''\n",
    "    \n",
    "    def __init__(self, model):\n",
    "        self._model = model\n",
    "\n",
    "    def evaluate(self, D: tsgm.dataset.Dataset) -> float:\n",
    "        X, y = D.Xy\n",
    "        \n",
    "        X_train , X_test, y_train, y_test = sklearn.model_selection.train_test_split(X, y, random_state=0)\n",
    "        y_train = keras.utils.to_categorical(y_train, 2)\n",
    "        self._model.fit(X_train, y_train)\n",
    "        \n",
    "        y_pred = np.argmax(self._model.predict(X_test), 1)\n",
    "        return sklearn.metrics.accuracy_score(y_pred, y_test)\n",
    "\n",
    "\n",
    "seq_len, feat_dim, n_classes = *Xr.shape[1:], 2\n",
    "models = [tsgm.models.zoo[\"clf_cl_n\"](seq_len, feat_dim, n_classes, n_conv_lstm_blocks=i) for i in range(1, 4)]\n",
    "for m in models:\n",
    "    m.model.compile(loss='binary_crossentropy', optimizer='adam', metrics=['accuracy'])\n",
    "evaluators = [EvaluatorConvLSTM(m.model) for m in models]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9666729f",
   "metadata": {},
   "source": [
    "Instantiate a consistency metric object using the set of model, and evaluator wrapper class."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "03a9085c",
   "metadata": {},
   "outputs": [],
   "source": [
    "consistency_metric = tsgm.metrics.ConsistencyMetric(evaluators=evaluators)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "26479c48",
   "metadata": {},
   "outputs": [],
   "source": [
    "consistency_metric(d_real, d_syn)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "66ffbbe8",
   "metadata": {},
   "source": [
    "## Downstream Performance\n",
    "\n",
    "Downstream performance metric measures the quality of the generated time by **evaluating a particular downstream model on real dataset and real dataset augmented with synthetically generated data**."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "a8898ea8",
   "metadata": {},
   "outputs": [],
   "source": [
    "downstream_model = tsgm.models.zoo[\"clf_cl_n\"](seq_len, feat_dim, n_classes, n_conv_lstm_blocks=1).model\n",
    "downstream_model.compile(loss='binary_crossentropy', optimizer='adam', metrics=['accuracy'])\n",
    "\n",
    "evaluator = EvaluatorConvLSTM(downstream_model)\n",
    "\n",
    "downstream_perf_metric = tsgm.metrics.DownstreamPerformanceMetric(evaluator)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "99429db6",
   "metadata": {},
   "outputs": [],
   "source": [
    "print(downstream_perf_metric(d_real, d_syn))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "34ed8ea9",
   "metadata": {},
   "source": [
    "## Privacy: Membership Inference Attack Metric\n",
    "\n",
    "`tsgm.metrics.PrivacyMembershipInferenceMetric` measures the possibility of membership inference attacks using synthetic data.\n",
    "The evaluation procedure is following:  \n",
    "    1. Split the historical data into training and hold-out sets ($D_{tr}$ and $D_{test}$),  \n",
    "    2. Train a generative model on $D_{train}$ and generate a synthetic dataset $\\hat{D}$,  \n",
    "    3. Train a one-class classification (OCC) model on synthetic data $\\hat{D}$ and evaluate it on $D_{tr}$ and $D_{test}$,  \n",
    "    4. Use the precision of the OCC model as the target score."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1e23f96b",
   "metadata": {},
   "source": [
    "Let's define an attacker model. For the demonstration purposes, we will define a one class SVM classifier."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "8573b996",
   "metadata": {},
   "outputs": [],
   "source": [
    "class FlattenTSOneClassSVM:\n",
    "    def __init__(self, clf):\n",
    "        self._clf = clf\n",
    "\n",
    "    def fit(self, X):\n",
    "        X_fl = X.reshape(X.shape[0], -1)\n",
    "        self._clf.fit(X_fl)\n",
    "\n",
    "    def predict(self, X):\n",
    "        X_fl = X.reshape(X.shape[0], -1)\n",
    "        return self._clf.predict(X_fl)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "dcfe6a93",
   "metadata": {},
   "outputs": [],
   "source": [
    "attacker = FlattenTSOneClassSVM(sklearn.svm.OneClassSVM())\n",
    "privacy_metric = tsgm.metrics.PrivacyMembershipInferenceMetric(\n",
    "    attacker=attacker\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "905227aa",
   "metadata": {},
   "outputs": [],
   "source": [
    "Xr, yr = tsgm.utils.gen_sine_vs_const_dataset(10, 100, 20, max_value=2, const=1)\n",
    "d_test = tsgm.dataset.Dataset(Xr, yr)\n",
    "\n",
    "privacy_metric(d_real, d_syn, d_test)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.8"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}