notebooks/116-sparsity-optimization/116-sparsity-optimization.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "cacbe6b4",
   "metadata": {
    "id": "rQc-wXjqrEuR"
   },
   "source": [
    "# Accelerate Inference of Sparse Transformer Models with OpenVINO™ and 4th Gen Intel&reg; Xeon&reg; Scalable Processors\n",
    "This tutorial demonstrates how to improve performance of sparse Transformer models with [OpenVINO](https://docs.openvino.ai/) on 4th Gen Intel® Xeon® Scalable processors. It uses a pre-trained model from the [Hugging Face Transformers](https://huggingface.co/transformers/) library and shows how to convert it to the OpenVINO™ IR format and run inference on a CPU using a dedicated runtime option that enables sparsity optimizations. It also demonstrates how to get more performance stacking sparsity with 8-bit quantization. To simplify the user experience, the [Hugging Face Optimum](https://huggingface.co/docs/optimum) library is used to convert the model to OpenVINO™ IR format and quantize it using [Neural Network Compression Framework](https://github.com/openvinotoolkit/nncf). It consists of the following steps:\n",
    "\n",
    "- Install prerequisites\n",
    "- Download and quantize sparse BERT model from a public source using the OpenVINO integration with Hugging Face Optimum.\n",
    "- Compare sparse 8-bit vs. dense 8-bit inference performance.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7bef22e9",
   "metadata": {},
   "source": [
    "## Prerequisites"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "4fc9afb5",
   "metadata": {},
   "outputs": [],
   "source": [
    "!pip install optimum[openvino] datasets"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4d6b41e6-132b-40da-b3b9-91bacba29e31",
   "metadata": {},
   "source": [
    "## Imports"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "771388d6",
   "metadata": {},
   "outputs": [],
   "source": [
    "from functools import partial\n",
    "from pathlib import Path\n",
    "\n",
    "from datasets import load_dataset\n",
    "from transformers import AutoModelForSequenceClassification, AutoTokenizer\n",
    "\n",
    "from optimum.intel.openvino import OVQuantizer\n",
    "from optimum.intel.openvino import OVConfig"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7603a481",
   "metadata": {},
   "source": [
    "## Quantize the model using Hugging Face Optimum API\n",
    "The sparsity acceleration MatMul operations are only available cases when these operations are quantized into 8-bit precision. If the model is not in 8-bit precision, it can be quantized using either method available for OpenVINO models. For more details, please refer to the [Model Optimization Guide](https://docs.openvino.ai/latest/openvino_docs_model_optimization_guide.html). In this tutorial we use the Hugging Face Optimum API to quantize the model. The Hugging Face Optimum API is a high-level API that allows us to convert and quantize models from the Hugging Face Transformers library to the OpenVINO™ IR format. For more details refer to the [Hugging Face Optimum documentation](https://huggingface.co/docs/optimum/intel/optimization_ov)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "b897c926",
   "metadata": {},
   "outputs": [],
   "source": [
    "model_id = \"neuralmagic/oBERT-12-downstream-pruned-unstructured-90-mnli\"\n",
    "quantized_sparse_dir = Path(\"bert_90_sparse_quantized\")\n",
    "\n",
    "# Instantiate model and tokenizer in PyTorch and load them from the HF Hub\n",
    "torch_model = AutoModelForSequenceClassification.from_pretrained(model_id)\n",
    "tokenizer = AutoTokenizer.from_pretrained(model_id)\n",
    "\n",
    "\n",
    "def preprocess_function(examples, tokenizer):\n",
    "    \"\"\"\n",
    "    Define a function that tokenizes the data and returns it in the format expected by the model.\n",
    "    \n",
    "    :param: examples: a dictionary containing the input data which are the items from caliration dataset.\n",
    "            tokenizer: a tokenizer object that is used to tokenize the text data.\n",
    "    :returns:\n",
    "            the data that can be fed directly to the model.\n",
    "    \"\"\"\n",
    "    return tokenizer(\n",
    "        examples[\"premise\"], examples[\"hypothesis\"], padding=\"max_length\", max_length=128, truncation=True\n",
    "    )\n",
    "\n",
    "\n",
    "# Create quantization config (default) and OVQuantizer\n",
    "# OVConfig is a wrapper class on top of NNCF config. \n",
    "# Use \"compression\" field to control quantization parameters\n",
    "# For more information about the parameters refer to NNCF GitHub documentatioin\n",
    "quantization_config = OVConfig()\n",
    "quantizer = OVQuantizer.from_pretrained(torch_model, feature=\"sequence-classification\")\n",
    "\n",
    "# Instantiate a dataset and convert it to calibration dataset using HF API\n",
    "# The latter one produces a model input\n",
    "dataset = load_dataset(\"glue\", \"mnli\")\n",
    "calibration_dataset = quantizer.get_calibration_dataset(\n",
    "    \"glue\",\n",
    "    dataset_config_name=\"mnli\",\n",
    "    preprocess_function=partial(preprocess_function, tokenizer=tokenizer),\n",
    "    num_samples=100,\n",
    "    dataset_split=\"train\",\n",
    ")\n",
    "# Apply static quantization and export the resulting quantized model to OpenVINO IR format\n",
    "quantizer.quantize(\n",
    "    quantization_config=quantization_config, calibration_dataset=calibration_dataset, save_directory=quantized_sparse_dir\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a6830eb7",
   "metadata": {},
   "source": [
    "## Benchmark quantized dense inference performance\n",
    "Benchmark dense inference performance using parallel execution on four CPU cores to simulate a small instance in the cloud infrastructure. Sequense length is set to 16 which is common for multiple use cases, e.g. conversational AI."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "aa895f88",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Dump benchmarking config for dense inference\n",
    "with open(\"perf_config.json\", \"w\") as outfile:\n",
    "    outfile.write(\n",
    "        \"\"\"\n",
    "        {\n",
    "            \"CPU\": {\"NUM_STREAMS\": 4, \"INFERENCE_NUM_THREADS\": 4}\n",
    "        }\n",
    "        \"\"\"\n",
    "    )"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "5f6c7526",
   "metadata": {},
   "outputs": [],
   "source": [
    "!benchmark_app -m bert_90_sparse_quantized/openvino_model.xml -shape \"input_ids[1,16],attention_mask[1,16],token_type_ids[1,16]\" -load_config perf_config.json"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f9151b11",
   "metadata": {},
   "source": [
    "## Benchmark quantized sparse inference performance"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "ad77ae5f",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Dump benchmarking config for dense inference\n",
    "# \"CPU_SPARSE_WEIGHTS_DECOMPRESSION_RATE\" controls minimum sparsity rate for weights to consider \n",
    "# for sparse optimization at the runtime.\n",
    "with open(\"perf_config_sparse.json\", \"w\") as outfile:\n",
    "    outfile.write(\n",
    "        \"\"\"\n",
    "        {\n",
    "            \"CPU\": {\"NUM_STREAMS\": 4, \"INFERENCE_NUM_THREADS\": 4, \"CPU_SPARSE_WEIGHTS_DECOMPRESSION_RATE\": 0.8}\n",
    "        }\n",
    "        \"\"\"\n",
    "    )"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "1ddd8b10",
   "metadata": {},
   "outputs": [],
   "source": [
    "!benchmark_app -m bert_90_sparse_quantized/openvino_model.xml -shape \"input_ids[1,16],attention_mask[1,16],token_type_ids[1,16]\" -load_config perf_config_sparse.json"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cc1d4d61",
   "metadata": {},
   "source": [
    "## When this might be helpful"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "135c8526",
   "metadata": {},
   "source": [
    "This feauture can improve inference performance for models with sparse weights in the scenarios when the model is deployed to handle multiple requests in parallel asyncronously. It is especially helpful in the case of small sequence length, e.g. 32 and lower.\n",
    "\n",
    "For more details about asynchronous inference with OpenVINO please refer to the following documentation:\n",
    "- [Deployment Optimization Guide](https://docs.openvino.ai/latest/openvino_docs_deployment_optimization_guide_common.html#doxid-openvino-docs-deployment-optimization-guide-common-1async-api)\n",
    "- [Inference Request API](https://docs.openvino.ai/latest/openvino_docs_OV_UG_Infer_request.html#doxid-openvino-docs-o-v-u-g-infer-request-1in-out-tensors)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.10"
  },
  "vscode": {
   "interpreter": {
    "hash": "fe409241748dff4afe127d33bbdaaa11b54ce82261ed669fd8a0538ea98c62f7"
   }
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}