examples/sentence_similarity/bert_senteval.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Parallel Experimentation with BERT on AzureML"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![Impressions](https://PixelServer20190423114238.azurewebsites.net/api/impressions/nlp/examples/sentence_similarity/bert_senteval.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "[SentEval](https://github.com/facebookresearch/SentEval) is a widely used benchmarking tool developed by Facebook Research for evaluating general-purpose sentence embeddings. It provides a simple interface for evaluating embeddings on up to 17 supported downstream tasks (such as sentiment classification, natural language inference, semantic similarity, etc.) \n",
    "\n",
    "Due to the fact that different BERT layers capture different information, and that the choice of pooling layer and pooling strategy for the encoding is highly dependent on the final finetuning task, we use SentEval to evaluate different combinations of these encoding parameters on the STSBenchmark dataset. In this notebook, we aim to show an example of\n",
    "* running SentEval experiments with BERT encodings\n",
    "* running parallel jobs on AzureML compute targets for faster experimentation (extracting sequence encodings from BERT with 110M parameters is computationally expensive, even without finetuning. Each experiment could take an hour or more, depending on the specs of the machine, so running multiple experiments sequentially can quickly add up) "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 00 Global Settings"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "System version: 3.6.8 |Anaconda, Inc.| (default, Dec 30 2018, 01:22:34) \n",
      "[GCC 7.3.0]\n",
      "AzureML version: 1.0.57\n"
     ]
    }
   ],
   "source": [
    "import os\n",
    "import sys\n",
    "import pickle\n",
    "import shutil\n",
    "import itertools\n",
    "import glob\n",
    "import numpy as np\n",
    "import pandas as pd\n",
    "import seaborn as sns\n",
    "import matplotlib.pyplot as plt\n",
    "import scrapbook as sb\n",
    "\n",
    "import azureml\n",
    "from azureml.core import Experiment\n",
    "from azureml.data.data_reference import DataReference\n",
    "from azureml.train.dnn import PyTorch\n",
    "from azureml.widgets import RunDetails\n",
    "\n",
    "sys.path.append(\"../../\")\n",
    "from utils_nlp.azureml.azureml_utils import get_or_create_workspace, get_or_create_amlcompute\n",
    "from utils_nlp.models.bert.common import Language, Tokenizer\n",
    "from utils_nlp.models.bert.sequence_encoding import BERTSentenceEncoder, PoolingStrategy\n",
    "from utils_nlp.eval.senteval import SentEvalConfig\n",
    "\n",
    "%matplotlib inline\n",
    "print(\"System version: {}\".format(sys.version))\n",
    "print(\"AzureML version: {}\".format(azureml.core.VERSION))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "tags": [
     "parameters"
    ]
   },
   "outputs": [],
   "source": [
    "# azureml config\n",
    "subscription_id = \"YOUR_SUBSCRIPTION_ID\"\n",
    "resource_group = \"YOUR_RESOURCE_GROUP_NAME\"  \n",
    "workspace_name = \"YOUR_WORKSPACE_NAME\"  \n",
    "workspace_region = \"YOUR_WORKSPACE_REGION\"\n",
    "\n",
    "# path config\n",
    "CACHE_DIR = \"./temp\"\n",
    "LOCAL_UTILS = \"../../utils_nlp\"\n",
    "LOCAL_SENTEVAL = \"../../utils_nlp/eval/SentEval\"\n",
    "\n",
    "EXPERIMENT_NAME = \"NLP-SS-bert\"\n",
    "CLUSTER_NAME = \"eval-gpu\"\n",
    "MAX_NODES = None # we scale the number of nodes in the cluster automatically"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "os.makedirs(CACHE_DIR, exist_ok=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We evaluate 768-dimensional encodings from BERT with each combination of 12 BERT layers and 2 pooling strategies (mean and max) for a total of 24 experiments. To run a smaller number of experiments or customize the pooling layers/strategies of interest, edit `EXP_PARAMS`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "MODEL_PARAMS = {\n",
    "    \"num_gpus\": 1,\n",
    "    \"language\": Language.ENGLISH,\n",
    "    \"to_lower\": True,\n",
    "    \"max_len\": 128,\n",
    "    \"cache_dir\": CACHE_DIR\n",
    "}\n",
    "\n",
    "SENTEVAL_PARAMS = {\n",
    "    \"usepytorch\": True, \n",
    "    \"batch_size\": 128,\n",
    "    \"transfer_tasks\": [\"STSBenchmark\"]\n",
    "}\n",
    "\n",
    "EXP_PARAMS = {\n",
    "    \"layer_index\": range(12),\n",
    "    \"pooling_strategy\": [PoolingStrategy.MEAN, PoolingStrategy.MAX],\n",
    "}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 01 Set up AzureML resources"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We set up the following AzureML resources for this example:\n",
    "* A [Workspace](https://docs.microsoft.com/en-us/azure/machine-learning/service/concept-workspace), a centralized hub for all the artifacts you create when you use Azure Machine Learning service\n",
    "* An [Experiment](https://docs.microsoft.com/en-us/python/api/azureml-core/azureml.core.experiment.experiment?view=azure-ml-py), which acts a container for trials or model runs\n",
    "* A [Datastore](https://docs.microsoft.com/en-us/azure/machine-learning/service/how-to-access-data), a compute location-independent abstraction of data in Azure storage accounts\n",
    "\n",
    "The following cell looks to set up the connection to your [Azure Machine Learning service Workspace](https://docs.microsoft.com/azure/machine-learning/service/concept-azure-machine-learning-architecture#workspace). You can choose to connect to an existing workspace or create a new one. \n",
    "\n",
    "**To access an existing workspace:**\n",
    "1. If you have a `config.json` file, you do not need to provide the workspace information; you will only need to update the `config_path` variable that is defined above which contains the file.\n",
    "2. Otherwise, you will need to supply the following:\n",
    "    * The name of your workspace\n",
    "    * Your subscription id\n",
    "    * The resource group name\n",
    "\n",
    "**To create a new workspace:**\n",
    "\n",
    "Set the following information:\n",
    "* A name for your workspace\n",
    "* Your subscription id\n",
    "* The resource group name\n",
    "* [Azure region](https://azure.microsoft.com/en-us/global-infrastructure/regions/) to create the workspace in, such as `eastus2`. \n",
    "\n",
    "This will automatically create a new resource group for you in the region provided if a resource group with the name given does not already exist. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "ws = get_or_create_workspace(\n",
    "    subscription_id=subscription_id,\n",
    "    resource_group=resource_group,\n",
    "    workspace_name=workspace_name,\n",
    "    workspace_region=workspace_region,\n",
    ")\n",
    "exp = Experiment(workspace=ws, name=EXPERIMENT_NAME)\n",
    "ds = ws.get_default_datastore()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 02 Set up SentEval"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Run the bash script to download the data for auxiliary transfer tasks."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "data_path = os.path.join(LOCAL_SENTEVAL, \"data/downstream\")\n",
    "data_script = \"get_transfer_data.bash\"\n",
    "tokenizer_name = \"tokenizer.sed\"\n",
    "!cd $data_path && pwd  && chmod 777 $data_script && chmod 777 $tokenizer_name &&bash $data_script"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We upload the SentEval dependency to datastore. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "$AZUREML_DATAREFERENCE_29393ed34c4d4ab398be34e5f5952c2c"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "ds.upload(\n",
    "    src_dir=LOCAL_SENTEVAL,\n",
    "    target_path=os.path.join(EXPERIMENT_NAME, \"senteval\"),\n",
    "    overwrite=False,\n",
    "    show_progress=False,\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 03 Define experiment configurations\n",
    "We define a set of static configurations, which entails model parameters that will stay consistent across all experiments, in `SentEvalConfig`. We also define the parameter space that will vary across the experiments. We serialize the configuration objects and upload them to our datastore to make them accessible to all experiments."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "sc = SentEvalConfig(\n",
    "    model_params=MODEL_PARAMS,\n",
    "    senteval_params=SENTEVAL_PARAMS,\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "parameter_groups = list(itertools.product(*list(EXP_PARAMS.values())))\n",
    "if MAX_NODES is not None:\n",
    "    parameter_groups = parameter_groups[:MAX_NODES]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "$AZUREML_DATAREFERENCE_07723c6c94b7431b9f811c8be538d8b7"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "os.makedirs(os.path.join(CACHE_DIR, \"config\"), exist_ok=True)\n",
    "\n",
    "static_config = (\n",
    "    SentEvalConfig(model_params=MODEL_PARAMS, senteval_params=SENTEVAL_PARAMS),\n",
    "    os.path.join(CACHE_DIR, \"config\", \"static_config.pkl\"),\n",
    ")\n",
    "exp_configs = [\n",
    "    (\n",
    "        dict(zip(EXP_PARAMS.keys(), p)),\n",
    "        os.path.join(CACHE_DIR, \"config\", \"exp_config_{0:03d}.pkl\".format(i)),\n",
    "    )\n",
    "    for i, p in enumerate(parameter_groups)\n",
    "]\n",
    "\n",
    "configs = [static_config] + exp_configs\n",
    "for config in configs:\n",
    "    pickle.dump(config[0], open(config[1], \"wb\"))\n",
    "\n",
    "ds.upload_files(\n",
    "    [c[1] for c in configs],\n",
    "    target_path=\"{}/config\".format(EXPERIMENT_NAME),\n",
    "    overwrite=True,\n",
    "    show_progress=False,\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 04 Scale the compute target"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Scale the number of nodes in the compute target to the number of experiments we want to run."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Scaling compute target eval-gpu to 24 node(s)\n"
     ]
    }
   ],
   "source": [
    "compute_target = get_or_create_amlcompute(\n",
    "    workspace=ws,\n",
    "    compute_name=CLUSTER_NAME,\n",
    "    vm_size=\"STANDARD_NC6\",\n",
    "    min_nodes=0,\n",
    "    max_nodes=len(parameter_groups),\n",
    "    idle_seconds_before_scaledown=300,\n",
    "    verbose=False,\n",
    ")\n",
    "\n",
    "print(\n",
    "    \"Scaling compute target {0} to {1} node(s)\".format(\n",
    "        CLUSTER_NAME, len(parameter_groups)\n",
    "    )\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 05 Define the execution script"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here we define the script to be executed for each experiment on the remote compute target. We deserialize the configuration objects from the datastore to specify the model parameters for the experiment, and run the SentEval evaluation engine with that model for the STSBenchmark transfer task.\n",
    "\n",
    "As specified in the SentEval repo, we implement the **batcher** function, which transforms a batch of text sentence into sentence embeddings.\n",
    "\n",
    "After running SentEval, we serialize the output."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "src_dir = os.path.join(CACHE_DIR, EXPERIMENT_NAME)\n",
    "os.makedirs(src_dir, exist_ok=True)\n",
    "if not os.path.exists(os.path.join(src_dir, \"utils_nlp\")):\n",
    "    shutil.copytree(\n",
    "        LOCAL_UTILS,\n",
    "        os.path.join(src_dir, \"utils_nlp\"),\n",
    "        ignore=shutil.ignore_patterns(\"__pycache__\", \"SentEval\"),\n",
    "    )"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Writing ./temp/NLP-SS-bert/run.py\n"
     ]
    }
   ],
   "source": [
    "%%writefile $src_dir/run.py\n",
    "import pickle\n",
    "import argparse\n",
    "import os\n",
    "from utils_nlp.eval.senteval import SentEvalConfig\n",
    "from utils_nlp.models.bert.sequence_encoding import BERTSentenceEncoder\n",
    "\n",
    "\n",
    "def prepare_output(output_dir, config_file):\n",
    "    os.makedirs(output_dir, exist_ok=True)\n",
    "    out = os.path.join(\n",
    "        output_dir,\n",
    "        \"results_{}.pkl\".format(config_file.split(\"/\")[-1].split(\".\")[0][-3:]),\n",
    "    )\n",
    "    return out\n",
    "\n",
    "\n",
    "def batcher(params, batch):\n",
    "    sentences = [\" \".join(s).lower() for s in batch]\n",
    "    embeddings = params[\"model\"].encode(\n",
    "        sentences, batch_size=params[\"batch_size\"], as_numpy=True\n",
    "    )\n",
    "    return embeddings\n",
    "\n",
    "\n",
    "if __name__ == \"__main__\":\n",
    "    parser = argparse.ArgumentParser()\n",
    "    parser.add_argument(\"--data_dir\", type=str, dest=\"data_dir\")\n",
    "    parser.add_argument(\n",
    "        \"--static_config\",\n",
    "        type=str,\n",
    "        dest=\"static_config\",\n",
    "        help=\"Filename of serialized static config object\",\n",
    "    )\n",
    "    parser.add_argument(\n",
    "        \"--exp_config\",\n",
    "        type=str,\n",
    "        dest=\"exp_config\",\n",
    "        help=\"Filename of serialized experiment config object\",\n",
    "    )\n",
    "    parser.add_argument(\n",
    "        \"--output_dir\",\n",
    "        type=str,\n",
    "        dest=\"output_dir\",\n",
    "        help=\"Directory to write serialized results to\",\n",
    "    )\n",
    "    args = parser.parse_args()\n",
    "\n",
    "    # Import senteval\n",
    "    sys.path.insert(0, args.data_dir)\n",
    "    import senteval\n",
    "\n",
    "    # Deserialize configs\n",
    "    static_config = pickle.load(open(args.static_config, \"rb\"))\n",
    "    exp_config = pickle.load(open(args.exp_config, \"rb\"))\n",
    "\n",
    "    # Update senteval params for this experiment\n",
    "    params = static_config.senteval_params\n",
    "    params[\"model\"] = BERTSentenceEncoder(**static_config.model_params)\n",
    "    for k, v in exp_config.items():\n",
    "        setattr(params[\"model\"], k, v)\n",
    "    params[\"task_path\"] = \"{}/data\".format(args.data_dir)\n",
    "\n",
    "    # Run the senteval engine\n",
    "    se = senteval.engine.SE(params, batcher)\n",
    "    results = se.eval(params[\"transfer_tasks\"])\n",
    "\n",
    "    # Pickle the output\n",
    "    output_file = prepare_output(args.output_dir, args.exp_config)\n",
    "    print(\"Pickling to {}\".format(output_file))\n",
    "    pickle.dump(results, open(output_file, \"wb\"))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 06 Run the experiments in parallel"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We iterate through the experiment parameter combinations and submit each job to AmlCompute as a `PyTorch` estimator. Since we explicitly set `node_count=1` and `process_count_per_node=1` in the estimator, the jobs will run in parallel."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Scaling compute target eval-gpu to 24 node(s)\n"
     ]
    }
   ],
   "source": [
    "runs = []\n",
    "for i in range(len(parameter_groups)):\n",
    "    est = PyTorch(\n",
    "        source_directory=src_dir,\n",
    "        script_params={\n",
    "            \"--data_dir\": ds.path(\"{}/senteval\".format(EXPERIMENT_NAME)).as_mount(),\n",
    "            \"--static_config\": ds.path(\n",
    "                \"{0}/{1}/{2}\".format(\n",
    "                    EXPERIMENT_NAME, \"config\", static_config[1].split(\"/\")[-1]\n",
    "                )\n",
    "            ).as_mount(),\n",
    "            \"--exp_config\": ds.path(\n",
    "                \"{0}/{1}/{2}\".format(\n",
    "                    EXPERIMENT_NAME, \"config\", exp_configs[i][1].split(\"/\")[-1]\n",
    "                )\n",
    "            ),\n",
    "            \"--output_dir\": \"./outputs\",\n",
    "        },\n",
    "        compute_target=compute_target,\n",
    "        entry_script=\"run.py\",\n",
    "        inputs=[\n",
    "            DataReference(\n",
    "                datastore=ds, path_on_datastore=\"outputs\"\n",
    "            ).as_upload(\n",
    "                path_on_compute=os.path.join(\"./outputs/results_{0:03d}.pkl\".format(i))\n",
    "            )\n",
    "        ],\n",
    "        node_count=1,\n",
    "        process_count_per_node=1,\n",
    "        use_gpu=True,\n",
    "        framework_version=\"1.1\",\n",
    "        conda_packages=[\"numpy\", \"pandas\"],\n",
    "        pip_packages=[\n",
    "            \"scikit-learn==0.20.3\",\n",
    "            \"azureml-sdk==1.0.53\",\n",
    "            \"pytorch-pretrained-bert>=0.6\",\n",
    "            \"cached-property==1.5.1\",\n",
    "        ],\n",
    "    )\n",
    "\n",
    "    run = exp.submit(est)\n",
    "    runs.append(run)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Each run object is collected in `runs`, so we can monitor any run via a Jupyter widget for debugging."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "#RunDetails(runs[0]).show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Alternatively, block until the runs are complete."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [],
   "source": [
    "map(lambda r: r.wait_for_completion(), runs)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Finally, we pull down the serialized outputs of each experiment from the datastore and inspect the metrics for analysis."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "24"
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "ds.download(\n",
    "    target_path=CACHE_DIR,\n",
    "    prefix=\"outputs\",\n",
    "    show_progress=False,\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here we aggregate the outputs from each SentEval experiment to plot the distribution of Pearson correlations reported across the different encodings. We can see that for the STS Benchmark downstream task, the first layer achieves the highest Pearson correlation on the test dataset. As suggested in [bert-as-a-service](https://github.com/hanxiao/bert-as-service), this can be interpreted as a representation that is closer to the original word embedding."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "results = [\n",
    "    pickle.load(open(f, \"rb\"))\n",
    "    for f in sorted(glob.glob(os.path.join(CACHE_DIR, \"outputs\", \"*.pkl\")))\n",
    "]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# For testing\n",
    "sb.glue(\"pearson\", results[0][\"STSBenchmark\"][\"pearson\"])\n",
    "sb.glue(\"mse\", results[0][\"STSBenchmark\"][\"mse\"])\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Text(0.5, 1, 'Pearson correlations of BERT sequence encodings on STS Benchmark')"
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAiIAAACcCAYAAABC68AGAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjAsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+17YcXAAAgAElEQVR4nOydd3gVxfrHP++eEwglQCoplIRqAQERUUApKmJBvViwK157vVbAgoiC5VqwKzbEAnYERZoUlSJFEaVICem9kUZIzjnz+2M34bSEAwmG3N98nmef5+zOOzPfM/vu7rszs7uilEKj0Wg0Go2mMTAaW4BGo9FoNJr/v+hARKPRaDQaTaOhAxGNRqPRaDSNhg5ENBqNRqPRNBo6ENFoNBqNRtNo6EBEo9FoNBpNo6EDkf8niMhMEXmqHvlLRaRLQ2qqDyJym4hkW7rCG1uP5uhDROJFRImI3Vr/QUSua2xdGl+891Uj1D9MRNIao27NUR6IiEiSiOyzLjbZIvKBiLRubF3/64jIChG50X2bUqq1UiqxsTS5IyJBwIvASEtXvld69Umt1M133rDyVdu4+1b18pqVdr2IOK1txSLyh4icb6Wd5mZf5lVPqYh0+ifbQhM4SqlzlFIfNraOw0VE/i0i20WkxPLp70UkxAqwqv2vSkQq3dbfsvI+LCJ7rG1pIvJZHfW4HxuFVj0d/7l/qvn/xlEdiFiMVkq1Bk4EBgCPNmThjRWBHy7+9Da1/9AAtAeCgS0HsWtn+U5v4FTgDq/00VYgU73c6Za2xsrbDngDmCMi7ZRSP1fbA8e712MtKfX+dxqNFyIyFJgGXKGUCgGOBT6HmgCr2ic/AZ5z88dbrV6ga4AzLZuTgB8PUmX1eTcGyAZePTL/rPH5f3j+POpoCoEIAEqpdOAHoBeAiLQVkfdEJFNE0kXkKRGxWWldRWSZiOSLSJ6IfCIi7arLsiL+8SKyGSgTEbu1nm7dbfwtImdYts1FZLqIZFjLdBFpbqUNs+4u7heRHEvLuNr+g4iEWb06Gdadxly3tJtEZJeIFIjIPBGJdUtTInKHiOwEdtax7RgRWWKV8beIXFaLjlAR+U5Eci0d34lIByttKnAa8JpXL4ESkW5ubT/Lyp8sIo+KiGGlXS8iv4jI81bZe0TkHLe6rxeRRKud94jIVbVo9NvuItID+NsyKxKRZbW1dzVKqRxgCXDcwWz95HUBHwGtgO6Hmh+gDt8yRGSCiOy2fPVzEQlzy3eN1b75IvKI5bdnWmkeQ23i1bUsIrEi8pW1j/aIyN1uaZOtumZZmraIyElu6R1F5Gsrb361D1hpN4jINmvfLhKRznX871NEZLWIFInZqzTMLW2FiDwpIqssDYtFJMItfYhb3lQRud7aXpfv2Sy/yxORROA8Lz01PX0B+GmCiPxkaVsqIq+LyMdWWrCIfGy1TZGIrBeR9rW0wbFWvUVWO1/gljbTKvd7q55fRaRrLc05ADM4/h1AKVWglPpQKVVSW/t75V2klNpt5c1SSs0IIB9KqQrgS9yOHes4fF5EUsTsmXlLRFpYaXWeE0WkhYi8YO27vdY+aOFW5VVWuXki8ohbvski8oXV7iUi8qeI9BCRiVY9qSIy0s1+nOWnJWKeb25xS6vWOF5EsoAPvP+3iNwtIlvFOi9qjjBKqaN2AZIwo3iAjph3wE9a63OBtzEvEFHAOuAWK60bcBbQHIgEfgKme5W7ySqzBdATSAVirfR4oKv1ewqw1qojEljtpmEY4LBsgoBzgXIgtJb/8z3wGRBq2Q+1to8A8jB7fZpj3n385JZPYV5Iw4AW/rZZ7ZAKjAPsVll5wPGW/UzgKet3OHAx0BIIAb4A5rrVtwK40Uu7ArpZv2cB31p544EdwL+ttOuBKuAmwAbcBmQAYmksBnpatjHV+vy0VV3tHm/psdeS1yMdiAX+AG7w51t+8l8P/GL9tmH2pFQCUXXVU0tZdfnWf6z/2MHa728Ds62044BS4HQr7UVMX6s+Hmr2p5svplm/DWAjMAloBnQBEoGzrfTJQAWmv9qAp4G1bv/3D+Ala38FA0OstIuAXZh343bM3snVtfzvOCDfqsPAPB7zgUg3H9sN9MD03xXAM1ZaJ6AEuALzOAkH+gbge7cC2zGP6zBguZcfrMDya+rwUyt9DfC81X5DMP32YyvtFmA+5vFjA/oDbfy0QZDVXg9b5Yyw/ldPt31YAJxstecnwJxa2vM0YB/wBDAYaF6LnYdfWNuutup5ELM3xHYI592WwIfALLf06cA8q41DrLZ4OpBzIvC6tR/irLYbhOnf8da+esfyhz7AfuBYL58922qrWcAe4BGrnpuAPW4azwO6Yp53hloaTvTS+KxVdws8j5/HgN+wfFUvR35pdAF1ijMPiFKgCEjG7CJvgdk1vx/romzZXgEsr6Wci4Dfvcp1vyh1A3KAM4Egr7y7gXPd1s8Gkqzfw6yTg90tPQc4xY+GGMCFnyAFeA+zO7V6vTXmSTLeWlfACK88HtuAscDPXjZvA49bv2fidYJys+sLFLqtr6CWQMQ6eewHjnNLuwVYYf2+HtjlltbSyhuNeWErwgyCWvjTEmC7xxNYIFJkLQozkGnjZuPuW9XLTW7/wWFtq7L28WV11FNXIFKXb20DzvDykSrME+0k3C5KVttVElggMhBI8aprIvCB9XsysNQt7Thgn/X7VCDX33/C7JH8t9u6gXmC7+zHdjzwkde2RcB1bj72qFva7cBCN63f+CnzYL63DLjVLW0kdQcitflpJ2v/t3RL/5gDgcgNlj+dcBAfPg3IAgy3bbOByW778F23tHOB7XWUdw7mRb8I03dfxCuo8PYLt+1XAUuBMsyAcEId9SRx4NhwYAZova00scro6mZ/KlYQQB3nRMtf9gF96jiWOrhtWwdc7uazS9zSRlsabdZ6iJW/XS3/aS5wj5vGSiDY6/hJt9r0F6BtXftWLw27NIWhmYuUUu2UUp2VUrcrpfYBnTGj4Eyry7MI86IbBSAiUSIyR8zu8GLMk0iEV7mp1T+UUrsw704nAzlW3uqhkVjMIKiaZGtbNflKKYfbejlmIOFNR6BAKVXoJ82jDqVUKebJIs6f3lq2dQYGVreH1SZXYZ5YPRCRliLyttU9WozZY9ROrKGtgxCBeXfn3SbuWrPc/ku59bO1UqoMM2C6FXPffS8ix9RSz8HaPRAilFLtMC8yq4CFXunVvlW9vOOWttbKG4p593faIdYNHNS3OgPfuO2vbYATM9COxdNHqy8ggdAZiPXyhYetcqvJcvtdDgSLOVbeEUj28mn3cl92K7MA88IUV4vtpV4ahmAGW7VpqD5uOmIGot4czPc82szLzh9+/dQqp8BtG17lfoQZVM0Rc9jwOXGbCO1GLJCqzOE9f3o9NFD7uaNa4w9KqdGYPREXYgZTN9Zm75X3E6XUmZhznm4FpojI2XVkucjy/+bAncBKEYnG7J1sCWx0268Lre3V1HZOjMDsYfO3b6upqz2y3X7vA/KUUk63dartReQcEVkr5jB1EWaQ534NyFXmsJM77YCbMXt39tahUdPANIVAxB+pmHdGEW4XkTZKqerJg09jRscnKKXaYHZNilcZymNFqU+VUkMwT6AKs9sOzLuBzm6mnaxth6M5TNzmqrjhUYeItMLsjk6vTa+fbanASq8La2ul1G1+8t2POWQw0Gqf06urrqOuavIw79q92yTdv7mXYKUWKaXOwrwgbcfsivVHQ7U7VvA6EzhV3OYhBJi3FPNu/RoR6XeY9dfmW6nAOV77LFiZ86EyMS/IgBk8YvpENWWYF4Rq3APOVMw7VPdyQ5RS5wYgNxXoJP4n8KViDn+6l9tCKbW6FtuPvGxbKaWeCVCDv7kSB/M9jzaz0g6HTMxj1b19a8pVSlUppZ5QSh2HObRwPnCtn3IygI5izWHxo/ewUEq5lFI/YvYA9TrEvFVKqS+AzYHkVUo5lVJfYwbIQzD3wT7MIdXq/dpWmRNbD0Ye5vBKbfNgGgQx5/B9hTm01t4KqBbgeQ3wd44rxNyXH4jI4COpUeNJkwxElFKZwGLgBRFpI+akv65iziwHs5uuFHMyYxzm2GitiEhPERlhOXAF5oFWHWnPBh4VkUjrIjYJs4flcDT/ALwh5mTRIBGpDgA+BcaJSF9LwzTgV6VU0iFU8R3QQ8wJjkHWMkBEjvVjG2L9xyIxJ0c+7pWejTmvwN//cGLO1p8q5qODnYH7CKBNRKS9iFxgBVr7MfeRsxbzBml3q97mmE8NZBF4r0INynw8+F1Lw6HWXZdvvYXZjp0t20gRudBK+xI4X8xJm80wx9zdj9dNwLliToCOxux1qWYdUGxNxmsh5iTOXiIyIADJ6zAvxM+ISCsxJ2ZWn5TfAiaKyPGW3rYicmkt5XwMjBaRs636g61JgoFM/vsEOFNELhNzInm4iPQNwPc+B+4WkQ4iEgpMCKAuH5RSycAGYLKINBORUzGHArD+93AR6W31IBZjBkf+/PhXzIDxIet4HGaVM+dQNYnIhSJyuXXuEBE5GXPuw9oA8l4vIudZbWaIOSn3eEvfwfKK5ZOhwDard+cd4CURqe6BjjtI7wpQM/H7feBFMSdT20TkVOvYaEiaYfbk5AIO6/+OrDtLjcYVmD3J34jIwAbWpamFJhmIWFyL6XBbMSPZLznQ7fsE5mTNvZgTRL8+SFnNgWcwI/YszCGeh620pzBPSpuBPzEnMR3ui8GuwTxpbcccN/0PgHV38xhmFJ+Jecdw+aEUrMzZ8yOtfBnW/6iejOXNdMy5NnmYJzLvIYuXgUvEfJrgFT/578I8wSZijqd+inmCORgGZm9MBma3/lDM3gZ/NES7F4lIKWZgdSpwgVLK/U5ovni+A+SbOsqajnnhP+EQNdTlWy9jDvssFpESzH0xEEAptQVzkuynmD5RCLi/cOkjzEmlSZhBec17IawL9mjMuT97rLrfBdoeTKxb3m5AilXnWCvtG0yfmiPmkN5fmPMW/JWTijl88DDmBSEV84bgoOccZT4CfS6mrxRgBl19rOS6fO8dzCGTPzD95WDHfV1chekz+Zh+9xlm8Axm79OXmEHINmAlfoJkpVQlcAFmG+VhznG7Vim1/TD0FGJOyNxp1fsx8F+l1CcB5C3G3A8pmPM+ngNuU0r9Ukee+daxUwxMxZzbU/24/HjMSbhrLT9YitnDGggPYB7P6zH37bM08HXIOhfejRmYFgJXYh5ngeZfgjnpf56I9G9IbRr/iOd5WaPRHK2ISBLmZMulja3l/xtivgBsu1LKu/dQo9HUk6bcI6LRaDRHBGtYs6s1lDEKs3dn7sHyaTSaQ0e/UU6j0Wh8icYc2gnHHJ66TVkvE9NoNA2LHprRaDQajUbTaOihGY1Go9FoNI2GDkQ0Go1Go9E0Gkd8jkhVXuJRN/bjKjis92IdUVxrvmtsCT6onJzGluAXVRTId77+WaTjob709chTuXJzY0vwoWjb0TctbU6h3+/VNTqRtb1hpxFZbis/uNE/zG37j8776cFZX3q/RPOI4u9aGxTR5R/VcLgcfWcFjUaj0Wg0h4azqrEVHDY6ENFoNBqNpomjnP4+D9U00IGIRqPRaDRNnSYciBydg2sajUaj0WgCp2q/7xIAIjJKRP4WkV0i4vf7TNZ3n7aKyBYR+dRt+7Mi8pe1jHXbniAiv4rIThH5zPpeVq3oQESj0Wg0miaOcjp8loNhfbjxdczvIR0HXCEix3nZdAcmAoOtL9z/x9p+HuY33fpifiPrQRFpY2V7FnhJKdUd83s//65Lhw5ENBqNRqNp6jirfJeDczKwSymVaH2kcQ7m5wzcuQl4XSlVCKCUqn6c8jhgpVLKoZQqw/zY5CgREWAE5ochAT4ELqpLhA5ENBqNRqNp6jgqfZeDE4f5Zexq0qxt7vQAeojIKhFZa317CczA4xwRaSkiEcBwoCPmZxGKlFKOOsr0QE9W1Wg0Go2miaP89ICIyM3AzW6bZiilZrib+CvKa90OdAeGAR2An0Wkl1JqsYgMAFYDucAawBFgmT4VaDQajUajacr46QGxgo4ZvsY1pGH2YlTTAfB+42casFYpVQXsEZG/MQOT9UqpqcBUAGsS604gD2gnInarV8RfmR7ooRmNRqPRaJo6hzdHZD3Q3XrKpRlwOTDPy2Yu5rAL1hBMDyBRRGwiEm5tPwE4AViszC/pLgcusfJfB3xblwjdI6LRaDQaTVPnMN4jopRyiMidwCLABryvlNoiIlOADUqpeVbaSBHZCjiBB5VS+SISjDlMA1AMXO02L2Q8MEdEngJ+B96rS4cORDQajUajaeKoAN8b4pNPqQXAAq9tk9x+K+A+a3G3qcB8csZfmYmYT+QEhA5ENBqNRqNp6jThN6vqQESj0Wg0mqaODkQ0Go1Go9E0GlUBvTfkqEQHIhqNRqPRNHV0j4hGo9FoNJpGQwciGo1Go9FoGg09NKPRaDQajabRcDobW8FhowMRjUaj0WiaOg49NKPRaDQajaaxcAT0SvejkqMuEPll7Qaemf4WTpeLi0eP4sZrLvNIf/blt1n322YAKvbvp6CwiDWLvjwyWjb+ybPvzMblUow56zT+fem5PjaLfl7Pm7O/RRB6JHTk2QfNDx2+NPMLflpv6rzl8tGMOi3gl8zVyard2Ty3ZDMupfhXn87cMKinr6atabz983YQ6BHVlmcuGmBqWvYXP+/KQinFKQlRPHTWCViv560XRnwvmp1xJYjg2PwzjnULfGxsPQcQNOhCQOHKSaXye/M7TEFDL8XW5QQQwZm0lapln9ZbD4Cte1+anTcODAPHhh+p+mmur02vU2l2xmWgFK6sZPZ//jJGwvE0O+/6A/8tIpb9n03HuW19vTWtSsrlvyu24XLBRb06cMPJXXxsFv+dyVtrd5n+FBnC0+f2AWD6T3/z855cFIqBncJ5aNixDbLvAOx9BtDi2jvBsFG5/Hv2z5vtYxN0yjCCL74OAGfybspfewr7cX1pcc0dNTZGbCfKX51C1YZV9dbUYtBJhD10OxgGpd/8wN4PPvOxaTnydNrdci2gqNyRSN7EpwGwRUcS8fj92NpHglLk3PUIjozsemtKGHoCZz5+DYbN4I85K1j75nwfm2POG8iQe8eglCJnWwrz736jJq1Z6xbc9OOz7Fi0gSWTZtVbD0DcsBM45QlT09+zV7D5dV9NCecPpN99Y0ApCralsOJOU9O45FkUbje/AF+ans/SG15sEE0nDO3HNY/fgGEzWDFnKfPf/MbHZuB5gxhz71iUUqRsS+KNu6cTHhfJf95+CMMwsAXZWDxzAcs+WdwgmgDaDe9LlyfHgc0g+5MfSX/N95wQfsGpdHrgMlBQtiWJHbe/DMCg9M8o25YCQGV6Htuue7bBdDUYukekYXA6nTz1wuu8M30a0VERjL3xHoYPGUjXhM41NuPvuaXm9ydffMu2nbuPkBYX0976hBlP3k/78FCuuO9Jhg3sS9dOsTU2yRnZvPfl98x6biJtWrciv6gYgJ/W/8G23Sl88cpkKqsc3DDxWYb0703rli3qp8mleHrRH7x1xWDat2nBVR8sZ2j3GLpGtjmgqaCU99fsYOa1p9OmRTMKyszX/m5Ky2dTWj5f3HgGAOM+WsmGlDwGdI6slyZEaHbW1ez//AVUSQHB10zCuXsTKv/AxxalXRRBA8+l4tNpsL8cWoYAYMR2xYjrRsVM823Cza+ciNGxJ67Uv+upyaDZ6H9T8cGTqOICgm97Gse2DajctAMm4dEEDf0X+95+FCrKoJXZhq49W6h47UHTqEVrWt73Ks5df9RPD+a+e2bZVt4cM4D2IcFc9ekahnaNomt46xqb5MIy3l+fyMyxp9AmOIiCcmvfZRSyKaOQz68ZDMC4z9eyMa2AkzqG11sXYtBi3D2UTXsQV34uIVPfomrjalzpyTUmRnQczS+8ktLJd6HKSpE27QBwbN1EycSbzGJahRAy/WOqNm+ovybDIGziXWTfOh5Hdh6xn7xG+co1VCWm1JjYO8XR9oYryLr+P7hKSjFC29WkRT41nqJ3P6Vi7W9Ii2BQdX6BPCDEEEY+eR1zrnqGkqwCrp83hZ1LN5K/84Cfh8a359Q7RvPRmCfYX1xOy/A2HmWcfv8lpPy6vd5a3DUNeuo6Fl75DGWZBVzw/RRSFm+kyE1Tm4T29LlzNN/96wkq95YT7KbJWVHJ3LMfaTA9piaD6568iWeueoKCrHymzHuOjUvXk7HzwLHXPj6G0XeM4YkxD1NeXEab8LYAFOUU8sSYiTgqHTRvGcwzi6fz25L1FOUU1l+YYdDl6RvZctkUKjML6LPwGQoWb2DfjgO6ghOi6XDXGDaPfhTn3jKCIg60lauikj/OfLD+Oo4gqgnPETmqvr7757YddOoQS8e4GIKCgjjnjKEs+3ltrfYLlq7k3DOHHREtf+1MpFNMFB2iIwkKsjPq9JNZ/uvvHjZfLfqJseeOoE3rVgCEtzMdd3dqJif16oHdZqNlcHN6JnRk1ca/6q8po4COoa3oENqKIJvB2cd1YMXOTA+brzclMbZ/F9q0aAZAWKvmAAhQ6XBR5XRR6XTicCrCrbT6YMR0QRXmoPbmgsuJY/uv2Lr19bCx9xlK1e/LzCAEoLykJk1sQWCzgy0IDDuqrLj+mjp0w1WQhSrMAacD5+ZV2I89yVPTSWfi+HWhGYQA+KnX3usUnDt+b5DZ6H9lFdGxXUs6tGtp7rue0azY7XmX/s2faVzWpxNtgoMACGvptu+cLqpcLiqdLhxOVZNWX2zdjsGVlYErJxOcDirXLCPopMEeNs1GnE/l4rmoslIAVHGRTzlBA4fi2LQOKg/vexfuNO/VE0dqBo70LHA4KFu0gpbDBnnYhIw5h5LP5uEqMTW5Ck1NQV06gc1GxdrfTK37KlAV9dcU07crhUnZ7E3NxVXlZOv8tXQ/q7+HTZ8rhrNx1lL2F5t+Xp5/wKfa94qnZUQbkn76s95aqons25XipGxKUkxNid+updNIT009rxzO1g+XUrnX1FSRX//jqy669u1GdlImuanZOKscrJ3/C/3P8uwNHn7FmSydtZDyYvPYK87fC4CzyoGj0ryrD2pmR4yG6fEDCOnXjYo9WexPyUFVOcidu4qwswd42LS/+kyyPliIc6+pqyrvyLZVg1NV5bsEgIiMEpG/RWSXiEyoxeYyEdkqIltE5FNr23AR2eS2VIjIRVbaTBHZ45bW11+51RxVPSI5uXlERx24Q28fFcGfW/zfHWdkZZOemcXA/n2OiJbs/CLaR4Qd0BIeyp879njYJKdnAXDtQ0/jdLm47YoLGNK/Nz3jO/DWnPlcc+FIKvZXsm7zdrp0jKW+5JRUEN3mQK9K+5AW/JnhebeQXGCemK+btRKXS3HraccyuGt7+nQIZ0DnCM585QdAMbZ/F7pEeN6xHQ7Suh2qpKBmXZUUYsR4DjlIaHsMwH7lRBCDqlXf4kr6C1fGbpyp22lx20sg4PhtGaogk/oibcJQe/MPaCouwOjY3cPGiIjBBQTf/KSpadkXOHdu8rCx9x5M1Srfru7DIad0P+1D3PZd62D+ytrrYZNcZJ4Ar5+zFpdS3HJqNwbHR9InNpSTOoZx1ozloGBs3050cetJqQ9GaASu/JyadVd+LvZux3rY2KI7ANB68qtgGFR8NRPHH55DVUGDhrP/+y8aRJMtKgJHVm7NuiM7j+a9j/Gsr7OpKXrmdMQwKHprFvtWbyCocwdcJaVEvvA49rhoKn79jcKX3wOXq16aQqJDKck84OclmQXE9uvqYROWEA3A1V9NQgyDX6Z/zZ6Vm0GEMx69ivn3vkn84OPrpcOdljGhlLlpKs8qINJLU1tL0/nfTEJsBr+9+DXpK8whY1vzIC74fgrK6WLz6/NJXrSx3ppCo8MpyDxw7BVk5tO1n+exF51gngsnfTUNwzD4evpnbF5p3uSFxYTzwAeP0D4+htnTPmyY3hCgWUwYlRl5NeuVmfmEnOipq0UXU1fveU8hNoOU5z+naLl5TjCaN6PPomdRDidpr35DwcL6D9U2OIcxNCMiNuB14CwgDVgvIvOUUlvdbLoDE4HBSqlCEYkCUEotB/paNmHALsB9LO1BpVRA8yYCDkREZBAQ755HKdUwA5015fmr17/tD0tXMnLYEGw2W0NKqFOMtxan00VKRjbvTXuQ7LxCrp/wLF+/NoVBJ/bir51JXPvQ04S2DaHPMV2x2erf+eSvg9m7eZwuFykFpbx71WnklOxj3Ec/8eVNZ1BUXkliXgmL7xoFwK2zf2FjSh79O0XUU5W/HeSpVAwbhLZn/5znkJBQml8xgYoPHkNahGCExbDvrfsBaH7Z/RhJPXCl7TgCkrxaz7BhRMRQ8e5kpG04wTdNYd8r90GFeecoIe0wojvh3Fn/YZlAdTpdipSiMt659GRySiu44fNf+fKaIRRWVLKnoIxFNw4D4Nav17MxrYD+HcJ8yzxkDQfff9hsGNFxlD75H4ywSFo//golD41DlZuBk7QLw9axC47NDXRy9qfJe//ZbNg7xZF14/3YoyKJ/uBFMi65CWw2gvv1JuPyW3Fk5RD57KO0vmAkpXMX1leUH02eq4bdRlh8NJ+OnUpITBhXffEY742cwPH/Gszu5Zs8ApmGwVeTdzOJ3UabhGi+v3QqrWLCOP/rx/j6jAlUFpfz2cB7KM8uIqRTJOd89jAF21MpSc7xKbN+ivB1J7uN6PhYpo59jLCYcB77YioTRt5DeXE5BZn5PDzqPtpFhXLvOxNYt2ANxXl7/ZV6iML8tZXXecpuo0VCDH+NeZxmseH0nvskvw+7F2dxORv630pldiHNO0XR66vJlG9LoSK5/vOOGpSqw5ojcjKwy/paLiIyB7gQ2OpmcxPwulKqEEAp5c9JLgF+UEqVH46IgK6OIvIR8DwwBBhgLSfVYX+ziGwQkQ3vzvKd+FYb7aMiyMo5cCeUnZNHZIT/cfAflq7knLOGBVz2odI+IpTsvAMnjuz8QiLD2vnYDB/YlyC7nQ7RkcTHtSfFmhR389jz+eKVycx48n6Ugs6x7euvKSSYrOJ9BzSV7CMyJNjLpgXDesQQZEvCwWMAACAASURBVDOIa9eK+LAQUgrKWLYjgxPiwmjZzE7LZnYGd4lmc3r9T4yqtBAJOXBBlJBQVKln172rpADnzt/B5UTtzUMVZGGEtsfW/UScmYlQtR+q9uNM/BMj1ncC5yFr2luAtD3gN9ImDFXs+V9VcT7OretNTYU5qLwMjPCYmnRbr0E4tq4DV8OMu0a1bk52idu+K60g0mtoLKp1MMO6Rpn7rm1L4kNbkVJUzvJdOfSObntg38VH8mem7/DI4eAqyMUIj6pZN8IjcRXm+9g4NqwCpxNXbhbOzFQMq5cEIOiU4VSt/6XB3mPgzM7FHn2gZ9TePgJnbr6XTR77VqwBhxNHRhZVSWnYO8XhzM6j8u9d5rCO00X58tU0O7a7dxWHTElWASExB/w8JCaMkmzPu/WSzAJ2LtmIy+Fkb2ouBYmZhMZHE3diN0687ixu++Ulhj9yJb3GnMbQ8WPrrak8s4BWbppaRodRnuWpqSyzgJRFG1EOJ6WpuezdnUkbq5ekPNv0oZKUXDLXbCO8V2fqS0FWPmExB469sJhwCrM9j72CzHw2LlmH0+EkNzWHzMR0ouM9e4yLcgpJ35FKz5P9fmX+kKnMyKdZ7IGbrmYx4VR6tVVlRj4Fi9ajHE72p+Swb3cGLbqY54RKa1/vT8lh7+ottOqd0CC6GhSn03c5OHFAqtt6mrXNnR5ADxFZJSJrRWSUn3IuB7wv9lNFZLOIvCQidY4lB3qbfhJmt8ztSqm7rOXu2oyVUjOUUicppU668dorAqwCeh3Tg5S0DNIysqiqquKHH1cyfMgpPnZ7ktMoLimlb69j/ZTSMBzfPYHkjGzSsnKpqnKw8Kd1DDvZc5hr+Cn9WPenOXRUuLeE5IxsOkRH4nS6KCo2h0h27EllR1Iqp/arf5fs8bGhpBSWkl5URpXTxaKtaQztHuNhM7xHLOuTzWCusHw/yQWldGjXkpg2LdmYkofDZc4T2ZiSR5eIkHprcmXuQULbI20jwLBhP2Ygzl2eQxzOnb9j62R1rbdojYRG4yrKRRXnY+vYE8QAw4atY09Ufv2HZlzpuzDCY5DQKLDZsZ0wGMd2z0mUzq3rMbpY+6RlCBIeg6vgwB2O/YTBOP74pd5aqjk+ui0pheWk7y03993fWQzrEuVhM7xbFOtTzZN24b5KkgvLiWvbguiQYDamFdbsu9/SCkgIa5ihGefu7RjRcRiR0WCz0+zUEVRtXO1hU7XhF+zH9wNAQtpgi+lgzimxaDZoBFWrf2wQPQD7t/yNvVMc9thosNtpdfYwyleu8bApX76K4AHmsKzRrg1BneNwpGWyf8vfGCGtMULNCZDBJ/elKjHZp45DJfOPRMISomnbMRIjyMZxo09h15LfPGx2LN5Ip1PNC2eL0NaEJURTlJLD/Hve5M1B/+HNIfeyfOqn/PX1z6x81vcpoEMl949E2iRE09rS1OXCU0jx0pS8aCMxg0xNzUNb06ZLNCXJOTRr2xKjmb1me/sBPSjakV5vTYl/7CI6IYbIjlHYguycMnoIvy3x7CnbuHgdx53aC4DWoSFEJ8SSk5JFWHQ4Qc3NuW0t27Si+0nHkLm7/poASjbtokWXGJp3ikKC7EReNJiCxZ668heuo+1gU5c9LIQWXWKoSM7G1rYVYrWVPSyENgOOodxtkuvRgnI4fRb3TgFrudkrWwBdotiB7sAw4ArgXRGpuSsXkRigN7DILc9E4BjMToswYHxd2gMdmvkLiAbqf5WoA7vdxsP33sYt9z2K0+nkX+ePpFuXzrz2ziyOP6YHw08zg5IFS1dwzplDG+zxRb9abDYevvUqbnv8JZwuFxedOYRuneN4/eO5HNc9nuED+zL4xF6s+X0LF93+KIZhcN+4S2nXpjX7K6u4fsIzALRq2YKn778JewMMIdkNgwkj+3DbnFW4XHBhn850i2zDGyu3clxMKMN6xDCoSxRr9mQz5u2lGIZw74hetGvZnDOPiWNdUi6XvvMjgjCoa5RPEHNYKBeVSz+m+SX3mY/K/vkLKj+DoMEX4cpKwrl7E66kv1AJxxM87ilQLqpWfg4VZTh3bMDofCzB46aAAmfSnzh3N8BQiMtF5fz3CL7+ERADx2/LUTlpBJ0xFlf6bpzbN+DcuQlbtz60uOcl037hR7DPDB6lXSTSLgJX0taDVBQ4dsNg/IjjuP3rDbiU4sLjO9A1IoQ3Vu/kuPZtGdY1ikGdI1iTnMeYD3/GJsJ/Tu9JuxbNOLN7NOtT87nsI/Ox2EHxEQztGnWQGgPE5WLfzFdoNfE5MAwqV/yAKy2J4EvG4djzN46Nq3H8sR577wGE/PcD0/6Tt1Cl5kQ+I6I9Rngkjm0NOITldFHwzGu0f/Np8/HdbxdRtTuZdrddx/6tO9i3cg37Vm8g+NT+xH71LrhcFL70Dq695iTogpdmEP32cyBC5badlHzl+zj5oaKcLhZP+pCxsx5CbAabP19J3s50TrvvYjI372HX0t/Ys3IzCaf35salz+Jyulg+bTYVRaX1rrsuTWse+5BRnzyEGAY7PltJ0Y50TnzgYvL+2EPKkt9IX7GZDqf3ZsyyZ1EuF+ufms3+olKi+ndn8LM3oFwuxDDY/Pp8j6dtDheX08WHk97loVmTMGwGKz//kfSdqVx83+Xs2byb35auZ/PK3+l9eh+eXfoyLqeL2dM+pLSolF5DunLlo9ehlDmSsmDGt6T9nXLwSgPB6SLx4Xc5fvajYDPImb2MfX+n0emhsZRu2k3B4g0ULd9Eu6F96PfTSyini6QpH+EoLCXkpJ50/e/N4FJgCGmvfuPxtM1Rg5/JqUqpGcCMOnKlAR3d1jsA3o6QBqxVSlUBe0Tkb8zApDqSuwz4xkqvrrc6VtgvIh8AD9QlXbzHyfwaiVRPSlkH1ExBV0pdcLC8VXmJ9X92roFxFdT/gGtoXGu+a2wJPqic+o0XHylUUcnBjf5hpAEmIzc0lSs3N7YEH4q2HVXz4wGYU1j/YdMjQeRR+DTmctthTQE4oty2/6h6+LOGwVlfHrk7ZT+UPXaZz7W21ZOf16lBROzADuAMIB0zuLhSKbXFzWYUcIVS6joRiQB+B/oqpfKt9LXARGvyanWeGKVUppi9BS8BFUopv0/kQOA9IpMDtNNoNBqNRvMPoxyHHrkqpRwicifmsIoNeF8ptUVEpgAblFLzrLSRIrIVcGI+DVMdhMRj9qis9Cr6ExGJxBz62QTcWpeOgAIRpZR3JRqNRqPRaI4WDu+pGZRSC4AFXtsmuf1WwH3W4p03Cd/JrSilRhyKhkCfmjlFRNaLSKmIVIqIU0Sa2NteNBqNRqP530Q5XD5LUyHQoZnXMB/P+QLzCZprMSeraDQajUajaWwOY2jmaCHgmWNKqV0iYlNKOYEPRGT1QTNpNBqNRqM54qiq//1ApFxEmgGbROQ5zMd4Wx05WRqNRqPRaAKmCQ3FeBPoc0/XWLZ3AmWYs2QvPlKiNBqNRqPRBM7//BwRpVSyiLQAYpRSTxxhTRqNRqPRaA4BVdl0Ag9vAn1qZjTms8ALrfW+IjLvSArTaDQajUYTGMqhfJamQqBDM5Mxv9JXBKCU2oT5JV6NRqPRaDSNjKpUPktTIdDJqg6l1N4j+W0XjUaj0Wg0h4c6vPeZHRUE/NE7EbkSsIlId+BuQD++q9FoNBrNUUBTDkQCHZq5Czge84N3nwJ7gXuOlCiNRqPRaDSB46z0XZoKgQYix1mLHQgGLuTAJ4A1Go1Go9E0IsopPksgiMgoEflbRHaJiN8v5IrIZSKyVUS2iMinbts7ichiEdlmpcdb2xNE5FcR2Skin1nvIauVQIdmPgEeAP4Cmu4zQhqNRqPR/A/ichz6HE4RsQGvA2cBacB6EZmnlNrqZtMdmAgMVkoVikiUWxGzgKlKqSUi0poD8cGzwEtKqTki8hbwb+DN2nQE2iOSq5Sar5Tao5RKrl4C/bMajUaj0WiOHM4qw2cJgJOBXUqpRKVUJTAHc8TDnZuA15VShQBKqRwAETkOsCullljbS5VS5WI+1TIC+NLK/yFwUV0iAu0ReVxE3gV+xJwnglXx1wHm12g0Go1Gc4RwBTgU40UckOq2ngYM9LLpASAiqwAbMFkptdDaXiQiXwMJwFJgAhAKFClVM302zaqnVgINRMYBxwBBHOh6UYAORDQajUajaWT8BSIicjNws9umGUqpGe4mforyfgGJHegODAM6AD+LSC9r+2lAPyAF+Ay4HvD3stM6X2oSaCDSRynVO0BbjUaj0Wg0/yD+hmKsoGOGr3UNaZjfjqumA5Dhx2atUqoK2CMif2MGJmnA70qpRAARmQucArwPtBMRu9Ur4q9MDwKdI7LWGg/SaDQajUZzlOF0GT5LAKwHultPuTQDLse3R2MuMBxARCIwh2QSrbyhIhJp2Y0AtiqlFLAcuMTafh3wbV0iAg1EhgCbrEd8NovInyKyOcC8Go1Go9FojiAup/gsB8PqsbgTWARsAz5XSm0RkSkicoFltgjIF5GtmAHGg0qpfKWUE/Np2h9F5E/MYZ53rDzjgftEZBcQDrxXl45Ah2ZGBWin0Wg0Go3mH8bhsB1WPqXUAmCB17ZJbr8VcJ+1eOddApzgZ3si5hM5ARFQIKIf1dVoNBqN5ujF6Wq634ILtEfksCm7699HuopDZuOyqIMb/cPMCN5/cKN/mKqj9N11FznaNLYEH2YZ2xtbgg9RRrvGluBDoauisSX40MoobmwJfllVuquxJfjQ0h7c2BJ8mFOc3dgS/PJPf/rFpQMRjUaj0Wg0jUWV8/CGZo4GdCCi0Wg0Gk0Tx6l0j4hGo9FoNJpGwqECfQj26EMHIhqNRqPRNHEcukdEo9FoNBpNY+H0+7b2poEORDQajUajaeJU6UBEo9FoNBpNY+EUHYhoNBqNRqNpJPTQjEaj0Wg0mkajqgn3iDTd5300Go1Go9EA4BDfJRBEZJT1QdtdIjKhFpvLRGSriGwRkU+90tqISLqIvOa2bYVV5iZrqfN15rpHRKPRaDSaJs7hDM2IiA14HTgLSAPWi8g8pdRWN5vuwERgsFKq0E9Q8SSw0k/xVymlNgSiQ/eIaDQajUbTxKkS3yUATgZ2KaUSlVKVwBzgQi+bm4DXlVKFAEqpnOoEEekPtAcW10e7DkQ0Go1Go2niHObQTByQ6raeZm1zpwfQQ0RWichaERkFICIG8ALwYC1lf2ANyzwmUvcEFj00o9FoNBpNE8fp51IvIjcDN7ttmqGUmuFu4qco5bVuB7oDw4AOwM8i0gu4GliglEr1E2dcpZRKF5EQ4CvgGmBWbdp1IKLRaDQaTROnys82K+iY4SepmjSgo9t6ByDDj81apVQVsEdE/sYMTE4FThOR24HWQDMRKVVKTVBKpVv1l1iTW0+mjkBED81oNBqNRtPEOcyhmfVAdxFJEJFmwOXAPC+bucBwABGJwByqSVRKXaWU6qSUigceAGYppSaIiN2yQ0SCgPOBv+oSoXtENBqNRqNp4jgPI49SyiEidwKLABvwvlJqi4hMATYopeZZaSNFZKtVzYNKqfw6im0OLLKCEBuwFHinLh06ENFoNBqNpokT4FMyPiilFgALvLZNcvutgPuspbYyZgIzrd9lQP9D0aADEY1Go9FomjgOnzmmTQcdiGg0Go1G08Q53B6RowEdiGg0Go1G08Rx6h4RjUaj0Wg0jYUemqkn9j4DaHHtnWDYqFz+PfvnzfaxCTplGMEXXweAM3k35a89hf24vrS45o4aGyO2E+WvTqFqw6oG0RU2vA89nroesRlkfLKM5Fe/9bGJuuAUujxwKUopSrcms+W2VwEYkTGb0m0pAFSk57H52v82iKY+Q/tx/eM3YtgMls1Zwrdvfu1jc8p5g7n03stRSpG8LYlX736RiLhI7n97AoZhYAuysXDm9yz9ZFGDaOo39ERuePxGDJuNpXMW882bX/nYDDpvMGPvvQKlIGnbHqbf/QKRcZE89PZES5OdBTO/Y/EnCxtEU+ywExgw5RrEMNg1ewV/vT7fx6bz6IH0uW8MKEXh1hR+vvMNAK5OmUXRdvNlg2Xp+Swf92KDaBow7CRun3wrhs3GD7N/YM4bn/vYDD3/dK6992qUgsRtiUy76xmi4qKYPGMShs3Abrczd+a3fPfx9w2iCeCEof249vF/Y9gMls9Zynw/PjXwvEFcfO/lYPnU63e/RERcJPe+PR4xDOxBNhbNXMCPDeRT/Yf159bJt2LYDBbOXsgXb3zhY3Pa+adx9b1Xo5QicVsiz931HFFxUTw649Gatpo3cx4LPl7gp4ZDp+/QExln+fmPcxYz14+fn3reYC679wqw/Pzlu18gIi6SBy0/twfZ+aEB/Xz4GUOY8sxEbDYbn876ktemv+tjM/qiUTww4Q6UUmz5azt33PQQx/c+hmdemERISGucLicvP/82875pGE2njxjEY9MewGbY+Ozjb3j7lZk+NudeeBZ3P3QLSim2b9nBvbc8wrG9ejDlvw/TOqQVLqeLN156j+/n1uvN4R6cPXIYL744BZth8P4Hs3nuv6/72FxyyWgmPXYfSik2b97KNdfeCcD38z9m4MATWbVqPRf+67oG09SQVOlApB6IQYtx91A27UFc+bmETH2Lqo2rcaUn15gY0XE0v/BKSiffhSorRdq0A8CxdRMlE28yi2kVQsj0j6naHNA3dg6OIfR85gZ+v2wq+zPyGbDoafIWbaBsR3qNSYuEaOLvvogNoyfh2FtGUESbmjRnRSXrzhjfMFosxDC44clbmHrV4+Rn5fP0vP+yYek60nem1dhEx8dw0R0XM2nMBMqKy2gT3haAwpxCHhszHkelg+Ytg3l+8StsXLKOwpzCemkyDIObnryFJ66aRH5WPs/Ne4H1S9eRtvPAW4Nj4mMYc8elPDxmPGXFZbR10zRxzEM4Kh0Etwxm+uJXWb9kHYU5BfXSJIYwcOp1LLniGcozCzh3wRRSF29k784D7+kJSWhP7ztHs/CiJ6jcW05wuOe++27kI/XS4I1hGNz11B2Mv3IiuZl5vP7dq6xespaUnSk1NnHxsVxxx1juGXMfpXtLaWe1U0FOAff8616qKqsIbhnMu0vfZs2SNeRn16+dwPSpcU/ezNNXTSY/K5+n5j3Hb3586sI7LuaJMRN9fOrxMRNqfOq5xS+zcck6ihrAp+546g4evvJh8jLzePm7l/l1ya8ebRUbH8vYO8Zy/5j7Kd1bWuNTBTkF3P+v+2va6q2lb7F2yVoK6tlWhmFw45O3MOWqSRRk5fPMvBfY4OXn0ZafP2r5eXU7FeUU8oibn7/YQH5uGAbTnn+UsRfdSGZGNj8s/4zFPyxnx9+7a2wSunTmrvtu4oKzr2Lv3mLCI8IA2Fe+j7tvnciexGTaR0eyaMWXrFi2iuK9JfXWNPnZ8Vx3ye1kZWTzzZKP+XHhSnbt2FNjE9+lI7feM47Lzh1H8d4SwiNCTU37KnjwjsdISkwlKjqCb3/8hJ+WraakuLRemqp1vfLyVEadewVpaZmsXbOA+d8tZtu2nTU23bolMP6hOzl96EUUFe0lMjK8Ju2FF9+iZcsW3HTj1fXWcqQ4nMd3jxYCeqGZiAT72RbREAJs3Y7BlZWBKycTnA4q1ywj6KTBHjbNRpxP5eK5qDLTIVVxkU85QQOH4ti0Dir3N4Qs2pzYjX17sqlIzkFVOcmeu5qIUQM8bOKuPoO0Dxbj2FsGQFVecYPUXRvd+nYnOymTnNRsnFUOVs//hQFnDfSwOeOKkSyetYCyYlNTcf5eAJxVDhyVDgCCmgVhGA0zs6lb3+5kJmWSnZqNo8rBL/N/5mQvTWdecTYLZ31fo2mvpcnhpsneLAgxGub9euH9ulKSlE1pSi6uKidJ366l49meT5N1v3I422cupXJvOQAV+Ud23/Xs25OMpAwyU7JwVDlYMW8Fg0ee6mFz7pXn8O2H8ynda/p5kVs7VVWa701s1iwIo4HaCXx9as38X+h/1skeNsOvOIvFs344qE9JA/lUj749yEjKIMtqq5XzVnLKyFM8bEZdOYr5bm21109bNaSmbn27k2W1k6PKwar5P/sce95+XnyE/bxf/94kJaaQkpxGVVUV3371A2efO8LD5qrrLmHmO5+yd6/p3/l5ZvCTuDuZPYnmzV52Vi55efmEh4fVW1OfE3uRvCeN1OR0qqocfPfNIs48Z5iHzdhrxvDx+5/XBD35eWbgmrQ7haREM7DLycojP7ewJkipLycP6Mfu3Uns2ZNCVVUVn3/+LReMPtvD5sZ/X8mbb86kqMjcb7m5B16VsWz5L5SU1D8gOpI4UT5LUyHQHpH1InKTUmotgIhcDDyN+Ya1emGERuDKr/mYH678XOzdjvWwsUV3AKD15FfBMKj4aiaOP9Z72AQNGs7+7327bw+X4OgwKjIOOOL+jHzanNjNw6Zl1xgA+s+fgtgMEv/7BQXL/zD/V/MgBiyahnK6SHp1Lnk/1L+nJiw6jPzMvJr1/Mx8uvXr7mETkxALwJSvnsYwDL6YPoc/Vv4OQHhMBOM/eJTo+Bg+njaz3r0hAOHR4V6a8ujer6eHTayladpXz2IYBp9Nn83vK3+r0fTIB5OIiY/hw2kf1PsuEaBldChlGQfKKc8sIKJfVw+bNl2iARg1dxJiM/jjha/JWLEZAFvzIM5dMAXldPHXa/NJXbSx3poiosPJycitWc/NzOOYfsd42HToYvr59K9fxGYzmPXSx6xfYfpNZEwkUz+cQmx8LDOmvtsgvSEAoV4+VZCZT7d+nod1tU89/tU0DMPgq+mfsdnyqbCYcB764FHax8fw6bQP690bAhARHUGuW1vlZebR08un4rqY3+V6/uvnsdlsfPzSx2xcYe6niJgIpnw4hZj4GN6b+l69e0MAwqLDyQvQz5+y/Pzz6bPZ5ObnD38wiej4GD5qID+PjmlPenpWzXpmRhb9+p/gYdO1WzwA3y78GJvNxgvPvM7yH3/xsOl7Ym+aBQWRtCeF+tI+JpLMjAOasjJy6NO/l4dNQtdOAHz+/fsYNhuvPPc2Py1b7WFzQr/jCWoWRPKeNBqC2LhoUtMO9IimpWdy8oB+Hjbdu3cB4KcVc7HZbEx58gUWLV7RIPX/E1ThamwJh02ggciVwPsisgKIBcKBEXXmCBS/H+XziuRsNozoOEqf/A9GWCStH3+FkofGocrNOw9pF4atYxccm9f7KashdXmZ2A1adInmt389QfPYMPp/+wS/Dn0AR3E5q068g8rsQoI7R3Hil49RtjWVfcnZ9ZPk7/tEXk1l2A2i42N4YuyjhMWE88QX03hg5D2UF5eRn5nHQ6P+Q2hUKA+8M5FfF6xmb97eemny+80k5SnKZrcRGx/DY2MfJjwmgqlfPM09I++q0XTfqLsJjQpjwjsPs2bBavbm+fZ4HZIif/vOp51stEmIZtElU2kVE8bZ3zzGvBETqCou56uT72FfdhGtO0Uy8vOHKdyeSmlyjm+Z9dbk1U42G3EJcdx/2YNExkTw0lcvcOOZt1BWXEZuZi43j7yN8PZhPPHuZH76/meK6tlO4N+nlJcuw24jOj6Gp8Y+RlhMOJO+mMr4kfdQXlxOQWY+E0bdS7uoUO63fKq4vj4V0CnBbKvxl40nIiaC5796nlvPvJWy4jLyMvO4feTthLUPY9K7k/jl+1/q3VaBtJPNbiMmPobHLT9/8ounudfNz++3/PyhI+jn3vfANpuNLl07c/H51xMT1565Cz5i+KALa3ojotpH8Orbz3DPbRN9/k9DafI9H9iJ79KRKy+8mejYKOZ89x7nDLm0Zggmsn0EL7z5JA/e8XiDaKpNl3fZdpudbt0SGHHmJXToEMOKZd/Qp9+Imt6ko52m1APiTUB9hEqpP4GpwK2Y75y/UylVa6gqIjeLyAYR2TBzl/f3czxxFeRihEcdEBQeiasw38fGsWEVOJ24crNwZqZiWL0kAEGnDKdq/S/gbLhRsorMfIJjD4wRNo8NZ3+W591eRUYBeQs3oBxOKlJyKd+dQYsuZi9JZbZpW5GcQ+HqrYT0jq+3pvysfMJjDoyIhceEU+h1t1eQmc+GJetwOpzkpuaQkZhBTHyMh01hTiFpO1I55uTjGkBTnpemCJ870PzMPNYt+RWnw0lOajbpienE+mgqIHVHCsc1gKayzAJaxR7oZm4ZE0Z5dqGPTerijSiHk9LUXIp3Z9Imwewl2ZdtXiBKU3LJWrONsF6d660pNzOPqNjImvXImAjys/N9bFYvXoPT4SQrNZvU3Wl0SPD8Ind+dgFJO5LpfbLnXebhUuDlU2G1+NRGN5/KTMwgOj7Ww6Yop5C0HSkN4lN5mXlEurVVhJ+2ysvMY43VVtmp2aTtTiPOq60KsgtI3pFMrwZoq/ysPCK8/Ny7nfIz81jv5ucZiel+jj3Tz49tgHbKzMgiLi66Zj0mNprszBwvm2wWLliGw+EgNTmd3buSSOhi+nPrkFZ8/PlbPPvUK/y2YXO99YDZAxITe0BTdGwU2Vm5XjbZLP1hJQ6Hg7SUDPbsSibe6iVp3boV785+mRenvcGmjX82iCaA9LRMOnY44LMd4mLIzPS8MUxLz2TevMU4HA6SklLZsWM33bslNJiGI01THpoJdI7Ie8B/gBOAccB8EbmjNnul1Ayl1ElKqZOu7xZbmxkAzt3bMaLjMCKjwWan2akjqNro2U1XteEX7Meb3WgS0gZbTAdzTolFs0EjqFr9YyB/JWBKft9Nyy7RBHeKRIJstL9oEHmLPIdXcn9YT+jg4wEICguhZZcY9iVnY2/bCmlmr9ne7uSelO2ofxfj7j92Ep0QQ2THKGxBdgaNHsKGJes8bNYv/pXjTzVPvCGhIcQkxJKdkk1YdDhBzZsB0KpNK3qcdAwZu+sOEgNh1x87iUmIJapje+xBdoaMPo31S371sFm3+Fd6nXpCjabYxax8dQAAD6BJREFUhFiyUrIJjw6nmZumY046lvTd6T51HCr5mxIJSYimdcdIjCAb8ReeQuri3zxsUhduJHqQeTFo/n/t3X10VPWdx/H3Z0JQkSKIokKogAaVg4KIHhREfKx2VXQtLHp8WNsF3dVWVqsVDoLV6tFW21rKsUZBoAI+IAqiPIQI6KooBAjyYALlKUFaXAWEditk5rt/3Bs6JCHcwpA7ke/rnHsyM/nNzSc3NzO/+T3dFk1p1uFEdm7cQuNjmpAI/3ZHtGhKq3M7sr3s4DOVlpTSpl0bTgyPU59r+/Bh4YK9ynw4+0O6nt8FgGYtmpHXIY/NGzZz3InH0fjI4Dg1PaYpnbt3omJtZpqsq59T51/Ti+LCvVsWF83+mE7nnwn845zaUus5dQabM/D3Kyspo3W71pwQHquLrr2IBdWO1UezP6JL2rFq06FNrceqU4aOVfXzvGeE8zz9f6/6ef55Bo7T0sXLaX/KybQ9uQ25ubn0veEqZs2Yu1eZmW8X0fPCYMzPscc2p8MpJ7NxfTm5ubmMeWkkr708lelTMzPTCWDZkhW069CWvO+2Jje3EVdf/z2KZs7fq0zhO/Po0as7AC2ObU77U75L+fpN5OY24tnxT/PGK28zY9qcjGUCWLhoKaee2p527dqSm5tL//59eWv63jNypk2bSZ8+FwDQsmUL8vM7sDYD3VX1ZbdZja2hiNo1sxz4j3DN+XWSegCZmdOYSvF/Y3/H0UN+CYkEu+bNIFWxniN/cDuV60qpLP6QypKFNDrzXL7zqxeD8hP+gO0MmssSx51AouXxVK4qyUicKpZMUTpkDGe/PBRyEmyeNI+/llbQ4YF+fF2ylv+dVcxXc0to2ecserz3NJZKseaRCVRu3ckx3Tty+lMDsZShhFg/cupes20OVCqZYszw5xk6fgSJnBzmvTqHitXl9Lv3RtYuW0PxnIWUzF/CWb278vSckaSSKSY8Ppad23ZwZq8u3DLs9qCZVGJ6wVTKSzfs/4dGyPTC8OcYPv5hEjkJil6dQ/nqcgbcexN/WraGhXM+Ycn8xXTp3ZVn5vyeVDLFuDDTKb26ctuwH+7JNLXgTTZmIJMlU3wybByXTXwgmL77yny2l22iy09v4MuSdVQULubzectofdGZXDv3SSyZovjRSXyzdSfHd8+nxxM/xCyFlGD579/aa7bNwRynkQ+N4omXHg+mpL4ymw1lG7jtvlspW1bGR4ULWDhvEef07sboogJSqRQFjz3P19t20O3CfO58aGDVYeK15yaz7rP1B52pKtfY4c/z4PgRJHISzHu1iE2ry/lBeE4tnrOQZeE59cs5vyOVTDHx8XHs3LaDzr26cPOwf8fMkMTbBW9SXnrwL9ypZIpnH3qWX7z0C3Jycpj9ymw2lm3klvtuoWxZGR8XfkzxvGK69e7Gc0XPkUwlGf3YaHZs28GpF57KwIcG7sk05bkprM/Asao6z4eF5/m74f/ev4Xn+aI5n7A0PM9/E57nfwzP87PC87wq07QMnefJZJKh9z/GpNefJycnwcsvvUHZZ2u4f+jdlCxZwewZc5lb9D9cdMkFzF/wFslkkkeHP8XWrdu5of819LjgHFoc25z+N10PwOD/GsqKTz876Ew/f/BJxr42ikQiweSJ01hdupbBD97Jp0tXUjTzPd5790N6XdyDmR9MJpVM8sTDv2Xb1u307fd9zj3/bJq3OIYbBlwDwAM/HsGq5WUZOVb3DB7GO29PJCeRYOy4V1i5soyHR/yURcUlTJ9eyKzZ87j8sotYVjKXZDLJz4Y8yldfBS2p896dwmmnnUrTpk1Yv3YRg+64j9mF8/fzU+tXsgGPEVGm+uD2ZduNF2ddtaz43Vb7L1TPCo7MzGyfTMrWwU/XVTbbf6F6Nj7xxf4L1bNWiSZxR6hha+rvcUeo4ehEbtwRavXB9jVxR6ihSaMaEyhjt+Hrgxt7d6hU7tpUr4uu9zu5b4332tc2TN1vBklXAs8QXCn3BTN7opYy/YGHCYYhlZjZTZJOBqaEz8sFRprZH8Ly5xBcBO8oggvq3WN1VDYitYhIyieYJdMJ2HMmmlmHKM93zjnn3KFTaf/8B0dJOcAo4HKggmCG7DQzW5lWJh8YAvQ0s62Sqj7JbwYuMLNvJDUFlofP/Rx4FhgELCCoiFwJzNhXjqgT2l8Md1xJMFh1PPDHyL+tc8455w6ZSqzGFsF5wBozW2tmu4CXgb7VygwERpnZVgAz2xJ+3WVmVU35RxDWJySdBDQzs4/CVpDxwHV1hYhaETnKzIoIunI2mNnDZGr6rnPOOecOSqUla2zpM1jDbVC1p7UBytPuV4SPpesIdJT0gaQFYVcOAJLaSloW7uPJsDWkTbifuva5l6iDVf8uKQGslnQ3sAnIvoEWzjnn3GGotum6ZlYAFNTxtAir9tAIyAf6AHnA+5I6m9k2MysHzpLUGnhT0uSI+9xL1BaRwUAT4CfAOcDNwK0Rn+ucc865QyhpqRpbBBVA27T7eUD1aYIVwFQz221m64BSgorJHmFLyArgwrB8Xtq3a9vnXqJWRIxgTMg0oDtBU83zEZ/rnHPOuUNotyVrbBEsBPIltZfUGBhA8D6f7k2CsaFV15jrCKyVlCfpqPDxFkBPoNTMNgM7JPVQsKTtrUDNS9enido1MwG4H/gUsnROp3POOXeYOpCVVM2sMhxuMYtgGu4YM1sh6RFgkZlNC793haSVBBf5vd/MvpR0OfC0JCPojnkqXIUd4D/5x/TdGdQxYwaiV0S+CAM555xzLstE7IqpwczeIZhim/7Y8LTbBtwbbullCglWW69tn4uAyNdWiFoRGSHpBaAI2LPylplNifqDnHPOOXdoVEbrislKUSsitwOnE6yeVlXtMoJV1ZxzzjkXowNtEckGUSsiXczszEOaxDnnnHMHpCFfaybqrJkFkg7+utXOOeecy7jdqWSNraGI2iLSC7hN0jqCMSIiGMNS60AV55xzztWfw6Fr5sr9F3HOOedcHL71FREz23CogzjnnHPuwFRaZdwRDljUFhHnnHPOZalvfYuIc84557JXMuUVEeecc87F5HBY0Mw555xzWaoht4hEXUfEOeecc1mqMpWssUUh6UpJpZLWSHpwH2X6S1opaYWkiWmPz5S0TdL0auXHSlonaWm4da0rg7eIOOeccw3cgQxWlZQDjAIuByqAhZKmmdnKtDL5wBCgp5ltldQqbRe/ApoAd9Sy+/vNbHKUHN4i4pxzzjVwyVSqxhbBecAaM1trZruAl4G+1coMBEaZ2VYAM9tS9Q0zKwJ2HGx2r4g455xzDVwylayxRdAGKE+7XxE+lq4j0FHSB5IWSIq6wOljkpZJ+o2kI+oqeMi7ZppPmqtM7UvSIDMrONj9XJqJMKFvc6ZMysZMkLlct2YiTCgbj5VniiYbM0F25vJMmbV716Ya77WSBgGD0h4qqPb71fb+bNXuNwLygT5AHvC+pM5mtq2OOEOAPwONgQLgZ8Aj+yrc0FpEBu2/SL3zTNFkYybIzlyeKRrPFF025vJMh5iZFZhZ97SteiWrAmibdj8P+LyWMlPNbLeZrQNKCSomdf3czRb4BniRoAtonxpaRcQ555xzmbEQyJfUXlJjYAAwrVqZN4GLASQdR9BVs7aunUo6Kfwq4DpgeV3lfdaMc845dxgys0pJdwOzgBxgjJmtkPQIsMjMpoXfu0LSSiBJMBvmSwBJ7wOnA00lVQA/MrNZwARJxxN0/SwF7qwrR0OriGRj351niiYbM0F25vJM0Xim6LIxl2fKAmb2DvBOtceGp9024N5wq/7cC/exz0v+mQwKfoZzzjnnXP3zMSLOOeeci02DqIhEWYK2vkkaI2mLpDoH4dQnSW0lzZW0KlyK954syHSkpE8klYSZfh53piqSciQtqb48cVwkrZf0abgk8qK481SR1FzSZEmfhefW+THnOS1t6eilkr6WNDjOTGGu/w7P8eWSJkk6Mgsy3RPmWRHnMart9VLSsZIKJa0Ov7bIgkz9wmOVktS9PvMczrK+IpK2BO1VQCfgRkmd4k0FwFgg6sIu9aUSuM/MzgB6AHdlwbH6BrjEzLoAXYErJfWIOVOVe4BVcYeo5mIz62pm2fQi+Aww08xOB7oQ8zEzs9LwGHUFzgH+BrwRZyZJbYCfAN3NrDPBwL8BMWfqTLAq5nkEf7erw+W64zCWmq+XDwJFZpYPFIX34860HPhX4L16znJYy/qKCNGWoK13ZvYe8FXcOdKFc7cXh7d3ELxhVF8lr74zmZntDO/mhlvsA5Mk5QH/ArwQd5ZsJqkZ0BsYDWBmu/azkFF9uxT4k5ltiDsIweD/oyQ1Irj+RvX1GOrbGcACM/ubmVUC84Hr4wiyj9fLvsC48PY4gmmesWYys1VmVlqfOVzDqIhEWYLWVSOpHXA28HG8SfZ0gSwFtgCFZhZ7JuC3wANANl0724DZkorDFRGzQQfgC+DFsBvrBUlHxx0qzQBgUtwhzGwT8BSwEdgMbDez2fGmYjnQW1JLSU2A77P34lVxO8HMNkPwIQpotZ/y7luqIVREoixB69JIagq8Dgw2s6/jzmNmybAZPQ84L2wyjo2kq4EtZlYcZ45a9DSzbgTdkHdJ6h13IIJP+d2AZ83sbOCv1H8Teq3CBZiuBV7LgiwtCD7htwdaA0dLujnOTGa2CngSKARmAiUE3bfOZZWGUBGJsgStC0nKJaiETDCzKXHnSRc26c8j/rE1PYFrJa0n6Oq7RNJL8UYCM/s8/LqFYMxDncsi15MKoCKtFWsyQcUkG1wFLDazv8QdBLgMWGdmX5jZbmAKcEHMmTCz0WbWzcx6E3RDrI47U5q/pK3AeRJBi6k7DDWEikiUJWgde5bTHQ2sMrNfx50HQNLxkpqHt48ieMH+LM5MZjbEzPLMrB3B+fSumcX66VXS0ZK+U3UbuIL9LItcH8zsz0C5pNPChy4FVsYYKd2NZEG3TGgj0ENSk/D/8FKyYCC0pFbh1+8SDMLMluMFwev4beHt24CpMWZxMcr6lVX3tQRtzLGQNIngaoTHhUvbjjCz0fGmoidwC/BpOCYDYGi4cl5cTgLGhbOfEsCrZpYV02WzzAnAG8F7GI2AiWY2M95Ie/yYYMnmxgTXmLg95jyEYx4uB+6IOwuAmX0saTKwmKD7YwnZsUrn65JaAruBu8xsaxwhanu9BJ4AXpX0I4KKXL8syPQVMBI4Hnhb0lIz+1595joc+cqqzjnnnItNQ+iacc4559y3lFdEnHPOORcbr4g455xzLjZeEXHOOedcbLwi4pxzzrnYeEXEOeecc7HxiohzzjnnYuMVEeecc87F5v8BSdOXlkeqA20AAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 720x144 with 2 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "if len(results) == 24:\n",
    "    df = pd.DataFrame(\n",
    "        np.reshape(\n",
    "            [r[\"STSBenchmark\"][\"pearson\"] for r in results],\n",
    "            (len(EXP_PARAMS[\"layer_index\"]), len(EXP_PARAMS[\"pooling_strategy\"])),\n",
    "        ).T,\n",
    "        index=[s.value for s in EXP_PARAMS[\"pooling_strategy\"]],\n",
    "        columns=EXP_PARAMS[\"layer_index\"],\n",
    "    )\n",
    "    fig, ax = plt.subplots(figsize=(10, 2))\n",
    "\n",
    "    sns.heatmap(df, annot=True, fmt=\".2g\", ax=ax).set_title(\n",
    "        \"Pearson correlations of BERT sequence encodings on STS Benchmark\"\n",
    "    )"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python (nlp_gpu)",
   "language": "python",
   "name": "nlp_gpu"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.8"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}