notebooks/notebook2.ipynb

{
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "0Oe730o8rzyn"
      },
      "source": [
        "# PyTerrier Tutorial Notebook 2\n",
        "\n",
        "This notebook provides experiences to attendees for building transformer pipelines in [PyTerrier](https://github.com/terrier-org/pyterrier). All experiments are conducted using the [CORD19 corpus](https://www.ncbi.nlm.nih.gov/pmc/articles/PMC7251955/) and the [TREC Covid test collection](https://ir.nist.gov/covidSubmit/).\n",
        "\n",
        "This notebook is divided in two, following the structure of the slides. In particular, in Part A you will experience: \n",
        " - understand the PyTerrier data model;\n",
        " - understand the general notion of PyTerrier transformers and operators for combining transformers into new transformers;\n",
        " - undertake experiments on the TREC Covid test collection.\n",
        "\n",
        "In Part B, you will experience:\n",
        " - construct and learn learning-to-rank pipelines;\n",
        " - evaluate and analyse learning-to-rank pipelines.\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "0edvI3LDtOym"
      },
      "source": [
        "# Setup\n",
        "\n",
        "We install PyTerrier, as well as the [LightGBM](https://lightgbm.readthedocs.io/en/latest/) and [FastRank](https://github.com/jjfiv/fastrank) learning-to-rank implementations, and we will initialise PyTerrier."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "mkCdrWdRzS64"
      },
      "outputs": [],
      "source": [
        "%pip install -q --upgrade fastrank lightgbm==3.1.1\n",
        "%pip install -q python-terrier"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "SWc4UQgX6vDD"
      },
      "outputs": [],
      "source": [
        "import pyterrier as pt\n",
        "if not pt.started():\n",
        "  pt.init()\n",
        "cord19 = pt.datasets.get_dataset('irds:cord19/trec-covid')\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "p18d1tpJvD_5"
      },
      "source": [
        "# Indexing\n",
        "\n",
        "For this notebook, we'll be using a Terrier index with term position information. We could create a new index, using the following code (note the additional `blocks=True` kwarg):\n",
        "```python\n",
        "import os\n",
        "\n",
        "pt_index_path = './terrier_cord19_blocks'\n",
        "\n",
        "if not os.path.exists(pt_index_path + \"/data.properties\"):\n",
        "    # create the index, using the IterDictIndexer indexer \n",
        "    indexer = pt.index.IterDictIndexer(pt_index_path, blocks=True)\n",
        "\n",
        "    # we give the dataset get_corpus_iter() directly to the indexer\n",
        "    # while specifying the fields to index and the metadata to record\n",
        "    index_ref = indexer.index(cord19.get_corpus_iter(), \n",
        "                              fields=('abstract',), \n",
        "                              meta=('docno',))\n",
        "\n",
        "else:\n",
        "    # if you already have the index, use it.\n",
        "    index_ref = pt.IndexRef.of(pt_index_path + \"/data.properties\")\n",
        "```\n",
        "\n",
        "However, its just as quick to use the pre-built index from the Terrier Data Repository. We use the ['terrier_stemmed_positions'](http://data.terrier.org/trec-covid.dataset.html#terrier_stemmed_positions) index variant."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "C9E6oLubIeI4"
      },
      "outputs": [],
      "source": [
        "index = pt.IndexFactory.of(\n",
        "    pt.get_dataset('trec-covid').get_index('terrier_stemmed_positions')\n",
        ")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "OiM8I_fkUcQ9"
      },
      "source": [
        "## Transformers & Operators\n",
        "\n",
        "You'll have noted that BatchRetrieve has a `transform()` method that takes as input a dataframe, and returns another dataframe, which is somehow a *transformation* of the earlier dataframe (e.g., a retrieval transformation). In fact, `BatchRetrieve` is just one of many similar objects in PyTerrier, which we call [transformers](https://pyterrier.readthedocs.io/en/latest/transformer.html) (represented by the `pt.Transformer` class).\n",
        "\n",
        "Let's give a look at a `BatchRetrieve` transformer, starting with one for the TF_IDF weighting model."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "UqMt-9UlTqLa"
      },
      "outputs": [],
      "source": [
        "tfidf = pt.BatchRetrieve(index, wmodel=\"TF_IDF\")\n",
        "\n",
        "# check tfidf is a transformer...\n",
        "print(isinstance(tfidf, pt.Transformer))"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "lAR09Eff7lUE"
      },
      "outputs": [],
      "source": [
        "# this prints the type hierarchy of the TF_IDF class\n",
        "tfidf.__class__.__mro__"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "GW3afDgPukvU"
      },
      "source": [
        "The interesting capability of all transformers is that they can be combined using Python operators (this is called *operator overloading*).\n",
        "\n",
        "Concretely, imagine that you want to chain transformers together – e.g. rank documents first by Tf then re-ranked the *exact same* documents by TF_IDF. We can do this using the `>>` operator – we call this *composition*, or \"*then*\"."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "e9HkXxtgug15"
      },
      "outputs": [],
      "source": [
        "# this is our first retrieval transformer\n",
        "# it transform a queries dataframe to a results dataframe\n",
        "tf = pt.BatchRetrieve(index, wmodel=\"Tf\")\n",
        "\n",
        "tf( cord19.get_topics(variant='title').head(1) )"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 1,
      "metadata": {
        "id": "uHHIn-rMvk_5"
      },
      "outputs": [
        {
          "ename": "NameError",
          "evalue": "name 'tf' is not defined",
          "output_type": "error",
          "traceback": [
            "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
            "\u001b[0;31mNameError\u001b[0m                                 Traceback (most recent call last)",
            "\u001b[0;32m/var/folders/9d/73m8d3j55q520slwcmnp74cw0000gn/T/ipykernel_35102/2847231192.py\u001b[0m in \u001b[0;36m<module>\u001b[0;34m\u001b[0m\n\u001b[1;32m      1\u001b[0m \u001b[0;31m# now let's define a pipeline\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m----> 2\u001b[0;31m \u001b[0mpipeline\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mtf\u001b[0m \u001b[0;34m>>\u001b[0m \u001b[0mtfidf\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m      3\u001b[0m \u001b[0mprint\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0misinstance\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mtfidf\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mpt\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mTransformer\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m      4\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m      5\u001b[0m \u001b[0mpipeline\u001b[0m\u001b[0;34m(\u001b[0m \u001b[0mcord19\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mget_topics\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mvariant\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;34m'title'\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mhead\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;36m1\u001b[0m\u001b[0;34m)\u001b[0m \u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
            "\u001b[0;31mNameError\u001b[0m: name 'tf' is not defined"
          ]
        }
      ],
      "source": [
        "# now let's define a pipeline \n",
        "pipeline = tf >> tfidf\n",
        "print(isinstance(tfidf, pt.Transformer))\n",
        "\n",
        "pipeline( cord19.get_topics(variant='title').head(1) )"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "r1yCXIB5w1Qb"
      },
      "source": [
        "There are a number of PyTerrier operators – there are more examples in the [PyTerrier documentation on operators](https://pyterrier.readthedocs.io/en/latest/operators.html)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "NaYavGvnxDk5"
      },
      "source": [
        "## Practice Task – Pipeline Construction\n",
        "\n",
        "Create a ranker that performs the following:\n",
        " - obtains the top 10 highest scoring documents by term frequency (`wmodel=\"Tf\"`)\n",
        " - obtains the top 10 highest scoring documents by TF.IDF (`wmodel=\"TF_IDF\"`)\n",
        " - reranks only those documents found in EITHER of the previous retrieval settings using BM25.\n",
        "\n",
        "by making use of PyTerrier operators combining different BatchRetrieve instances.\n",
        "\n",
        "How many documents are retrieved by this full pipeline for the query `\"chemical\"`.\n",
        "> If you obtain the correct solution, the document with docno `\"8hykq71k\"` should have a score of $12.646089$ for query `\"chemical\"`.\n",
        "\n",
        "Hints:\n",
        " - choose careully your [PyTerrier operators](https://pyterrier.readthedocs.io/en/latest/operators.html)\n",
        " - you should not need to perform any Pandas dataframe operations"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "XnsRdT4WvspD"
      },
      "outputs": [],
      "source": [
        "# YOUR SOLUTION GOES HERE\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "-jgcfi900J7z"
      },
      "outputs": [],
      "source": [
        "#@title See Solution\n",
        "pipe = ((tf %10) | (tfidf % 10)) >> pt.BatchRetrieve(index, wmodel='BM25')\n",
        "pipe.search('chemical')"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "4g8V7Zzpy3g1"
      },
      "source": [
        "## Other transformers in the toolbox\n",
        "\n",
        "Lets start creating more interesting retrieval pipelines. We'll define these, but firstly lets recap on the PyTerrier datamodel:\n",
        " - $Q$: a set of queries\n",
        " - $D$: a set of documents\n",
        " - $R$: a set of retrieved documents for a set of queries\n",
        "\n",
        "Lets use three: \n",
        " - `pt.BatchRetrieve(index, wmodel=\"BM25\")` - input $Q$ or $R$ (retrieval or reranking), output $R$\n",
        " - `pt.rewrite.SDM()` (sequential dependence proximity model) - input $Q$, output $Q$. \n",
        " - `pt.rewrite.Bo1QueryExpansion(index)` - input $R$, output $Q$.\n",
        "\n",
        "\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "5nJHxh7A2dPP"
      },
      "outputs": [],
      "source": [
        "bm25 = pt.BatchRetrieve(index, wmodel=\"BM25\")\n",
        "sdm = pt.rewrite.SDM()\n",
        "qe = pt.rewrite.Bo1QueryExpansion(index)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "6F_YTyOQyV-A"
      },
      "source": [
        "Lets see how `sdm` applies to a given query. This generates a query in an Indri-like query language that Terrier (c.f. `pt.BatchRetrieve()`) can understand.\n",
        " - `#combine()` - is used for weighting sub-expressions\n",
        " - `#1() - matches as a phrase, i.e. how many times does the constituent words exactly match as a phrase\n",
        " - `#uw8()` and `#uw12()` look for how many times the constituent words appear in unordered windows of 8 or 12 tokens.\n",
        " - finally, the weighting model is overridden for these query terms.\n",
        "  "
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "EsNzb6kAyV-A"
      },
      "outputs": [],
      "source": [
        "sdm.search(\"chemical reactions\").iloc[0][\"query\"]"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "IVmde7NpJdTT"
      },
      "source": [
        "## Practice Task – Experimenting with Pipelines\n",
        "\n",
        "Conduct an [Experiment](https://pyterrier.readthedocs.io/en/latest/experiments.html) comparing sequential dependence model and Bo1 query expansion on TREC CORD19 with the BM25 baseline. You will need to construct appropriate pipelines, by considering the input and output datatypes of the `bm25`, `sdm` and `qe`. \n",
        "\n",
        "Which approaches result in significant increases in NDCG and MAP? Is NDCG@10 also improved?"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "TF24TZSb3bOo"
      },
      "outputs": [],
      "source": [
        "topics = cord19.get_topics(variant='title')\n",
        "qrels = cord19.get_qrels()"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "eqXox-g-JpUd"
      },
      "outputs": [],
      "source": [
        "# YOUR SOLUTION GOES HERE\n",
        "from pyterrier.measures import *"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "xi9YoMa402CU"
      },
      "outputs": [],
      "source": [
        "#@title See Solution\n",
        "from pyterrier.measures import *\n",
        "pt.Experiment(\n",
        "  #these are our 3 pipelines\n",
        "  [bm25,\n",
        "   bm25 >> qe >> bm25,\n",
        "   sdm >> bm25],\n",
        "  topics,\n",
        "  qrels,\n",
        "  eval_metrics=[MAP, nDCG, nDCG@10],\n",
        "  # we declare BM25 as the baseline to obtain significance testing\n",
        "  baseline=0,\n",
        "  names=[\"BM25\", \"BM25 >> QE >> BM25\", \"SDM >> BM25\"]\n",
        ")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "0aJrYP1TSLwY"
      },
      "source": [
        "# Part B - Learning to Rank\n",
        "\n",
        "In this part of the notebook, you will experience constructing, learning, evaluating and analysing learning to rank pipelines.\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "CSi3svV24mCs"
      },
      "source": [
        "Firstly, lets split out topics into train, validation and test sets. TREC Covid only has 50 topics, which isnt a lot for training(!). We'll split this 30 for training 5 for validation and 15 for evaluation. We will also examine statistical significance, even if this is artificial for 15 topics.\n",
        "\n",
        "We're only going to-rank the top 10 documents for each query - hopefully learning to rank can help to re-rank the top 10 documents to be more effective."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "INvVKQz6K7SB"
      },
      "outputs": [],
      "source": [
        "RANK_CUTOFF = 10\n",
        "SEED=42\n",
        "\n",
        "from sklearn.model_selection import train_test_split\n",
        "\n",
        "tr_va_topics, test_topics = train_test_split(topics, test_size=15, random_state=SEED)\n",
        "train_topics, valid_topics =  train_test_split(tr_va_topics, test_size=5, random_state=SEED)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "vftc_y-65VfA"
      },
      "source": [
        "## Feature Set\n",
        "\n",
        "Lets define our feature set.  We're going to have a total of 7 features:\n",
        "\n",
        "1.   the BM25 abstract score;\n",
        "2.   sequential dependence model, scored by BM25;\n",
        "3.   does the abstract contain 'coronavirus covid', scored by BM25;\n",
        "4.   the BM25 score on the title (even though we didnt index it earlier!);\n",
        "5.   was the paper released/published in 2020? Recent papers were more useful for this task;\n",
        "6.   does the paper have a DOI, i.e. is it a formal publication?\n",
        "7.   the coordinate match score for the query - i.e. how many query terms appear in the abstract.\n",
        "\n",
        "Several of these feature require additional metadata `[\"title\", \"date\", \"doi\"]`. Fortunately, the TREC Covid dataset allows us to obtain more metadata after indexing. We use `pt.text.get_text(cord19, [\"title\", \"date\", \"doi\"])` to retrieve these extra metadata columns.\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "AYgwwrsPTGP1"
      },
      "outputs": [],
      "source": [
        "ltr_feats1 = (bm25 % RANK_CUTOFF) >> pt.text.get_text(cord19, [\"title\", \"date\", \"doi\"]) >> (\n",
        "    pt.transformer.IdentityTransformer()\n",
        "    ** # sequential dependence\n",
        "    (sdm >> bm25)\n",
        "    ** # score of text for query 'coronavirus covid'\n",
        "    (pt.apply.query(lambda row: 'coronavirus covid') >> bm25)\n",
        "    ** # score of title (not originally indexed)\n",
        "    (pt.text.scorer(body_attr=\"title\", takes='docs', wmodel='BM25') ) \n",
        "    ** # date 2020\n",
        "    (pt.apply.doc_score(lambda row: int(\"2020\" in row[\"date\"])))\n",
        "    ** # has doi\n",
        "    (pt.apply.doc_score(lambda row: int( row[\"doi\"] is not None and len(row[\"doi\"]) > 0) ))\n",
        "    ** # abstract coordinate match\n",
        "    pt.BatchRetrieve(index, wmodel=\"CoordinateMatch\")\n",
        ")\n",
        "\n",
        "# for reference, lets record the feature names here too\n",
        "fnames=[\"BM25\", \"SDM\", 'coronavirus covid', 'title', \"2020\", \"hasDoi\", \"CoordinateMatch\"]"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "nu3M9ujw6trC"
      },
      "source": [
        "Lets see the output for a particular query. We can see that we now have extra document metadata columns `[\"title\", \"date\", \"doi\"]`, as well as the all-important `\"features\"` columns. This makes dataframe have type $R_f$. Indeed,  it is this column that we use for learning."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "OMG9kK9AUWI4"
      },
      "outputs": [],
      "source": [
        "ltr_feats1.search(\"coronovirus origin\")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "3vQNbSSgJCmC"
      },
      "source": [
        "We can also look at the raw features values (in this case for the first ranked document). Note that the BM25 in the \"score\" column above is also the first value in the feature array (20.54), because we used an identity transformer."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "EzVDLL8pI_gR"
      },
      "outputs": [],
      "source": [
        "ltr_feats1.search(\"coronovirus origin\").iloc[0][\"features\"]"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Od8P-N4E7QTk"
      },
      "source": [
        "## Learning \n",
        "\n",
        "In this part of the the notebook, we apply three different learning to rank techniques:\n",
        "\n",
        " - coordinate ascent from FastRank, a listwise linear technique\n",
        " - random forests from `scikit-learn`, a pointwise regression tree technique\n",
        " - LambdaMART from LightGBM, a listwise regression tree technique\n",
        "\n",
        "In each case, we take our feature pipeline, `ltr_feats1`, and we compose it (`>>`) with the learned model. We use `pt.ltr.apply_learned_model()` which knows how to deal with different learners.\n",
        "\n",
        "The full pipeline is then fitted (learned) using `.fit()`, specifying the training topics and qrels. Importantly, the preceeding stages of the pipeline (retrieval and feature calculation) are applied to the training topics in order to obtained the results, which are then passed to the learning to rank technique. LightGBM has early stopping enabled, which uses a validation topics set – similarly the validation topics are transformed into validation results.\n",
        "\n",
        "Finally, `%time` is notebook \"magic command\" which displays how long learning takes for each technique. Learning for each technique takes < 30 seconds."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "OFPZ39eSYIXV"
      },
      "outputs": [],
      "source": [
        "import fastrank\n",
        "\n",
        "train_request = fastrank.TrainRequest.coordinate_ascent()\n",
        "\n",
        "params = train_request.params\n",
        "params.init_random = True\n",
        "params.normalize = True\n",
        "params.seed = 1234567\n",
        "\n",
        "ca_pipe = ltr_feats1 >> pt.ltr.apply_learned_model(train_request, form='fastrank')\n",
        "\n",
        "%time ca_pipe.fit(train_topics, cord19.get_qrels())"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "T0SpZ13wUagq"
      },
      "outputs": [],
      "source": [
        "from sklearn.ensemble import RandomForestRegressor\n",
        "\n",
        "rf = RandomForestRegressor(n_estimators=400, verbose=1, random_state=SEED, n_jobs=2)\n",
        "\n",
        "rf_pipe = ltr_feats1 >> pt.ltr.apply_learned_model(rf)\n",
        "\n",
        "%time rf_pipe.fit(train_topics, cord19.get_qrels())"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "u0jb1jw_VdWU"
      },
      "outputs": [],
      "source": [
        "import lightgbm as lgb\n",
        "\n",
        "# this configures LightGBM as LambdaMART\n",
        "lmart_l = lgb.LGBMRanker(\n",
        "    task=\"train\",\n",
        "    silent=False,\n",
        "    min_data_in_leaf=1,\n",
        "    min_sum_hessian_in_leaf=1,\n",
        "    max_bin=255,\n",
        "    num_leaves=31,\n",
        "    objective=\"lambdarank\",\n",
        "    metric=\"ndcg\",\n",
        "    ndcg_eval_at=[10],\n",
        "    ndcg_at=[10],\n",
        "    eval_at=[10],\n",
        "    learning_rate= .1,\n",
        "    importance_type=\"gain\",\n",
        "    num_iterations=100,\n",
        "    early_stopping_rounds=5\n",
        ")\n",
        "\n",
        "lmart_x_pipe = ltr_feats1 >> pt.ltr.apply_learned_model(lmart_l, form=\"ltr\", fit_kwargs={'eval_at':[10]})\n",
        "\n",
        "%time lmart_x_pipe.fit(train_topics, cord19.get_qrels(), valid_topics, cord19.get_qrels())"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "b7rNqB7sBKFB"
      },
      "source": [
        "## Evaluation\n",
        "\n",
        "Lets now compare our ranking pipelines on our 15 topics with the BM25 baseline. In all cases, we're ranking only 10 results per query, so MAP will be lower. \n",
        "\n",
        "We'll report mean response time (`\"mrt\"`) as well as MAP, NDCG and NDCG@10 measures."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "P0EhTwejVqev"
      },
      "outputs": [],
      "source": [
        "pt.Experiment(\n",
        "    [bm25 % RANK_CUTOFF, ca_pipe, rf_pipe, lmart_x_pipe],\n",
        "    test_topics,\n",
        "    qrels, \n",
        "    names=[\"BM25\",  \"BM25 + CA(7f)\", \"BM25 + RF(7f)\", \"BM25 + LMart(7f)\"],\n",
        "    baseline=0,\n",
        "    eval_metrics=[\"map\", \"ndcg\", \"ndcg_cut_10\", \"mrt\"])"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "LmGe5rniC_Pn"
      },
      "source": [
        "Thats really interesting – all three learned models could improve NDCG@10 over BM25, but the coordinate ascent model improved the most (significantly so on all three metrics, but again on only 15 queries). Coordinate Ascent improved upto 10 queries."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "4ov0JjWGEbrT"
      },
      "source": [
        "## Analysis\n",
        "\n",
        "Lets start to analyse our learned models. Two things we could do is either a feature ablation study, or to evaluate the performance of each feature independently. To to that, we compose the feature pipeline (`ltr_feats1`) with `pt.ltr.feature_to_score(i)` for some feature number $i$."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "7nCb9U7uvszV"
      },
      "outputs": [],
      "source": [
        "pt.Experiment(\n",
        "    [ltr_feats1 >> pt.ltr.feature_to_score(i) for i in range(len(fnames))],\n",
        "    test_topics,\n",
        "    qrels, \n",
        "    names=fnames,\n",
        "    eval_metrics=[\"map\", \"ndcg\", \"ndcg_cut_10\", \"num_rel_ret\"])"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "rU7YYKFSFeke"
      },
      "source": [
        "Interesting, we observe that the 'coronavirus covid' feature achieved NDCG@10 of 0.572172. On average, this outperforms some of the regression tree-based learners. That coordinate ascent could outperform the strongest feature gives some credence to the appropriateness of such a linear learner in  environments without lots of training data."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "VjaWBEo8HA20"
      },
      "source": [
        "Next, we analyse the actual learned models. For the coordinate ascent model, we plot the feature weights (note the log-scale y-axis); while for the regression-tree based techniques, we report the feature importance."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "CdOP1b8Cjp44"
      },
      "outputs": [],
      "source": [
        "import matplotlib.pyplot as plt, numpy as np\n",
        "\n",
        "fig, (ax0, ax1, ax2) = plt.subplots(1, 3, figsize=(10, 6))\n",
        "\n",
        "ax0.bar(np.arange(len(fnames)), ca_pipe[1].model.to_dict()['Linear']['weights'])\n",
        "ax0.set_xticks(np.arange(len(fnames)))\n",
        "ax0.set_xticklabels(fnames, rotation=45, ha='right')\n",
        "ax0.set_title(\"Coordinate Ascent\\n Feature Weights\")\n",
        "ax0.set_yscale('log')\n",
        "\n",
        "ax1.bar(np.arange(len(fnames)), rf.feature_importances_)\n",
        "ax1.set_xticks(np.arange(len(fnames)))\n",
        "ax1.set_xticklabels(fnames, rotation=45, ha='right')\n",
        "ax1.set_title(\"Random Forests\\n Feature Importance\")\n",
        "\n",
        "ax2.bar(np.arange(len(fnames)), lmart_l.feature_importances_)\n",
        "ax2.set_xticks(np.arange(len(fnames)))\n",
        "ax2.set_xticklabels(fnames, rotation=45, ha='right')\n",
        "ax2.set_title(\"$\\lambda$MART\\n Feature Importance\")\n",
        "\n",
        "fig.show()\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "1HhGDiPLHO7v"
      },
      "source": [
        "## Practice Task – Concatenation\n",
        "\n",
        "Our learned model has low recall because only 10 documents are re-ranked. Lets make a small function, `append_baseline()`, that can append the BM25 baselines results to the output of the learned model. This is defined using [transformer operators](https://pyterrier.readthedocs.io/en/latest/operators.html) (`^` and `%`).\n",
        "\n",
        "As an exercise, apply `append_baseline()` to each of the learned model pipelines defined above, and report the MAP and NDCG computed on all 1000 ranked results. \n",
        "\n",
        "Which of the learned models result in significantly improved MAP and NDCG?\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "yypYHpf-nX7w"
      },
      "outputs": [],
      "source": [
        "def append_baseline(system, baseline, max_results = 1000):\n",
        "    return (system ^ baseline) % max_results"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "-BWu-iEUyV-F"
      },
      "outputs": [],
      "source": [
        "#YOUR SOLUTION GOES HERE\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "vZfkePNh2GWH"
      },
      "outputs": [],
      "source": [
        "#@title See Solution\n",
        "pt.Experiment(\n",
        "    [bm25] + [append_baseline(x, bm25) for x in [ca_pipe, rf_pipe, lmart_x_pipe]],\n",
        "    test_topics,\n",
        "    qrels, \n",
        "    names=[\"BM25\",  \"BM25 + CA(7f)\", \"BM25 + RF(7f)\", \"BM25 + LMart(7f)\"],\n",
        "    baseline=0,\n",
        "    eval_metrics=[\"map\", \"ndcg\", \"ndcg_cut_10\", \"mrt\"])"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "qLm3m_IXIIQG"
      },
      "source": [
        "Hopefully, you should see MAP improved, while NDCG@10 is preserved."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "That's it folks - join us after lunch for Part 3!"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": []
    }
  ],
  "metadata": {
    "colab": {
      "collapsed_sections": [],
      "name": "CIKM 2021 Tutorial Notebook - Part 2.ipynb",
      "provenance": [],
      "toc_visible": true
    },
    "kernelspec": {
      "display_name": "Python 3.7.11 ('pyterrier')",
      "language": "python",
      "name": "python3"
    },
    "language_info": {
      "codemirror_mode": {
        "name": "ipython",
        "version": 3
      },
      "file_extension": ".py",
      "mimetype": "text/x-python",
      "name": "python",
      "nbconvert_exporter": "python",
      "pygments_lexer": "ipython3",
      "version": "3.7.11"
    },
    "vscode": {
      "interpreter": {
        "hash": "0f0a989db2f537f9f241eb219508f09c114912537e09b1f30dabb5252b40d5e7"
      }
    }
  },
  "nbformat": 4,
  "nbformat_minor": 0
}