Data Leakage.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "9d249187",
   "metadata": {},
   "source": [
    "Data leakage can ruin our models in subdle ways. So, this is one of the most important concepts for practicing data scientists."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "830cd304",
   "metadata": {},
   "source": [
    "## What is a data leakage?"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "de7baa3b",
   "metadata": {},
   "source": [
    "Data leakage (or leakage) happens when your training data contains information about the target, but similar data will not be available when the model is used for prediction. "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0362b515",
   "metadata": {},
   "source": [
    "This leads to high performance on the training set (and possibly even the validation data), but the model will perform poorly in production.\n",
    "\n",
    "In other words, leakage causes a model to look accurate until you start making decisions with the model, and then the model becomes very inaccurate."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3d205db7",
   "metadata": {},
   "source": [
    "Data leakage can happen at any step in the project life cycle of a data science project. Two of them being,\n",
    "\n",
    "1. Target Leakage\n",
    "2. train-test contamination"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "10bcb34f",
   "metadata": {},
   "source": [
    "## 1. What is Target Leakage?"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "81188aa7",
   "metadata": {},
   "source": [
    "Target leakage occurs when your predictors include data that will not be available at the time you make predictions or some future data."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c4cc4269",
   "metadata": {},
   "source": [
    "#### How to prevent it?"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "31e6b181",
   "metadata": {},
   "source": [
    "It is important to think about target leakage in terms of the timing or chronological order that data becomes available, not merely whether a feature helps make good predictions."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "bd24fefa",
   "metadata": {},
   "source": [
    "An example will be helpful. Imagine you want to predict who will get sick with pneumonia. The top few rows of your raw data look like this:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "1fd4a56c",
   "metadata": {},
   "outputs": [],
   "source": [
    "got_pneumonia age weight male took_antibiotic_medicine...\n",
    "False 65 100 False False...\n",
    "False 72 130 True False...\n",
    "True 58 100 False True..."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9cadd72b",
   "metadata": {},
   "source": [
    "The model would see that anyone who has a value of False for took_antibiotic_medicine didn't have pneumonia. Since validation data comes from the same source as training data, the pattern will repeat itself in validation, and the model will have great validation (or cross-validation) scores.\n",
    "\n",
    "But the model will be very inaccurate when subsequently deployed in the real world, because even patients who will get pneumonia won't have received antibiotics yet when we need to make predictions about their future health."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e2915efd",
   "metadata": {},
   "source": [
    "![](dl.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "371bdb4f",
   "metadata": {},
   "source": [
    "At the time of prediction, we won't be knowing if the patient will likely to have pneumonia or not. So, \"took_antibiotic_medicine\" will not make any sense at the time of prediction as the patient has not yet been diagnoised with pneumonia yet. Or, the patient has not become yet a patient at the time of prediction for the likelihood of the disease."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1e68cc5c",
   "metadata": {},
   "source": [
    "Using \"took_antibiotic_medicine\" will be equivalent to using future data to predict the present"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e85811e9",
   "metadata": {},
   "source": [
    "## 2. Train-Test Contamination"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6662b0d4",
   "metadata": {},
   "source": [
    "Recall that validation is meant to be a measure of how the model does on data that it hasn't considered before. You can corrupt this process in subtle ways if the validation data affects the preprocessing behavior. \n",
    "\n",
    "This is sometimes called train-test contamination."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "693ee03e",
   "metadata": {},
   "source": [
    "For example, imagine you run preprocessing (like fitting an imputer for missing values) before calling train_test_split(). The end result? Your model may get good validation scores, giving you great confidence in it, but perform poorly when you deploy it to make decisions."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f7ab04f8",
   "metadata": {},
   "source": [
    "Another example would be performing scaling like StandardScalar on our dataset before splitting it. Then feature scaling will carry some information of test set into the training set or we can say possibility of data leakage"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "084fd97e",
   "metadata": {},
   "source": [
    "If your validation is based on a simple train-test split, exclude the validation data from any type of fitting, including the fitting of preprocessing steps. This is easier if you use scikit-learn pipelines. When using cross-validation, it's even more critical that you do your preprocessing inside the pipeline!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "fc7e3ec3",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Number of rows in the dataset: 1319\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>reports</th>\n",
       "      <th>age</th>\n",
       "      <th>income</th>\n",
       "      <th>share</th>\n",
       "      <th>expenditure</th>\n",
       "      <th>owner</th>\n",
       "      <th>selfemp</th>\n",
       "      <th>dependents</th>\n",
       "      <th>months</th>\n",
       "      <th>majorcards</th>\n",
       "      <th>active</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>0</td>\n",
       "      <td>37.66667</td>\n",
       "      <td>4.5200</td>\n",
       "      <td>0.033270</td>\n",
       "      <td>124.983300</td>\n",
       "      <td>True</td>\n",
       "      <td>False</td>\n",
       "      <td>3</td>\n",
       "      <td>54</td>\n",
       "      <td>1</td>\n",
       "      <td>12</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>0</td>\n",
       "      <td>33.25000</td>\n",
       "      <td>2.4200</td>\n",
       "      <td>0.005217</td>\n",
       "      <td>9.854167</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>3</td>\n",
       "      <td>34</td>\n",
       "      <td>1</td>\n",
       "      <td>13</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>0</td>\n",
       "      <td>33.66667</td>\n",
       "      <td>4.5000</td>\n",
       "      <td>0.004156</td>\n",
       "      <td>15.000000</td>\n",
       "      <td>True</td>\n",
       "      <td>False</td>\n",
       "      <td>4</td>\n",
       "      <td>58</td>\n",
       "      <td>1</td>\n",
       "      <td>5</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>0</td>\n",
       "      <td>30.50000</td>\n",
       "      <td>2.5400</td>\n",
       "      <td>0.065214</td>\n",
       "      <td>137.869200</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>0</td>\n",
       "      <td>25</td>\n",
       "      <td>1</td>\n",
       "      <td>7</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>0</td>\n",
       "      <td>32.16667</td>\n",
       "      <td>9.7867</td>\n",
       "      <td>0.067051</td>\n",
       "      <td>546.503300</td>\n",
       "      <td>True</td>\n",
       "      <td>False</td>\n",
       "      <td>2</td>\n",
       "      <td>64</td>\n",
       "      <td>1</td>\n",
       "      <td>5</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   reports       age  income     share  expenditure  owner  selfemp  \\\n",
       "0        0  37.66667  4.5200  0.033270   124.983300   True    False   \n",
       "1        0  33.25000  2.4200  0.005217     9.854167  False    False   \n",
       "2        0  33.66667  4.5000  0.004156    15.000000   True    False   \n",
       "3        0  30.50000  2.5400  0.065214   137.869200  False    False   \n",
       "4        0  32.16667  9.7867  0.067051   546.503300   True    False   \n",
       "\n",
       "   dependents  months  majorcards  active  \n",
       "0           3      54           1      12  \n",
       "1           3      34           1      13  \n",
       "2           4      58           1       5  \n",
       "3           0      25           1       7  \n",
       "4           2      64           1       5  "
      ]
     },
     "execution_count": 1,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import pandas as pd\n",
    "\n",
    "# Read the data\n",
    "data = pd.read_csv('datasets/AER_credit_card_data.csv', \n",
    "                   true_values = ['yes'], false_values = ['no'])\n",
    "\n",
    "# Select target\n",
    "y = data.card\n",
    "\n",
    "# Select predictors\n",
    "X = data.drop(['card'], axis=1)\n",
    "\n",
    "print(\"Number of rows in the dataset:\", X.shape[0])\n",
    "X.head()\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "0925a0af",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Cross-validation accuracy: 0.981052\n"
     ]
    }
   ],
   "source": [
    "# cross-validating\n",
    "\n",
    "from sklearn.pipeline import make_pipeline, Pipeline\n",
    "from sklearn.ensemble import RandomForestClassifier\n",
    "from sklearn.model_selection import cross_val_score\n",
    "\n",
    "my_pipeline = make_pipeline(RandomForestClassifier(n_estimators=100))\n",
    "\n",
    "# my_pipeline = Pipeline(\n",
    "#     steps=[\n",
    "#         ('model', RandomForestClassifier(n_estimators=100))\n",
    "#     ]\n",
    "# )\n",
    "\n",
    "cv_scores = cross_val_score(my_pipeline, X, y,\n",
    "                          cv=5,\n",
    "                          scoring='accuracy')\n",
    "\n",
    "print(\"Cross-validation accuracy: %f\" % cv_scores.mean())"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ac345228",
   "metadata": {},
   "source": [
    "you'll find that it's very rare to find models that are accurate 98% of the time. It happens, but it's uncommon enough that we should inspect the data more closely for target leakage.\n",
    "\n",
    "Here is a summary of the data, which you can also find under the data tab:"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "eba378be",
   "metadata": {},
   "source": [
    "1. card: 1 if credit card application accepted, 0 if not\n",
    "\n",
    "\n",
    "2. reports: Number of major derogatory reports\n",
    "\n",
    "\n",
    "3. age: Age n years plus twelfths of a year\n",
    "\n",
    "\n",
    "4. income: Yearly income (divided by 10,000)\n",
    "\n",
    "\n",
    "5. share: Ratio of monthly credit card expenditure to yearly income\n",
    "\n",
    "\n",
    "6. expenditure: Average monthly credit card expenditure\n",
    "\n",
    "\n",
    "7. owner: 1 if owns home, 0 if rents\n",
    "\n",
    "\n",
    "8. selfempl: 1 if self-employed, 0 if not\n",
    "\n",
    "\n",
    "9. dependents: 1 + number of dependents\n",
    "\n",
    "\n",
    "10. months: Months living at current address\n",
    "\n",
    "\n",
    "11. majorcards: Number of major credit cards held\n",
    "\n",
    "\n",
    "12. active: Number of active credit accounts"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7f1eb4c3",
   "metadata": {},
   "source": [
    "A few variables look suspicious. For example, does expenditure mean expenditure on this card or on cards used before appying?\n",
    "\n",
    "At this point, basic data comparisons can be very helpful:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "83f8546c",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Fraction of those who did not receive a card and had no expenditures: 1.00\n",
      "Fraction of those who received a card and had no expenditures: 0.02\n"
     ]
    }
   ],
   "source": [
    "expenditures_cardholders = X.expenditure[y]\n",
    "expenditures_noncardholders = X.expenditure[~y]\n",
    "\n",
    "print('Fraction of those who did not receive a card and had no expenditures: %.2f' \\\n",
    "      %((expenditures_noncardholders == 0).mean()))\n",
    "print('Fraction of those who received a card and had no expenditures: %.2f' \\\n",
    "      %(( expenditures_cardholders == 0).mean()))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b4aebad4",
   "metadata": {},
   "source": [
    "As shown above, everyone who did not receive a card had no expenditures, while only 2% of those who received a card had no expenditures. It's not surprising that our model appeared to have a high accuracy. \n",
    "\n",
    "But this also seems to be a case of target leakage, where expenditures probably means expenditures on the card they applied for."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a9538bf9",
   "metadata": {},
   "source": [
    "Since share is partially determined by expenditure, it should be excluded too. The variables active and majorcards are a little less clear, but from the description, they sound concerning. In most situations, it's better to be safe than sorry if you can't track down the people who created the data to find out more."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "6bfe23c7",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Cross-val accuracy: 0.830165\n"
     ]
    }
   ],
   "source": [
    "# Drop predictors from dataset which might lead to data leakage\n",
    "potential_leaks = ['expenditure', 'share', 'active', 'majorcards']\n",
    "X2 = X.drop(potential_leaks, axis=1)\n",
    "\n",
    "# Evaluate the model without suspicious predictors \n",
    "cv_scores = cross_val_score(my_pipeline, X2, y, \n",
    "                            cv=5,\n",
    "                            scoring='accuracy')\n",
    "\n",
    "print(\"Cross-val accuracy: %f\" % cv_scores.mean())"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "bb1bf3aa",
   "metadata": {},
   "source": [
    "We can find that the accuracy is very much lower than before which might seem disappointing. But we can expect it to be right about 80% of the time when used on new applications. \n",
    "\n",
    "Whereas, the previous model might do much worse than that on new data. (Inspite of high cross-validation scores)."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2d2bea7a",
   "metadata": {},
   "source": [
    "## Points to keep in mind"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "70c69d63",
   "metadata": {},
   "source": [
    "1. Data Leakage can be a huge mistake in many data science applications.\n",
    "\n",
    "\n",
    "2. Separation of training and validation data carefully before doing any pre-processing on the data can prevent train-test contamination. (Pipelines can help in implementing this separation).\n",
    "    \n",
    "    \n",
    "    2.1. Always fit on training data and use the frozen parameter to transform the test data\n",
    "    \n",
    "    \n",
    "    2.2. Don't fit on test data.\n",
    "\n",
    "\n",
    "3. Caution, common sense, domain expertise and EDA can help identify target leakage."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "fa85c5d8",
   "metadata": {},
   "source": [
    "## Examples"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "00f2a2e5",
   "metadata": {},
   "source": [
    "### The Data Science of Shoelaces\n",
    "\n",
    "Nike has hired you as a data science consultant to help them save money on shoe materials. Your first assignment is to review a model one of their employees built to predict how many shoelaces they'll need each month. The features going into the machine learning model include:\n",
    "\n",
    "\n",
    "1. The current month (January, February, etc)\n",
    "\n",
    "\n",
    "2. Advertising expenditures in the previous month\n",
    "\n",
    "\n",
    "3. Various macroeconomic features (like the unemployment rate) as of the beginning of the current month\n",
    "\n",
    "\n",
    "4. The amount of leather they ended up using in the current month\n",
    "\n",
    "\n",
    "The results show the model is almost perfectly accurate if you include the feature about how much leather they used. But it is only moderately accurate if you leave that feature out. You realize this is because the amount of leather they use is a perfect indicator of how many shoes they produce, which in turn tells you how many shoelaces they need.\n",
    "\n",
    "\n",
    "Do you think the leather used feature constitutes a source of data leakage? If your answer is \"it depends,\" what does it depend on?\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7db04264",
   "metadata": {},
   "source": [
    "Answer:\n",
    "\n",
    "This is tricky, and it depends on details of how data is collected (which is common when thinking about leakage). Would you at the beginning of the month decide how much leather will be used that month? If so, this is ok. But if that is determined during the month, you would not have access to it when you make the prediction. If you have a guess at the beginning of the month, and it is subsequently changed during the month, the actual amount used during the month cannot be used as a feature (because it causes leakage)."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7eef41a0",
   "metadata": {},
   "source": [
    "### Return of the Shoelaces\n",
    "\n",
    "You have a new idea. You could use the amount of leather Nike ordered (rather than the amount they actually used) leading up to a given month as a predictor in your shoelace model.\n",
    "\n",
    "\n",
    "Does this change your answer about whether there is a leakage problem? If you answer \"it depends,\" what does it depend on?"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "dd2bef85",
   "metadata": {},
   "source": [
    "Answer:\n",
    "\n",
    "This could be fine, but it depends on whether they order shoelaces first or leather first. If they order shoelaces first, you won't know how much leather they've ordered when you predict their shoelace needs. If they order leather first, then you'll have that number available when you place your shoelace order, and you should be ok."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a59ad44c",
   "metadata": {},
   "source": [
    "### Getting Rich With Cryptocurrencies?\n",
    "\n",
    "You saved Nike so much money that they gave you a bonus. Congratulations.\n",
    "\n",
    "\n",
    "Your friend, who is also a data scientist, says he has built a model that will let you turn your bonus into millions of dollars. Specifically, his model predicts the price of a new cryptocurrency (like Bitcoin, but a newer one) one day ahead of the moment of prediction. His plan is to purchase the cryptocurrency whenever the model says the price of the currency (in dollars) is about to go up.\n",
    "\n",
    "\n",
    "The most important features in his model are:\n",
    "\n",
    "\n",
    "1. Current price of the currency\n",
    "2. Amount of the currency sold in the last 24 hours\n",
    "3. Change in the currency price in the last 24 hours\n",
    "4. Change in the currency price in the last 1 hour\n",
    "5. Number of new tweets in the last 24 hours that mention the currency\n",
    "\n",
    "The value of the cryptocurrency in dollars has fluctuated up and down by over 100 dollars in the last year, and yet his model's average error is less than 1 dollar. He says this is proof his model is accurate, and you should invest with him, buying the currency whenever the model says it is about to go up.\n",
    "\n",
    "\n",
    "Is he right? If there is a problem with his model, what is it?"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b569bd34",
   "metadata": {},
   "source": [
    "Answer:\n",
    "\n",
    "There is no source of leakage here. These features should be available at the moment you want to make a predition, and they're unlikely to be changed in the training data after the prediction target is determined. But, the way he describes accuracy could be misleading if you aren't careful. If the price moves gradually, today's price will be an accurate predictor of tomorrow's price, but it may not tell you whether it's a good time to invest. For instance, if it is  100 dollars today, a price of 100 dollars tomorrow may seem accurate, even if it can't tell you whether the price is going up or down from the current price. A better prediction target would be the change in price over the next day. If you can consistently predict whether the price is about to go up or down (and by how much), you may have a winning investment opportunity.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8f77bcee",
   "metadata": {},
   "source": [
    "### Preventing Infections\n",
    "\n",
    "An agency that provides healthcare wants to predict which patients from a rare surgery are at risk of infection, so it can alert the nurses to be especially careful when following up with those patients.\n",
    "\n",
    "\n",
    "You want to build a model. Each row in the modeling dataset will be a single patient who received the surgery, and the prediction target will be whether they got an infection.\n",
    "\n",
    "\n",
    "Some surgeons may do the procedure in a manner that raises or lowers the risk of infection. But how can you best incorporate the surgeon information into the model?\n",
    "\n",
    "\n",
    "You have a clever idea.\n",
    "\n",
    "\n",
    "Take all surgeries by each surgeon and calculate the infection rate among those surgeons.\n",
    "For each patient in the data, find out who the surgeon was and plug in that surgeon's average infection rate as a feature.\n",
    "Does this pose any target leakage issues? Does it pose any train-test contamination issues?"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "06f82330",
   "metadata": {},
   "source": [
    "Answer:\n",
    "    \n",
    "This poses a risk of both target leakage and train-test contamination (though you may be able to avoid both if you are careful).\n",
    "\n",
    "\n",
    "You have target leakage if a given patient's outcome contributes to the infection rate for his surgeon, which is then plugged back into the prediction model for whether that patient becomes infected. You can avoid target leakage if you calculate the surgeon's infection rate by using only the surgeries before the patient we are predicting for. Calculating this for each surgery in your training data may be a little tricky.\n",
    "\n",
    "\n",
    "You also have a train-test contamination problem if you calculate this using all surgeries a surgeon performed, including those from the test-set. The result would be that your model could look very accurate on the test set, even if it wouldn't generalize well to new patients after the model is deployed. This would happen because the surgeon-risk feature accounts for data in the test set. Test sets exist to estimate how the model will do when seeing new data. So this contamination defeats the purpose of the test set."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "575c8301",
   "metadata": {},
   "source": [
    "### Housing Prices\n",
    "\n",
    "You will build a model to predict housing prices. The model will be deployed on an ongoing basis, to predict the price of a new house when a description is added to a website. Here are four features that could be used as predictors.\n",
    "\n",
    "\n",
    "1. Size of the house (in square meters)\n",
    "\n",
    "2. Average sales price of homes in the same neighborhood\n",
    "\n",
    "3. Latitude and longitude of the house\n",
    "\n",
    "4. Whether the house has a basement\n",
    "\n",
    "You have historic data to train and validate the model.\n",
    "\n",
    "\n",
    "Which of the features is most likely to be a source of leakage?\n",
    "\n",
    "[Hint](): Which of these features might be updated in a database after the house is sold? That's the one to worry about."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3db2a011",
   "metadata": {},
   "source": [
    "Answer:\n",
    "\n",
    "2 is the source of target leakage. Here is an analysis for each feature:\n",
    "\n",
    "\n",
    "1. The size of a house is unlikely to be changed after it is sold (though technically it's possible). But typically this will be available when we need to make a prediction, and the data won't be modified after the home is sold. So it is pretty safe.\n",
    "\n",
    "\n",
    "2. We don't know the rules for when this is updated. If the field is updated in the raw data after a home was sold, and the home's sale is used to calculate the average, this constitutes a case of target leakage. At an extreme, if only one home is sold in the neighborhood, and it is the home we are trying to predict, then the average will be exactly equal to the value we are trying to predict. In general, for neighborhoods with few sales, the model will perform very well on the training data. But when you apply the model, the home you are predicting won't have been sold yet, so this feature won't work the same as it did in the training data.\n",
    "\n",
    "\n",
    "3. These don't change, and will be available at the time we want to make a prediction. So there's no risk of target leakage here.\n",
    "\n",
    "\n",
    "4. This also doesn't change, and it is available at the time we want to make a prediction. So there's no risk of target leakage here."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "a7e3e4ca",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "id": "44d426b4",
   "metadata": {},
   "source": [
    "Leakage is a hard and subtle issue. You should be proud if you picked up on the issues in these examples.\n",
    "\n",
    "Now you have the tools to make highly accurate models, and pick up on the most difficult practical problems that arise with applying these models to solve real problems."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "e6bbd788",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "id": "89156e10",
   "metadata": {},
   "source": [
    "#### Author: Piyush Kumar\n",
    "[Github](https://github.com/styles3544)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "45e06047",
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}