Ethical AI Lab 1 - Exercise.ipynb

{
 "cells": [
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# COMP0195: Accountable, Transparent, and Responsible Artificial Intelligence (22/23) - Lab 1"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Learning Goals\n",
    "\n",
    "##### - Finding the distribution of features in datasets with exploratory analysis\n",
    "##### - Identifying potential bias by documenting the differences between privileged and under-privileged groups\n",
    "##### - Finding the prevelence between features and target variables \n",
    "##### - Training a Machine Learning model \n",
    "##### - Evaluating a Machine Learning model with respect to its performance with privileged and under-privileged groups in       dataset\n",
    "##### - Confirming bias in the dataset and identifying the bias type\n"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### In this lab we will be workng with the UCI Adult Dataset. Please download the dataset from here (https://archive.ics.uci.edu/ml/datasets/adult) and save it as a csv."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 65,
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import pandas as pd\n",
    "import matplotlib.pyplot as plt\n",
    "\n",
    "from sklearn import svm\n",
    "from sklearn.ensemble import RandomForestClassifier\n",
    "from sklearn.metrics import accuracy_score,confusion_matrix\n",
    "from sklearn.metrics import classification_report\n",
    "from sklearn.feature_selection import mutual_info_classif"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "names = ['age','workclass','fnlwgt','education','education-num',\n",
    "        'marital-status','occupation','relationship','race','sex',\n",
    "        'capital-gain','capital-loss','hours-per-week','native-country',\n",
    "        'y']\n",
    "\n",
    "#Load dataset\n",
    "df = pd.read_csv(r\"C:\\Users\\alich\\OneDrive\\Documents\\UCL CS\\Ethical AI\\adult.csv\",names=names,na_values='?')\n",
    "df = df.dropna()"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Data Analysis"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### We will first do some exploratory data analysis to get a better understadning of the distribution of sensitive variables like race and gender that can make the dataset biased."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#TASK 1\n",
    "#Get the popluation count by race\n",
    "\n",
    "################\n",
    "\n",
    "counts = \n",
    "\n",
    "################\n",
    "\n",
    "labels = counts.index\n",
    "\n",
    "#Plot pie chart\n",
    "plt.pie(counts, startangle=90)\n",
    "plt.legend(labels, loc=2,fontsize=8)\n",
    "plt.title(\"Race\",size=20)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#TASK 2\n",
    "\n",
    "#Get the popluation count by Sex\n",
    "\n",
    "################\n",
    "\n",
    "counts =\n",
    "\n",
    "################\n",
    "\n",
    "labels = counts.index\n",
    "\n",
    "#Plot pie chart\n",
    "plt.pie(counts, startangle=90)\n",
    "plt.legend(labels, loc=2,fontsize=8)\n",
    "plt.title(\"Sex\",size=20)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Question: What do you notice in the distribution of different values of race and gender?"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Defining protected features\n"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##### Before we move on, we will need to define our protected features. We do this by creating binary variables using the sensitive attributes. We define the variable so that 1 represents a privileged group and 0 represents an unprivileged group."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# We define these features using the code below. For race, we define \n",
    "# the protected feature so that “White” is the privileged group. \n",
    "# That is the variable has a value of 1 if the person is white and \n",
    "# 0 otherwise. For sex, “Male” is the privileged group. \n",
    "\n",
    "df_fair = df[['race','sex']]\n",
    "\n",
    "#Define protected features \n",
    "df_fair['priv_race'] = [1 if r==' White' else 0 for r in df_fair['race']]\n",
    "df_fair['priv_sex'] = [1 if s==' Male' else 0 for s in df_fair['sex']]\n",
    "\n",
    "#Define target variable \n",
    "df_fair['y'] =  [1 if y == ' >50K'else 0 for y in df['y']]"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Bias and Prevalence"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### For a target variable, prevalence is the proportion of the positive cases to overall cases. Where a positive case is when the target variable has a value of 1. Our dataset has an overall prevalence of 24.8%. That is roughly 1/4 of the people in our dataset earn above $50K. We can also use prevalence as an indicator of bias in our dataset. Ideally, in datasets with no bias, prevalence of all features, irrespective of their values should be the same."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# TASK 3\n",
    "\n",
    "#Calculate the prevelance\n",
    "#Look into the groupby operation for pandas\n",
    "\n",
    "####\n",
    "#Calculate the prevelance of target variable of y\n",
    "prev = \n",
    "print(prev)\n",
    "\n",
    "####\n",
    "#Calculate the prevelance of priv_race with respect to y\n",
    "prev_race = \n",
    "print(prev_race)\n",
    "\n",
    "####\n",
    "#Calculate the prevelance of priv_Sex with respect to y\n",
    "prev_sex = \n",
    "print(prev_sex)\n",
    "\n",
    "####\n",
    "#Calculate the combined prevelance of priv_race and priv_sex with respect to y\n",
    "prev_comb = \n",
    "print(prev_comb)\n",
    "\n",
    "\n"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Question: What do you notice about the difference in prevelence between privileged and underpriveleged race and sex?"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### At this point, you should be asking yourself why we have these large differences in prevalence. The dataset was built using United States census data from 1994. The country has a history of discrimination based on gender and race. Ultimately, the target variable reflects this discrimination. In this sense, prevalence can be used to understand the extent to which historical injustice is embedded in our target variable."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#Target variable \n",
    "y = [1 if y == ' >50K' else 0 for y in df['y']]\n",
    "\n",
    "#Model features\n",
    "X = df[['age','education-num','hours-per-week']]\n",
    "X['marital-status'] = [1 if x==' Married-civ-spouse' else 0 for x in df['marital-status']] \n",
    "X['native-country'] = [1 if x==' United-States' else 0 for x in df['native-country']] \n",
    "\n",
    "occ_groups = {\n",
    "    ' Priv-house-serv':0,' Other-service':0,' Handlers-cleaners':0,\n",
    "    ' Farming-fishing':1,' Machine-op-inspct':1,' Adm-clerical':1,\n",
    "    ' Transport-moving':2,' Craft-repair':2,' Sales':2, ' ?': -1,\n",
    "    ' Armed-Forces':3,' Tech-support':3,' Protective-serv':3,\n",
    "    ' Prof-specialty':4,' Exec-managerial':4}\n",
    "\n",
    "X['occupation'] = [occ_groups[x] for x in df['occupation']]\n",
    "\n",
    "X.head()"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Fairness Metrics\n",
    "\n",
    "#### Different metrics can be used to evaluate the fairness of an AI tool. These include equality of opportunity, statistical parity, predictive parity and calibration. You can find more details about these fairness metrics in this paper from Pratyush and Vilasenor (2020) https://arxiv.org/pdf/2001.07864.pdf. \n",
    "\n",
    "#### In this exercise we will be using equality of opportunity as a metric to evaluate the fairness of ML algorithms. You will train two algorithms (Random Forest and SVM) and evaluate/compare their accuracies acorss different groups. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "\n",
    "#TASK 4\n",
    "#Training a Random forest Classifier\n",
    "\n",
    "model = \n",
    "model.fit(X, y)\n",
    "\n",
    "#Get predictions\n",
    "y_pred = model.predict(X)\n",
    "df_fair['y_pred'] = y_pred\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 81,
   "metadata": {},
   "outputs": [],
   "source": [
    "\n",
    "def fairness_metrics(df):\n",
    "    \n",
    "# TASK 5\n",
    "# Make a Confusion Matrix of real values and predicted values\n",
    "    cm = \n",
    "    TN, FP, FN, TP = cm.ravel()\n",
    "    \n",
    "    N = TP+FP+FN+TN #Total population\n",
    "    ACC = (TP+TN)/N #Accuracy\n",
    "    \n",
    "    return np.array([ACC])  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "\n",
    "\n",
    "#TASK 6\n",
    "#Calculate the fairness metrics for privileged and under-privileed group in race \n",
    "fm_race_1 =\n",
    "fm_race_0 = \n",
    "\n",
    "#Get ratio of fairness metrics\n",
    "fm_race = fm_race_0/fm_race_1\n",
    "\n",
    "print(fm_race_0)\n",
    "print(fm_race_1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#TASK 7\n",
    "\n",
    "#Calculate the fairness metrics for privileged and under-privileed group in sex \n",
    "fm_sex_1 = \n",
    "fm_sex_0 = "
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Question: Do you notice any differences in accuracy between the privileged and under-privileged race and gender in the Random Forest algorithm? What kind of bias do you think this would be from the (Baker and Hawn, 2021), and why? \n",
    "\n",
    "##### Baker and Hawn (2021): https://edarxiv.org/pbmvz/"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#TASK 8\n",
    "#Training a Support Vector Machine Model\n",
    "\n",
    "model1 = \n",
    "model1.fit(X, y)\n",
    "\n",
    "#Get predictions\n",
    "y_pred_svm = model1.predict(X)\n",
    "df_fair['y_pred_svm'] = y_pred_svm"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def fairness_metrics_2(df):\n",
    "    \"\"\"Calculate fairness for subgroup of population\"\"\"\n",
    "    \n",
    "# Make a Confusion Matrix of real values and target values\n",
    "    cm=confusion_matrix(df['y'],df['y_pred_svm'])\n",
    "    TN, FP, FN, TP = cm.ravel()\n",
    "    \n",
    "    N = TP+FP+FN+TN #Total population\n",
    "    ACC = (TP+TN)/N #Accuracy\n",
    "    \n",
    "    return np.array([ACC])  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#TASK 9\n",
    "#Calculate the fairness metrics for privileged and under-privileed group in race for SVM\n",
    "fm_race_1_svm = fairness_metrics_2(df_fair[df_fair.priv_race==1])\n",
    "fm_race_0_svm = fairness_metrics_2(df_fair[df_fair.priv_race==0])\n",
    "\n",
    "#Get ratio of fairness metrics\n",
    "fm_race_svm = fm_race_0_svm/fm_race_1_svm\n",
    "\n",
    "print(fm_race_0_svm)\n",
    "print(fm_race_1_svm)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#TASK 10\n",
    "#Calculate the fairness metrics for privileged and under-privileed group in sex for SVM\n",
    "fm_sex_1_svm = fairness_metrics_2(df_fair[df_fair.priv_sex==1])\n",
    "fm_sex_0_svm = fairness_metrics_2(df_fair[df_fair.priv_sex==0])\n",
    "\n",
    "#Get ratio of fairness metrics\n",
    "fm_sex_svm = fm_sex_0_svm/fm_sex_1_svm\n",
    "\n",
    "print(fm_sex_0_svm)\n",
    "print(fm_sex_1_svm)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Question: What do you notice as the difference in fairness between SVM and Random Forest Algorithm? "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "base",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.12"
  },
  "orig_nbformat": 4,
  "vscode": {
   "interpreter": {
    "hash": "bb2560baf818c11029a225c330940475feae786f05463d7ccf2e31c29c63e44b"
   }
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}