logistic_reg_demo.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 机器学习实现之模型评估二"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "目录\n",
    "\n",
    "逻辑回归模型\n",
    "皮马印第安人糖尿病数据集\n",
    "使用精确率进行模型评估的局限性\n",
    "混淆矩阵、模型衡量指标及其意义"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##皮马印第安人糖尿病数据集\n",
    "\n",
    "基于数据集中包括的某些诊断测量来诊断性的预测患者是否患有糖尿病\n",
    "输入的特征有：患者的怀孕次数，葡萄糖量，血压，皮褶厚度，胰岛素水平，糖尿病谱系功能，年龄等\n",
    "输出结果：是否患有糖尿病\n",
    "数据来源：https://www.kaggle.com/uciml/pima-indians-diabetes-database"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "任务：选取怀孕次数，胰岛素水平，体重指数，年龄这四个特征对是否患有糖尿病进行预测"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Pregnancies</th>\n",
       "      <th>Glucose</th>\n",
       "      <th>BloodPressure</th>\n",
       "      <th>SkinThickness</th>\n",
       "      <th>Insulin</th>\n",
       "      <th>BMI</th>\n",
       "      <th>DiabetesPedigreeFunction</th>\n",
       "      <th>Age</th>\n",
       "      <th>Outcome</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>6</td>\n",
       "      <td>148</td>\n",
       "      <td>72</td>\n",
       "      <td>35</td>\n",
       "      <td>0</td>\n",
       "      <td>33.6</td>\n",
       "      <td>0.627</td>\n",
       "      <td>50</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>1</td>\n",
       "      <td>85</td>\n",
       "      <td>66</td>\n",
       "      <td>29</td>\n",
       "      <td>0</td>\n",
       "      <td>26.6</td>\n",
       "      <td>0.351</td>\n",
       "      <td>31</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>8</td>\n",
       "      <td>183</td>\n",
       "      <td>64</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>23.3</td>\n",
       "      <td>0.672</td>\n",
       "      <td>32</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>1</td>\n",
       "      <td>89</td>\n",
       "      <td>66</td>\n",
       "      <td>23</td>\n",
       "      <td>94</td>\n",
       "      <td>28.1</td>\n",
       "      <td>0.167</td>\n",
       "      <td>21</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>0</td>\n",
       "      <td>137</td>\n",
       "      <td>40</td>\n",
       "      <td>35</td>\n",
       "      <td>168</td>\n",
       "      <td>43.1</td>\n",
       "      <td>2.288</td>\n",
       "      <td>33</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   Pregnancies  Glucose  BloodPressure  SkinThickness  Insulin   BMI  \\\n",
       "0            6      148             72             35        0  33.6   \n",
       "1            1       85             66             29        0  26.6   \n",
       "2            8      183             64              0        0  23.3   \n",
       "3            1       89             66             23       94  28.1   \n",
       "4            0      137             40             35      168  43.1   \n",
       "\n",
       "   DiabetesPedigreeFunction  Age  Outcome  \n",
       "0                     0.627   50        1  \n",
       "1                     0.351   31        0  \n",
       "2                     0.672   32        1  \n",
       "3                     0.167   21        0  \n",
       "4                     2.288   33        1  "
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# 数据预处理\n",
    "\n",
    "import pandas as pd\n",
    "path = 'dataset/pima_data.csv'\n",
    "pima = pd.read_csv(path)\n",
    "pima.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [],
   "source": [
    "# X, y的赋值\n",
    "feature_names = ['Pregnancies','Insulin','BMI','Age']\n",
    "X = pima[feature_names]\n",
    "y = pima.Outcome"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(768, 4)\n",
      "(768,)\n"
     ]
    }
   ],
   "source": [
    "# 维度的确认\n",
    "print(X.shape)\n",
    "print(y.shape)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [],
   "source": [
    "#数据分离\n",
    "from sklearn.model_selection import train_test_split\n",
    "X_train, X_test, y_train, y_test = train_test_split(X, y, random_state = 0)#test_size(test/train)默认为0.25"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "G:\\anaconda3\\envs\\sklearn\\lib\\site-packages\\sklearn\\linear_model\\logistic.py:432: FutureWarning: Default solver will be changed to 'lbfgs' in 0.22. Specify a solver to silence this warning.\n",
      "  FutureWarning)\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "LogisticRegression(C=1.0, class_weight=None, dual=False, fit_intercept=True,\n",
       "                   intercept_scaling=1, l1_ratio=None, max_iter=100,\n",
       "                   multi_class='warn', n_jobs=None, penalty='l2',\n",
       "                   random_state=None, solver='warn', tol=0.0001, verbose=0,\n",
       "                   warm_start=False)"
      ]
     },
     "execution_count": 22,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# 模型训练\n",
    "from sklearn.linear_model import LogisticRegression\n",
    "logreg = LogisticRegression()\n",
    "logreg.fit(X_train , y_train)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [],
   "source": [
    "# 基于测试数据集的结果预测\n",
    "y_pred = logreg.predict(X_test)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.6927083333333334\n"
     ]
    }
   ],
   "source": [
    "# 使用准确率进行评估\n",
    "from sklearn import metrics\n",
    "print(metrics.accuracy_score(y_test,y_pred))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "预测准确率的局限性：\n",
    "无法真实反映模型针对各个分类的预测准确度\n",
    "准确率可以方便的用于衡量模型的整体预测效果，但无法反应细节信息，具体表现：\n",
    "没有体现数据的实际分布情况\n",
    "没有体现模型错误预测的类型\n",
    "空准确率：当模型总是预测比例较高的类别，其预测准确率的数值"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0    130\n",
       "1     62\n",
       "Name: Outcome, dtype: int64"
      ]
     },
     "execution_count": 34,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# 确认正负样本数量\n",
    "y_test.value_counts()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "所以可见其中负样本的比例较高，高达0.67  由此可见即使全部直接预测为负样本即可达到0.67的准确率，所以前面的百分之69的准确率局限性较大"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[[118  12]\n",
      " [ 47  15]]\n",
      "118 12 47 15\n"
     ]
    }
   ],
   "source": [
    "# 计算并展示混淆矩阵\n",
    "confusion = metrics.confusion_matrix(y_test,y_pred)\n",
    "print(confusion)\n",
    "\n",
    "#四个因子赋值\n",
    "TN = confusion[0,0]\n",
    "FP = confusion[0,1]\n",
    "FN = confusion[1,0]\n",
    "TP = confusion[1,1]\n",
    "print(TN,FP,FN,TP)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "true: [1 0 0 1 0 0 1 1 0 0 1 1 0 0 0 0 1 0 0 0 1 1 0 0 0]\n",
      "pred: [0 0 0 0 0 0 0 1 0 1 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0]\n"
     ]
    }
   ],
   "source": [
    "# 部分预测结果与实际情况对比展示\n",
    "print(\"true:\",y_test.values[0:25])\n",
    "print(\"pred:\",y_pred[0:25])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 混淆矩阵指标计算"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "准确率：整体样本中，预测正确样本数的比例\n",
    "Accuracy = （TP + TN）/(TP + TN + FP + FN)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.6927083333333334\n"
     ]
    }
   ],
   "source": [
    "accuracy = (TP + TN)/(TP + TN + FP + FN)\n",
    "print(accuracy)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "错误率：整体样本中，预测错误的比例\n",
    "Misclassification Rate = （FP + FN）/(TP + TN + FP + FN)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.3072916666666667\n"
     ]
    }
   ],
   "source": [
    "mis_rate = (FP + FN)/(TP + TN + FP + FN)\n",
    "print(mis_rate)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "召回率：正样本中，预测正确的比例\n",
    "recall = TP/(TP + FN)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.24193548387096775\n"
     ]
    }
   ],
   "source": [
    "recall = TP/(TP + FN)\n",
    "print(recall)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "特异度：负样本中，预测正确的比例\n",
    "Specificity = TN/(TN + FP)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.9076923076923077\n"
     ]
    }
   ],
   "source": [
    "specificity = TN/(TN + FP)\n",
    "print(specificity)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "精确率：预测结果为正样本中，预测正确的比例\n",
    "Precision = TP/(TP + FP)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.5555555555555556\n"
     ]
    }
   ],
   "source": [
    "precision = TP/(TP + FP)\n",
    "print(precision)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "F1分数：综合Precision和Recall的判断指标"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.3370786516853933\n"
     ]
    }
   ],
   "source": [
    "f1_score = 2 * precision * recall/(precision + recall)\n",
    "print(f1_score)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "结论：\n",
    "\n",
    "分类任务中，相比单一的预测准确率，混淆矩阵提供了更全面的模型评估信息\n",
    "通过混淆矩阵，我们可以计算出多样性的模型表现衡量指标，从而更好地选择模型\n",
    "\n",
    "\n",
    "哪个衡量指标更关键？\n",
    "\n",
    "衡量指标的选择取决于应用场景\n",
    "垃圾邮件检测（正样本判断为“垃圾邮件”）：希望普通邮件（负样本）不要被判断为垃圾邮件（正样本），需要关注精确率和召回率\n",
    "异常交易检测（正样本为“异常交易”）：希望所有的异常交易都被检测到，需要关注特异度"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.4"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}