diff --git a/temp/model+test.html b/temp/model+test.html new file mode 100644 index 0000000..d8f0e41 --- /dev/null +++ b/temp/model+test.html @@ -0,0 +1,15899 @@ + + + + + +model+test + + + + + + + + + + + + + + + + + + + + + + +
+
+ +
+ +
+
+
+

Summary of the data set

The data set used in this project is the results of a chemical analysis of the Portuguese "Vinho Verde" wine, conducted by Paulo Cortez, University of Minho, Guimarães, +Portugal A. Cerdeira, F. Almeida, T. +Matos and J. Reis, Viticulture Commission of the Vinho Verde Region(CVRVV), Porto, Portugal @2009. It was sourced from the UCI Machine +Learning Repository.

+

There are two datasets for red and white wine samples. For each wine sample observation , the inputs contains measurements of various objective physicochemical tests, and the output is the median wine quality ratings given by experts on the scale from 0 (very bad) and 10 (very excellent).The author notes that data on grape types, wine brand, wind selling price among other are not available due to privacy and logistics issues. There are 1599 observations for red wine and 4898 observations of white wine.

+ +
+
+
+
+

Data Import

+
+
+
+ +
+ +
+
+
+

After importing the downloaded data, the below tables show the summary statistics of all numeric features in the white wine data set.

+ +
+
+
+ +
+ +
+ + + + +
+ +
+
+
+

Similar table for red wine data set

+ +
+
+
+ +
+ +
+ + + + +
+ +
+
+
+

Base on the brief summary of the data above, there is no missing value, all the features have numeric values, hence there is no major preprocessing needed. We decide to combine the two data sets of red wine and white wine to consider wine type (i.e. red or wine) as another possible features that could link to wine quality. Below is the combined data set.

+ +
+
+
+ +
+ +
+ + + + +
+ +
+
+ +
+ +
+
+ +
+ +
+
+ +
+ +
+ + + + +
+ +
+
+ +
+ +
+ + + + +
+ +
+
+
+

PreProcessor

+
+
+
+ +
+ +
+
+ +
+ +
+
+
+ + +

Model Selection

+
+
+
+
+

Stacking Classifier

+
+
+
+ +
+ +
+
+ +
+ +
+
+
+

ALL other classifiers

+
+
+
+ +
+ +
+ + + + +
+ +
+
+
+

Plotting results

All classifiers

+
+
+
+ +
+ +
+ + + + +
+ +
+
+
+

Plotting stability across cv folds

+
+
+
+ +
+ +
+
+ +
+ +
+ + + + +
+ +
+
+ +
+ +
+
+ +
+ +
+ + + + +
+ +
+
+
+

Here, we can observe that Random forests are more consistent across CV folds

+
+
+
+
+

HyperParameter Optimization

+
+
+
+ +
+ +
+
+ +
+ +
+
+ +
+ +
+ + + + +
+ +
+
+
+

Interpreting Our model

+
+
+
+ +
+ +
+ + + + +
+ +
+
+ +
+ +
+ + + + +
+ +
+ + + + + + + + + diff --git a/temp/model+test.ipynb b/temp/model+test.ipynb new file mode 100644 index 0000000..29151a2 --- /dev/null +++ b/temp/model+test.ipynb @@ -0,0 +1,1602 @@ +{ + "cells": [ + { + "cell_type": "code", + "execution_count": 185, + "metadata": {}, + "outputs": [], + "source": [ + "import os\n", + "\n", + "%matplotlib inline\n", + "import string\n", + "from collections import deque\n", + "\n", + "import matplotlib.pyplot as plt\n", + "import numpy as np\n", + "import pandas as pd\n", + "import altair as alt\n", + "\n", + "# data\n", + "from sklearn import datasets\n", + "from sklearn.compose import ColumnTransformer, make_column_transformer\n", + "from sklearn.dummy import DummyClassifier, DummyRegressor\n", + "from sklearn.ensemble import RandomForestClassifier, RandomForestRegressor\n", + "from sklearn.feature_extraction.text import CountVectorizer\n", + "\n", + "# Feature selection\n", + "from sklearn.feature_selection import RFE, RFECV\n", + "from sklearn.impute import SimpleImputer\n", + "\n", + "# classifiers / models\n", + "from sklearn.linear_model import RidgeClassifier\n", + "from sklearn.linear_model import LogisticRegression\n", + "\n", + "# other\n", + "from sklearn.metrics import accuracy_score, log_loss, make_scorer, mean_squared_error\n", + "from sklearn.model_selection import (\n", + " GridSearchCV,\n", + " RandomizedSearchCV,\n", + " ShuffleSplit,\n", + " cross_val_score,\n", + " cross_validate,\n", + " train_test_split,\n", + ")\n", + "from sklearn.pipeline import Pipeline, make_pipeline\n", + "from sklearn.preprocessing import (\n", + " OneHotEncoder,\n", + " OrdinalEncoder,\n", + " PolynomialFeatures,\n", + " StandardScaler,\n", + ")\n", + "from sklearn.neighbors import KNeighborsClassifier\n", + "from sklearn.neural_network import MLPClassifier\n", + "from sklearn.neighbors import NearestCentroid\n", + "from sklearn.discriminant_analysis import QuadraticDiscriminantAnalysis\n", + "\n", + "from sklearn.model_selection import cross_val_predict\n", + "from sklearn.metrics import plot_precision_recall_curve, plot_roc_curve\n", + "from sklearn.metrics import accuracy_score, log_loss, make_scorer, mean_squared_error, confusion_matrix\n", + "\n", + "\n", + "#New import statements to add\n", + "from sklearn.svm import SVC\n", + "from sklearn.ensemble import VotingClassifier\n", + "from catboost import CatBoostClassifier\n", + "from sklearn.metrics import (plot_confusion_matrix)" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## Summary of the data set\n", + "The data set used in this project is the results of a chemical analysis of the Portuguese \"Vinho Verde\" wine, conducted by [Paulo Cortez, University of Minho, Guimarães,\n", + "Portugal](http://www3.dsi.uminho.pt/pcortez) A. Cerdeira, F. Almeida, T.\n", + "Matos and J. Reis, Viticulture Commission of the Vinho Verde Region(CVRVV), Porto, Portugal @2009. It was sourced from the [UCI Machine\n", + "Learning Repository](https://archive.ics.uci.edu/ml/datasets/wine+quality).\n", + "\n", + "There are two datasets for red and white wine samples. For each wine sample observation , the inputs contains measurements of various objective physicochemical tests, and the output is the median wine quality ratings given by experts on the scale from 0 (very bad) and 10 (very excellent).The author notes that data on grape types, wine brand, wind selling price among other are not available due to privacy and logistics issues. There are 1599 observations for red wine and 4898 observations of white wine." + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "# Data Import" + ] + }, + { + "cell_type": "code", + "execution_count": 71, + "metadata": { + "jupyter": { + "source_hidden": true + } + }, + "outputs": [], + "source": [ + "red_wine = pd.read_csv('data/raw/winequality-red.csv')\n", + "white_wine = pd.read_csv('data/raw/winequality-white.csv')" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "jupyter": { + "source_hidden": true + } + }, + "source": [ + "After importing the downloaded data, the below tables show the summary statistics of all numeric features in the white wine data set. " + ] + }, + { + "cell_type": "code", + "execution_count": 72, + "metadata": { + "collapsed": true, + "jupyter": { + "outputs_hidden": true, + "source_hidden": true + } + }, + "outputs": [ + { + "data": { + "text/html": [ + "
\n", + "\n", + "\n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + "
fixed acidityvolatile aciditycitric acidresidual sugarchloridesfree sulfur dioxidetotal sulfur dioxidedensitypHsulphatesalcoholquality
count4898.0000004898.0000004898.0000004898.0000004898.0000004898.0000004898.0000004898.0000004898.0000004898.0000004898.0000004898.000000
mean6.8547880.2782410.3341926.3914150.04577235.308085138.3606570.9940273.1882670.48984710.5142675.877909
std0.8438680.1007950.1210205.0720580.02184817.00713742.4980650.0029910.1510010.1141261.2306210.885639
min3.8000000.0800000.0000000.6000000.0090002.0000009.0000000.9871102.7200000.2200008.0000003.000000
25%6.3000000.2100000.2700001.7000000.03600023.000000108.0000000.9917233.0900000.4100009.5000005.000000
50%6.8000000.2600000.3200005.2000000.04300034.000000134.0000000.9937403.1800000.47000010.4000006.000000
75%7.3000000.3200000.3900009.9000000.05000046.000000167.0000000.9961003.2800000.55000011.4000006.000000
max14.2000001.1000001.66000065.8000000.346000289.000000440.0000001.0389803.8200001.08000014.2000009.000000
\n", + "
" + ], + "text/plain": [ + " fixed acidity volatile acidity citric acid residual sugar \\\n", + "count 4898.000000 4898.000000 4898.000000 4898.000000 \n", + "mean 6.854788 0.278241 0.334192 6.391415 \n", + "std 0.843868 0.100795 0.121020 5.072058 \n", + "min 3.800000 0.080000 0.000000 0.600000 \n", + "25% 6.300000 0.210000 0.270000 1.700000 \n", + "50% 6.800000 0.260000 0.320000 5.200000 \n", + "75% 7.300000 0.320000 0.390000 9.900000 \n", + "max 14.200000 1.100000 1.660000 65.800000 \n", + "\n", + " chlorides free sulfur dioxide total sulfur dioxide density \\\n", + "count 4898.000000 4898.000000 4898.000000 4898.000000 \n", + "mean 0.045772 35.308085 138.360657 0.994027 \n", + "std 0.021848 17.007137 42.498065 0.002991 \n", + "min 0.009000 2.000000 9.000000 0.987110 \n", + "25% 0.036000 23.000000 108.000000 0.991723 \n", + "50% 0.043000 34.000000 134.000000 0.993740 \n", + "75% 0.050000 46.000000 167.000000 0.996100 \n", + "max 0.346000 289.000000 440.000000 1.038980 \n", + "\n", + " pH sulphates alcohol quality \n", + "count 4898.000000 4898.000000 4898.000000 4898.000000 \n", + "mean 3.188267 0.489847 10.514267 5.877909 \n", + "std 0.151001 0.114126 1.230621 0.885639 \n", + "min 2.720000 0.220000 8.000000 3.000000 \n", + "25% 3.090000 0.410000 9.500000 5.000000 \n", + "50% 3.180000 0.470000 10.400000 6.000000 \n", + "75% 3.280000 0.550000 11.400000 6.000000 \n", + "max 3.820000 1.080000 14.200000 9.000000 " + ] + }, + "execution_count": 72, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "white_wine.describe()" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "jupyter": { + "source_hidden": true + } + }, + "source": [ + "Similar table for red wine data set" + ] + }, + { + "cell_type": "code", + "execution_count": 73, + "metadata": { + "collapsed": true, + "jupyter": { + "outputs_hidden": true, + "source_hidden": true + } + }, + "outputs": [ + { + "data": { + "text/html": [ + "
\n", + "\n", + "\n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + "
fixed acidityvolatile aciditycitric acidresidual sugarchloridesfree sulfur dioxidetotal sulfur dioxidedensitypHsulphatesalcoholquality
count1599.0000001599.0000001599.0000001599.0000001599.0000001599.0000001599.0000001599.0000001599.0000001599.0000001599.0000001599.000000
mean8.3196370.5278210.2709762.5388060.08746715.87492246.4677920.9967473.3111130.65814910.4229835.636023
std1.7410960.1790600.1948011.4099280.04706510.46015732.8953240.0018870.1543860.1695071.0656680.807569
min4.6000000.1200000.0000000.9000000.0120001.0000006.0000000.9900702.7400000.3300008.4000003.000000
25%7.1000000.3900000.0900001.9000000.0700007.00000022.0000000.9956003.2100000.5500009.5000005.000000
50%7.9000000.5200000.2600002.2000000.07900014.00000038.0000000.9967503.3100000.62000010.2000006.000000
75%9.2000000.6400000.4200002.6000000.09000021.00000062.0000000.9978353.4000000.73000011.1000006.000000
max15.9000001.5800001.00000015.5000000.61100072.000000289.0000001.0036904.0100002.00000014.9000008.000000
\n", + "
" + ], + "text/plain": [ + " fixed acidity volatile acidity citric acid residual sugar \\\n", + "count 1599.000000 1599.000000 1599.000000 1599.000000 \n", + "mean 8.319637 0.527821 0.270976 2.538806 \n", + "std 1.741096 0.179060 0.194801 1.409928 \n", + "min 4.600000 0.120000 0.000000 0.900000 \n", + "25% 7.100000 0.390000 0.090000 1.900000 \n", + "50% 7.900000 0.520000 0.260000 2.200000 \n", + "75% 9.200000 0.640000 0.420000 2.600000 \n", + "max 15.900000 1.580000 1.000000 15.500000 \n", + "\n", + " chlorides free sulfur dioxide total sulfur dioxide density \\\n", + "count 1599.000000 1599.000000 1599.000000 1599.000000 \n", + "mean 0.087467 15.874922 46.467792 0.996747 \n", + "std 0.047065 10.460157 32.895324 0.001887 \n", + "min 0.012000 1.000000 6.000000 0.990070 \n", + "25% 0.070000 7.000000 22.000000 0.995600 \n", + "50% 0.079000 14.000000 38.000000 0.996750 \n", + "75% 0.090000 21.000000 62.000000 0.997835 \n", + "max 0.611000 72.000000 289.000000 1.003690 \n", + "\n", + " pH sulphates alcohol quality \n", + "count 1599.000000 1599.000000 1599.000000 1599.000000 \n", + "mean 3.311113 0.658149 10.422983 5.636023 \n", + "std 0.154386 0.169507 1.065668 0.807569 \n", + "min 2.740000 0.330000 8.400000 3.000000 \n", + "25% 3.210000 0.550000 9.500000 5.000000 \n", + "50% 3.310000 0.620000 10.200000 6.000000 \n", + "75% 3.400000 0.730000 11.100000 6.000000 \n", + "max 4.010000 2.000000 14.900000 8.000000 " + ] + }, + "execution_count": 73, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "red_wine.describe()" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "jupyter": { + "source_hidden": true + } + }, + "source": [ + "Base on the brief summary of the data above, there is no missing value, all the features have numeric values, hence there is no major preprocessing needed. We decide to combine the two data sets of red wine and white wine to consider wine type (i.e. red or wine) as another possible features that could link to wine quality. Below is the combined data set." + ] + }, + { + "cell_type": "code", + "execution_count": 74, + "metadata": { + "collapsed": true, + "jupyter": { + "outputs_hidden": true, + "source_hidden": true + } + }, + "outputs": [ + { + "data": { + "text/html": [ + "
\n", + "\n", + "\n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + "
fixed acidityvolatile aciditycitric acidresidual sugarchloridesfree sulfur dioxidetotal sulfur dioxidedensitypHsulphatesalcoholqualitytype
07.00.2700.3620.70.04545.0170.01.001003.000.458.86white
16.30.3000.341.60.04914.0132.00.994003.300.499.56white
28.10.2800.406.90.05030.097.00.995103.260.4410.16white
37.20.2300.328.50.05847.0186.00.995603.190.409.96white
47.20.2300.328.50.05847.0186.00.995603.190.409.96white
..........................................
15946.20.6000.082.00.09032.044.00.994903.450.5810.55red
15955.90.5500.102.20.06239.051.00.995123.520.7611.26red
15966.30.5100.132.30.07629.040.00.995743.420.7511.06red
15975.90.6450.122.00.07532.044.00.995473.570.7110.25red
15986.00.3100.473.60.06718.042.00.995493.390.6611.06red
\n", + "

6497 rows × 13 columns

\n", + "
" + ], + "text/plain": [ + " fixed acidity volatile acidity citric acid residual sugar chlorides \\\n", + "0 7.0 0.270 0.36 20.7 0.045 \n", + "1 6.3 0.300 0.34 1.6 0.049 \n", + "2 8.1 0.280 0.40 6.9 0.050 \n", + "3 7.2 0.230 0.32 8.5 0.058 \n", + "4 7.2 0.230 0.32 8.5 0.058 \n", + "... ... ... ... ... ... \n", + "1594 6.2 0.600 0.08 2.0 0.090 \n", + "1595 5.9 0.550 0.10 2.2 0.062 \n", + "1596 6.3 0.510 0.13 2.3 0.076 \n", + "1597 5.9 0.645 0.12 2.0 0.075 \n", + "1598 6.0 0.310 0.47 3.6 0.067 \n", + "\n", + " free sulfur dioxide total sulfur dioxide density pH sulphates \\\n", + "0 45.0 170.0 1.00100 3.00 0.45 \n", + "1 14.0 132.0 0.99400 3.30 0.49 \n", + "2 30.0 97.0 0.99510 3.26 0.44 \n", + "3 47.0 186.0 0.99560 3.19 0.40 \n", + "4 47.0 186.0 0.99560 3.19 0.40 \n", + "... ... ... ... ... ... \n", + "1594 32.0 44.0 0.99490 3.45 0.58 \n", + "1595 39.0 51.0 0.99512 3.52 0.76 \n", + "1596 29.0 40.0 0.99574 3.42 0.75 \n", + "1597 32.0 44.0 0.99547 3.57 0.71 \n", + "1598 18.0 42.0 0.99549 3.39 0.66 \n", + "\n", + " alcohol quality type \n", + "0 8.8 6 white \n", + "1 9.5 6 white \n", + "2 10.1 6 white \n", + "3 9.9 6 white \n", + "4 9.9 6 white \n", + "... ... ... ... \n", + "1594 10.5 5 red \n", + "1595 11.2 6 red \n", + "1596 11.0 6 red \n", + "1597 10.2 5 red \n", + "1598 11.0 6 red \n", + "\n", + "[6497 rows x 13 columns]" + ] + }, + "execution_count": 74, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "white_wine['type'] = 'white'\n", + "red_wine['type'] = 'red'\n", + "wine_df = pd.concat([white_wine, red_wine], axis = 0)\n", + "wine_df" + ] + }, + { + "cell_type": "code", + "execution_count": 75, + "metadata": { + "jupyter": { + "source_hidden": true + } + }, + "outputs": [], + "source": [ + "bins = (1, 4, 6, 9)\n", + "rating_groups = ['poor','normal','excellent']\n", + "wine_df['quality'] = pd.cut(wine_df['quality'], bins = bins, labels = rating_groups)" + ] + }, + { + "cell_type": "code", + "execution_count": 76, + "metadata": { + "jupyter": { + "source_hidden": true + } + }, + "outputs": [], + "source": [ + "train_df, test_df = train_test_split(wine_df,test_size = 0.2 , random_state = 123)\n", + "\n", + "#train_df['type'] = train_df['type'].astype('category')\n", + "#test_df['type'] = test_df['type'].astype('category')\n", + "X_train = train_df.drop(columns = ['quality'], axis=1)\n", + "y_train = train_df['quality']\n", + "\n", + "X_test = test_df.drop(columns = ['quality'], axis=1)\n", + "y_test = test_df['quality']\n" + ] + }, + { + "cell_type": "code", + "execution_count": 77, + "metadata": { + "collapsed": true, + "jupyter": { + "outputs_hidden": true, + "source_hidden": true + } + }, + "outputs": [ + { + "data": { + "text/plain": [ + "normal 3967\n", + "excellent 1028\n", + "poor 202\n", + "Name: quality, dtype: int64" + ] + }, + "execution_count": 77, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "y_train.value_counts()" + ] + }, + { + "cell_type": "code", + "execution_count": 78, + "metadata": { + "collapsed": true, + "jupyter": { + "outputs_hidden": true, + "source_hidden": true + } + }, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "\n", + "Int64Index: 5197 entries, 1554 to 3582\n", + "Data columns (total 13 columns):\n", + " # Column Non-Null Count Dtype \n", + "--- ------ -------------- ----- \n", + " 0 fixed acidity 5197 non-null float64 \n", + " 1 volatile acidity 5197 non-null float64 \n", + " 2 citric acid 5197 non-null float64 \n", + " 3 residual sugar 5197 non-null float64 \n", + " 4 chlorides 5197 non-null float64 \n", + " 5 free sulfur dioxide 5197 non-null float64 \n", + " 6 total sulfur dioxide 5197 non-null float64 \n", + " 7 density 5197 non-null float64 \n", + " 8 pH 5197 non-null float64 \n", + " 9 sulphates 5197 non-null float64 \n", + " 10 alcohol 5197 non-null float64 \n", + " 11 quality 5197 non-null category\n", + " 12 type 5197 non-null object \n", + "dtypes: category(1), float64(11), object(1)\n", + "memory usage: 533.0+ KB\n" + ] + } + ], + "source": [ + "train_df.info()" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "# PreProcessor" + ] + }, + { + "cell_type": "code", + "execution_count": 207, + "metadata": {}, + "outputs": [], + "source": [ + "numeric_features = ['fixed acidity', 'volatile acidity', 'citric acid', 'residual sugar', \n", + " 'chlorides', 'free sulfur dioxide','total sulfur dioxide', 'density', 'pH', 'sulphates', 'alcohol']\n", + "binary_features = ['type']\n", + "\n", + "numeric_transformer = make_pipeline(SimpleImputer(), StandardScaler())\n", + "binary_transformer = make_pipeline(OneHotEncoder(drop=\"if_binary\", dtype=int))\n", + "\n", + "preprocessor = ColumnTransformer(\n", + " transformers = [\n", + " ('num', numeric_transformer, numeric_features),\n", + " ('bin', binary_transformer, binary_features)\n", + " ]\n", + ")" + ] + }, + { + "cell_type": "code", + "execution_count": 211, + "metadata": {}, + "outputs": [], + "source": [ + "#DataStructure to store results\n", + "results={}\n", + "# helper function from lectures 573, UBC MDS\n", + "def mean_std_cross_val_scores(model, X_train, y_train, **kwargs):\n", + " \"\"\"\n", + " Returns mean and std of cross validation\n", + " \"\"\"\n", + " scores = cross_validate(model, \n", + " X_train, y_train, n_jobs=-1, \n", + " **kwargs) \n", + " \n", + " mean_scores = pd.DataFrame(scores).mean()\n", + " #std_scores = pd.DataFrame(scores).std()\n", + " out_col = []\n", + "\n", + " for i in range(len(mean_scores)): \n", + " out_col.append(mean_scores[i])\n", + "\n", + " return pd.Series(data = out_col, index = mean_scores.index)" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "\n", + " \n", + "# Model Selection\n", + " \n", + "" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "### Stacking Classifier" + ] + }, + { + "cell_type": "code", + "execution_count": 212, + "metadata": {}, + "outputs": [], + "source": [ + "#Stacking Classifier\n", + "scoring_metric = {'f1_micro'}\n", + "pipe_rf = make_pipeline(preprocessor, RandomForestClassifier(bootstrap=False, max_depth=20,\n", + " max_features='sqrt', n_estimators=1800,\n", + " random_state=123))\n", + "pipe_catboost = make_pipeline(preprocessor, CatBoostClassifier(verbose=0, random_state=123))\n", + "pipe_svc = make_pipeline(preprocessor, SVC(probability=True))\n", + "classifiers = {\n", + " \"svm\": pipe_svc,\n", + " 'random forest' : pipe_rf,\n", + " 'CatBoost' : pipe_catboost,\n", + "}" + ] + }, + { + "cell_type": "code", + "execution_count": 213, + "metadata": {}, + "outputs": [], + "source": [ + "#Testing stacking model\n", + "stacking_model = StackingClassifier(list(classifiers.items()), \n", + " final_estimator=RandomForestClassifier())\n", + "\n", + "results['StackingClf'] = mean_std_cross_val_scores(stacking_model, X_train, y_train, return_train_score=True, scoring=scoring_metric)" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "#### ALL other classifiers" + ] + }, + { + "cell_type": "code", + "execution_count": 214, + "metadata": {}, + "outputs": [ + { + "data": { + "text/html": [ + "
\n", + "\n", + "\n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + "
StackingClfRidgeClassifierRandom ForestKNNMLP ClassifierNearest CentroidQDA
fit_time172.5207580.03120318.6037350.02517814.3872230.0145310.020165
score_time0.7251420.0096270.5309300.1113330.0125450.0124880.011142
test_f1_micro0.8343280.7791050.8431770.7977690.8079610.5020230.715223
train_f1_micro0.9989900.7823271.0000000.8531850.9873490.5040400.723205
\n", + "
" + ], + "text/plain": [ + " StackingClf RidgeClassifier Random Forest KNN \\\n", + "fit_time 172.520758 0.031203 18.603735 0.025178 \n", + "score_time 0.725142 0.009627 0.530930 0.111333 \n", + "test_f1_micro 0.834328 0.779105 0.843177 0.797769 \n", + "train_f1_micro 0.998990 0.782327 1.000000 0.853185 \n", + "\n", + " MLP Classifier Nearest Centroid QDA \n", + "fit_time 14.387223 0.014531 0.020165 \n", + "score_time 0.012545 0.012488 0.011142 \n", + "test_f1_micro 0.807961 0.502023 0.715223 \n", + "train_f1_micro 0.987349 0.504040 0.723205 " + ] + }, + "execution_count": 214, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "#Model selection plot \n", + "\n", + "classifiers_plot = {\n", + " \"RidgeClassifier\": RidgeClassifier(random_state=123),\n", + " \"Random Forest\":RandomForestClassifier(bootstrap=False, max_depth=20,\n", + " max_features='sqrt', n_estimators=1800,\n", + " random_state=123),\n", + " \"KNN\": KNeighborsClassifier(n_neighbors=5),\n", + " \"MLP Classifier\":MLPClassifier(alpha=0.05, hidden_layer_sizes=(50, 100, 50),\n", + " learning_rate='adaptive', max_iter=1000,random_state=123),\n", + " \"Nearest Centroid\": NearestCentroid(),\n", + " \"QDA\" :QuadraticDiscriminantAnalysis()\n", + "}\n", + "for (name, model) in classifiers_plot.items():\n", + " pipe_iter = make_pipeline(preprocessor, model)\n", + " results[name] = mean_std_cross_val_scores(pipe_iter, X_train, y_train, return_train_score=True, scoring=scoring_metric)\n", + "pd.DataFrame(results)" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "# Plotting results\n", + "### All classifiers" + ] + }, + { + "cell_type": "code", + "execution_count": 220, + "metadata": {}, + "outputs": [ + { + "data": { + "text/html": [ + "\n", + "
\n", + "" + ], + "text/plain": [ + "alt.Chart(...)" + ] + }, + "execution_count": 220, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "#alt.Chart(results).encode\n", + "plot_results = pd.DataFrame(results).T\n", + "plot_results =plot_results.reset_index()\n", + "bar = alt.Chart(plot_results).mark_bar().encode(\n", + " alt.X('test_f1_micro', axis=alt.Axis(title='F1 Micro score')),\n", + " alt.Y('index', sort='-x', axis=alt.Axis(title='Classifier')),\n", + ").properties(\n", + " width=alt.Step(40) # controls width of bar.\n", + ")\n", + "bar" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## Plotting stability across cv folds" + ] + }, + { + "cell_type": "code", + "execution_count": 256, + "metadata": {}, + "outputs": [], + "source": [ + "scores_rf = cross_validate(pipe_rf, X_train, y_train, \n", + " return_train_score=True,scoring = scoring_metric, n_jobs=-1, cv=20 )" + ] + }, + { + "cell_type": "code", + "execution_count": 265, + "metadata": {}, + "outputs": [ + { + "data": { + "text/html": [ + "\n", + "
\n", + "" + ], + "text/plain": [ + "alt.Chart(...)" + ] + }, + "execution_count": 265, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "plot_rf = pd.DataFrame(scores_rf)\n", + "bar = alt.Chart(plot_rf).mark_bar().encode(\n", + " x= alt.X('test_f1_micro', axis=alt.Axis(title='F1 Micro score'), bin=alt.Bin(maxbins=6)),\n", + " y= alt.Y('count()'),\n", + ")\n", + "bar" + ] + }, + { + "cell_type": "code", + "execution_count": 267, + "metadata": {}, + "outputs": [], + "source": [ + "#pipe_mlp = make_pipeline(preprocessor, model)\n", + "scores_sml = cross_validate(stacking_model, X_train, y_train, \n", + " return_train_score=True,scoring = scoring_metric, n_jobs=-1, cv=20 )" + ] + }, + { + "cell_type": "code", + "execution_count": 270, + "metadata": {}, + "outputs": [ + { + "data": { + "text/html": [ + "\n", + "
\n", + "" + ], + "text/plain": [ + "alt.Chart(...)" + ] + }, + "execution_count": 270, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "plot_sml = pd.DataFrame(scores_sml)\n", + "alt.Chart(plot_sml).mark_bar().encode(\n", + " x= alt.X('test_f1_micro', axis=alt.Axis(title='F1 Micro score'), bin=alt.Bin(maxbins=6)),\n", + " y= alt.Y('count()'),\n", + ")" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "#### Here, we can observe that Random forests are more consistent across CV folds" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "# HyperParameter Optimization" + ] + }, + { + "cell_type": "code", + "execution_count": 160, + "metadata": {}, + "outputs": [], + "source": [ + "param_dist = {\n", + " 'randomforestclassifier__max_depth': [10, 20, 30, 40, 50, 60, 70, 80, 90, 100,300,500,450,200,250, 600, 700],\n", + " 'randomforestclassifier__bootstrap': [True, False],\n", + " 'randomforestclassifier__max_features': ['auto', 'sqrt'],\n", + " 'randomforestclassifier__min_samples_leaf': [1, 2, 4],\n", + " 'randomforestclassifier__min_samples_split': [2, 5, 10],\n", + " 'randomforestclassifier__n_estimators': [200, 400, 600, 800, 1000, 1200, 1400, 1600, 1800, 2000]\n", + "}" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "random_search = RandomizedSearchCV(pipe_rf, param_distributions=param_dist, n_jobs=-1, n_iter=20, cv=10, scoring = scoring_metric)\n", + "random_search.fit(X_train, y_train)" + ] + }, + { + "cell_type": "code", + "execution_count": 271, + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "Best cv score from grid search: 0.829\n" + ] + }, + { + "data": { + "text/plain": [ + "{'randomforestclassifier__n_estimators': 1800,\n", + " 'randomforestclassifier__min_samples_split': 2,\n", + " 'randomforestclassifier__min_samples_leaf': 1,\n", + " 'randomforestclassifier__max_features': 'sqrt',\n", + " 'randomforestclassifier__max_depth': 20,\n", + " 'randomforestclassifier__bootstrap': False}" + ] + }, + "execution_count": 271, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "print(\"Best cv score from grid search: %.3f\" % random_search.best_score_)\n", + "random_search.best_params_" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## Interpreting Our model" + ] + }, + { + "cell_type": "code", + "execution_count": 273, + "metadata": {}, + "outputs": [ + { + "data": { + "text/plain": [ + "0.8523076923076923" + ] + }, + "execution_count": 273, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "best_model_pipe = random_search.best_estimator_\n", + "best_model_pipe.fit(X_train, y_train)\n", + "best_model_pipe.score(X_test, y_test)" + ] + }, + { + "cell_type": "code", + "execution_count": 275, + "metadata": {}, + "outputs": [ + { + "data": { + "image/png": "iVBORw0KGgoAAAANSUhEUgAAAVwAAAEGCAYAAAApAy29AAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjMuMywgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/Il7ecAAAACXBIWXMAAAsTAAALEwEAmpwYAAAsAklEQVR4nO3deXxU1fn48c+TBAgQSIAEwh5k30RZFEFFZRHQFkUUEVy//SnWpVYtSt0KtnWvVkSRWit1QxEUrChbRUBAWSUssshOEBKEALJm8vz+uDdhErNMYObOZPq8fc0rc+8999xzL+MzZ84951xRVYwxxoReTLgLYIwx/yss4BpjjEcs4BpjjEcs4BpjjEcs4BpjjEfiwl2A8qJStSStmlwv3MWIWBXj7Lu7NPWrx4e7CBFt27atZGVlyZnkEVu9sWrO0YDS6tHMGara90yOV1YWcANUNbkevZ54J9zFiFhpyVXDXYSIN7pvy3AXIaJ1P7/zGeehOUep1PK6gNIeWzk2+YwPWEYWcI0xUURAIvfXlgVcY0z0ECAmNtylKJYFXGNMdJEzagYOKQu4xpgoYk0KxhjjHavhGmOMBwSr4RpjjDfEarjGGOMZ66VgjDFesJtmxhjjDcGaFIwxxjNWwzXGGC9Yk4IxxnhDgFi7aWaMMd6wNlxjjPGCNSkYY4x3rIZrjDEesRquMcZ4QGxorzHGeMeG9hpjjBfsppkxxnjHmhSMMcYDNh+uMcZ4xZoUjDHGO3bTzBhjPGJtuMYY4wGxJgVjjPGO1XCNMcYbYgHXGGNCz3nCjgVcY4wJPREkxgKuCVD7utUZ1qUBMQJfbdrHf9bsKbC9VZ0E7uvRlMzDxwFYuuMAU9N/zN8uAqP7tWL/kZP8be4PnpbdK9s2bmPeZ/NQVdp0akPnizsXmW7Pzj1MGj+Jvtf1pVm7ZgAcP3qcOZ/MYd/efQhCz6t7UrdRXS+LH3azF65l5Asf4cvN5cYB3fj9LX3CXaSgshpukInIW8B/VPUjEZkLPKiqS08jn0uAE6q6MKgFPE0icNN5DXl2zkZ+OnKSUf1asnxnNhnZxwqk27D3cLHB9PJWtcnIPkblCpHbF/FM5ObmMvfTuVx1y1UkVE/gg3EfcFars6hZu+Yv0i2cuZBGzRoVWD9v+jwaN29M/yH98eX4yDmZ42Hpw8/ny+UPz37Ix6/cTb06SVx283P0u7g9rc6Kni+dSA64kdt/whuXAN3CXYg8TWtVZe+h42QePoEvV1m8dT8dGyQGvH+NKhXoUK86czdlhbCU4bVn5x6SaiWRWDOR2LhYWrRvweZ1m3+RbtXiVTRt25TKCZXz1504doKMrRm06dQGgNi4WCpVruRZ2SPBsjVbOathMmkNkqlYIY6BvTsy/atV4S5WUIlIQK8A8ukrIutFZJOIPFzE9kQR+VREvhORNSJya2l5hjTgisgwEflWRFaKyOsicr6IrBKReBGp6haynYjEisjzIpLubr/H3b+TiHwlIstEZIaIlPg1LCJ9RGSRiCwXkUkikuCu3yoio9z16SLSSkTSgOHA793yXRTKaxGIGlUqsO/Iifzln46cpEaVCr9I1yylKn++ohUPXNqU+onx+euHdmrAByt2oZ6UNjx+PvgzCYkJ+csJiQkcPnS4QJrDBw/zw7ofaNelXYH12fuzia8az+yPZ/P+2PeZ88kcTp446Um5I8XuzGzq16mRv1yvTg12Z2aHsURBJmV4lZSNSCwwFugHtAGGiEibQsnuAtaqagecytsLIlKxpHxDFnBFpDUwGOiuqucAPqAlMA34M/As8I6qrgZuB5oA56rq2cC7IlIBGAMMUtVOwJvAX0o4XjLwKNBLVTsCS4H7/ZJkuetfw2mC2AqMA15U1XNUdX7QTj6YCkXPrT8d4fcfr+bRz75n1vpMftfjLADOqV+dQ8dy2PrT0TAU0jtaxNeJFPq/Z/70+XTv052YmIIf79zcXDJ3Z9K+S3uG3DWEChUqsGzespCWN9KoFnH9IvcXeJkJgdVuA6jhngdsUtXNqnoCmAgMKJRGgWriZJYA/ASU2EYVyjbcnkAnYIl7cpWBvcBoYAlwDLjXTdsLGKeqOQCq+pOItAPaAbPc/WOB3SUcryvON9HXbvqKwCK/7VPcv8uAgYGcgIjcjvNlQJVaqYHsckb2HzlJrSqnviBrVqnA/qMFa2DHTubmv1+VcZDYGCGhUizNUxI4t0EiZ9evToXYGCpXiOWO7mm8/vXWkJfbSwnVEzicfapGezj7MFWrVS2QZu+uvXzx4RcAHDtyjG0btiExQmrDVBKqJ5Da0Pm3bNq2Kcvm/28F3Hq1k9i1Z3/+csae/aQmB95sVR4U/qItQbKI+N/7Ga+q49339YEdftt2AucX2v8VnApkBlANGKyquZQglAFXgAmqOrLASpFUnG+DCkA88LObtvBXrwBrVPWCMhxvlqoOKWb7cfevjwDP27344wFqNmkT8l/qm/f9TJ1qlUiuWpH9R0/SNa0Gry3YWiBNYnwc2cecL9GzalUhRoTDx31MWpnBpJUZgNOToX/rOlEXbAHq1K/DgX0HyN6fTUK1BDakb+Dyay8vkObmB27Ofz9ryiyatGhC0zZNAacJYn/mfmqk1GDn5p3UTCl4sy3adWzTmB+2Z7JtVxZ1aycxZdZy/vHkLeEuVlCV4aZZlqoW3cWl6EaHwjHgcmAlcBnQFKdyOF9VDxZ3wFAG3DnAVBF5UVX3ikhNnG+BMcBjOE0IzwB3AzOB4SIyV1Vz3LTrgRQRuUBVF7lNDC1UdU0xx1sMjBWRZqq6SUSqAA1UdUMJZTwEVA/K2QZBrsK/l+xgRM9miAjzftjHruxjXNo8GYAvN2bRpVENLmuRTK4qJ3KUsfO3hLnU3oqJjaHHlT2YNmEaubm5tOnYhlp1apH+bToA7c9rX+L+Pa7owcyPZuLz+aheozq9BvbyotgRIy4ulmdHXMc1947F51OG/rorrZtGTw+FQNpnA7QTaOi33ACnJuvvVuBpddppNonIFqAV8G1xmYYs4KrqWhF5FJgpIjHASWAqkKOq77mN0gtF5DLgDaAFsEpETgL/UNVXRGQQ8LKIJLplfQkoMuCqaqaI3AK8LyJ5t54fBUoKuJ8CH4nIAOCeSGjHXZVxkBHT1hZY9+XGU70OZm/IZPaGzBLz+H7PYb7fc7jENOVZWos00lqkFVhXXKDtPbB3geWUuikMvnNwqIpWLvTp3pY+3duGuxghE6RuYUuA5iLSBNgFXA/cUCjNdpym0/kiUgfnHtUvu8z4CWk/XFX9APigmG0+CraJ3E/Bm1yo6krg4iL2vcXv/SV+7/8LdCkifZrf+6U4dxRxa79nl3oixphyIe+m2Zlyf2nfDczAuX/0pqquEZHh7vZxwJPAWyKSjlOvfkhVS+yTWS4HPhhjTHGCNbRXVacD0wutG+f3PgMo0zA9C7jGmOghkT3SzAKuMSaqWMA1xhiPWMA1xhgPBOumWahYwDXGRJfIjbcWcI0xUUTKNLTXcxZwjTFRxZoUjDHGK5Ebby3gGmOii9VwjTHGA4E+zSFcLOAaY6KKBVxjjPGIPSbdGGM8YjVcY4zxgk1eY4wx3hAi+6GYFnCNMVHEeikYY4xnYuymmTHGeECsScEYYzwhWA3XGGM8YzVcY4zxiN00M8YYL1gbrjHGeEMQm4DcGGO8YjVcY4zxiLXhGmOMF6wN1xhjvOHMpRC5EdcCrjEmqkRwvLWAa4yJLjbSzBhjvGDz4UaHxjWq8I/BHcJdjIiV2u134S5CxBt1+ZhwFyGiaRDysPlwjTHGMzYfrjHGeCaC460FXGNMFBG7aWaMMZ6wfrjGGOOhSA64kTutjjHGnAaRwF6l5yN9RWS9iGwSkYeLSXOJiKwUkTUi8lVpeVoN1xgTVYJRwxWRWGAs0BvYCSwRkWmqutYvTRLwKtBXVbeLSO3S8rUarjEmegRYuw0gJp8HbFLVzap6ApgIDCiU5gZgiqpuB1DVvaVlagHXGBM1nAnIA3sBySKy1O91u19W9YEdfss73XX+WgA1RGSuiCwTkZtKK581KRhjokpM4E0KWarauZhtRWVSeDBcHNAJ6AlUBhaJyGJV3VDcAS3gGmOiSpA6KewEGvotNwAyikiTpao/Az+LyDygA1BswLUmBWNM1BB38ppAXqVYAjQXkSYiUhG4HphWKM1U4CIRiRORKsD5wLqSMrUarjEmqgRjoJmq5ojI3cAMIBZ4U1XXiMhwd/s4VV0nIl8Aq4Bc4A1VXV1SvsUGXBEZQwkT+KjqvadxHsYYE1LBGtqrqtOB6YXWjSu0/BzwXKB5llTDXVqm0hljTJgJTk+FSFVswFXVCf7LIlLVbRw2xpiIFcFz15R+00xELhCRtbiNwSLSQUReDXnJjDGmrAK8YRau+RYC6aXwEnA5sA9AVb8DLg5hmYwx5rQFay6FUAiol4Kq7ij0jeALTXGMMeb0CWUa+OC5QALuDhHpBqjbH+1eSulrZowx4RLJE5AH0qQwHLgLZxzxLuAcd9kYYyJKoM0JEdukoKpZwFAPymKMMWcskpsUAumlcJaIfCoimSKyV0SmishZXhTOGGPKSgJ8hUMgTQrvAR8CdYF6wCTg/VAWyhhjTld57xYmqvq2qua4r3coYcivMcaEi9NLIbBXOJQ0l0JN9+2X7vN8JuIE2sHAZx6UzRhjykYkonsplHTTbBlOgM0r/R1+2xR4MlSFMsaY0xXJT+0taS6FJl4WxBhjzlRek0KkCmikmYi0A9oA8XnrVPXfoSqUMcacrnJZw80jIk8Al+AE3OlAP2ABYAHXGBNxIjfcBtZLYRDOQ9J+VNVbcZ7ZUymkpTLGmNMgArExEtArHAJpUjiqqrkikiMi1YG9gA18CKL/Ll7HYy9NwefLZeivunLPTb0LbFdVHn1xCnMWraVyfAX+/uhQzm7ZkGPHT3LVb1/mxMkccny5XHlpB0b8pj8Az4z/jC/mpxMTE0NyUgJ/f3QoqSmJ4Ti9oOt5QWueemAQsTExvD11IS9NmFVge2K1yrzy2DCaNEjm2ImT3PPku6z7YTcA1RMq8/KjN9C6aV1U4Z4n32VJ+pZwnEZQzV60lj++MBlfbi43DriA+27uU2C7qjLyhcnMWriGyvEVGfv4MDq0ahjQvmPemcMTL3/CxplPUSspwbNzOl2R3KQQSA13qYgkAf/A6bmwHPg2lIXymohsFZHkcBzb58tl5POTeO+FO5j33kg+nr2c9Vt+LJBmzqK1bN6ZyaIPH+X5h67noecmAVCpYhyTx9zNf//9EHMmjODLxd+zbPVWAH47tCdfvv0wcyaMoHf3tvztX194fWohERMjPDfiOq793at0ve7PXNOnEy2bpBZI88Ctl5O+YScX3vAUdz7xNk89MCh/29MPDGLOorWcf+2fueiGp35xrcsjny+XEc9O4sO/38miDx5h8oxlfL95d4E0sxeu5Ycde1k6+XFeHHk9DzzzQUD77tyzn7nffE+D1BqentOZiOS5FEoNuKr6W1U94D7Lpzdws9u0EBFEpFw/CHPF2m00aZBC4/rJVKwQx1W9OjJjfnqBNDPmr+a6vl0QETq1S+Pg4aPsycpGRKhaxWndOZnjIyfHl/9BqlY1//4mR46dCN8nLMg6tU1j844stu3ax8kcH1NmLad/j7MLpGnZJJV5S9YDsHHbHhrVrUlKzWpUqxpPt3Ob8vbURYBzzQ4ePur5OQTbsjXbaNIgmTT3MzSwTyc+n1fwMzR9XjrX9z8PEaFL+yYcPHSUH7OyS933kRenMOqeARFda/QnCDES2CscShr40LGkbaq6PFiFEJE04HOcm3HdcGYlGwC0BMYBVYAfgNtUdb+IzAUWAt2BaSLyK2AF0AlIAW4CRgLtgQ9U9VH3OJ/gPGs+Hvi7qo4P1jmcrt2Z2dSrk5S/XDclieVrtxVKc6BQmkR2Z2ZTJzkRny+XPrc9z5admdw68CI6tk3LT/fUuP8w6YslVKsaz+RX7gnxmXijbkoiu/bsz1/O2LOfTu3SCqRZvXEXV156Dou/20zHNo1pmFqTerWT8OXmknXgMGOfGEa75vVZuW4HI1/4yPlCKsd2Zx6gfp1TNdB6tZNYtmZrwTR7f5lm997sEvf9fF46dVMSadeiQUjLH1RhrL0GoqQa7gslvJ4PQVmaA2NVtS1wALgGpyfEQ6p6NpAOPOGXPklVe6jqC+7yCVW9GCdAT8WZQrIdcIuI1HLT3KaqnYDOwL1+64skIreLyFIRWbovKzM4Z1mIFjFKuvAHRosYSJ1X44iNjWHOhBGs+GQUK9ZtY90PGflpRg6/kuWfjOKayzvz5uR5QS13uBRV0yp8fV6aMIuk6lWY9+7D3D64B6s27MTnyyUuNpYOLRvy5kfz6THsGY4cO859t/T+RX7lTZGfj0L36ov7nBW375FjJ3jhXzP44x1XBKuYnonkuRRKGvhwqZcFAbao6kr3/TKgKU5Q/cpdNwFn4pw8HxTaf5r7Nx1Yo6q7AURkM06tdh9OkL3aTdcQJ8jvK65Abg14PMC5HTuHZP6IeilJZOw5kL+8O/MAqckFb27Vq104TTapydULpEmsVoVu5zbjy2++p3XTegW2Xd27E8MefD3/hlp5llG4planBj9mZRdIc+jnY9w9+p385e+mjmJbxj4qx1cgY+8Blq1xfkFMm7OS+24u/wG3Xu2kgrX+vQd+cYO0Xu0aRaY5cdJX5PqtO7PYnrGPi4Y+nb/+khufZfa/HqROoc9eJBEgNoKruIHcNPPKcb/3PiCplPSFnyCct39uobxygTgRuQToBVygqh1wmiDiCbNzWjdi885MtmXs48TJHD6ZvZw+F7YrkKbPhe348IslqCrLVm+lWtV46iQnkrX/MNmHjgBw9PgJ5i/dQLPGtQHYvGNv/v4zFqymWeM63p1UCC1fu42mjVJoVK8WFeJiGdi7I5/PW1UgTfWEylSIiwXgpqu6sXDFJg79fIy9+w6xa8/+/Gt0cZeWUXHTrGObRmzekcm2XVmcOJnDlJnL6HtR+wJp+l3UjonTv0VVWZK+heoJ8aQmJxa7b5tm9dgw4ym+mzqK76aOol7tJOa+PSKig22ecjl5TQTIBvaLyEWqOh+4EfiqlH1KkgjsV9UjItIK6BqMQp6puLhY/nr/NQz5/Wv4fLkMubIrrc6qy4SPFwBw89UX0qtbG+YsWkvXa5+kcnxFXnrkBgD27svm3iffxZebS26u8uue59KnuxOs//Lap2zatpeYGKFBak2eHXFd2M4xmJy76h8y+eW7iI0V3p22mO83/8itAy8E4F9TFtCySSqv/elGfLm5rN/yI/c8+W7+/iOen8T40bdQsUIsW3dlcZdfTbi8iouL5dk/XMuge1/Fl6sM/VVXWjety78mO5+hW6+5kN7d2zJr4Vo6DRxN5fgKvPLYsBL3Lc8ieWivaFGNOF4Xwrlp9h9VbecuPwgkAJ9w6qbZZuBWv5tmD6rqUjd9/rJbk31QVa/034bT1PAJzqOC1uPcXPuTqs4Vka1AZ/fpFkU6t2Nnnfv1N0E86+iS2u134S5CxPvp2zHhLkJE6961C8uXLT2jcJnavJ0O/dvkgNL+7detlqlq5zM5XlkFMrRXcB6xc5aqjhaRRkCqqgatL66qbsW5wZW37H9T7hc1UVW9pLhlVZ0LzC0mbb9ijp9WhuIaYyJYJNdwA2nDfRW4ABjiLh8CxoasRMYYcwYieeBDIG2456tqRxFZAeD+pK8Y4nIZY0yZCRAXwb0UAgm4J0UkFvexOiKSgnPn3xhjIk4Ex9uAAu7LwMdAbRH5C87sYY+GtFTGGHMaJIzDdgNRasBV1XdFZBnOFI0CXKWq60JeMmOMOQ0RHG8D6qXQCDgCfOq/TlW3h7JgxhhzOiK5l0IgTQqfcephkvFAE5x+rG1DWC5jjCkzgbBNLh6IQJoUCowRdGcRu6OY5MYYEz5hHLYbiDIP7VXV5SLSJRSFMcaYM1V4prRIEkgb7v1+izFARyA0cxUaY8wZCOZj0kWkL/B3IBZ4Q1WfLiZdF2AxMFhVPyopz0BquNX83ufgtOkGNljZGGM8FoyA6449GIvzlJudwBIRmaaqa4tI9wwwI5B8Swy4bmYJqvqH0yq1McZ4LEiTi58HbFLVzW6eE3GeQrO2ULp7cCqgATWzFjuXgojEqaoPpwnBGGMinvOY9MBepagP7PBb3umu8zuW1AeuxpnRMCAl1XC/xQm2K0VkGs7TFvIn/VbVKYEexBhjvFKGkWbJIrLUb3m833MOi8qk8Fy2L+E8AswXaK06kDbcmjiPobmMU/1xFbCAa4yJKGW8aZZVwny4O3Eew5WnAZBRKE1nYKIbbJOB/iKSo6qfFHfAkgJubbeHwmpOBdo84Z+13BhjihCkob1LgOYi0gTnKeLXAzf4J1DVJqeOKW/hPEThk5IyLSngxuI8dSGQqrUxxkQAISYI/XBVNUdE7sbpfRALvKmqa0RkuLs94HZbfyUF3N2qOvp0MjXGmHAQgjd5japOB6YXWldkoFXVWwLJs6SAG7nDNYwxpigCcRE8trekgNvTs1IYY0wQBLOGGwrFBlxV/cnLghhjTDCU6wnIjTGmPIngeGsB1xgTPYTAHkUeLhZwjTHRQ6xJwRhjPOGMNLOAa4wxnojccGsB1xgTZSK4gmsB1xgTTSRY8+GGhAVcY0zUsF4KxhjjIbtpFgUkwrubhF31lHCXIOIdPJoT7iJENF9uECYhlKA9YickLOAaY6KGNSkYY4yHrIZrjDEeidxwawHXGBNFBIi1Gq4xxngjguOtBVxjTDQRJIIbFSzgGmOiitVwjTHGA063sMiNuBZwjTHRQ6yGa4wxnonkEaEWcI0xUcOZgDzcpSieBVxjTFSxXgrGGOORCG5RsIBrjIkuVsM1xhgPWBuuMcZ4RcR6KRhjjFciN9xawDXGRBGnSSFyQ64FXGNMVInccGsB1xgTbSI44lrANcZEFWtSMMYYj0RuuLWAa4yJNhEccSP5icLGGFMmQt4zH0r/r9S8RPqKyHoR2SQiDxexfaiIrHJfC0WkQ2l5Wg3XGBM9gjQfrojEAmOB3sBOYImITFPVtX7JtgA9VHW/iPQDxgPnl5Sv1XCNMVFFAnyV4jxgk6puVtUTwERggH8CVV2oqvvdxcVAg9IytYBrjIkigkhgr1LUB3b4Le901xXn/4DPS8vUmhSMMVGlDE0KySKy1G95vKqOz8umiPRa9PHkUpyAe2FpB7SAa4yJGgE2F+TJUtXOxWzbCTT0W24AZPzieCJnA28A/VR1X2kHtCYFY0x0CU4j7hKguYg0EZGKwPXAtAKHEWkETAFuVNUNgRTNarjGmKgSjAnIVTVHRO4GZgCxwJuqukZEhrvbxwGPA7WAV9024ZwSasyABdyI8N9Fa3nkpSn4fLkM+/UF3HtT7wLbVZVHXpzM7IVrqRxfkTGPDeXslg3ZtWc/d49+m737DhETI9w4oBu3D74EgKdf/4zP56cTEyMk10hgzKPDSE1JDMPZBV/PLk156q6+xMbE8Pb05bw08esC2xMT4nnlD7+mSb2aHDuRwz3PTWXd1kwAxjz4ay7v2oKsAz/T7TevhaP4ITH3m3WMHvMxvlxl8BXn89uhvQpsV1VGvfwxX36zjsqVKvD8yCG0a+H8Ys4+dJSHn5vI+i0/IsCzDw2hU7s01mzcxSN/m8TxEyeJi43hyd8P4pzWjcNwdmUTrJG9qjodmF5o3Ti/978BflOWPK1JIcx8vlweemES7/9tOAve/yNTZi1j/ZbdBdLMWbSWzTsy+WbSY7zw8GBGPPshAHGxMYy692q+nvgIn//jft6cPD9/37uGXcZX7zzMl/9+iD7d2/H8m194fm6hEBMjPHdvf64d+S5dbxvLNZe1o2Xj5AJpHrjhItI37eHC/zeOO5/+mKfu6pu/7f0ZKxk08h2vix1SPl8uj780mbeevZ1ZEx5i2pwVbNz6Y4E0c79Zx5admcx994/89cHreORvH+VvGzVmCj3Oa81/3x7J52/+gWaN6wDw9Lhp/O7my/n8n3/g/tv68dS4Tz09r9Pi9sMN5BUOFnBdbkdnzy1fu40mDVJIq59MxQpxXN2rI1/MSy+Q5vN56VzX7zxEhM7tmpB9+Ch7srKpk5zI2S2dWkpC1XhapNVhd2Y2ANWqVs7f/8jR4xH9YL2y6NSqPpt3/cS23Qc4mZPLlC/X0L9bqwJpWjZOZt6KzQBs3LGPRqlJpNSoCsDC9O3sP3jU83KH0sp122lcP5lG9ZzP0K8uO5eZC1YXSDNzwWoGXt4FEaFj2zQOHT7K3n3ZHPr5GN9+t5nBVzj99StWiCOxmvvZEeHwkWMAHDx8jDq1yscvpGCNNAuFchlwRSRNRL4XkQnusLqPRKSKiPQUkRUiki4ib4pIJTd9ceu3isjjIrIAuDYc5/Jj5gHq107KX65bOyk/aJ5Kk029OqfS1Ev5ZZrtu/eRvmEXndqe+sn313H/4ZwBjzN55jIe+n/9Q1J+r9VNrsauzIP5yxmZB6mbXK1AmtWb93DlRa0B6NiyHg3rJFEvubqn5fTSnqwD1PP/DKUksicru1Ca7AJpUlOS+DEzm+0Z+6iVlMCDT79P//97noeenciRo8cBeOLuq3nqtWlcMGgUf31tGiNuv8KL0zkjgtVwQ6UlTr+5s4GDwP3AW8BgVW2P0z59p4jEF7XeL59jqnqhqk70svB5tIiefYU7ZWsR3f/8kxw+cpzbRv6TJ+8bWKBm+8fhV7Jy6miu6dOJf340P2hlDqeiaiaFr+FL7y8gKSGeea/fwe1Xn8eqjbvx+XI9KqH3ivwM/SJNUZ8hwefzsXrjToYN6M70fz5I5fiKvPbeHADemfo1j919FYs+eoLH7hrAQ8+G5X+RMgvSSLOQKM8Bd4eq5t0teQfoCWzx654xAbgYJzAXtT7PB8UdQERuF5GlIrI0KzMzuKV31a2dxK69B/KXd+89QGqh2ljdlCQy9pxKk5F5gNRk5+fdyRwft/3xn1xzeWeuvKTouTMG9unMZ3O/C3rZwyEj6yD1U05dn3op1flx36ECaQ4dOcHdz03j4jteZ/jTn5CcVJVtP+4vnFXUSE1JIsP/M5SZTe3kxBLT/Jh5gDrJ1UlNSSI1JZFz2zi/jPr36MDqDTsBmDxjCX0vPhuAKy49h+/WbQ/tiQRLBEfc8hxwixz1UYTSLu3PxR5AdbyqdlbVzskpKYGXrAzObd2IzTsy2ZaxjxMnc/h49nIuv6h9gTR9L2rPh59/i6qydPUWqleNp05yIqrKfX95jxaN63DnkMsK7LN5x9789zMWpNOsce2QlN9ry7/fRdP6tWiUmkSFuBgGXtqWzxeuL5CmetVKVIhzPto39e/IwlXbOHTkRDiK64kOrRqydWcmO3Y7n6FP/7uC3t3bFkjTu3tbpsxYgqqyfM1WqlWtTO1aidSuVZ16KUn8sN35vHy9fCPN01IBqF2rOotX/gDAwuUbSWsQmv8Hgi3GfXJvaa9wKM/dwhqJyAWquggYAswG7hCRZqq6CbgR+Ar4HkgrYn1EiIuL5ekHBjH4vlfx5eZyw5VdaXVWXd6asgCAWwZeSK9ubZi9cA3nXTuaKpUq8vdHhwLwzarNTPpiCa2b1uPSm54B4JHhV9KrW1uefPVTfti+FxGhYWoNnhsxOGznGEy+XGXEmOlMfmYYsTHCu5+v5Pttmdx6ZScA/vWfZbRsnMJrD12FL1dZvy2Te54/1V/9jUcG0r1DGrUSq7B64u95esJc3vl8RbhOJyji4mIZfd813PTg6/hyc7mu//m0aFKXd6Y6PwCHDejOpV3b8OXidfS44S9UrlSR5x6+Pn//P/3uGu7789ucPOmjYb1aPP/wEACe/sNgRo35mBxfLpUqxvHUg9eF5fzKKpLvD0tRbTuRTkTScPrHzQO6ARtxAukFwPM4XyRLgDtV9biI9Cxm/Vags6pmlXbMjp0661dffxuCs4kOtS9/MtxFiHhbP/1juIsQ0fr06Mp3K5adUbxs16GjTpm5IKC0LVOrLittoEKwlecabq6qDi+0bg5wbuGEqlrc+rTQFM0YEw55E5BHqvIccI0xpqAwdvkKRLkMuKq6FWgX7nIYYyJPBMfb8hlwjTGmaAFNLh42FnCNMVElguOtBVxjTPQI5yiyQFjANcZElwiOuBZwjTFRxbqFGWOMR6wN1xhjvCAQYwHXGGO8ErkR1wKuMSZq5E1AHqks4BpjokoEx1sLuMaY6GI1XGOM8YgN7TXGGI9Ebri1gGuMiSLhfCJvICzgGmOiio00M8YYr0RuvLWAa4yJLhEcby3gGmOiSfgegR4IC7jGmKgR6SPNYsJdAGOM+V9hNVxjTFSJ5BquBVxjTFSxbmHGGOMFG/hgjDHeiPSbZhZwjTFRxZoUjDHGI5Fcw7VuYcaYqCIBvkrNR6SviKwXkU0i8nAR20VEXna3rxKRjqXlaQHXGBNdghBxRSQWGAv0A9oAQ0SkTaFk/YDm7ut24LXSimYB1xgTNQSIEQnoVYrzgE2qullVTwATgQGF0gwA/q2OxUCSiNQtKVNrww3QiuXLsqpXjt0W7nL4SQaywl2ICBZx1yc18U/hLkJhkXaNGp9pBsuXL5tRuYIkB5g8XkSW+i2PV9Xx7vv6wA6/bTuB8wvtX1Sa+sDu4g5oATdAqpoS7jL4E5Glqto53OWIVHZ9SheN10hV+wYpq6KqwHoaaQqwJgVjjPmlnUBDv+UGQMZppCnAAq4xxvzSEqC5iDQRkYrA9cC0QmmmATe5vRW6AtmqWmxzAliTQnk2vvQk/9Ps+pTOrlExVDVHRO4GZgCxwJuqukZEhrvbxwHTgf7AJuAIcGtp+YpqiU0OxhhjgsSaFIwxxiMWcI0xxiMWcMsxEXlLRAa57+eKyGl18RGRS0SkW3BLV76JyFaRgPtzGhMQC7gG4BIgagKuiNjN4CBzh7qaM2QB1wMiMkxEvhWRlSLyuoic7052ES8iVUVkjYi0E5FYEXleRNLd7fe4+3cSka9EZJmIzCht+KCI9BGRRSKyXEQmiUiCu36riIxy16eLSCsRSQOGA793y3dRyC9IAEQkTUTWicg/3OszU0Qqi8g5IrLYvT4fi0gNN/1cEfmriHwF/M5dflFE5rn5dBGRKSKyUUT+7HecT9zrukZEbg/bCQeRe+2+F5EJ7nX6SESqiEhPEVnh/tu/KSKV3PTFrd8qIo+LyALg2rCeVJSwgBtiItIaGAx0V9VzAB/QEqcP35+BZ4F3VHU1zgQYTYBzVfVs4F0RqQCMAQapaifgTeAvJRwvGXgU6KWqHYGlwP1+SbLc9a8BD6rqVmAc8KKqnqOq84N28meuOTBWVdsCB4BrgH8DD7nXJx14wi99kqr2UNUX3OUTqnoxzvlNBe4C2gG3iEgtN81t7nXtDNzrt768a4kzVPVs4CDOZ+AtYLCqtsfpEnqniMQXtd4vn2OqeqGqTvSy8NHKfnqFXk+gE7BEnAkzKgN7gdE4nauPAfe6aXsB41Q1B0BVfxKRdjhBYpa7fywljNUGuuLMbvS1m74isMhv+xT37zJg4BmeW6htUdWV7vtlQFOcoPqVu24CMMkv/QeF9s/rqJ4OrMnrlC4im3FGCO3DCbJXu+ka4gT5fcE8iTDZoapfu+/fAR7DuZ4b3HUTcL6Avixm/UvucuFras6ABdzQE2CCqo4ssFIkFUgAKgDxwM9u2qLGa69R1QvKcLxZqjqkmO3H3b8+Iv/f/7jfex+QVEr6n4vZP7dQXrlAnIhcgvMld4GqHhGRuTj/FtEg0A72pU2bVfiamjNgTQqhNwcYJCK1AUSkpog0xhnl8xjwLvCMm3YmMDzvpo+I1ATWAykicoG7roKItC3heIuB7iLSzE1fRURalFLGQ0C10zo7b2UD+/3amW8EviohfWkSgf1usG2F8+sgWjTK+8wAQ4DZQFre54JT1+77YtabELCAG2KquhanTXWmiKwCZgE3Azmq+h7wNNBFRC4D3gC2A6tE5DvgBncuzkHAM+66lZTQo0BVM4FbgPfd4y0GWpVSzE+BqyPpplkJbgaec8/tHJymmdP1BU5NdxXwJM61ihbrgJvdc6sJvIgz9HSSiKTj1PLHqeqxotaHqcxRz4b2GhNl3J4n/1HVduEuiynIarjGGOMRq+EaY4xHrIZrjDEesYBrjDEesYBrjDEesYBrgkJEfG63stXu/A1VziAv/1nQ3hCRNiWkPa2ZzqSY2cCKW18ozeEyHutPIvJgWctooo8FXBMsR925GNoBJ3AmxMknpznblKr+xu3LXJxLiKKZzkx0s4BrQmE+0MytfX4pIu8B6eLMhvaciCxxZ7G6A0Acr4jIWhH5DKidl5H4zfMrIn3FmensOxGZI0XMdCYiKSIy2T3GEhHp7u5bS5wZx1aIyOuUPqS1xJnEROQFtyxzRCTFXddURL5w95nvjl4zJl+kj6U35Yw7LLkfzigugPOAdqq6xQ1a2araxZ0C8GsRmQmcizO7VXugDrAWZ1Y0/3xTgH8AF7t51XQn9xkHHFbV59107+HMfLZARBrhPASwNc6sYgtUdbSIXIEzM1tpbnOPURln8qHJqroPqAosV9UHRORxN++7cYZrD1fVjSJyPvAqcNlpXEYTpSzgmmCpLCIr3ffzgX/i/NT/VlW3uOv7AGfntc/izGXQHLgYeF9VfUCGiPy3iPy7AvPy8lLVn4opRy+gjTtTGkB1EanmHmOgu+9nIrI/gHMqbiaxXE7NovUOMEWcOYe74QyRzdu/UgDHMP9DLOCaYDnqzvebzw08/rNNCXCPqs4olK4/pc9uVdRMakWJwZn962gRZQl4lE8ZZxJT97gHCl8DY/xZG67x0gycSa8rAIhICxGpCswDrnfbeOsClxax7yKgh4g0cfet6a4vPNPZTJyf97jpznHfzgOGuuv6ATVKKWtJM4nF4EwoBHADTlPFQWCLiFzrHkNEpEMpxzD/YyzgGi+9gdM+u1xEVgOv4/zK+hjYiDNR+GsUMT2gOwva7Tg/37/j1E/6wjOd3Qt0dm/KreVUb4lRwMUishynaWN7KWUtaSaxn4G2IrIMp402b8ayocD/ueVbAwwI4JqY/yE2l4IxxnjEarjGGOMRC7jGGOMRC7jGGOMRC7jGGOMRC7jGGOMRC7jGGOMRC7jGGOOR/w/GlaD+hNNcAAAAAABJRU5ErkJggg==\n", + "text/plain": [ + "
" + ] + }, + "metadata": { + "needs_background": "light" + }, + "output_type": "display_data" + } + ], + "source": [ + "from sklearn.metrics import (plot_confusion_matrix)\n", + "plot_confusion_matrix(best_model_pipe, X_test, y_test, cmap = plt.cm.Blues, normalize='true')\n", + "predictions_m = best_model_pipe.predict(X_test)\n", + "cm = confusion_matrix(y_test, predictions_m)" + ] + } + ], + "metadata": { + "kernelspec": { + "display_name": "Python [conda env:573]", + "language": "python", + "name": "conda-env-573-py" + }, + "language_info": { + "codemirror_mode": { + "name": "ipython", + "version": 3 + }, + "file_extension": ".py", + "mimetype": "text/x-python", + "name": "python", + "nbconvert_exporter": "python", + "pygments_lexer": "ipython3", + "version": "3.8.6" + } + }, + "nbformat": 4, + "nbformat_minor": 4 +}