Assignment_3.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "\n",
    "_You are currently looking at **version 1.1** of this notebook. To download notebooks and datafiles, as well as get help on Jupyter notebooks in the Coursera platform, visit the [Jupyter Notebook FAQ](https://www.coursera.org/learn/python-text-mining/resources/d9pwm) course resource._\n",
    "\n",
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Assignment 3\n",
    "\n",
    "In this assignment you will explore text message data and create models to predict if a message is spam or not. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "5572\n"
     ]
    }
   ],
   "source": [
    "import pandas as pd\n",
    "import numpy as np\n",
    "\n",
    "spam_data = pd.read_csv('spam.csv')\n",
    "\n",
    "spam_data['target'] = np.where(spam_data['target']=='spam',1,0)\n",
    "#spam_data['str_length'] = spam_data['text'].str.len()\n",
    "spam_data.head(10)\n",
    "print(len(spam_data))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "4179\n"
     ]
    }
   ],
   "source": [
    "from sklearn.model_selection import train_test_split\n",
    "\n",
    "\n",
    "X_train, X_test, y_train, y_test = train_test_split(spam_data['text'], \n",
    "                                                    spam_data['target'],\n",
    "                                                    random_state=0)\n",
    "print(len(X_train))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Question 1\n",
    "What percentage of the documents in `spam_data` are spam?\n",
    "\n",
    "*This function should return a float, the percent value (i.e. $ratio * 100$).*"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def answer_one(): \n",
    "    spams = [i for i in spam_data['target'] if i == 1] \n",
    "    return len(spams) * 100 / len(spam_data['target'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "13.406317300789663"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "answer_one()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Question 2\n",
    "\n",
    "Fit the training data `X_train` using a Count Vectorizer with default parameters.\n",
    "\n",
    "What is the longest token in the vocabulary?\n",
    "\n",
    "*This function should return a string.*"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from sklearn.feature_extraction.text import CountVectorizer\n",
    "\n",
    "def answer_two():\n",
    "    longest = ''\n",
    "    maxi = 0\n",
    "    vect = CountVectorizer().fit(X_train)\n",
    "    for x in vect.vocabulary_.keys():\n",
    "        if len(x) > maxi:\n",
    "            longest = x\n",
    "            maxi = len(x)\n",
    "    return longest"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'com1win150ppmx3age16subscription'"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "answer_two()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Question 3\n",
    "\n",
    "Fit and transform the training data `X_train` using a Count Vectorizer with default parameters.\n",
    "\n",
    "Next, fit a fit a multinomial Naive Bayes classifier model with smoothing `alpha=0.1`. Find the area under the curve (AUC) score using the transformed test data.\n",
    "\n",
    "*This function should return the AUC score as a float.*"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from sklearn.naive_bayes import MultinomialNB\n",
    "from sklearn.metrics import roc_auc_score\n",
    "\n",
    "def answer_three():\n",
    "    vect = CountVectorizer().fit(X_train)\n",
    "    transformed = vect.transform(X_train)\n",
    "    clf = MultinomialNB(alpha = 0.1)\n",
    "    clf.fit(transformed, y_train)\n",
    "    predictions = clf.predict(vect.transform(X_test))\n",
    "    # print('AUC: ', roc_auc_score(y_test, predictions))\n",
    "    return roc_auc_score(y_test, predictions)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.97208121827411165"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "answer_three()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Question 4\n",
    "\n",
    "Fit and transform the training data `X_train` using a Tfidf Vectorizer with default parameters.\n",
    "\n",
    "What 20 features have the smallest tf-idf and what 20 have the largest tf-idf?\n",
    "\n",
    "Put these features in a two series where each series is sorted by tf-idf value and then alphabetically by feature name. The index of the series should be the feature name, and the data should be the tf-idf.\n",
    "\n",
    "The series of 20 features with smallest tf-idfs should be sorted smallest tfidf first, the list of 20 features with largest tf-idfs should be sorted largest first. \n",
    "\n",
    "*This function should return a tuple of two series\n",
    "`(smallest tf-idfs series, largest tf-idfs series)`.*"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from sklearn.feature_extraction.text import TfidfVectorizer\n",
    "\n",
    "def answer_four():\n",
    "    vect = TfidfVectorizer().fit(X_train)\n",
    "    X_train_vectorized = vect.transform(X_train)\n",
    "    feature_names = np.array(vect.get_feature_names())\n",
    "    sorted_tfidf_index = X_train_vectorized.max(0).toarray()[0].argsort()\n",
    "    tf_idf_val = X_train_vectorized.max(0).toarray()[0]\n",
    "    # print(\"here you are\",tf_idf_val)\n",
    "    small_val = [tf_idf_val[i] for i in sorted_tfidf_index[:20]]\n",
    "    #print(small_val)\n",
    "    large_val = [tf_idf_val[i] for i in sorted_tfidf_index[:-21:-1]]\n",
    "    small_name = [feature_names[i] for i in sorted_tfidf_index[:20]]\n",
    "    large_name = [feature_names[i] for i in sorted_tfidf_index[:-21:-1]]\n",
    "    small = list(zip(small_val, small_name))\n",
    "    #print(small)\n",
    "    large = list(zip(large_val, large_name))\n",
    "    small = sorted(small, key=lambda element: (element[0], element[1]))\n",
    "    large = sorted(large, key=lambda element: (element[0], element[1]))\n",
    "    large = large[::-1]\n",
    "    \n",
    "    small_vals = [x for x,y in small]\n",
    "    small_data = [y for x,y in small]\n",
    "    large_vals = [x for x,y in large]\n",
    "    large_data = [y for x,y in large]\n",
    "    \n",
    "    res1 = pd.Series(index = small_data, data = small_vals)\n",
    "    res2 = pd.Series(index = large_data, data = large_vals)\n",
    "    \n",
    "    print(small)\n",
    "    print(large[::-1])\n",
    "    #     print(sorted_tfidf_index[:20])\n",
    "    #     print(\"small\", small_id)\n",
    "    #     large_id = feature_names[sorted_tfidf_index[:-21:-1]]\n",
    "    #     print(large_id)\n",
    "    #     print('Smallest tfidf:\\n{}\\n'.format(feature_names[sorted_tfidf_index[:20]]))\n",
    "    #     print('Largest tfidf: \\n{}'.format(feature_names[sorted_tfidf_index[:-21:-1]]))\n",
    "\n",
    "    #     sorted_coef_index = model.coef_[0].argsort()\n",
    "    #     print('Smallest Coefs:\\n{}\\n'.format(feature_names[sorted_coef_index[:10]]))\n",
    "    #     print('Largest Coefs: \\n{}'.format(feature_names[sorted_coef_index[:-11:-1]]))\n",
    "\n",
    "    return  res1, res2"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[(0.074474523543075904, 'aaniye'), (0.074474523543075904, 'athletic'), (0.074474523543075904, 'chef'), (0.074474523543075904, 'companion'), (0.074474523543075904, 'courageous'), (0.074474523543075904, 'dependable'), (0.074474523543075904, 'determined'), (0.074474523543075904, 'exterminator'), (0.074474523543075904, 'healer'), (0.074474523543075904, 'listener'), (0.074474523543075904, 'organizer'), (0.074474523543075904, 'pest'), (0.074474523543075904, 'psychiatrist'), (0.074474523543075904, 'psychologist'), (0.074474523543075904, 'pudunga'), (0.074474523543075904, 'stylist'), (0.074474523543075904, 'sympathetic'), (0.074474523543075904, 'venaam'), (0.091249611095134395, 'diwali'), (0.091249611095134395, 'mornings')]\n",
      "[(0.93270157742555138, 'blank'), (0.98016589937757925, 'tick'), (1.0, '146tf150p'), (1.0, '645'), (1.0, 'anything'), (1.0, 'anytime'), (1.0, 'beerage'), (1.0, 'done'), (1.0, 'er'), (1.0, 'havent'), (1.0, 'home'), (1.0, 'lei'), (1.0, 'nite'), (1.0, 'ok'), (1.0, 'okie'), (1.0, 'thank'), (1.0, 'thanx'), (1.0, 'too'), (1.0, 'where'), (1.0, 'yup')]\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "(aaniye          0.074475\n",
       " athletic        0.074475\n",
       " chef            0.074475\n",
       " companion       0.074475\n",
       " courageous      0.074475\n",
       " dependable      0.074475\n",
       " determined      0.074475\n",
       " exterminator    0.074475\n",
       " healer          0.074475\n",
       " listener        0.074475\n",
       " organizer       0.074475\n",
       " pest            0.074475\n",
       " psychiatrist    0.074475\n",
       " psychologist    0.074475\n",
       " pudunga         0.074475\n",
       " stylist         0.074475\n",
       " sympathetic     0.074475\n",
       " venaam          0.074475\n",
       " diwali          0.091250\n",
       " mornings        0.091250\n",
       " dtype: float64, yup          1.000000\n",
       " where        1.000000\n",
       " too          1.000000\n",
       " thanx        1.000000\n",
       " thank        1.000000\n",
       " okie         1.000000\n",
       " ok           1.000000\n",
       " nite         1.000000\n",
       " lei          1.000000\n",
       " home         1.000000\n",
       " havent       1.000000\n",
       " er           1.000000\n",
       " done         1.000000\n",
       " beerage      1.000000\n",
       " anytime      1.000000\n",
       " anything     1.000000\n",
       " 645          1.000000\n",
       " 146tf150p    1.000000\n",
       " tick         0.980166\n",
       " blank        0.932702\n",
       " dtype: float64)"
      ]
     },
     "execution_count": 39,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "answer_four()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Question 5\n",
    "\n",
    "Fit and transform the training data `X_train` using a Tfidf Vectorizer ignoring terms that have a document frequency strictly lower than **3**.\n",
    "\n",
    "Then fit a multinomial Naive Bayes classifier model with smoothing `alpha=0.1` and compute the area under the curve (AUC) score using the transformed test data.\n",
    "\n",
    "*This function should return the AUC score as a float.*"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from sklearn.naive_bayes import MultinomialNB\n",
    "from sklearn.metrics import roc_auc_score\n",
    "from sklearn.feature_extraction.text import TfidfVectorizer\n",
    "def answer_five():\n",
    "    vect = TfidfVectorizer(min_df=3).fit(X_train)\n",
    "    X_train_vectorized = vect.transform(X_train)\n",
    "    clf = MultinomialNB(alpha = 0.1)\n",
    "    clf.fit(X_train_vectorized, y_train)\n",
    "    predictions = clf.predict(vect.transform(X_test))\n",
    "    return roc_auc_score(y_test, predictions)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.94162436548223349"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "answer_five()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Question 6\n",
    "\n",
    "What is the average length of documents (number of characters) for not spam and spam documents?\n",
    "\n",
    "*This function should return a tuple (average length not spam, average length spam).*"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def answer_six():\n",
    "    lst1 = [len(x) for x,y in zip(spam_data['text'], spam_data['target']) if y == 1]\n",
    "    lst2 = [len(x) for x,y in zip(spam_data['text'], spam_data['target']) if y == 0]\n",
    "    return (sum(lst2)/len(lst2), sum(lst1)/len(lst1))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(71.02362694300518, 138.8661311914324)"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "answer_six()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<br>\n",
    "<br>\n",
    "The following function has been provided to help you combine new features into the training data:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def add_feature(X, feature_to_add):\n",
    "    \"\"\"\n",
    "    Returns sparse feature matrix with added feature.\n",
    "    feature_to_add can also be a list of features.\n",
    "    \"\"\"\n",
    "    from scipy.sparse import csr_matrix, hstack\n",
    "    return hstack([X, csr_matrix(feature_to_add).T], 'csr')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Question 7\n",
    "\n",
    "Fit and transform the training data X_train using a Tfidf Vectorizer ignoring terms that have a document frequency strictly lower than **5**.\n",
    "\n",
    "Using this document-term matrix and an additional feature, **the length of document (number of characters)**, fit a Support Vector Classification model with regularization `C=10000`. Then compute the area under the curve (AUC) score using the transformed test data.\n",
    "\n",
    "*This function should return the AUC score as a float.*"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 53,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from sklearn.svm import SVC\n",
    "from sklearn import svm\n",
    "from sklearn.naive_bayes import MultinomialNB\n",
    "from sklearn.metrics import roc_auc_score\n",
    "from sklearn.feature_extraction.text import TfidfVectorizer\n",
    "def answer_seven():\n",
    "    vect1 = TfidfVectorizer(min_df=5).fit(X_train)\n",
    "    X_train_vectorized_1 = vect1.transform(X_train)\n",
    "    X_train_vectorized_1 = add_feature(X_train_vectorized_1, [len(i) for i in X_train])\n",
    "    X_test_vectorized = vect1.transform(X_test)\n",
    "    X_test_vectorized = add_feature(X_test_vectorized,[len(i) for i in X_test])\n",
    "    # print(X_train_vectorized_1)\n",
    "    clfrSVM = svm.SVC(C=10000)\n",
    "    clfrSVM.fit(X_train_vectorized_1, y_train)\n",
    "    predictions = clfrSVM.predict(X_test_vectorized)\n",
    "    return  roc_auc_score(y_test, predictions)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 54,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.95813668234215565"
      ]
     },
     "execution_count": 54,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "answer_seven()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Question 8\n",
    "\n",
    "What is the average number of digits per document for not spam and spam documents?\n",
    "\n",
    "*This function should return a tuple (average # digits not spam, average # digits spam).*"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def answer_eight():\n",
    "    import re\n",
    "    lst11 = [len(re.sub(\"[^0-9]\", \"\", x)) for x,y in zip(spam_data['text'], spam_data['target']) if y == 1]\n",
    "    lst22 = [len(re.sub(\"[^0-9]\", \"\", x)) for x,y in zip(spam_data['text'], spam_data['target']) if y == 0]\n",
    "    return (sum(lst22)/len(lst22), sum(lst11)/len(lst11))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(0.2992746113989637, 15.759036144578314)"
      ]
     },
     "execution_count": 34,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "answer_eight()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Question 9\n",
    "\n",
    "Fit and transform the training data `X_train` using a Tfidf Vectorizer ignoring terms that have a document frequency strictly lower than **5** and using **word n-grams from n=1 to n=3** (unigrams, bigrams, and trigrams).\n",
    "\n",
    "Using this document-term matrix and the following additional features:\n",
    "* the length of document (number of characters)\n",
    "* **number of digits per document**\n",
    "\n",
    "fit a Logistic Regression model with regularization `C=100`. Then compute the area under the curve (AUC) score using the transformed test data.\n",
    "\n",
    "*This function should return the AUC score as a float.*"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 51,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from sklearn.linear_model import LogisticRegression\n",
    "from sklearn.svm import SVC\n",
    "from sklearn import svm\n",
    "from sklearn.metrics import roc_auc_score\n",
    "from sklearn.feature_extraction.text import TfidfVectorizer\n",
    "import re\n",
    "def answer_nine():\n",
    "    vect1 = TfidfVectorizer(min_df=5, ngram_range=(1,3)).fit(X_train)\n",
    "    X_train_vectorized_1 = vect1.transform(X_train)\n",
    "    X_train_vectorized_1 = add_feature(X_train_vectorized_1, [len(i) for i in X_train])\n",
    "    X_train_vectorized_1 = add_feature(X_train_vectorized_1,[len(re.sub(\"[^0-9]\", \"\", x)) for x in X_train])\n",
    "    X_test_vectorized = vect1.transform(X_test)\n",
    "    X_test_vectorized = add_feature(X_test_vectorized,[len(i) for i in X_test])\n",
    "    X_test_vectorized = add_feature(X_test_vectorized,[len(re.sub(\"[^0-9]\", \"\", x)) for x in X_test])\n",
    "    model = LogisticRegression(C=100)\n",
    "    model.fit(X_train_vectorized_1, y_train)\n",
    "    predictions = model.predict(X_test_vectorized)\n",
    "    return roc_auc_score(y_test, predictions)#Your answer here"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 52,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.96533283533945646"
      ]
     },
     "execution_count": 52,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "answer_nine()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Question 10\n",
    "\n",
    "What is the average number of non-word characters (anything other than a letter, digit or underscore) per document for not spam and spam documents?\n",
    "\n",
    "*Hint: Use `\\w` and `\\W` character classes*\n",
    "\n",
    "*This function should return a tuple (average # non-word characters not spam, average # non-word characters spam).*"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 58,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def answer_ten():\n",
    "    import re\n",
    "    regx = re.compile('\\W')\n",
    "    #     result = regx.findall(s)\n",
    "    #     lst111 = [len(re.sub(\"[0-9a-zA-Z_]\", \"\", x)) for x,y in zip(spam_data['text'], spam_data['target']) if y == 1]\n",
    "    #     lst222 = [len(re.sub(\"[0-9a-zA-Z_]\", \"\", x)) for x,y in zip(spam_data['text'], spam_data['target']) if y == 0]\n",
    "    lst111 = [len(regx.findall(x)) for x,y in zip(spam_data['text'], spam_data['target']) if y == 1]\n",
    "    lst222 = [len(regx.findall(x)) for x,y in zip(spam_data['text'], spam_data['target']) if y == 0]\n",
    "    return (sum(lst222)/len(lst222), sum(lst111)/len(lst111))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 59,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(17.29181347150259, 29.041499330655956)"
      ]
     },
     "execution_count": 59,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "answer_ten()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Question 11\n",
    "\n",
    "Fit and transform the training data X_train using a Count Vectorizer ignoring terms that have a document frequency strictly lower than **5** and using **character n-grams from n=2 to n=5.**\n",
    "\n",
    "To tell Count Vectorizer to use character n-grams pass in `analyzer='char_wb'` which creates character n-grams only from text inside word boundaries. This should make the model more robust to spelling mistakes.\n",
    "\n",
    "Using this document-term matrix and the following additional features:\n",
    "* the length of document (number of characters)\n",
    "* number of digits per document\n",
    "* **number of non-word characters (anything other than a letter, digit or underscore.)**\n",
    "\n",
    "fit a Logistic Regression model with regularization C=100. Then compute the area under the curve (AUC) score using the transformed test data.\n",
    "\n",
    "Also **find the 10 smallest and 10 largest coefficients from the model** and return them along with the AUC score in a tuple.\n",
    "\n",
    "The list of 10 smallest coefficients should be sorted smallest first, the list of 10 largest coefficients should be sorted largest first.\n",
    "\n",
    "The three features that were added to the document term matrix should have the following names should they appear in the list of coefficients:\n",
    "['length_of_doc', 'digit_count', 'non_word_char_count']\n",
    "\n",
    "*This function should return a tuple `(AUC score as a float, smallest coefs list, largest coefs list)`.*"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 68,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from sklearn.linear_model import LogisticRegression\n",
    "from sklearn.metrics import roc_auc_score\n",
    "from sklearn.feature_extraction.text import CountVectorizer\n",
    "import re\n",
    "def answer_eleven():\n",
    "    regx = re.compile('\\W')\n",
    "    vect1 = CountVectorizer(min_df=5, ngram_range=(2,5), analyzer='char_wb').fit(X_train)\n",
    "    X_train_vectorized_1 = vect1.transform(X_train)\n",
    "    X_train_vectorized_1 = add_feature(X_train_vectorized_1, [len(i) for i in X_train])\n",
    "    X_train_vectorized_1 = add_feature(X_train_vectorized_1,[len(re.sub(\"[^0-9]\", \"\", x)) for x in X_train])\n",
    "    X_train_vectorized_1 = add_feature(X_train_vectorized_1,[len(regx.findall(x)) for x in X_train])\n",
    "    X_test_vectorized = vect1.transform(X_test)\n",
    "    X_test_vectorized = add_feature(X_test_vectorized,[len(i) for i in X_test])\n",
    "    X_test_vectorized = add_feature(X_test_vectorized,[len(re.sub(\"[^0-9]\", \"\", x)) for x in X_test])\n",
    "    X_test_vectorized = add_feature(X_test_vectorized,[len(regx.findall(x)) for x in X_test])\n",
    "    model = LogisticRegression(C=100)\n",
    "    model.fit(X_train_vectorized_1, y_train)\n",
    "    predictions = model.predict(X_test_vectorized)\n",
    "    AUC = roc_auc_score(y_test, predictions)\n",
    "    sorted_coef_index = model.coef_[0].argsort()\n",
    "    res_small = sorted_coef_index[:10]\n",
    "    res_large = sorted_coef_index[:-11:-1]\n",
    "    return AUC, sorted(sorted_coef_index[:10]), sorted(sorted_coef_index[:-11:-1], reverse = True)\n",
    "    "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 69,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(0.97885931107074342,\n",
       " [1143, 1189, 1310, 1620, 2907, 3218, 3227, 4471, 4520, 8245],\n",
       " [16315, 15949, 15855, 10911, 10589, 8804, 6160, 5158, 2893, 658])"
      ]
     },
     "execution_count": 69,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "answer_eleven()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "coursera": {
   "course_slug": "python-text-mining",
   "graded_item_id": "Pn19K",
   "launcher_item_id": "y1juS",
   "part_id": "ctlgo"
  },
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.2"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}