English_Learner_Project.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "provenance": [],
      "authorship_tag": "ABX9TyNmZHrra5niPWbW+W1CvbZg"
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    },
    "language_info": {
      "name": "python"
    }
  },
  "cells": [
    {
      "cell_type": "markdown",
      "source": [
        "Writing is a foundational skill. Sadly, it's one few students are able to hone, often because writing tasks are infrequently assigned in school. A rapidly growing student population, students learning English as a second language, known as English Language Learners (ELLs), are especially affected by the lack of practice. While automated feedback tools make it easier for teachers to assign more writing tasks, they are not designed with ELLs in mind.\n",
        "\n",
        "Existing tools are unable to provide feedback based on the language proficiency of the student, resulting in a final evaluation that may be skewed against the learner. Data science may be able to improve automated feedback tools to better support the unique needs of these learners."
      ],
      "metadata": {
        "id": "V-VYdxlxvQKb"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "!pip install afinn"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "ZB265e9w3-J3",
        "outputId": "59786aa5-3446-4067-a541-0613d91181bf"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Looking in indexes: https://pypi.org/simple, https://us-python.pkg.dev/colab-wheels/public/simple/\n",
            "Collecting afinn\n",
            "  Downloading afinn-0.1.tar.gz (52 kB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m52.6/52.6 KB\u001b[0m \u001b[31m2.9 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25h  Preparing metadata (setup.py) ... \u001b[?25l\u001b[?25hdone\n",
            "Building wheels for collected packages: afinn\n",
            "  Building wheel for afinn (setup.py) ... \u001b[?25l\u001b[?25hdone\n",
            "  Created wheel for afinn: filename=afinn-0.1-py3-none-any.whl size=53447 sha256=7c9d6cff91fae0e5e3d7f8a4433f5e8edfbcc54646d45971822cea438d46401c\n",
            "  Stored in directory: /root/.cache/pip/wheels/f6/6f/c3/b305c5107a17618f2938a067d5ffcbb556909d82398762089e\n",
            "Successfully built afinn\n",
            "Installing collected packages: afinn\n",
            "Successfully installed afinn-0.1\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "V6wteLA_sFMD"
      },
      "outputs": [],
      "source": [
        "import pandas as pd\n",
        "import numpy as np\n",
        "import re\n",
        "import os\n",
        "import matplotlib.pyplot as plt\n",
        "import seaborn as sns\n",
        "from afinn import Afinn"
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "data = pd.read_csv('train.csv')"
      ],
      "metadata": {
        "id": "a4zyIoLDudC2"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "data"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 6914
        },
        "id": "crOHTqJEunN_",
        "outputId": "632e196f-6ae4-4430-b2dd-aa123c02e71c"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "           text_id                                          full_text  \\\n",
              "0     0016926B079C  I think that students would benefit from learn...   \n",
              "1     0022683E9EA5  When a problem is a change you have to let it ...   \n",
              "2     00299B378633  Dear, Principal\\n\\nIf u change the school poli...   \n",
              "3     003885A45F42  The best time in life is when you become yours...   \n",
              "4     0049B1DF5CCC  Small act of kindness can impact in other peop...   \n",
              "...            ...                                                ...   \n",
              "3906  FFD29828A873  I believe using cellphones in class for educat...   \n",
              "3907  FFD9A83B0849  Working alone, students do not have to argue w...   \n",
              "3908  FFDC4011AC9C  \"A problem is a chance for you to do your best...   \n",
              "3909  FFE16D704B16  Many people disagree with Albert Schweitzer's ...   \n",
              "3910  FFED00D6E0BD  Do you think that failure is the main thing fo...   \n",
              "\n",
              "      cohesion  syntax  vocabulary  phraseology  grammar  conventions  \n",
              "0          3.5     3.5         3.0          3.0      4.0          3.0  \n",
              "1          2.5     2.5         3.0          2.0      2.0          2.5  \n",
              "2          3.0     3.5         3.0          3.0      3.0          2.5  \n",
              "3          4.5     4.5         4.5          4.5      4.0          5.0  \n",
              "4          2.5     3.0         3.0          3.0      2.5          2.5  \n",
              "...        ...     ...         ...          ...      ...          ...  \n",
              "3906       2.5     3.0         3.0          3.5      2.5          2.5  \n",
              "3907       4.0     4.0         4.0          4.0      3.5          3.0  \n",
              "3908       2.5     3.0         3.0          3.0      3.5          3.0  \n",
              "3909       4.0     4.5         4.5          4.0      4.5          4.5  \n",
              "3910       3.5     2.5         3.5          3.0      3.0          3.5  \n",
              "\n",
              "[3911 rows x 8 columns]"
            ],
            "text/html": [
              "\n",
              "  <div id=\"df-0a21a538-77c4-42aa-9492-6b8307ec3a55\">\n",
              "    <div class=\"colab-df-container\">\n",
              "      <div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>text_id</th>\n",
              "      <th>full_text</th>\n",
              "      <th>cohesion</th>\n",
              "      <th>syntax</th>\n",
              "      <th>vocabulary</th>\n",
              "      <th>phraseology</th>\n",
              "      <th>grammar</th>\n",
              "      <th>conventions</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>0016926B079C</td>\n",
              "      <td>I think that students would benefit from learn...</td>\n",
              "      <td>3.5</td>\n",
              "      <td>3.5</td>\n",
              "      <td>3.0</td>\n",
              "      <td>3.0</td>\n",
              "      <td>4.0</td>\n",
              "      <td>3.0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>0022683E9EA5</td>\n",
              "      <td>When a problem is a change you have to let it ...</td>\n",
              "      <td>2.5</td>\n",
              "      <td>2.5</td>\n",
              "      <td>3.0</td>\n",
              "      <td>2.0</td>\n",
              "      <td>2.0</td>\n",
              "      <td>2.5</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>00299B378633</td>\n",
              "      <td>Dear, Principal\\n\\nIf u change the school poli...</td>\n",
              "      <td>3.0</td>\n",
              "      <td>3.5</td>\n",
              "      <td>3.0</td>\n",
              "      <td>3.0</td>\n",
              "      <td>3.0</td>\n",
              "      <td>2.5</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>003885A45F42</td>\n",
              "      <td>The best time in life is when you become yours...</td>\n",
              "      <td>4.5</td>\n",
              "      <td>4.5</td>\n",
              "      <td>4.5</td>\n",
              "      <td>4.5</td>\n",
              "      <td>4.0</td>\n",
              "      <td>5.0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>0049B1DF5CCC</td>\n",
              "      <td>Small act of kindness can impact in other peop...</td>\n",
              "      <td>2.5</td>\n",
              "      <td>3.0</td>\n",
              "      <td>3.0</td>\n",
              "      <td>3.0</td>\n",
              "      <td>2.5</td>\n",
              "      <td>2.5</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>...</th>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3906</th>\n",
              "      <td>FFD29828A873</td>\n",
              "      <td>I believe using cellphones in class for educat...</td>\n",
              "      <td>2.5</td>\n",
              "      <td>3.0</td>\n",
              "      <td>3.0</td>\n",
              "      <td>3.5</td>\n",
              "      <td>2.5</td>\n",
              "      <td>2.5</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3907</th>\n",
              "      <td>FFD9A83B0849</td>\n",
              "      <td>Working alone, students do not have to argue w...</td>\n",
              "      <td>4.0</td>\n",
              "      <td>4.0</td>\n",
              "      <td>4.0</td>\n",
              "      <td>4.0</td>\n",
              "      <td>3.5</td>\n",
              "      <td>3.0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3908</th>\n",
              "      <td>FFDC4011AC9C</td>\n",
              "      <td>\"A problem is a chance for you to do your best...</td>\n",
              "      <td>2.5</td>\n",
              "      <td>3.0</td>\n",
              "      <td>3.0</td>\n",
              "      <td>3.0</td>\n",
              "      <td>3.5</td>\n",
              "      <td>3.0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3909</th>\n",
              "      <td>FFE16D704B16</td>\n",
              "      <td>Many people disagree with Albert Schweitzer's ...</td>\n",
              "      <td>4.0</td>\n",
              "      <td>4.5</td>\n",
              "      <td>4.5</td>\n",
              "      <td>4.0</td>\n",
              "      <td>4.5</td>\n",
              "      <td>4.5</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3910</th>\n",
              "      <td>FFED00D6E0BD</td>\n",
              "      <td>Do you think that failure is the main thing fo...</td>\n",
              "      <td>3.5</td>\n",
              "      <td>2.5</td>\n",
              "      <td>3.5</td>\n",
              "      <td>3.0</td>\n",
              "      <td>3.0</td>\n",
              "      <td>3.5</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "<p>3911 rows × 8 columns</p>\n",
              "</div>\n",
              "      <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-0a21a538-77c4-42aa-9492-6b8307ec3a55')\"\n",
              "              title=\"Convert this dataframe to an interactive table.\"\n",
              "              style=\"display:none;\">\n",
              "        \n",
              "  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n",
              "       width=\"24px\">\n",
              "    <path d=\"M0 0h24v24H0V0z\" fill=\"none\"/>\n",
              "    <path d=\"M18.56 5.44l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94zm-11 1L8.5 8.5l.94-2.06 2.06-.94-2.06-.94L8.5 2.5l-.94 2.06-2.06.94zm10 10l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94z\"/><path d=\"M17.41 7.96l-1.37-1.37c-.4-.4-.92-.59-1.43-.59-.52 0-1.04.2-1.43.59L10.3 9.45l-7.72 7.72c-.78.78-.78 2.05 0 2.83L4 21.41c.39.39.9.59 1.41.59.51 0 1.02-.2 1.41-.59l7.78-7.78 2.81-2.81c.8-.78.8-2.07 0-2.86zM5.41 20L4 18.59l7.72-7.72 1.47 1.35L5.41 20z\"/>\n",
              "  </svg>\n",
              "      </button>\n",
              "      \n",
              "  <style>\n",
              "    .colab-df-container {\n",
              "      display:flex;\n",
              "      flex-wrap:wrap;\n",
              "      gap: 12px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert {\n",
              "      background-color: #E8F0FE;\n",
              "      border: none;\n",
              "      border-radius: 50%;\n",
              "      cursor: pointer;\n",
              "      display: none;\n",
              "      fill: #1967D2;\n",
              "      height: 32px;\n",
              "      padding: 0 0 0 0;\n",
              "      width: 32px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert:hover {\n",
              "      background-color: #E2EBFA;\n",
              "      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "      fill: #174EA6;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert {\n",
              "      background-color: #3B4455;\n",
              "      fill: #D2E3FC;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert:hover {\n",
              "      background-color: #434B5C;\n",
              "      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n",
              "      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n",
              "      fill: #FFFFFF;\n",
              "    }\n",
              "  </style>\n",
              "\n",
              "      <script>\n",
              "        const buttonEl =\n",
              "          document.querySelector('#df-0a21a538-77c4-42aa-9492-6b8307ec3a55 button.colab-df-convert');\n",
              "        buttonEl.style.display =\n",
              "          google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "\n",
              "        async function convertToInteractive(key) {\n",
              "          const element = document.querySelector('#df-0a21a538-77c4-42aa-9492-6b8307ec3a55');\n",
              "          const dataTable =\n",
              "            await google.colab.kernel.invokeFunction('convertToInteractive',\n",
              "                                                     [key], {});\n",
              "          if (!dataTable) return;\n",
              "\n",
              "          const docLinkHtml = 'Like what you see? Visit the ' +\n",
              "            '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n",
              "            + ' to learn more about interactive tables.';\n",
              "          element.innerHTML = '';\n",
              "          dataTable['output_type'] = 'display_data';\n",
              "          await google.colab.output.renderOutput(dataTable, element);\n",
              "          const docLink = document.createElement('div');\n",
              "          docLink.innerHTML = docLinkHtml;\n",
              "          element.appendChild(docLink);\n",
              "        }\n",
              "      </script>\n",
              "    </div>\n",
              "  </div>\n",
              "  "
            ]
          },
          "metadata": {},
          "execution_count": 6
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "data_text = data['full_text']"
      ],
      "metadata": {
        "id": "k4SGzpI52Goy"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "afinn = Afinn(language='en')\n",
        "\n",
        "data['sentiment score'] = data['full_text'].apply(afinn.score)"
      ],
      "metadata": {
        "id": "dgTl48Vxustv"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [],
      "metadata": {
        "id": "WN6Yd-ynutxy"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [],
      "metadata": {
        "id": "KCGwDreout5P"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [],
      "metadata": {
        "id": "EJaNo91K8UeT"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [],
      "metadata": {
        "id": "Jpn5NHlM8UmN"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [],
      "metadata": {
        "id": "Re0Et4NT8Uqy"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [],
      "metadata": {
        "id": "KNrUNYG18Uuj"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [],
      "metadata": {
        "id": "kY-TCuRh8Uxj"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [],
      "metadata": {
        "id": "bIXhayOe8U0j"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [],
      "metadata": {
        "id": "GKQXVIHV8U38"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [],
      "metadata": {
        "id": "KCU9M7Y78U7M"
      },
      "execution_count": null,
      "outputs": []
    }
  ]
}