book/NLP_chapter2.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "provenance": [],
      "authorship_tag": "ABX9TyOq3I8esMSuvU66hJh7Ulmt",
      "include_colab_link": true
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    },
    "language_info": {
      "name": "python"
    }
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/giorgiosld/Natural-Language-Processing/blob/main/book/NLP_chapter2.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "# Accessing Text Corpora and Lexical Resources"
      ],
      "metadata": {
        "id": "h5oBpAvKpEuV"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Accessing Text Corpora\n",
        "\n",
        "Text corpus is a large body of text designt to contain a careful balance of material in one or more genres.\n"
      ],
      "metadata": {
        "id": "lQnzdTUWpCpd"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "import nltk\n",
        "\n",
        "nltk.download('gutenberg')\n",
        "nltk.corpus.gutenberg.fileids()\n",
        "\n",
        "emma = nltk.corpus.gutenberg.words(\"austen-emma.txt\")\n",
        "len(emma)\n",
        "\n",
        "# if you want see the concordancing of a text like in the first chapter\n",
        "emma2 = nltk.Text(nltk.corpus.gutenberg.words(\"austen-emma.txt\"))\n",
        "emma2.concordance(\"surprize\")\n",
        "\n",
        "# to be more coincise python allows...\n",
        "from nltk.corpus import gutenberg\n",
        "gutenberg.fileids()\n",
        "emma = gutenberg.words(\"austen-emma.txt\")\n"
      ],
      "metadata": {
        "id": "f6AeNt_g4Sq7"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "With the for loop we can display three statistics for each text like average word length, average sentence length and number of times each vocabulary item appears in the text. Is possible notice that the average word length is 4 (actually 3 but the function counts also space)."
      ],
      "metadata": {
        "id": "nlL6mtj4tjcQ"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# decomment this line the first time\n",
        "# nltk.download('punkt')\n",
        "\n",
        "for fileid in gutenberg.fileids():\n",
        "  n_chars = len(gutenberg.raw(fileid))\n",
        "  n_words = len(gutenberg.words(fileid))\n",
        "  n_sents = len(gutenberg.sents(fileid))\n",
        "  n_vocab = len(set(w.lower() for w in gutenberg.words(fileid)))\n",
        "  print(round(n_chars/n_words), round(n_words/n_sents), round(n_words/n_vocab), fileid)"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "vklXmKAPpgfi",
        "outputId": "b3bed656-4607-4c58-ba5f-d099d1ce4504"
      },
      "execution_count": 15,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stderr",
          "text": [
            "[nltk_data] Downloading package punkt to /root/nltk_data...\n",
            "[nltk_data]   Package punkt is already up-to-date!\n"
          ]
        },
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "5 25 26 austen-emma.txt\n",
            "5 26 17 austen-persuasion.txt\n",
            "5 28 22 austen-sense.txt\n",
            "4 34 79 bible-kjv.txt\n",
            "5 19 5 blake-poems.txt\n",
            "4 19 14 bryant-stories.txt\n",
            "4 18 12 burgess-busterbrown.txt\n",
            "4 20 13 carroll-alice.txt\n",
            "5 20 12 chesterton-ball.txt\n",
            "5 23 11 chesterton-brown.txt\n",
            "5 18 11 chesterton-thursday.txt\n",
            "4 21 25 edgeworth-parents.txt\n",
            "5 26 15 melville-moby_dick.txt\n",
            "5 52 11 milton-paradise.txt\n",
            "4 12 9 shakespeare-caesar.txt\n",
            "4 12 8 shakespeare-hamlet.txt\n",
            "4 12 7 shakespeare-macbeth.txt\n",
            "5 36 12 whitman-leaves.txt\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "We see that the raw() function is used to gives us the contents of the file without any linguistic processing. Instead sents() divides the text up into its sentences."
      ],
      "metadata": {
        "id": "Uv3fQwqg5ndl"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "macbeth_sentences = gutenberg.sents(\"shakespeare-macbeth.txt\")\n",
        "macbeth_sentences\n",
        "macbeth_sentences[1116]\n",
        "longest_len = max(len(s) for s in macbeth_sentences)\n",
        "[s for s in macbeth_sentences if len(s) == longest_len]"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "DObQ084R56L7",
        "outputId": "29db93b3-80f2-4ca4-baa7-c6ed1fa6679d"
      },
      "execution_count": 21,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "[['Doubtfull',\n",
              "  'it',\n",
              "  'stood',\n",
              "  ',',\n",
              "  'As',\n",
              "  'two',\n",
              "  'spent',\n",
              "  'Swimmers',\n",
              "  ',',\n",
              "  'that',\n",
              "  'doe',\n",
              "  'cling',\n",
              "  'together',\n",
              "  ',',\n",
              "  'And',\n",
              "  'choake',\n",
              "  'their',\n",
              "  'Art',\n",
              "  ':',\n",
              "  'The',\n",
              "  'mercilesse',\n",
              "  'Macdonwald',\n",
              "  '(',\n",
              "  'Worthie',\n",
              "  'to',\n",
              "  'be',\n",
              "  'a',\n",
              "  'Rebell',\n",
              "  ',',\n",
              "  'for',\n",
              "  'to',\n",
              "  'that',\n",
              "  'The',\n",
              "  'multiplying',\n",
              "  'Villanies',\n",
              "  'of',\n",
              "  'Nature',\n",
              "  'Doe',\n",
              "  'swarme',\n",
              "  'vpon',\n",
              "  'him',\n",
              "  ')',\n",
              "  'from',\n",
              "  'the',\n",
              "  'Westerne',\n",
              "  'Isles',\n",
              "  'Of',\n",
              "  'Kernes',\n",
              "  'and',\n",
              "  'Gallowgrosses',\n",
              "  'is',\n",
              "  'supply',\n",
              "  \"'\",\n",
              "  'd',\n",
              "  ',',\n",
              "  'And',\n",
              "  'Fortune',\n",
              "  'on',\n",
              "  'his',\n",
              "  'damned',\n",
              "  'Quarry',\n",
              "  'smiling',\n",
              "  ',',\n",
              "  'Shew',\n",
              "  \"'\",\n",
              "  'd',\n",
              "  'like',\n",
              "  'a',\n",
              "  'Rebells',\n",
              "  'Whore',\n",
              "  ':',\n",
              "  'but',\n",
              "  'all',\n",
              "  \"'\",\n",
              "  's',\n",
              "  'too',\n",
              "  'weake',\n",
              "  ':',\n",
              "  'For',\n",
              "  'braue',\n",
              "  'Macbeth',\n",
              "  '(',\n",
              "  'well',\n",
              "  'hee',\n",
              "  'deserues',\n",
              "  'that',\n",
              "  'Name',\n",
              "  ')',\n",
              "  'Disdayning',\n",
              "  'Fortune',\n",
              "  ',',\n",
              "  'with',\n",
              "  'his',\n",
              "  'brandisht',\n",
              "  'Steele',\n",
              "  ',',\n",
              "  'Which',\n",
              "  'smoak',\n",
              "  \"'\",\n",
              "  'd',\n",
              "  'with',\n",
              "  'bloody',\n",
              "  'execution',\n",
              "  '(',\n",
              "  'Like',\n",
              "  'Valours',\n",
              "  'Minion',\n",
              "  ')',\n",
              "  'caru',\n",
              "  \"'\",\n",
              "  'd',\n",
              "  'out',\n",
              "  'his',\n",
              "  'passage',\n",
              "  ',',\n",
              "  'Till',\n",
              "  'hee',\n",
              "  'fac',\n",
              "  \"'\",\n",
              "  'd',\n",
              "  'the',\n",
              "  'Slaue',\n",
              "  ':',\n",
              "  'Which',\n",
              "  'neu',\n",
              "  \"'\",\n",
              "  'r',\n",
              "  'shooke',\n",
              "  'hands',\n",
              "  ',',\n",
              "  'nor',\n",
              "  'bad',\n",
              "  'farwell',\n",
              "  'to',\n",
              "  'him',\n",
              "  ',',\n",
              "  'Till',\n",
              "  'he',\n",
              "  'vnseam',\n",
              "  \"'\",\n",
              "  'd',\n",
              "  'him',\n",
              "  'from',\n",
              "  'the',\n",
              "  'Naue',\n",
              "  'toth',\n",
              "  \"'\",\n",
              "  'Chops',\n",
              "  ',',\n",
              "  'And',\n",
              "  'fix',\n",
              "  \"'\",\n",
              "  'd',\n",
              "  'his',\n",
              "  'Head',\n",
              "  'vpon',\n",
              "  'our',\n",
              "  'Battlements']]"
            ]
          },
          "metadata": {},
          "execution_count": 21
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "### Brown Corpus\n",
        "In this example it has been used a dataset collected by the Brown Univesity for studying systematic differences between genres, a kind of linguistic inquiry known as **stylistic**.  "
      ],
      "metadata": {
        "id": "ZuEs7vSb6juu"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "import nltk\n",
        "\n",
        "nltk.download(\"brown\")\n",
        "\n",
        "brown.categories()\n",
        "brown.words(categories=\"news\")\n",
        "brown.words(fileids=[\"cg22\"])\n",
        "brown.sents(categories=[\"news\", \"editorial\", \"reviews\"])"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "yeCQnlBF75P5",
        "outputId": "4d9d1213-1965-4489-97b0-da9dcf378ce6"
      },
      "execution_count": 27,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stderr",
          "text": [
            "[nltk_data] Downloading package brown to /root/nltk_data...\n",
            "[nltk_data]   Package brown is already up-to-date!\n"
          ]
        },
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "[['The', 'Fulton', 'County', 'Grand', 'Jury', 'said', 'Friday', 'an', 'investigation', 'of', \"Atlanta's\", 'recent', 'primary', 'election', 'produced', '``', 'no', 'evidence', \"''\", 'that', 'any', 'irregularities', 'took', 'place', '.'], ['The', 'jury', 'further', 'said', 'in', 'term-end', 'presentments', 'that', 'the', 'City', 'Executive', 'Committee', ',', 'which', 'had', 'over-all', 'charge', 'of', 'the', 'election', ',', '``', 'deserves', 'the', 'praise', 'and', 'thanks', 'of', 'the', 'City', 'of', 'Atlanta', \"''\", 'for', 'the', 'manner', 'in', 'which', 'the', 'election', 'was', 'conducted', '.'], ...]"
            ]
          },
          "metadata": {},
          "execution_count": 27
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "Now compare genres in their usage of modal verbs producing the counts for a particular genre."
      ],
      "metadata": {
        "id": "mF2ScqoN8ncV"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "news_text = brown.words(categories=\"news\")\n",
        "fdist = nltk.FreqDist(w.lower() for w in news_text)\n",
        "modals = ['can', 'could', 'may', 'might', 'must', 'will']\n",
        "for m in modals:\n",
        "  print(f\"{m}: {fdist[m]}\")"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "OpXVVnaA8wIt",
        "outputId": "afce9e02-324f-4569-b4b0-1e205a7f014e"
      },
      "execution_count": 28,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "can: 94\n",
            "could: 87\n",
            "may: 93\n",
            "might: 38\n",
            "must: 53\n",
            "will: 389\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "The next step is to obtain the counts for each genre of interest using the conditional frequency distribution."
      ],
      "metadata": {
        "id": "Es7Ov3ps9gP3"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "cfd = nltk.ConditionalFreqDist((genre, word)\n",
        "                              for genre in brown.categories()\n",
        "                              for word in brown.words(categories=genre))\n",
        "genres = [\"news\", \"religion\", \"hobbies\", \"science_fiction\", \"romance\", \"humor\"]\n",
        "modals = [\"can\", \"could\", \"may\", \"might\", \"must\", \"will\"]\n",
        "cfd.tabulate(condition=genres, samples=modals)"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "cncenFBm93NI",
        "outputId": "a23e10c4-3255-40a6-f5d2-1bfde1f3bde6"
      },
      "execution_count": 29,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "                  can could   may might  must  will \n",
            "      adventure    46   151     5    58    27    50 \n",
            " belles_lettres   246   213   207   113   170   236 \n",
            "      editorial   121    56    74    39    53   233 \n",
            "        fiction    37   166     8    44    55    52 \n",
            "     government   117    38   153    13   102   244 \n",
            "        hobbies   268    58   131    22    83   264 \n",
            "          humor    16    30     8     8     9    13 \n",
            "        learned   365   159   324   128   202   340 \n",
            "           lore   170   141   165    49    96   175 \n",
            "        mystery    42   141    13    57    30    20 \n",
            "           news    93    86    66    38    50   389 \n",
            "       religion    82    59    78    12    54    71 \n",
            "        reviews    45    40    45    26    19    58 \n",
            "        romance    74   193    11    51    45    43 \n",
            "science_fiction    16    49     4    12     8    16 \n"
          ]
        }
      ]
    }
  ]
}