04_intro_to_llms/Homework_1.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Let's write an elementary tokenizer that uses words as tokens.\n",
    "\n",
    "We will use Mark Twain's _Life On The Mississippi_ as a test bed. The text is in the accompanying file 'Life_On_The_Mississippi.txt'\n",
    "\n",
    "Here's a not-terribly-good such tokenizer:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "('\\ufeffThe', 1)\n",
      "('Project', 79)\n",
      "('Gutenberg', 22)\n",
      "('eBook', 4)\n",
      "('of', 4469)\n",
      "('Life', 5)\n",
      "('on', 856)\n",
      "('the', 8443)\n",
      "('Mississippi', 104)\n",
      "('This', 127)\n",
      "('ebook', 2)\n",
      "('is', 1076)\n",
      "('for', 1017)\n",
      "('use', 34)\n",
      "('anyone', 4)\n",
      "('anywhere', 8)\n",
      "('in', 2381)\n",
      "('United', 36)\n",
      "('States', 26)\n",
      "('and', 5692)\n",
      "('most', 119)\n",
      "('other', 223)\n",
      "('parts', 5)\n",
      "('world', 40)\n",
      "('at', 676)\n",
      "('no', 325)\n",
      "('cost', 18)\n",
      "('with', 1053)\n",
      "('almost', 37)\n",
      "('restrictions', 2)\n",
      "('whatsoever.', 2)\n",
      "('You', 92)\n",
      "('may', 85)\n",
      "('copy', 12)\n",
      "('it,', 199)\n",
      "('give', 67)\n",
      "('it', 1382)\n",
      "('away', 107)\n",
      "('or', 561)\n",
      "('re-use', 2)\n",
      "('under', 112)\n",
      "('terms', 22)\n",
      "('License', 8)\n",
      "('included', 2)\n",
      "('this', 591)\n",
      "('online', 4)\n",
      "('www.gutenberg.org.', 4)\n",
      "('If', 85)\n",
      "('you', 813)\n",
      "('are', 361)\n",
      "('not', 680)\n",
      "('located', 9)\n",
      "('States,', 8)\n",
      "('will', 287)\n",
      "('have', 557)\n",
      "('to', 3518)\n",
      "('check', 4)\n",
      "('laws', 13)\n",
      "('country', 50)\n",
      "('where', 152)\n",
      "('before', 150)\n",
      "('using', 10)\n",
      "('eBook.', 2)\n",
      "('Title:', 1)\n",
      "('Author:', 1)\n",
      "('Mark', 2)\n",
      "('Twain', 2)\n",
      "('Release', 1)\n",
      "('date:', 1)\n",
      "('July', 7)\n",
      "('10,', 2)\n",
      "('2004', 1)\n",
      "('[eBook', 1)\n",
      "('#245]', 1)\n",
      "('Most', 4)\n",
      "('recently', 3)\n",
      "('updated:', 1)\n",
      "('January', 2)\n",
      "('1,', 2)\n",
      "('2021', 1)\n",
      "('Language:', 1)\n",
      "('English', 7)\n",
      "('Credits:', 1)\n",
      "('Produced', 2)\n",
      "('by', 623)\n",
      "('David', 2)\n",
      "('Widger.', 2)\n",
      "('Earliest', 2)\n",
      "('PG', 3)\n",
      "('text', 4)\n",
      "('edition', 3)\n",
      "('produced', 15)\n",
      "('Graham', 2)\n",
      "('Allan', 2)\n",
      "('***', 4)\n",
      "('START', 1)\n",
      "('OF', 16)\n",
      "('THE', 29)\n",
      "('PROJECT', 4)\n",
      "('GUTENBERG', 3)\n"
     ]
    }
   ],
   "source": [
    "wdict = {}\n",
    "with open('Life_On_The_Mississippi.txt', 'r') as L:\n",
    "    line = L.readline()\n",
    "    nlines = 1\n",
    "    while line:\n",
    "\n",
    "        words = line.split()\n",
    "        for word in words:\n",
    "            if wdict.get(word) is not None:\n",
    "                wdict[word] += 1\n",
    "            else:\n",
    "                wdict[word] = 1\n",
    "        line = L.readline()\n",
    "        nlines += 1\n",
    "\n",
    "nitem = 0 ; maxitems = 100\n",
    "for item in wdict.items():\n",
    "    nitem += 1\n",
    "    print(item)\n",
    "    if nitem == maxitems: break\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This is unsatisfactory for a few reasons:\n",
    "\n",
    "* There are non-ASCII (Unicode) characters that should be stripped (the so-called \"Byte-Order Mark\" or BOM \\ufeff at the beginning of the text);\n",
    "\n",
    "* There are punctuation marks, which we don't want to concern ourselves with;\n",
    "\n",
    "* The same word can appear capitalized, or lower-case, or with its initial letter upper-cased, whereas we want them all to be normalized to lower-case.\n",
    "\n",
    "Part 1 of this assignment: insert code in this loop to operate on the str variable 'line' so as to fix these problems before 'line' is split into words.\n",
    "\n",
    "A hint to one possible way to do this: use the 'punctuation' character definition in the Python 'string' module, the 'maketrans' and 'translate' methods of Python's str class, to eliminate punctuation, and the regular expression ('re') Python module to eliminate any Unicode---it is useful to know that the regular expression r'[^\\x00-x7f]' means \"any character not in the vanilla ASCII set.\n",
    "\n",
    "Part 2: Add code to sort the contents of wdict by word occurrence frequency.  What are the top 100 most frequent word tokens?  Adding up occurrence frequencies starting from the most frequent words, how many distinct words make up the top 90% of word occurrences in this \"corpus\"?\n",
    "\n",
    "For this part, the docs of Python's 'sorted' and of the helper 'itemgetter' from 'operator' reward study.\n",
    "\n",
    "Write your modified code in the cell below."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "pytorch.venv",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.12"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}