dicts_sets_numpy.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Recitation 3\n",
    "\n",
    "Welcome back! This recitation will cover the following topics:\n",
    "- Sets\n",
    "- Dictionaries\n",
    "- File I/O\n",
    "- Numpy"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Task 1\n",
    "\n",
    "Bee Movie is a 2007 American computer-animated comedy film produced by DreamWorks Animation and starring Jerry Seinfeld. The film centers on Barry B. Benson (Seinfeld), a honey bee who tries to sue the human race for exploiting bees after learning from his florist friend Vanessa Bloome (Zellweger) that humans sell and consume honey. [Source](https://en.wikipedia.org/wiki/Bee_Movie)\n",
    "\n",
    "Your bee studies class (BEE 210) has tasked you with homework telling you to break down the script of the movie. They have asked you to provide the following things:\n",
    "- A text file giving every word that appears in the film, along with the frequency of each word (return a dictionary as well) **NOTE:** Try to replace as much punctuation as you can, this includes question marks, periods, and colons.\n",
    "- Output the top ten words which appeared in the film along with their frequencies\n",
    "- Output giving the number of times the characters: Bee Gandhi, Ralph Lauren, Larry King, and bejesus appear in the script. If a character did not appear in the script, catch the error and output a nice sentance telling the user that the character did not appear."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# code to download the bee movie text\n",
    "!wget https://courses.cs.washington.edu/courses/cse163/21su/files/lectures/L04/bee-movie.txt"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Function that takes in a text file name and writes a \n",
    "# text file with the frequencies of each word also return the dictionary you create\n",
    "def find_frequencies(file_name: str = \"bee_frequencies.txt\", script_file: str = \"bee-movie.txt\") -> dict:\n",
    "  pass"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Answer\n",
    "\n",
    "```python\n",
    "\n",
    "def find_frequencies(file_name: str = \"bee_frequencies.txt\", script_file: str = \"bee-movie.txt\") -> dict:\n",
    "  freq_dict = {}\n",
    "\n",
    "  with open(script_file) as f:\n",
    "    for line in f:\n",
    "      line = line.strip()\n",
    "      line = line.replace(\":\", \"\").replace(\".\", \"\").replace(\"?\", \"\").replace(\")\", \"\").replace(\"(\", \"\").lower()\n",
    "      words = line.split()\n",
    "\n",
    "      for word in words:\n",
    "        if word in freq_dict:\n",
    "          freq_dict.update({word: freq_dict[word] + 1})\n",
    "        else:\n",
    "          freq_dict.update({word: 1})\n",
    "\n",
    "  ## Option 1\n",
    "  with open(file_name, \"w+\") as f:\n",
    "    for word in freq_dict:\n",
    "      f.write(word + \":\" + str(freq_dict[word]))\n",
    "      f.write('\\n')\n",
    "\n",
    "  ## Option 2\n",
    "  freq_list = list(freq_dict.items())\n",
    "  strip_quote = \"'\"\n",
    "  freq_list = [f\"{freq[0].strip(strip_quote)}:{str(freq[1])} \\n\" for freq in freq_list]\n",
    "  with open(file_name, \"w+\") as f:\n",
    "    f.writelines(freq_list)\n",
    "  \n",
    "  return freq_dict\n",
    "\n",
    "freq_dict = find_frequencies()\n",
    "```\n",
    "\n",
    "**Q:** What is the difference between the two options?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Function that takes in a text file of key/frequencies and \n",
    "# returns the top ten words that appear as a list\n",
    "def find_top_ten(file_name: str = \"bee_frequencies.txt\") -> list:\n",
    "  pass"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Answer\n",
    "\n",
    "```python\n",
    "\n",
    "def find_top_ten(file_name: str = \"bee_frequencies.txt\") -> list:\n",
    "    # Initialize an empty dictionary to store word frequencies\n",
    "    word_frequencies = {}\n",
    "\n",
    "    try:\n",
    "        # Open the file for reading\n",
    "        with open(file_name, 'r') as file:\n",
    "            # Read each line in the file\n",
    "            for line in file:\n",
    "                # Split the line into key and frequency using a space as the delimiter\n",
    "                key, frequency = line.strip().split(':')\n",
    "                # Store the word and its frequency in the dictionary\n",
    "                word_frequencies[key] = int(frequency)\n",
    "\n",
    "    except FileNotFoundError:\n",
    "        print(f\"File '{file_name}' not found.\")\n",
    "        return []\n",
    "\n",
    "    # Sort the dictionary by values (frequencies) in descending order\n",
    "    sorted_words = sorted(word_frequencies.items(), key=lambda x: x[1], reverse=True)\n",
    "\n",
    "    # Extract the top ten words from the sorted list\n",
    "    top_ten_words = [word for word, frequency in sorted_words[:10]]\n",
    "\n",
    "    return top_ten_words\n",
    "\n",
    "top_words = find_top_ten()\n",
    "print(\"Top Ten Words:\", top_words)\n",
    "```\n",
    "\n",
    "**Q:** How can we avoid common words such as \"the\" and \"of\" from appearing? HINT: These are called stop words in NLP"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Function that takes in a list of words and a frequency \n",
    "# dictionary and outputs the frequency of each word\n",
    "def find_words(to_find: list, freq_dict: dict):\n",
    "  pass"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Answer\n",
    "\n",
    "```python\n",
    "def find_words(to_find, freq_dict):\n",
    "    \n",
    "    for word in to_find:\n",
    "        # Use the get() method to retrieve the frequency from the dictionary\n",
    "        frequency = freq_dict.get(word, 0)\n",
    "        if frequency > 0:\n",
    "            print(f'{word} appeared {frequency} times in the script')\n",
    "        else:\n",
    "            print(f'The word: {word} did not appear in the script')\n",
    "\n",
    "words_to_find = [\"gandhi\", \"lauren\", \"bejesus\", \"larry\"]\n",
    "find_words(words_to_find, freq_dict)\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Task 2\n",
    "**Open Ended**\n",
    "\n",
    "You and your friend have created a new startup to motivate the Rutgers community, a noble cause! You guys want to broadcast a motivational quote every day to remind them of how great Rutgers is. Read in the quotes.txt file and choose a quote at random to broadcast.\n",
    "\n",
    "<img src=\"https://media.tenor.com/UF6EuB04PRQAAAAd/rutgers-greg-schiano.gif\" alt=\"meme\" width=\"200\"/>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!wget https://gist.githubusercontent.com/robatron/a66acc0eed3835119817/raw/0e216f8b6036b82de5fdd93526e1d496d8e1b412/quotes.txt"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Function that takes in a text file name and prints a \n",
    "# random line from said file\n",
    "# NOTE: The quotes from this txt file do not represent Rutgers' or your TA's views in ANY way\n",
    "def print_random_line(file_name: str = \"quotes.txt\"):\n",
    "  pass"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Answer\n",
    "```python\n",
    "import random\n",
    "\n",
    "def print_random_line(file_name=\"quotes.txt\"):\n",
    "    try:\n",
    "        # Open the file for reading\n",
    "        with open(file_name, 'r') as file:\n",
    "            # Read all lines from the file and store them in a list\n",
    "            lines = file.readlines()\n",
    "            \n",
    "            if not lines:\n",
    "                print(f\"The file '{file_name}' is empty.\")\n",
    "            else:\n",
    "                # Select a random line from the list\n",
    "                random_line = random.choice(lines)\n",
    "                # Print the random line\n",
    "                print(random_line.strip())\n",
    "\n",
    "    except FileNotFoundError:\n",
    "        print(f\"File '{file_name}' not found.\")\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Task 3\n",
    "\n",
    "Determine if a $9 x 9$ Sudoku board is valid. Only the filled cells need to be validated according to the following rules:\n",
    "\n",
    "Each row must contain the digits 1-9 without repetition.\n",
    "\n",
    "Each column must contain the digits 1-9 without repetition.\n",
    "\n",
    "Each of the nine 3 x 3 sub-boxes of the grid must contain the digits 1-9 without repetition.\n",
    "\n",
    "Note:\n",
    "- A Sudoku board (partially filled) could be valid but is not necessarily solvable.\n",
    "- Only the filled cells need to be validated according to the mentioned rules.\n",
    "\n",
    "For example, this board below:\n",
    "\n",
    "![example](https://upload.wikimedia.org/wikipedia/commons/thumb/f/ff/Sudoku-by-L2G-20050714.svg/250px-Sudoku-by-L2G-20050714.svg.png)\n",
    "\n",
    "is input as:\n",
    "\n",
    "```text\n",
    "board = \n",
    "[[\"5\",\"3\",\".\",\".\",\"7\",\".\",\".\",\".\",\".\"]\n",
    ",[\"6\",\".\",\".\",\"1\",\"9\",\"5\",\".\",\".\",\".\"]\n",
    ",[\".\",\"9\",\"8\",\".\",\".\",\".\",\".\",\"6\",\".\"]\n",
    ",[\"8\",\".\",\".\",\".\",\"6\",\".\",\".\",\".\",\"3\"]\n",
    ",[\"4\",\".\",\".\",\"8\",\".\",\"3\",\".\",\".\",\"1\"]\n",
    ",[\"7\",\".\",\".\",\".\",\"2\",\".\",\".\",\".\",\"6\"]\n",
    ",[\".\",\"6\",\".\",\".\",\".\",\".\",\"2\",\"8\",\".\"]\n",
    ",[\".\",\".\",\".\",\"4\",\"1\",\"9\",\".\",\".\",\"5\"]\n",
    ",[\".\",\".\",\".\",\".\",\"8\",\".\",\".\",\"7\",\"9\"]]\n",
    "```\n",
    "\n",
    "and is a valid board. Hint: look at the slicing section from the numpy lecture.\n",
    "\n",
    "You're better off solving this question on the website I got it from [source](https://leetcode.com/problems/valid-sudoku/)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!pip3 install numpy"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Answer\n",
    "\n",
    "```python\n",
    "import numpy as np\n",
    "\n",
    "def is_valid_sudoku(board: list):\n",
    "    board = np.array(board)\n",
    "    \n",
    "    # Check rows\n",
    "    for row in board:\n",
    "        if not is_valid_row(row):\n",
    "            return False\n",
    "    \n",
    "    # Check columns\n",
    "    for col in board.T:\n",
    "        if not is_valid_row(col):\n",
    "            return False\n",
    "    \n",
    "    # Check subgrids (3x3)\n",
    "    for i in range(0, 9, 3):\n",
    "        for j in range(0, 9, 3):\n",
    "            subgrid = board[i:i+3, j:j+3]\n",
    "            if not is_valid_row(subgrid.flatten()):\n",
    "                return False\n",
    "    \n",
    "    return True\n",
    "\n",
    "def is_valid_row(nums):\n",
    "    seen = set()\n",
    "    for num in nums:\n",
    "        if num != '.' and num in seen:\n",
    "            return False\n",
    "        seen.add(num)\n",
    "    return True\n",
    "\n",
    "# Example usage:\n",
    "board = [\n",
    "    [\"5\",\"3\",\".\",\".\",\"7\",\".\",\".\",\".\",\".\"],\n",
    "    [\"6\",\".\",\".\",\"1\",\"9\",\"5\",\".\",\".\",\".\"],\n",
    "    [\".\",\"9\",\"8\",\".\",\".\",\".\",\".\",\"6\",\".\"],\n",
    "    [\"8\",\".\",\".\",\".\",\"6\",\".\",\".\",\".\",\"3\"],\n",
    "    [\"4\",\".\",\".\",\"8\",\".\",\"3\",\".\",\".\",\"1\"],\n",
    "    [\"7\",\".\",\".\",\".\",\"2\",\".\",\".\",\".\",\"6\"],\n",
    "    [\".\",\"6\",\".\",\".\",\".\",\".\",\"2\",\"8\",\".\"],\n",
    "    [\".\",\".\",\".\",\"4\",\"1\",\"9\",\".\",\".\",\"5\"],\n",
    "    [\".\",\".\",\".\",\".\",\"8\",\".\",\".\",\"7\",\"9\"]\n",
    "]\n",
    "\n",
    "print(is_valid_sudoku(board))  # Should return True\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Task 4\n",
    "\n",
    "Numpy is often used for tasks involving matrix manipulation such as linear regression. Read in the house prices csv file in pandas, drop the columns that don't have numerical data (we can work with this in a later date when we get more advanced with pandas). Then store the labels (price of the home) in a numpy array called `Y`, and the features (all the other info we know about the properties) in a different array called `X`. Finally use the following formula to find the weights (importances) to give to the features, once we have those we can make predictions.\n",
    "\n",
    "$W = (X^{T} X)^{-1} X^{T} Y$\n",
    "\n",
    "NOTE: No need to write a function for this, just write the logic in a codeblock\n",
    "\n",
    "<img src=\"https://media.tenor.com/miz5HbQcXlYAAAAC/spongebob.gif\" alt=\"meme\" width=\"200\"/>\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!wget https://raw.githubusercontent.com/ywchiu/riii/master/data/house-prices.csv\n",
    "!pip3 install pandas"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Answer\n",
    "\n",
    "```python\n",
    "import pandas as pd\n",
    "import numpy as np\n",
    "\n",
    "# Read the CSV file into a Pandas DataFrame\n",
    "df = pd.read_csv('house-prices.csv').drop(['Brick', 'Neighborhood', 'Home'], axis=1)\n",
    "\n",
    "# Extract the labels (price of the homes) into a numpy array Y\n",
    "Y = df['Price'].values\n",
    "\n",
    "# Extract the features (all other numeric columns) into a different numpy array X\n",
    "X = df.drop(columns=['Price']).values\n",
    "\n",
    "# Perform linear regression using the provided formula\n",
    "X_with_bias = np.column_stack((X, np.ones(len(X)))) # need to add a constant value (bias term) this is the \"b\" in y = mx + b\n",
    "X_transpose = np.transpose(X_with_bias)\n",
    "X_transpose_dot_X = np.dot(X_transpose, X_with_bias)\n",
    "X_transpose_dot_Y = np.dot(X_transpose, Y)\n",
    "weights = np.dot(np.linalg.inv(X_transpose_dot_X), X_transpose_dot_Y)\n",
    "\n",
    "# The 'weights' variable now contains the importance (weights) to give to the features\n",
    "df.head()\n",
    "for attr, weight in zip(list(df.drop(columns=['Price']).columns), weights):\n",
    "    print(f'{attr} has {weight} importance')\n",
    "```\n",
    "\n",
    "This example is taken from the dataset\n",
    "\n",
    "```python\n",
    "price = weights[0]*1790 + weights[1]*2 + weights[2]*2 + weights[3]*2\n",
    "error = price - 114300\n",
    "\n",
    "print(error)\n",
    "```"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.7"
  },
  "orig_nbformat": 4
 },
 "nbformat": 4,
 "nbformat_minor": 2
}