Example-Prepare-Document.ipynb

{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# from https://en.wikipedia.org/wiki/Inflation"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "document_text = \"\"\"\n",
    "\n",
    "In economics, inflation (or less frequently, price inflation) is a general rise in the price level of an economy\n",
    "over a period of time.[1][2][3][4] When the general price level rises, each unit of currency buys fewer goods and\n",
    "services; consequently, inflation reflects a reduction in the purchasing power per unit of money – a loss of real\n",
    "value in the medium of exchange and unit of account within the economy.[5][6] The opposite of inflation is \n",
    "deflation, a sustained decrease in the general price level of goods and services. The common measure of inflation\n",
    "is the inflation rate, the annualised percentage change in a general price index, usually the consumer price \n",
    "index, over time.[7]\n",
    "\n",
    "Economists believe that very high rates of inflation and hyperinflation are harmful, and are caused by excessive\n",
    "growth of the money supply.[8] Views on which factors determine low to moderate rates of inflation are more \n",
    "varied. Low or moderate inflation may be attributed to fluctuations in real demand for goods and services, or \n",
    "changes in available supplies such as during scarcities.[9] However, the consensus view is that a long sustained \n",
    "period of inflation is caused by money supply growing faster than the rate of economic growth.[10][11]\n",
    "\n",
    "Inflation affects economies in various positive and negative ways. The negative effects of inflation include an \n",
    "increase in the opportunity cost of holding money, uncertainty over future inflation which may discourage \n",
    "investment and savings, and if inflation were rapid enough, shortages of goods as consumers begin hoarding out \n",
    "of concern that prices will increase in the future. Positive effects include reducing unemployment due to nominal \n",
    "wage rigidity,[12] allowing the central bank greater freedom in carrying out monetary policy, encouraging loans \n",
    "and investment instead of money hoarding, and avoiding the inefficiencies associated with deflation.\n",
    "\n",
    "Today, most economists favour a low and steady rate of inflation.[13] Low (as opposed to zero or negative) \n",
    "inflation reduces the severity of economic recessions by enabling the labor market to adjust more quickly in a \n",
    "downturn, and reduces the risk that a liquidity trap prevents monetary policy from stabilising the economy.[14] \n",
    "The task of keeping the rate of inflation low and stable is usually given to monetary authorities. Generally, \n",
    "these monetary authorities are the central banks that control monetary policy through the setting of interest \n",
    "rates, through open market operations, and through the setting of banking reserve requirements.[15]\n",
    "\"\"\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.feature_extraction.text import CountVectorizer"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "vectorizer = CountVectorizer()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "X_train = vectorizer.fit_transform([document_text])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "X_train"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "vectorizer.get_feature_names()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import re\n",
    "regex = re.compile('[^a-zA-Z]')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "alpha_text = regex.sub(' ', document_text)\n",
    "alpha_text = alpha_text.lower()\n",
    "alpha_text = ' '.join(alpha_text.split())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "alpha_text"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# remove stop words\n",
    "import nltk\n",
    "from nltk.corpus import stopwords\n",
    "nltk.download('stopwords')\n",
    "stop = stopwords.words('english')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "nostop_text = ' '.join(word.lower() for word in alpha_text.split() if word.lower() not in stop)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "nostop_text"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# lemmatized words\n",
    "nltk.download('wordnet')\n",
    "\n",
    "from nltk.stem.porter import PorterStemmer\n",
    "stemmer = PorterStemmer()\n",
    "\n",
    "# lemmatized words\n",
    "from nltk.stem import WordNetLemmatizer\n",
    "lemmatizer = WordNetLemmatizer()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "lemma_text =  ' '.join(lemmatizer.lemmatize(w) for w in nostop_text.split())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "lemma_text"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "stem_text =  ' '.join(stemmer.stem(w) for w in nostop_text.split())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "stem_text"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.7"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}