Data science - statistics (Python).ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Data science - quantitative analysis\n",
    "\n",
    "Data science can be used to numerical data as well.\n",
    "We do not need to modify it to suite computer analysis, so we can read the data directly in.\n",
    "Here we examine World Value Survey dataset.\n",
    "\n",
    "[Narrated code walkthrough](https://www.youtube.com/watch?v=b-vtqbhlBaQ)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "import numpy as np\n",
    "\n",
    "data = pd.read_csv( './data/wvs.csv').astype(np.float64)\n",
    "data['V10'].value_counts()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Supervised machine learning\n",
    "\n",
    "We use a general purpose library [scikit-learn](https://scikit-learn.org/stable/) in developing and workingwith these exercises.\n",
    "1. splitting the data into train and test datasets\n",
    "1. fit a model to the data\n",
    "1. examine the model performance using test dataset"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn import tree\n",
    "from sklearn.model_selection import train_test_split"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "y = data['V10'] ## data we try to predidct\n",
    "x = data.drop( 'V10', axis = 1) ## data we used to predict, let's remove the predictor from there\n",
    "\n",
    "## We simplify data used for predicting and only use V4, V5, V6, V7, V8 and V9\n",
    "x = x[['V4', 'V5', 'V6', 'V7', 'V8', 'V9']]\n",
    "\n",
    "x_train, x_test, y_train, y_test = train_test_split( x, y , test_size=0.3, random_state=42)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "y_train.value_counts()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "y_test.value_counts()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "model = tree.DecisionTreeClassifier()\n",
    "model = model.fit( x_train, y_train )"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "pred = model.predict( x_test )"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.metrics import *\n",
    "\n",
    "print( confusion_matrix( y_test, pred ) )\n",
    "print( classification_report( y_test, pred ) )"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Exercises\n",
    "\n",
    "* `V10` has several unwanted values: `-5`, `-2` and `-1`. Remove them from the data and rerun the analysis.\n",
    "* What other variables would you add to the analysis? Do they improve accuracy? See [survey documentation](https://www.worldvaluessurvey.org/WVSDocumentationWV6.jsp) for the meaning of variables.\n",
    "* What other methods than `DecisionTreeClassifier` exists (see [scikit-learn supervised learning documentation](https://scikit-learn.org/stable/supervised_learning.html) documentation)? Try out them. Do you get better results?\n",
    "* What does cross validation mean? Try out cross validation and test out these things."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Unsupervised machine learning\n",
    "\n",
    "Beyond seeking to classify data based on existing attributes, in some questions you may want to find groups of similar entries from the data.\n",
    "This is unsupervised machine learning; several methods excists for this."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.cluster import KMeans\n",
    "\n",
    "data_for_kmeans = data.iloc[1:500,].copy() ## slices the data to be smaller and easier\n",
    "\n",
    "model = KMeans(n_clusters=5, random_state=42)\n",
    "clusters = model.fit_predict( data_for_kmeans )"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "## add clusters to our data\n",
    "\n",
    "data_for_kmeans['clusters'] = clusters\n",
    "\n",
    "data_for_kmeans.groupby('clusters').agg('mean')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Exercises\n",
    "\n",
    "* Which cluster has highest number of data points\n",
    "* How does changing the number of clusters change there results"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.13"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}