01-pandas-features.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Data analysis\n",
    "\n",
    "In this exercise, we will look at how to load, manipulate and visualize data sets in Python. We will also look at how to calculate features on data sets.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Pandas\n",
    "A widely used Python package for data management and analysis is `pandas`. The basic data structure provided by this package is the data frame. A data frame is a table that can in principle be used in the same way as a table in a relational database, e.g. rows or columns (or both) can be selected."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import pandas as pd\n",
    "from scipy.io import arff\n",
    "import matplotlib.pyplot as plt\n",
    "\n",
    "data = arff.loadarff('S08.arff')\n",
    "df = pd.DataFrame(data[0])\n",
    "df"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Select rows and columns\n",
    "\n",
    "With pandas, certain rows and columns can be easily selected from the data frame. One option is to select columns by name.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "accx = df.loc[:,\"Sensor_T8_Acceleration_X\"]\n",
    "accx"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can also select rows and columns by their indexes. The `:` stands here for \"Select all rows\". If only a single column is selected, the result is of type `Series`, otherwise the result is again a data frame. Columns can also be selected via their index:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "acc = df.iloc[:,1:4]\n",
    "acc"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "lines can be accessed in the same way. For example, the following expression returns the first 5 lines:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df.iloc[0:5,:]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Both options can also be combined, e.g. like this:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df.loc[0:5,[\"Sensor_T8_Acceleration_X\", \"Sensor_T8_Acceleration_Y\"]]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Another practical option is to select rows or columns using Boolean expressions. The following expression, for example, returns all rows for which the value of `Sensor_T8_Acceleration_X` is less than 0.7."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df.loc[df.Sensor_T8_Acceleration_X < 0.7,:]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Insert values\n",
    "\n",
    "Inserting values into a data frame works in exactly the same way. The following expression sets all values in the `Sensor_T8_Acceleration_Y` column that are less than 0 to the value 0."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df.loc[df.Sensor_T8_Acceleration_Y < 0,\"Sensor_T8_Acceleration_Y\"] = 0\n",
    "df"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Apply\n",
    "In many cases, we want to apply a function to a complete row or column of data. The function `apply` allows us to do this. The following expression calculates the mean values per column:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df.iloc[:,1:31].apply(np.mean)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Exercise 1\n",
    "\n",
    "Calculate the distribution of the classes in `df`. (`collections.Counter`)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Plot the distribution of classes as a bar plot."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Plot some accelerometer axes (e.g. \"Sensor_RightForeArm_Acceleration_X\", \"Sensor_RightForeArm_Acceleration_Y\", \"Sensor_RightForeArm_Acceleration_Z\") as a line plot. The different axes should be displayed in different colors."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Exercise 2\n",
    "\n",
    "Next, we want to calculate some features on the data. For sequential data, features are typically calculated based on segments. This means that we first calculate a feature function (mean, ...) for rows 1 to n, then for n+1 to 2n, and so on. The segments can also overlap. \n",
    "\n",
    "Implement the function `feature`, which calculates a given statistical feature function (mean, ...) for a given window size, overlap and data set. Then calculate the mean, median and variance of the accelerometer data of the right foot with segment lengths of 128, 256 and 512. Use 50% overlap and plot the result.\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.11.4"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}