Analysis_and_Visualization_of_Data_with_R.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "# Analyzing ETCBC data with R"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "by Martijn Naaijer (m.naaijer@vu.nl)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "In this R course you learn how to analyze and visualize structured datasets obtained from the ETCBC database. The focus is on descriptive statistics.\n",
    "\n",
    "The course consists of three parts:\n",
    "\n",
    " 1. Basic operations\n",
    " 2. Moving data around\n",
    " 3. Analysis and visualization of data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "## 1. Basic operations"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "In this section you will learn how to make a vector and a matrix in R and how to do some basic calculations."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "At the end of section 1 you know the following functions:\n",
    "* c()\n",
    "* class()\n",
    "* sum()\n",
    "* mean()\n",
    "* cumsum()\n",
    "* cumprod()\n",
    "* seq()\n",
    "* rep()\n",
    "* length()\n",
    "* sort()\n",
    "* matrix()\n",
    "* colSums()\n",
    "* rowSums()\n",
    "* apply()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "### 1.1 Basic Operations"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "First look at some basic arithmetic operations. As in Python, comments are preceded by #."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "5 + 5 # a comment \n",
    "\n",
    "5 - 2\n",
    "\n",
    "5 * 7\n",
    "\n",
    "5 / 4\n",
    "\n",
    "5 ^ 5  # power\n",
    "\n",
    "5 ** 5  # also power\n",
    "\n",
    "sqrt(5)\n",
    "\n",
    "1 + 5 ; 1 + 6 # two statements on one line. It is better not to do this, it can easily make lines too long"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "1 == 2  # is equal to, result is a Boolean\n",
    "\n",
    "2 == 2 \n",
    "\n",
    "1 != 2 # is not equal to "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "3 > 1\n",
    "\n",
    "3 < 1"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "The assignment of a value to a variable works as follows "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "num <- 7  # use <- or ="
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "num"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "### 1.2 Vectors in R"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "In R, there is a special role for vectors. A vector is a sequence of elements. Very often these elements are numeric, but that does not have to be the case. How do we create a vector?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "num.vec1 <- 1:5 \n",
    "\n",
    "num.vec1\n",
    "\n",
    "num.vec2 <- 5:1\n",
    "\n",
    "num.vec2\n",
    "\n",
    "num.vec3 <- c(3, 2, 4, 1, 5)\n",
    "\n",
    "num.vec3"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "In R, the dot (.) has no special significance, in contrast to its use in Python. It is used generally between the parts of a variable name."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "You can use an index vector to get a slice of a vector. R is 1-based, which means that in R the first element of a vector is indexed with 1."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "# access members of a vector by indexing. The result is a new vector.\n",
    "\n",
    "num.vec2[2]\n",
    "\n",
    "num.vec2[2:4]\n",
    "\n",
    "num.vec2[c(2, 4)]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "In contrast to the Python list, an R vector contains only one data type."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "class(num.vec2)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "What happens if we try to mix data types?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "mix.vec <- c(1, 'etcbc', TRUE) # R interprets this as a character vector, so 1 and TRUE are characters here.\n",
    "\n",
    "class(mix.vec)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "### 1.3 Vectors: Basic Calculations "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "You can easily calculate some basic properties of the data in a vector."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "mean(num.vec2)\n",
    "\n",
    "sum(num.vec2)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "There are functions for the cumulative sum and the cumulative product."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "cumsum(num.vec2)\n",
    "\n",
    "cumprod(num.vec2)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "bool.vec <- c(TRUE, TRUE, FALSE) # short notation c(T, T, F)\n",
    "\n",
    "class(bool.vec)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "You can also use the functions sum() and mean() on a logical vector."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "sum(bool.vec) # number of TRUE's in the vector\n",
    "\n",
    "mean(bool.vec) # fraction of TRUE values in the vector "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "We create a character vector."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "ch.vec <- c('a', 'bb', 'ccc')\n",
    "\n",
    "class(ch.vec)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "With as.factor() you can convert a numeric or character vector into a factor variable. In data analysis you will often work with factor variables. A factor variable has a limited number of values (called levels in R)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "fac.vec <- as.factor(ch.vec)\n",
    "class(fac.vec)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "length() returns the number of elements in a vector. sort() sorts the numbers. decreasing is an optional argument, of which FALSE is the default value."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "# length() and sort()\n",
    "\n",
    "length(num.vec3)\n",
    "\n",
    "sort(num.vec3)\n",
    "\n",
    "sort(num.vec3, decreasing = T) # T is same as TRUE, F is same as FALSE"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "If you add a number to a vector, the number is added to each element. The same for multiplication. If you add two vectors, they are added element wise."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "num.vec1\n",
    "\n",
    "num.vec1 + 2\n",
    "\n",
    "num.vec1 * 2\n",
    "\n",
    "num.vec1 + num.vec3"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "Look what happens if you add two vectors of unequal length. There is also an, in this case harmless, warning."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "short.vec <- c(1, 2)\n",
    "\n",
    "num.vec1 + short.vec"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "### 1.4 Making vectors with seq() and rep()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "You can also make a vector with seq()."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "num.vec4 <- seq(1, 5)\n",
    "\n",
    "num.vec4"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "seq() has an optional argument 'by', which specifies the step size."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "num.vec5 <- seq(1, 5, by = 0.5)\n",
    "\n",
    "num.vec5"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "If you want to create a vector with repeating elements, you can use the function rep(). The second argument specifies how often the first argument should be repeated. Look also the mathematically important numbers pi and e (exp(1) in R)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "rep(5, 4) # the first argument is an integer\n",
    "\n",
    "c(rep(5, 4), rep(4, 5))\n",
    "\n",
    "c(rep(5, 4), pi, exp(1), 10, rep(4, 5))\n",
    "\n",
    "rep(c(1, 2), 3) # the first argument is a vector."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "### 1.5 Matrices in R"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "In R you can easily create a matrix. A matrix is a 2-dimensional object with a rectangular shape that contains only one data type. Often it is numeric. You can make it with a simple numeric vector and a specification of the number of rows or the number of columns."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "# creating matrices\n",
    "\n",
    "mat.A <- matrix(1:10, nrow = 5)\n",
    "\n",
    "class(mat.A)\n",
    "\n",
    "mat.A"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "mat.B <- matrix(1:10, ncol = 5)\n",
    "\n",
    "mat.B"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "Now there are more elements in the matrix than in the vector. How does R handle this?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "mat.C <- matrix(1:10, nrow = 5, ncol = 5)\n",
    "\n",
    "mat.C"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "In the examples above the matrix was filled by column. This is the default way, but you can also do it by row with byrow. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "mat.D <- matrix(1:10, ncol = 5, byrow = TRUE)\n",
    "\n",
    "mat.D"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "You calculate the sums of the rows and the columns with rowSums() and colSums()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "rowSums(mat.D)\n",
    "\n",
    "colSums(mat.D)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "### 1.6 The Function apply()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "For calculations on numeric matrices the function apply() can be very useful. It applies a function that you choose on all the rows or the columns of the matrix. The function has 3 arguments. The first one is the name of the matrix object, the second argument is the dimension, this is 1 (rows) or 2 (columns) and the third argument is the function that you want to apply. This could also be a function that you have written yourself"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "apply(mat.D, 1, mean)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "apply(mat.D, 2, mean)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "### 1.7 Making Your Own Functions"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "If you want to write a function in R you use the keyword function, and the function body is embraced by {}.\n",
    "The structure of an R function is as follows:\n",
    "\n",
    "my.function <- function(arg1, arg2, ...) {   \n",
    "&nbsp;&nbsp;function body  \n",
    "&nbsp;&nbsp;...  \n",
    "&nbsp;&nbsp;return(result)  \n",
    "\n",
    "}\n",
    "\n",
    "The return statement is not required in R. If you omit it, the function simply returns the last thing that it calculated. Indentation is not required either, but it is stringly recommended to indent the function body for readability. Below you see an example of a simple R function."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "sum.2.nums <- function(a, b) {\n",
    "  a + b\n",
    "}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "new.num <- sum.2.nums(4, 6) #here we call the function, which sums the two numbers.\n",
    "\n",
    "new.num"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "This is an example of apply() with a customized function as third argument. It subtracts the mean of the whole column for every element in that column. x is the argument of the function, in this case this is a column of the matrix.\n",
    "apply() is a member of the apply family of functions. In later sessions we will meet other members of the family, namely sapply() and tapply(). These are so called vectorized functions that perform calculations rapidly."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "apply(mat.D, 2, function (x) {x - mean(x)})"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "We have seen that various R functions have optional arguments, that have a default value if you omit them. If you want to know more about the arguments and the use of a function, you can use help(function.name) or ?function.name"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "help(length)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "?length"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "For many R problems there are good ad-hoc solutions. The most obvious is to to use Google. If you get an error message, copy it and paste the message in Google and follow the links. It is likely that you find the solution of your problem on [stackoverflow.com](https://stackoverflow.com) or similar websites. The same applies for visualizations in R. If you want to make a specific kind of visualization, for instance a barplot, and you want to browse a little bit to see what the possibilities are, simply type \"barplot r\" in Google, click on \"images\" and go the website where you will likely find example data and code. You can assume that every problem you encounter with R, has been encountered by others as well, so you will probably find the solution online."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "In R there does not seem to be a preferred way to do things, like there is in Python, although R programmers have certain preferences, such as the avoidance of for loops if possible. If you want to know more about good R style, take a look at [Google's R Style Guide](https://google.github.io/styleguide/Rguide.xml)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "## 2. Moving data around"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "At the end of this section you know the following functions:\n",
    "\n",
    "* cbind()\n",
    "* rbind()\n",
    "* data.frame()\n",
    "* t()\n",
    "* colnames()\n",
    "* nrow()\n",
    "* ncol()\n",
    "* subset()\n",
    "* ifelse()\n",
    "* write.csv()\n",
    "* paste()\n",
    "* attach()\n",
    "* detach()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "### 2.1 Combining Vectors"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "First, we create a vector called 'lexeem', consisting of the lexemes of the first three words of the books of Genesis, Exodus and Psalms."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "lexeem <- c('B', 'R>CJT',  'BR>[', 'W', '>LH', 'CM/', '>CR=/', 'H', '>JC/')\n",
    "\n",
    "lexeem"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "A second vector is created with the names of the book for each lexeme."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "boek <- c(rep('Genesis', 3), rep('Exodus', 3), rep('Psalms', 3))\n",
    "\n",
    "boek"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "These vectors are combined in a matrix by using the fuction cbind(). cbind stands for column bind. The resulting matrix is assigned to the variable 'words' and converted into a dataframe. In R we distinguish between a matrix and a dataframe. A matrix contains data of one specific type. Often a matrix contains only numeric data, but the words matrix contains character data. We will expand this dataset with numeric data, so we first make a data frame of the matrix. A data frame can contain more data types."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "words <- cbind(boek, lexeem)\n",
    "\n",
    "class(words)\n",
    "\n",
    "class(words[,1])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "words <- data.frame(words)\n",
    "\n",
    "class(words[,1])\n",
    "\n",
    "words"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "You can access a column of a data frame with $. Each column of the data frame is a vector."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "words$boek"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "The vectors can also be combined as rows of a matrix. The function t(), transpose, flips the axes of the matrix."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "words.horiz <- rbind(boek, lexeem)\n",
    "\n",
    "words.horiz\n",
    "\n",
    "t(words.horiz)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "### 2.2 Expanding and Saving a Dataset"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "We continue with the object 'words', because it has the most widely used data format: every column contains a variable, every row contains 1 observation. The header of a data frame can easily be changed by using the function colnames()."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "colnames(words) <- c('book', 'lexeme')\n",
    "\n",
    "words"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "We expand the dataset a bit with some extra columns. Often it is useful to have an enumeration of the rows. This can be done easily by creating a new column vector which we give the name 'enumeration'."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "words$enumeration <- 1:nrow(words)\n",
    "\n",
    "class(words$enumeration)\n",
    "\n",
    "words"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "Not only is there a function nrow(), there is also ncol()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "ncol(words)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "A column with identical values for each row can be added as follows:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "words$phase <- 'biblical'\n",
    "\n",
    "words"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "The function ifelse() can be used if you want to add a column with only two values. These values are often 0/1 or TRUE/FALSE. In this case we add a column with the values 'prose' and 'poetry'. ifelse() has three arguments. In this particular case it can be read as follows: if the column 'book' has one of the values 'Genesis'/'Exodus', the column 'genre' gets the value 'prose', else it gets the value 'poetry'."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "narrative <- c('Genesis', 'Exodus')\n",
    "\n",
    "words$genre <- ifelse(words$book %in% narrative, 'prose', 'poetry')\n",
    "\n",
    "words"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "A data-frame can easily be saved as csv file."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "write.csv(words, file = 'my_words.csv')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "### 2.3 Making Subsets"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "The following condition returns a boolean vector."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "words$book == 'Genesis'"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "If this vector is used as index of the dataframe a subset of the data can be made. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "words.gen <- words[words$book == 'Genesis',]\n",
    "\n",
    "words.gen"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "We can also use more conditions as index."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "words.gen.not.B <- words[words$book == 'Genesis' & words$lexeme != 'B',] # not unexpected, & means 'and'\n",
    "\n",
    "words.gen.not.B"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "words.gen.and.W <- words[words$book == 'Genesis' | words$lexeme == 'W',] # | means 'or'\n",
    "\n",
    "words.gen.and.W"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "In the case of numeric variables you can use >, <, >= and <=."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "words.high <- words[words$enumeration > 5,]\n",
    "\n",
    "words.high"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "You can also use the function subset() for making subsets of dataframes."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "words.subset <- subset(words, book == 'Genesis' & words$lexeme != 'B')\n",
    "\n",
    "words.subset"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "You can merge the values of two variables by using the function paste(). You use the argument sep to specify the separator."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "words$book.genre <- paste(words$book, words$genre, sep = '.')\n",
    "\n",
    "words"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "The resulting new variable is a character variable. If it should be a factor, you need to change this manually."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "class(words$book.genre)\n",
    "\n",
    "words$book.genre <- as.factor(words$book.genre)\n",
    "\n",
    "class(words$book.genre)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "You can remove a variable (a column) from the data frame in the following way."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "words$book.genre <- NULL\n",
    "\n",
    "words"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "Or by using a negative index."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "words[,-5]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "If you want to access specific column of a data frame you use the name of the data frame followed by $. If you want to simplify your code a little bit you can use the function attach(). I never use attach(), but if you like it, feel free to use it."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "attach(words)\n",
    "\n",
    "lexeme #you do not need to say words$lexeme"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "You can go back to the $-notation with detach()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "detach(words)\n",
    "lexeme"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "## 3. Analysis and Visualization of Data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "After this section you know the following functions:\n",
    "* getwd()\n",
    "* setwd()\n",
    "* read.csv()\n",
    "* dim()\n",
    "* str()\n",
    "* head()\n",
    "* tail()\n",
    "* subset()\n",
    "* sapply()\n",
    "* stripchart()\n",
    "* table()\n",
    "* unique()\n",
    "* barplot()\n",
    "* boxplot()\n",
    "* mosaicplot()\n",
    "* mosaic() from the vcd package\n",
    "* assoc() from the vcd package\n",
    "* tapply()\n",
    "* hist()\n",
    "* abline()\n",
    "* rug()\n",
    "* melt() from the reshape package\n",
    "* dist()\n",
    "* hclust()\n",
    "* plot()\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "### 3.1 Importing and Inspecting a Dataset"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "If we want to import data from a separate file, in our case the file 'all_words.csv', we need to go to the right directory with setwd(). getwd() returns the present working directory.\n",
    "\n",
    "Source of the file: https://shebanq.ancient-data.org/hebrew/query?id=1875 The query on which this file is based is very simple. I have done this to show that even with a very simple query it is possible to make a nice analysis with some effort."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "If the data are in a subdirectory called /Shebanq_Course_Files you uncomment and execute the following script. If they are in the working directory, you can import the data into R directly."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "#getwd()\n",
    "\n",
    "setwd('Shebanq_Course_Files') # You can do this because 'Shebanq_Course_Files' is a subfolder of the present working directory.\n",
    "                              # otherwise use the complete path: setwd('/mnt/shared/martijn/R_Course/Shebanq_Course_Files')\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "The dataset can be imported now with the function read.csv(). The argument 'header' has TRUE as default value, so in this case you did not have to use it."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "dat <- read.csv('all_words.csv', header = TRUE) \n",
    "#you can also use: dat <- read.table('he_locale.csv', sep = ',', header = TRUE)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "You **always** inspect a dataset in various ways,\n",
    "first with the function dim(), dimensions, which returns the number of rows and the number of columns in a dataset."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "dim(dat)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "You can also get these numbers separately."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "nrow(dat)\n",
    "\n",
    "ncol(dat)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "With str() you get the structure of a dataset."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "str(dat)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "What are the column names?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "colnames(dat)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "The function head() returns the first 6 rows of the dataframe (default), which you can change in the second argument."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "head(dat)\n",
    "\n",
    "head(dat, 3)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "It is a good idea to inspect the bottom of your dataset as well."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "tail(dat)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "The csv files that you download from the SHEBANQ website are an example of tidy data. This means that there is one observation on each row and one type of information (variable) in each column. If you want to read more about tidy data, you can read [this article](https://www.jstatsoft.org/article/view/v059i10). If you make a dataset with Text-Fabric, you should also try to make accorsing to these principles."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "### 3.2 Additional Counts and Manipulations"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "You can make a subset of the data frame by specifying which columns and rows you want to select."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "dat[1:10, 1:5]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "A very important function for the kind of data we have (many categorical variables) is table(). Table counts how many each level of a categorical variable occurs."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "table(dat$book)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "You can give table() two or three arguments, each argument is a categorical variable."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "table(dat$book, dat$state)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "A different way of making subsets is by using the function subset()."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "first.chapters <- subset(dat, dat$chapter < 2) #first argument is the data object, second argument is a condition.\n",
    "\n",
    "dim(first.chapters)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "Again, we enumerate the dataset."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "dat$enumeration <- 1:nrow(dat)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "We create an extra variable 'genre', which makes a distinction assigns the value 'prose', 'poetry', or 'prophecy' on the basis of the book in which a word occurs."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "prose <- c('Genesis', 'Exodus', 'Leviticus', 'Numeri', 'Deuteronomium', 'Josua', 'Judices', 'Samuel_I', 'Samuel_II', 'Reges_I', 'Reges_II', 'Jona', 'Ruth', 'Esther', 'Daniel', 'Esra', 'Nehemia', 'Chronica_I', 'Chronica_II')\n",
    "poetry <- c('Psalmi', 'Iob', 'Proverbia', 'Canticum', 'Ecclesiastes', 'Threni')\n",
    "prophecy <- c('Jesaia', 'Jeremia', 'Ezechiel', 'Hosea', 'Joel', 'Amos', 'Obadia', 'Micha', 'Nahum', 'Habakuk', 'Zephania', 'Haggai', 'Sacharia', 'Maleachi')\n",
    "\n",
    "dat$genre <- ifelse(dat$book %in% prose, 'prose', \n",
    "             ifelse(dat$book %in% poetry, 'poetry', 'prophecy'))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "We fix the order of the books in the data frame. If we skip this step, in many analyses the order will be alphabetic. If you want that, that is good, but here we maintain the canonical order."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "unique(dat$book)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "dat$book <- factor(dat$book, levels = unique(dat$book))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "In the following analyses we work with the 39 books of the Hebrew Bible. However, you might also want to work with 36 books only, if you decide that 1 and 2 Samuel are only one book of Samuel, and likewise with Kings and Chronicles. If you want this, you should add an extra column in which this distinction is not made. First we split the names of the books with the function strsplit(), which splits a string on the character that is specified as second argument of the function. Then we use the function sapply(). This function takes a vector as first argument, and a function as second argument and returns a vector, which we call book2. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "book.split <- strsplit(as.character(dat$book), '_')\n",
    "\n",
    "dat$book2 <- sapply(book.split, function(x) return(x[1])) # the function returns only the first element\n",
    "    \n",
    "unique(dat$book2)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "### 3.3 The Scatterplot"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "A basic type of visualization is the scatterplot. It shows the relationship between two numeric variables.  \n",
    "Suppose we have two vectors of equal length, called x and y. Imagine that x is the time and y is place (for instance of a car). Each element in x corresponds with an element in y with the same index."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "x <- 1:10\n",
    "\n",
    "y <- 2 * x + 5\n",
    "\n",
    "x\n",
    "y"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "A scatterplot is made easily."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "plot(x, y)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "This plot is clear, but we can give it a bit more structure and information. You want to know more? Look at [QuickR](http://www.statmethods.net/advgraphs/axes.html)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "plot(x, y, \n",
    "     main = \"Relationship between time and place of a car\",\n",
    "     xlab = \"Time (in seconds)\",\n",
    "     ylab = \"Place (in metres)\",\n",
    "     xlim = c(0, 10),\n",
    "     ylim = c(0, 20))\n",
    "\n",
    "axis(x, at = 1:10)\n",
    "lines(x, y)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "### 3.4. A Scatterplot of Biblical Data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "We make a scatterplot to visualize the frequencies of the verbal tenses in the Song of Songs."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "song <- dat[dat$book == 'Canticum',] # select the book\n",
    "song.tenses <- table(song$chapter, song$tense) # make a table with frequencies of tenses per chapter. \n",
    "song.tenses <- as.data.frame.matrix(song.tenses) # make a data frame.\n",
    "\n",
    "class(song.tenses)\n",
    "\n",
    "song.tenses$chap <- 1:8 \n",
    "\n",
    "plot(rownames(song.tenses), song.tenses$impf)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "In the figure above you see that with a simple lines it rapidly becomes difficult to distinguish the different lines. Therefore we decide to use different colors and line types for the lines and different symbols for the points. Make sure you understand all the arguments used in the functions lines() and legend(). One way to do that is to change them and see what happens. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "#First we make an empty plot and we specify the colors, linetypes and symbols\n",
    "plot(c(1, 8), c(0,25), type=\"n\", xlab=\"Chapter\", ylab=\"Frequency\" ) \n",
    "colors <- rainbow(8) \n",
    "linetype <- c(1:8) \n",
    "plchar <- seq(1,1+8,1)\n",
    "\n",
    "\n",
    "# we add the lines using a for loop. Every i selects a different column in song.tenses \n",
    "for (i in 1:8) { \n",
    "  tense <- song.tenses[,i]\n",
    "  lines(song.tenses$chap, tense, type=\"b\", lwd=1.5,\n",
    "    lty=linetype[i], col=colors[i], pch=plchar[i]) \n",
    "} \n",
    "\n",
    "# add a title\n",
    "title(\"Distribution of verbal tenses in the Song of Songs\")\n",
    "\n",
    "# add a legend, the first two arguments specify the location of the legend in the plot.\n",
    "legend(1, 25, colnames(song.tenses)[1:8], cex=0.7, col=colors, pch=plchar, lty=linetype, title=\"Tense\")\n",
    "\n",
    "#You may find the resulting plot a bit too 'crowded'. In that case you could decide to make a separate plot for each tense or you \n",
    "#could for instance make 2 plots for finite and nonfinite verb forms.\n",
    "\n",
    "#Suppose you do not want to plot the absolute frequencies of the verbal forms, but the relative frequencies (or fractions).\n",
    "#How would you do that? Hint: use apply()."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "### 3.5 The Stripchart"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "he.local <- dat[dat$uvf == 'H',]\n",
    "\n",
    "dim(he.local)\n",
    "\n",
    "head(he.local)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "In the cell above a new object 'he.local' is created. It is a subset of the object 'dat'. If we would have said:\n",
    "\n",
    "dat <- dat[dat$uvf == 'H',]\n",
    "\n",
    "we would have created a subset of the original object with the same name. In that case we would not have had access anymore to the original data. Therefore it is advisable to give subsets a new name."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "A stripchart is a very simple visualisation, but it can be useful. We make a stripchart of the place in the Hebrew Bible of all the occurrences of local he. One occurrence is represented by a square. We use the variable enumeration as indicator of the place of a local he in the text. Look what happens if the argument 'method' is removed. 'main' is used to add a header to the chart."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "stripchart(he.local$enumeration, method = 'jitter', main = 'Stripchart of he locale in Biblical Hebrew')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "Now we make a stripchart of the words MMLKH/ and MLKWT/, both meaning 'kingdom'. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "kingdom = c('MMLKH/', 'MLKWT/')\n",
    "\n",
    "dat.king <- dat[dat$lexeme.t %in% kingdom,]\n",
    "\n",
    "dat.king$lexeme.t <- factor(dat.king$lexeme.t)\n",
    "\n",
    "dim(dat.king)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "stripchart(dat.king$enumeration ~ dat.king$lexeme.t, \n",
    "           method = 'jitter', \n",
    "           xlim = c(0, 450000), \n",
    "           col=c(\"blue\", \"green\")[dat.king$lexeme.t])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "### 3.6 The Mosaicplot"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "A very useful kind of visualization for **categorical data** is the mosaicplot. In it, you get an impression of which variant of a feature is over- or underrepresented in specific groups."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "We study this by looking at the distribution of MLKWT/ and MMLKH/ in Early Biblical Hebrew (EBH) and Late Biblical Hebrew LBH. The variation between these two lexemes is one of the classic examples of supposed linguistic change in Biblical Hebrew. MMLKH/ occurs more in EBH and MLKWT/ occurs more in LBH.  \n",
    "First a new column is created with the name 'phase', in which the typical narrative EBH books get the value 'EBH' and the typical narrative LBH books get the value 'LBH'. The other books get the value 'other'. A table is created using this new column and the lexeme column, after which the function mosaicplot() is applied to the table."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "EBH <- c('Genesis', 'Exodus', 'Leviticus', 'Numeri', 'Deuteronomium', 'Josua', 'Judices', 'Samuel_I', 'Samuel_II', 'Reges_I', 'Reges_II')\n",
    "LBH <- c('Esther', 'Daniel', 'Esra', 'Nehemia', 'Chronica_I', 'Chronica_II')\n",
    "\n",
    "dat.king$phase <- ifelse(dat.king$book %in% EBH, 'EBH', \n",
    "                   ifelse(dat.king$book %in% LBH, 'LBH', 'other')\n",
    "                  )\n",
    "\n",
    "kingdom.table <- table(dat.king$lexeme.t, dat.king$phase)\n",
    "mosaicplot(kingdom.table, main = 'Distibution of \"Kingdom\" in the Hebrew Bible', col = c(3, 5, 6))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "In this course we work mainly with the base R package. However, in R you will work very often with functions that come from packages that you have to download from CRAN (Comprehensive R Archive Network). These packages are made by your fellow R users and developers. You can download a package with install.package('package.name'). On your own computer you need to do this only once. In every session that you want to use functions from that package you have to load the package with library(package.name). First we install and load the reshape package. In it you can find functions with which you can manipulate the shape of data frames."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "With the package vcd you can make beautiful mosaicplots.First use the function structable."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "install.packages('vcd')\n",
    "library(vcd)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "mosaic(structable(kingdom.table), shade = T, varnames = F)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "With the function assoc() from the package vcd you can make an associationplot, which is related to the mosaicplot."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "assoc(kingdom.table, shade = T, legend = T)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "### 3.7 The Barplot"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "he.count <- table(he.local$book)\n",
    "he.count"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "unique(he.local$book)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "he.count <- table(he.local$book)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "barplot(he.count, las = 2)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "### 3.8 The Cleveland Dotplot"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "Sometimes a so called Cleveland dot plot is en elegant alternative of the barplot. You can make it by using the function dotchart()."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "dotchart(he.count)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "### 3.9 Again the Barplot"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "Now we know how many times local he occurs in each book. However, longer books tend to have more cases of local he than shorter books, so comparing them is relatively hard. Therefore, we divide the number of local he's by the total number of words per book. First we select the Hebrew words (in other words, we remove the Aramaic parts)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "unique(dat$language)\n",
    "dat.h <- dat[dat$language == 'hbo',]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "Then all Hebrew words are counted per book."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "word.count <- table(dat.h$book)\n",
    "word.count"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "Finally, we divide."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "he.count/word.count"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "barplot(he.count/word.count, las = 2, main = 'Fraction of words with local he in Biblical Hebrew')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "In various studies the length or complexity of linguistic units like phrases or clauses plays a role. First we count the number of clauses in each book."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "colnames(dat.h) \n",
    "clause.count <- tapply(dat.h$clause_a, (dat.h$book), function(x) length(unique(x)))\n",
    "#clause.count <- tapply(dat.h$clause_a, (dat.h$book.ord), max) does not work properly, because a number of clauses have been removed from Daniel and Esra"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "Then a data frame is made and a column name is added."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "clause.count <- as.data.frame(clause.count)\n",
    "\n",
    "colnames(clause.count) <- c('clauses')\n",
    "    \n",
    "clause.count"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "The number of words in every book is counted and then the average number of words per clause in each book."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "words.tab <- as.data.frame(table(dat.h$book))\n",
    "\n",
    "colnames(words.tab) <- c('book', 'words')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "words.per.clause <- words.tab$words/clause.count$clauses\n",
    "\n",
    "rownames(words.per.clause)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "We add genre and manipulate the structure of the dataframe a bit."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "words.per.clause <- as.data.frame(words.per.clause)\n",
    "\n",
    "nrow(words.per.clause)\n",
    "\n",
    "words.per.clause$book <- unique(dat$book)\n",
    "\n",
    "words.per.clause$genre <- ifelse(rownames(words.per.clause) %in% prose, 'prose', \n",
    "             ifelse(rownames(words.per.clause) %in% poetry, 'poetry', 'prophecy'))\n",
    "\n",
    "rownames(words.per.clause) <- NULL\n",
    "words.per.clause"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true,
    "deletable": true,
    "editable": true
   },
   "source": [
    "We can make a barplot of this result, and color the genres."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "barplot(words.per.clause$words.per.clause, las = 2, col = c(rep('red', 11), rep('blue', 7), 'red', rep('blue', 7), rep('green', 3), 'red', rep('green', 3), rep('red', 11)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "### 3.10 The Boxplot"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true,
    "deletable": true,
    "editable": true
   },
   "source": [
    "We have discoverd the mean of the number of words per clause per book. However, there is more to discover than just the mean. How want to know more about the variation within each book, for which a boxplot offers a good solution. For more information on box plots check the [Wikipedia page](https://en.wikipedia.org/wiki/Box_plot)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "install.packages('reshape')\n",
    "library(reshape)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "dat.h.prose <- dat.h[dat.h$genre == 'prose', ]\n",
    "\n",
    "genex.tab <- (table(factor(dat.h.prose$book), dat.h.prose$clause_a.))\n",
    "\n",
    "genex.tab"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "With the function melt() we transform the data frame genex.tab. From a \"wide\" format we go to a \"long\" data format."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "genex.melt <- melt(genex.tab)\n",
    "\n",
    "genex.ready <- subset(genex.melt, genex.melt[ , 3] != 0) \n",
    "\n",
    "genex.ready"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "Now the boxplot is created."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "boxplot(genex.ready[,3] ~ genex.ready[,1], las = 2, ylim = c(0, 20))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "poet <- dat.h[dat.h$genre == 'poetry',]\n",
    "head(poet)\n",
    "table(poet$book, poet$clause_a.)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "### 3.11 Sampling and the Histogram"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "In various publications it is suggested that the article H occurs in lower frequencies in poetry than in prose. A natural way to investigate the variation within a dataset is by using a technique called sampling. Sampling from datasets plays an important role in various statistical techniques, like the bootstrap. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "source": [
    "The function sample() selects a number of elements from a vector randomly. This can be done with or without replacement:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "sample(1:6, 2, replace = T) #this is equal to rolling a die twice\n",
    "sample(1:6, 2, replace = F) #this is not equal to rolling a die twice. Why not?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "A histogram shows the distribution of a numeric variable. Suppose we roll a die 1000 times. With a histogram you can visualize the frequency of each number of eyes."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "die.rolls <- sample(1:6, 1000, replace = T)\n",
    "hist(die.rolls, breaks=seq(0.5,6.5,1)) # the breaks argument makes the figure clearer."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "In this experiment we take a random sample of 500 words from the books of Genesis and Psalms. In each sample we count how often the article H occurs in it and the result is stored in a vector. This is repeated 500 times. At the end we compare the books by making a histogram of the two vectors. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "h.sampler <- function(book, nsamp) {\n",
    "    # this function counts the number of 'H's in a random sample of size nsamp and returns it\n",
    "    rand.vec <- sample(1:nrow(book), nsamp, replace = T)\n",
    "    gen.sample <- book[rand.vec,]\n",
    "    h.count <- nrow(gen.sample[gen.sample$lexeme.t == 'H',])    \n",
    "    }\n",
    "\n",
    "\n",
    "par(mfrow = c(1, 2)) # with par(mfrow = ) we specify the organization of the plots. c(1, 2) means: 1 row, 2 columns, \n",
    "                     # in other words, 2 plots next to each other\n",
    "\n",
    "books <- c('Genesis', 'Psalmi') #the books are specified\n",
    "\n",
    "for (bo in books) {\n",
    "    book.subset <- dat[dat$book == bo,]\n",
    "    h.count.vec <- numeric() # a numeric vector is initiated\n",
    "    \n",
    "    for (i in 1:500) {\n",
    "        count <- h.sampler(book.subset, 500)\n",
    "        h.count.vec[i] <- count # the count is added to the vector \n",
    "    }\n",
    "hist(h.count.vec, col = 'lightblue')\n",
    "abline(v = mean(h.count.vec)) # abline() with v draws vertical line. h = horizontal line\n",
    "rug(h.count.vec) # sometimes this is an interesting addition to the plot. It adds the individual measurements to the x-axis.\n",
    "}\n",
    "\n",
    "par(mfrow = c(1, 1)) # we restore the original organization of the plots."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true,
    "deletable": true,
    "editable": true
   },
   "source": [
    "### 3.12 Hierarchical clustering"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "Hierarchical clustering is a multivariate technique, in which groups are made of similar objects. Similar objects are objects with more or less the same values for the features in the dataset. Characteristic of hierarchical clustering is that groups of nested clusters are formed in one supercluster. In a dataset one calculates the distances between the objects using the function dist()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "First we remove proper nouns."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "dat.h.2 <- dat.h[dat.h$part.of.speech != 'nmpr',]\n",
    "dat.h.2 <- droplevels(dat.h.2)\n",
    "head(dat.h.2)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "lex <- table(dat.h.2$book, dat.h.2$lexeme.t)\n",
    "lexem.binary <- apply(lex, 2, function(x) {ifelse(x > 0, 1, 0)})\n",
    "select.lex <- lexem.binary[,colSums(lexem.binary) > 1 & colSums(lexem.binary) < 11]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "In the next cell the distance object lex.dist is calculated. Check what it looks like. The function dist() has an argument 'method'. In this argument you specify the method with which the distance is measured. For binary data (1 (present) or 0 (absent)), you use the method 'binary', in general. Other methods are 'euclidian', 'manhattan', ''"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "lex.dist <- dist(select.lex, method = 'binary')\n",
    "plot(hclust(lex.dist))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "There is a lot of interesting literature on clustering, for instance in corpus linguistics:  \n",
    "Moisl, H., Cluster Analysis for Corpus Linguistics, Mouton De Gruyter, 2015.  \n",
    "\n",
    "And a practical introduction on clustering with R:\n",
    "\n",
    "Kassambara, A., Practical Guide to Cluster Analysis in R: Unsupervised Machine Learning, sthda.com, 2017."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "### 3.13 The Heatmap"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "An alternative way to visualize these data is using a heatmap. A heatmap is a rectangular grid in which in each cell the value of 1 variable of 1 object is represented by a color. Some heatmap functions also perform a clustering, like the one that is used here."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "heatmap(select.lex, labCol = '')\n",
    "\n",
    "#with labCol = '', the labels of the lexemes are removed. Because there are so many, they are unreadable."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "### 3.14 Merging dataframes"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "It may occur that you use data from different sources, which you want to use together in your research. Then it useful to merge different dataframes."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "First we make two dataframes consisting of a number of fictive books. dat.frame.A has information about the size of the books and dat.frame.B has information about the genre."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "books.1 <- LETTERS[seq(from = 1, to = 10)]\n",
    "num.of.words <- 1000 * 1:10\n",
    "\n",
    "dat.frame.A <- as.data.frame(cbind(books.1, num.of.words))\n",
    "\n",
    "books.2 <- LETTERS[seq(from = 6, to = 15)]\n",
    "genre <- c(rep('prose', 5), rep('poetry', 5))\n",
    "\n",
    "dat.frame.B <- as.data.frame(cbind(books.2, genre))\n",
    "\n",
    "dat.frame.A\n",
    "\n",
    "dat.frame.B"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "These dataframes can be merged in different ways, but we use the vaiable book as key for merging. Check carefully what the following four scripts do. If you download different csv files from SHEBANQ, you can use the feature \"monad\" as key for merging dataframes."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "merge(dat.frame.A, dat.frame.B, by.x = 'books.1', by.y = 'books.2') # this is an inner join"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "merge(dat.frame.A, dat.frame.B, by.x = 'books.1', by.y = 'books.2', all = T) # this is an outer join"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "merge(dat.frame.A, dat.frame.B, by.x = 'books.1', by.y = 'books.2', all.x = T) # this is a left join"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "merge(dat.frame.A, dat.frame.B, by.x = 'books.1', by.y = 'books.2', all.y = T) # this is a right join"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "### 3.15 Finally..."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "Most of the code in this course is based on the base R package. However, many things can be done with other packages. There are many packages with specialized statistical tools, but also packages for general operations. The package dplyr is useful for data wrangling and ggplot2 can be used for making beautiful visualizations. \n",
    "\n",
    "Look here for cheatsheets:  \n",
    "https://www.rstudio.com/wp-content/uploads/2015/02/data-wrangling-cheatsheet.pdf  \n",
    "https://www.rstudio.com/wp-content/uploads/2015/03/ggplot2-cheatsheet.pdf  \n",
    "\n",
    "More interesting cheatsheets by R-studio can be found here:  \n",
    "https://www.rstudio.com/resources/cheatsheets/"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "Of course there is much more to learn about R. As you might know already, doing things with R is more important than reading about R. There are many good practical online courses for free or a bit of money.  \n",
    "\n",
    "You can find some here:  \n",
    "\n",
    "https://www.coursera.org/courses?languages=en&query=data+sience+r  \n",
    "https://www.edx.org/course?search_query=r  \n",
    "https://www.udemy.com/courses/search/?q=r&src=ukw  \n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "Are you looking for books on language analysis and R?  \n",
    "\n",
    "Baayen, R.H., Analyzing Linguistic Data: A Practical Introduction to Statistics using R, Cambridge University Press, 2008.  \n",
    "Gries, S.Th., Quantitative Corpus Linguistics with R: A Practical Introduction, Routledge, 2016, 2nd edition.  \n",
    "Levshina, N., How to do Linguistics with R: Data exploration and statistical analysis, John Benjamins, 2015.  "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "Finally, if you want to receive a daily newsletter about R and statistical analyses and visualizations, subscribe on www.r-bloggers.com . This is both for beginners and more advanced users of R."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "R",
   "language": "R",
   "name": "ir"
  },
  "language_info": {
   "codemirror_mode": "r",
   "file_extension": ".r",
   "mimetype": "text/x-r-source",
   "name": "R",
   "pygments_lexer": "r",
   "version": "3.4.1"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}