From d326675b65796b8ab92282cb59b9e5932edf044f Mon Sep 17 00:00:00 2001 From: jimbolepore Date: Mon, 30 Apr 2018 17:30:22 -0700 Subject: [PATCH] James Lepore HW 1 --- .../Stats412_HW1_James_Lepore_003933437.Rmd | 240 +++++++++ .../Stats412_HW1_James_Lepore_003933437.html | 510 ++++++++++++++++++ 2 files changed, 750 insertions(+) create mode 100644 week-2/problem-set-1/Stats412_HW1_James_Lepore_003933437.Rmd create mode 100644 week-2/problem-set-1/Stats412_HW1_James_Lepore_003933437.html diff --git a/week-2/problem-set-1/Stats412_HW1_James_Lepore_003933437.Rmd b/week-2/problem-set-1/Stats412_HW1_James_Lepore_003933437.Rmd new file mode 100644 index 0000000..7553430 --- /dev/null +++ b/week-2/problem-set-1/Stats412_HW1_James_Lepore_003933437.Rmd @@ -0,0 +1,240 @@ +--- +title: "Stats 412 Homework 1" +author: 'James Lepore (UID: 003933437)' +date: "4/30/2018" +output: html_document +highlight: pygments +--- + +```{r setup, include=FALSE} +knitr::opts_chunk$set(echo = TRUE) + +library(MASS) +library(ROCR) +``` + +### The Sound of Gunfire, Off in the Distance +Our first dataset this week comes from a study of the causes of civil wars.[^1] The data +can be read into from a csv posted online by using the following command. + +```{r} +war <- read.csv("http://www.stat.cmu.edu/~cshalizi/uADA/15/hw/06/ch.csv", row.names = 1) +summary(war) +``` + +Every row of the data represents a combination of a country and of a five year interval — the +first row is Afghanistan, 1960, really meaning Afghanistan, 1960–1965. The +variables are: + +- The country name; +- The year; +- An indicator for whether a civil war began during that period: 1 indicates a +civil war has begun, the code of NA means an on-going civil war, 0 means peace. +- Exports, really a measure of how dependent the country’s economy is on com- modity exports; +- Secondary school enrollment rate for males, as a percentage; +- Annual growth rate in GDP; +- An index of the geographic concentration of the country’s population (which would be 1 if the entire population lives in one city, and 0 if it evenly spread across the territory); +- The number of months since the country’s last war or the end of World War II, whichever is more recent; +- The natural logarithm of the country’s population; +- An index of social “fractionalization”, which tries to measure how much the +country is divided along ethnic and/or religious lines; +- An index of ethnic dominance, which tries to measure how much one ethnic +group runs affairs in the country. + +Some of these variables are NA for some countries. + +### 1 +**Estimate**: Fit a logistic regression model for the start of civil war on all other variables except country and year (yes, this makes some questionable assumptions about independent observations); include a quadratic term for exports. Report the coefficients and their standard errors, together with R’s p-values. Which ones are found to be significant at the 5% level? + +```{r} +glm.fit <- glm(start ~ . + I(exports^2), family=binomial(link="logit"), data=war[,-c(1,2)]) +summary(glm.fit) +``` + +All the coefficients except for the index for ethnic "dominance" are significant at the $\alpha = .05$ level. + +### 2 +**Interpretation**: All parts of this question refer to the logistic regression model you just fit. + + 1. What is the model’s predicted probability for a civil war in India in the period beginning 1975? What probability would it predict for a country just like India in 1975, except that its male secondary school enrollment rate was 30 points higher? What probability would it predict for a country just like India in 1975, except that the ratio of commodity exports to GDP was 0.1 higher? + +```{r} +(india.1975 <- war[which(war$country=="India" & war$year==1975),]) +glm.fit$fitted.values[toString(which(war$country=="India" & war$year==1975))] +``` + +The model’s predicted probability for a civil war in India in the period beginning 1975 is 35.04%. + +```{r} +india.hyp.1 <- india.1975 +india.hyp.1$schooling = india.hyp.1$schooling + 30 + +predict.glm(glm.fit, india.hyp.1, type="response") +``` + +If everything else remains the same for India in the period beginning 1975, but the male secondary school enrollment rate increases by 30 percentage points, then the predicted probability of war decreases by more than half to 17.31%. + +```{r} +india.hyp.2 <- india.1975 +india.hyp.2$exports = india.hyp.2$exports + .1 + +predict.glm(glm.fit, india.hyp.2, type="response") +``` + +If everything else remains the same for India in the period beginning 1975, except the ratio of commodity exports to GDP was 0.1 higher, then the predicted probability of war almost doubles to 69.61%. + + 2. What is the model’s predicted probability for a civil war in Nigeria in the period beginning 1965? What probability would it predict for a country just like Nigeria in 1965, except that its male secondary school enrollment rate was 30 points higher? What probability would it predict for a country just like Nigeria in 1965, except that the ratio of commodity exports to GDP was 0.1 higher? + +```{r} +(nigeria.1965 <- war[which(war$country=="Nigeria" & war$year==1965),]) +glm.fit$fitted.values[toString(which(war$country=="Nigeria" & war$year==1965))] +``` + +The model’s predicted probability for a civil war in Nigeria in the period beginning 1965 is 17.10%. + +```{r} +nigeria.hyp.1 <- nigeria.1965 +nigeria.hyp.1$schooling = nigeria.hyp.1$schooling + 30 + +predict.glm(glm.fit, nigeria.hyp.1, type="response") +``` + +If everything else remains the same for Nigeria in the period beginning 1965, but the male secondary school enrollment rate increases by 30 percentage points, then the predicted probability of war decreases by more than half to 7.41%. + +```{r} +nigeria.hyp.2 <- nigeria.1965 +nigeria.hyp.2$exports = nigeria.hyp.2$exports + .1 + +predict.glm(glm.fit, nigeria.hyp.2, type="response") +``` + +If everything else remains the same for Nigeria in the period beginning 1965, except the ratio of commodity exports to GDP was 0.1 higher, then the predicted probability of war almost doubles to 33.10%. + + 3. In the parts above, you changed the same predictor variables by the same amounts. If you did your calculations properly, the changes in predicted probabilities are not equal. Explain why not. (The reasons may or may not be the same for the two variables.) + +The primary reason that the changes in predicted probabilities are not equal in magnitude is that the coefficients of a logistic regression model with a logit link measure changes in log odds. To put it in perhaps a simpler light, when the coefficient of a logit is small in magnitude (i.e. less < .1), the coefficient can actually be thought of as an approximation of the percent change for every one unit increase. The percent change in predicted probability is obviously dependent on the baseline and starting input values. India in 1975 (35.04%) and Nigeria in 1965 (17.09%) are starting from two very different places, and the male secondary school enrollment rates are also quite different (36% for India versus 7% for Nigeria). This is in contrast to a linear regression where the coefficients measure the linear effect of a one unit increase in the input (meaning the base and starting value are irrelevant). That being said, even in a regular regression model we would still observe a different magnitude change in predicted probabilites for equivalent increases in the ratio of commodity exports to GDP. The reasoning why is quite similar. If you recall, we modeled the exports variable with an additional quadratic term, meaning the linear interpretation of the coefficients continues to no longer be applicable because once again the starting input values will matter. For example, a .1 increase from .026 (India 1975) will result in a .0152 increase in the quadratic input. Whereas a .1 increase from .123 (Nigeria 1965) will result in a .0346 quadratic input (more than double the magnitude). + +### 3 +**Confusion**: Logistic regression predicts a probability of civil war for each country and period. Suppose we want to make a definite prediction of civil war or not, that is, to classify each data point. The probability of misclassification is minimized by predicting war if the probability is ≥ 0.5, and peace otherwise. + + 1. Build a 2 × 2 *confusion matrix* (a.k.a. “classification table” or “contigency table”) which counts: the number of outbreaks of civil war correctly predicted by the logistic regression; the number of civil wars not predicted by the model; the number of false predictions of civil wars; and the number of correctly predicted absences of civil wars. (Note that some entries in the table may be zero.) + +```{r} +pred.labels <- rep("Peace", length(glm.fit$fitted.values)) +pred.labels[glm.fit$fitted.values>=0.5] = "War" +war$start.labels[war$start==0] = "Peace" +war$start.labels[war$start==1] = "War" +(cm <- table(pred.labels, war[complete.cases(war),]$start.labels)) +``` + + 2. What fraction of the logistic regression’s predictions are incorrect, i.e. what is the misclassification rate? (Note that this is if anything too kind to the model, since it’s looking at predictions to the same training data set). + +```{r} +mean(pred.labels!=war[complete.cases(war),]$start.labels) +``` + +The misclassification rate for this training data set is 6.98%. + + 3. Consider a foolish (?) pundit who always predicts “no war”. What fraction of the pundit’s predictions are correct on the whole data set? What fraction are correct on data points where the logistic regression model also makes a prediction? + +```{r} +(naive.1 = table(war$start.labels)) +prop.table(naive.1) + +(naive.2 = table(war[complete.cases(war),]$start.labels)) +prop.table(naive.2) +``` + +A naive prediction (always picking "Peace") has a misclassification rate of about 6.69% for both the whole data set and just the observations with no missing data. Because civil war is a relatively unlikely event to occur, and because even when it does occur it is relatively difficult to predict, the naive approach actually does slightly better than the logistic regression in terms of accuracy. It is always a good idea to calculate the naive prediction accuracy so we have an idea of what the baseline is when evaluating models. + +### 4 +**Comparison**: Since this is a classification problem with only two classes, we can compare Logistic Regression right along side Discriminant Analysis. This will require some reading. (see Introduction to Statistical Learning pages 138-149) + + 1. Fit an Linear Discriminant Analysis (LDA) model using the same predictors that you used for your logistic regression model. What is the training misclassification rate? + +```{r} +lda.fit <- lda(start ~ ., data=war[,-c(1,2,12)]) +lda.fit + +lda.pred <- predict(lda.fit, war[complete.cases(war),]) + +(cm <- table(lda.pred$class, war[complete.cases(war),]$start)) + +mean(lda.pred$class!=war[complete.cases(war),]$start) +``` + +The training misclassification rate for the LDA is 6.40%. + + 2. Fit a Quadratic Discriminant Analysis (QDA) model using the very same predictors. What is the training misclassification rate? + +```{r} +qda.fit <- qda(start ~ ., data=war[,-c(1,2,12)]) +qda.fit + +qda.pred <- predict(qda.fit, war[complete.cases(war),]) + +(cm <- table(qda.pred$class, war[complete.cases(war),]$start)) + +mean(qda.pred$class!=war[complete.cases(war),]$start) +``` + +The training misclassification rate for the QDA is 6.40%. + + 3. How does the prediction accuracy of the three models compare? Why do you think this is? + +The prediction accuracy for both the LDA and QDA is exactly the same at 6.40% (44 errors distributed differently with the QDA more generously predicting civil wars). This misclassification rate is lower (albeit modestly) than both the logistic regression (48 errors) and naive approach (46 errors)! LDA and QDA can give more stable estimates when the sample size is small and assumptions can be made about the distributions of the input variables, and when the classes are fairly well-separated (which, given the relative infrequency of civil wars, is somewhat true in this case). + +### 5 +**ROC**: Construct an ROC curve for all three of your models. Plot the ROC curves of all three models on the same plot. + +```{r, fig.width=9, fig.height=7} +rocr.pred.glm <- prediction(glm.fit$fitted.values, war[complete.cases(war),]$start) +performance(rocr.pred.glm, "auc")@y.values[[1]] + +rocr.pred.lda <- prediction(lda.pred$posterior[,2], war[complete.cases(war),]$start) +performance(rocr.pred.lda, "auc")@y.values[[1]] + +rocr.pred.qda <- prediction(qda.pred$posterior[,2], war[complete.cases(war),]$start) +performance(rocr.pred.qda, "auc")@y.values[[1]] + +perf.glm <- performance(rocr.pred.glm,"tpr","fpr") +perf.lda <- performance(rocr.pred.lda,"tpr","fpr") +perf.qda <- performance(rocr.pred.qda,"tpr","fpr") + +plot(perf.glm, col="red", lwd=3, main="ROC Curve for GLM, LDA, and QDA") +plot(perf.lda, col="blue", lwd=3, add=TRUE) +plot(perf.qda, col="darkgreen", lwd=3, add=TRUE) +abline(a=0, b=1, col="black", lwd=2) + +legend("bottomright", legend=c("GLM", "LDA", "QDA"), +col=c("red", "blue", "darkgreen"), lty=1, lwd=3, +box.lty=0, cex=1.4) +``` + +Looking at the plot of the ROC curve, we can easily observe that the QDA has the largest area under the curve (AUC) of the three methods (0.8864621 versus 0.8597792 for the GLM, and 0.8315387 for the LDA). This result is important in model evaluation because it indicates which method yields the best balance between maximizing the true positive rate (sensitivity) while minimizing the false positive rate (1 - specificity) across different possible thresholds (naturally, any increase in sensitivity will correspond with a decrease in specificity). This area under the curve measures what is referred to as discrimination, which can be loosely defined as the ability of the model to correctly classify randomly drawn pairs from each of the classes. So, while the QDA and LDA both had the same misclassification rate, QDA appears to be the best fit for classification for this specific training set. Another consideration when doing classification is whether or not there is a preference for which type of error matters the most. In this case, I would lean towards saying that for planning purposes it is worse to predict peace when there is actually going to be a civil war than it is to predict civil war when there will actually be peace. With this in mind, it might be beneficial to take that into account when choosing an optimal threshold. When using the .5 threshold we used for the confusion matrices in the previous problem, the QDA had the lowest false negative rate which is perhaps another point in its favor. + +[^1]: Based on an exercise of Cosmo Shalizi's that uses data from Collier, Paul and Anke Hoeffler (2004). *Greed and Grievance in Civil War.* Oxford Economic Papers, 56: 563–595. URL: http://economics.ouls.ox.ac.uk/12055/1/2002-01text.pdf. + + +### 6 + +Fit a logistic regression using `y` as the response with `x1` and `x2` as indepedent variables. Does anything strange happen? Explain. + +```{r} +y<- c(0,0,0,0,1,1,1,1) +x1<-c(1,2,3,3,5,6,10,11) +x2<-c(3,2,-1,-1,2,4,1,0) + +glm.fit2 <- glm(y ~ x1 + x2, family=binomial(link="logit")) +summary(glm.fit2) +``` + +When we run a logistic regression on this sample data, we get the following warning message: "glm.fit: fitted probabilities numerically 0 or 1 occurred". This warning means that the data is separable, or, in other words, that there is a cut in one or more of the predictors that can perfectly predict the response. Looking at the data, we can identify where this separation occurs. When the independent variable "x1" is less than or equal to 3, the response variable "y" is always 0, and when the independent variable "x1" is greater than 3, the response variable "y" is always 1. This type of separation is called complete separation. When this occurs, there is no maximum likelihood estimate for the "x1" parameter and so the coefficients of the model want to go to infinity since the classification is perfect. In R, this behavior is represented with a large estimate and an enormous standard error. While perfect predictors can certainly provide useful information, that information must be evaluated externally and then those predictors must be taken out before running the logistic regression (or you can use a different method that handles these predictors such as an exact logistic regression or introducing a complexion penalty parameter). For example, let us suppose that we have reason to believe that values less than or equal to 3 in "x1" would not ***always*** lead to a 0 response, and this phenomenon in our sample data is simply due to the small sample size. In this case, we might be interested in knowing if "x2" has any predictive power. Dropping out "x1" results in the following: + +```{r} +glm.fit3 <- glm(y ~ x2, family=binomial(link="logit")) +summary(glm.fit3) +``` + +We no longer get a warning message and the coefficients and standard errors look normal, but we also see that "x2" is not a statistically significant predictor. We would now probably have to either gather more data so that we can observe cases that would result in no longer having perfect prediction (i.e. observations with "x1" less than or equal to 3 and a "y" of 1, and "x1" greater than 3 and a "y" of 0), or try to understand why "x1" might in fact be a perfect predictor (in which case, running fancy algorithms for prediction would be unneccessary assuming "x1" is easily obtained!) + diff --git a/week-2/problem-set-1/Stats412_HW1_James_Lepore_003933437.html b/week-2/problem-set-1/Stats412_HW1_James_Lepore_003933437.html new file mode 100644 index 0000000..3c3a7d2 --- /dev/null +++ b/week-2/problem-set-1/Stats412_HW1_James_Lepore_003933437.html @@ -0,0 +1,510 @@ + + + + + + + + + + + + + + +Stats 412 Homework 1 + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
+ + + + + + + + + + + + + + +
+

The Sound of Gunfire, Off in the Distance

+

Our first dataset this week comes from a study of the causes of civil wars.1 The data can be read into from a csv posted online by using the following command.

+
war <- read.csv("http://www.stat.cmu.edu/~cshalizi/uADA/15/hw/06/ch.csv", row.names = 1)
+summary(war)
+
##         country          year          start            exports      
+##  Afghanistan:   8   Min.   :1960   Min.   :0.00000   Min.   :0.0020  
+##  Algeria    :   8   1st Qu.:1969   1st Qu.:0.00000   1st Qu.:0.0515  
+##  Angola     :   8   Median :1978   Median :0.00000   Median :0.1090  
+##  Argentina  :   8   Mean   :1978   Mean   :0.06684   Mean   :0.1640  
+##  Australia  :   8   3rd Qu.:1986   3rd Qu.:0.00000   3rd Qu.:0.2030  
+##  Austria    :   8   Max.   :1995   Max.   :1.00000   Max.   :2.1390  
+##  (Other)    :1240                  NA's   :121       NA's   :125     
+##    schooling          growth             peace       concentration   
+##  Min.   :  0.30   Min.   :-22.0760   Min.   :  1.0   Min.   :0.0000  
+##  1st Qu.: 16.00   1st Qu.: -0.3135   1st Qu.:184.0   1st Qu.:0.4670  
+##  Median : 38.00   Median :  1.8740   Median :305.0   Median :0.5830  
+##  Mean   : 42.65   Mean   :  1.5631   Mean   :326.6   Mean   :0.5751  
+##  3rd Qu.: 66.00   3rd Qu.:  3.7222   3rd Qu.:472.0   3rd Qu.:0.7500  
+##  Max.   :147.00   Max.   : 14.4090   Max.   :592.0   Max.   :0.9710  
+##  NA's   :251      NA's   :370        NA's   :121     NA's   :160     
+##      lnpop       fractionalization   dominance    
+##  Min.   :10.62   Min.   :   4      Min.   :0.000  
+##  1st Qu.:14.25   1st Qu.: 171      1st Qu.:0.000  
+##  Median :15.48   Median : 936      Median :0.000  
+##  Mean   :15.35   Mean   :1784      Mean   :0.473  
+##  3rd Qu.:16.55   3rd Qu.:3162      3rd Qu.:1.000  
+##  Max.   :20.91   Max.   :6975      Max.   :1.000  
+##  NA's   :22      NA's   :128       NA's   :104
+

Every row of the data represents a combination of a country and of a five year interval — the first row is Afghanistan, 1960, really meaning Afghanistan, 1960–1965. The variables are:

+
    +
  • The country name;
    • +
  • The year;
    • +
  • An indicator for whether a civil war began during that period: 1 indicates a civil war has begun, the code of NA means an on-going civil war, 0 means peace.
    • +
  • Exports, really a measure of how dependent the country’s economy is on com- modity exports;
    • +
  • Secondary school enrollment rate for males, as a percentage;
    • +
  • Annual growth rate in GDP;
    • +
  • An index of the geographic concentration of the country’s population (which would be 1 if the entire population lives in one city, and 0 if it evenly spread across the territory);
    • +
  • The number of months since the country’s last war or the end of World War II, whichever is more recent;
    • +
  • The natural logarithm of the country’s population;
    • +
  • An index of social “fractionalization”, which tries to measure how much the country is divided along ethnic and/or religious lines;
    • +
  • An index of ethnic dominance, which tries to measure how much one ethnic group runs affairs in the country.
    • +
+

Some of these variables are NA for some countries.

+
+
+

1

+

Estimate: Fit a logistic regression model for the start of civil war on all other variables except country and year (yes, this makes some questionable assumptions about independent observations); include a quadratic term for exports. Report the coefficients and their standard errors, together with R’s p-values. Which ones are found to be significant at the 5% level?

+
glm.fit <- glm(start ~ . + I(exports^2), family=binomial(link="logit"), data=war[,-c(1,2)])
+summary(glm.fit)
+
## 
+## Call:
+## glm(formula = start ~ . + I(exports^2), family = binomial(link = "logit"), 
+##     data = war[, -c(1, 2)])
+## 
+## Deviance Residuals: 
+##     Min       1Q   Median       3Q      Max  
+## -1.3655  -0.3627  -0.1893  -0.0932   3.3636  
+## 
+## Coefficients:
+##                     Estimate Std. Error z value Pr(>|z|)    
+## (Intercept)       -1.307e+01  2.795e+00  -4.677 2.91e-06 ***
+## exports            1.894e+01  5.865e+00   3.229 0.001243 ** 
+## schooling         -3.156e-02  9.784e-03  -3.225 0.001259 ** 
+## growth            -1.152e-01  4.307e-02  -2.675 0.007466 ** 
+## peace             -3.713e-03  1.093e-03  -3.397 0.000681 ***
+## concentration     -2.487e+00  1.005e+00  -2.474 0.013357 *  
+## lnpop              7.677e-01  1.658e-01   4.632 3.63e-06 ***
+## fractionalization -2.135e-04  9.102e-05  -2.345 0.019020 *  
+## dominance          6.704e-01  3.535e-01   1.896 0.057920 .  
+## I(exports^2)      -2.944e+01  1.178e+01  -2.499 0.012449 *  
+## ---
+## Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
+## 
+## (Dispersion parameter for binomial family taken to be 1)
+## 
+##     Null deviance: 337.73  on 687  degrees of freedom
+## Residual deviance: 256.42  on 678  degrees of freedom
+##   (600 observations deleted due to missingness)
+## AIC: 276.42
+## 
+## Number of Fisher Scoring iterations: 7
+

All the coefficients except for the index for ethnic “dominance” are significant at the \(\alpha = .05\) level.

+
+
+

2

+

Interpretation: All parts of this question refer to the logistic regression model you just fit.

+
    +
  1. What is the model’s predicted probability for a civil war in India in the period beginning 1975? What probability would it predict for a country just like India in 1975, except that its male secondary school enrollment rate was 30 points higher? What probability would it predict for a country just like India in 1975, except that the ratio of commodity exports to GDP was 0.1 higher?
    2. +
+
(india.1975 <- war[which(war$country=="India" & war$year==1975),])
+
##     country year start exports schooling growth peace concentration
+## 500   India 1975     0   0.026        36  0.322   112         0.537
+##        lnpop fractionalization dominance
+## 500 20.23462              2937         0
+
glm.fit$fitted.values[toString(which(war$country=="India" & war$year==1975))]
+
##       500 
+## 0.3504199
+

The model’s predicted probability for a civil war in India in the period beginning 1975 is 35.04%.

+
india.hyp.1 <- india.1975
+india.hyp.1$schooling = india.hyp.1$schooling + 30
+
+predict.glm(glm.fit, india.hyp.1, type="response")
+
##     500 
+## 0.17309
+

If everything else remains the same for India in the period beginning 1975, but the male secondary school enrollment rate increases by 30 percentage points, then the predicted probability of war decreases by more than half to 17.31%.

+
india.hyp.2 <- india.1975
+india.hyp.2$exports = india.hyp.2$exports + .1
+
+predict.glm(glm.fit, india.hyp.2, type="response")
+
##       500 
+## 0.6961378
+

If everything else remains the same for India in the period beginning 1975, except the ratio of commodity exports to GDP was 0.1 higher, then the predicted probability of war almost doubles to 69.61%.

+
    +
  1. What is the model’s predicted probability for a civil war in Nigeria in the period beginning 1965? What probability would it predict for a country just like Nigeria in 1965, except that its male secondary school enrollment rate was 30 points higher? What probability would it predict for a country just like Nigeria in 1965, except that the ratio of commodity exports to GDP was 0.1 higher?
    2. +
+
(nigeria.1965 <- war[which(war$country=="Nigeria" & war$year==1965),])
+
##     country year start exports schooling growth peace concentration
+## 802 Nigeria 1965     1   0.123         7  1.916   232         0.539
+##        lnpop fractionalization dominance
+## 802 17.65479              6090         0
+
glm.fit$fitted.values[toString(which(war$country=="Nigeria" & war$year==1965))]
+
##       802 
+## 0.1709917
+

The model’s predicted probability for a civil war in Nigeria in the period beginning 1965 is 17.10%.

+
nigeria.hyp.1 <- nigeria.1965
+nigeria.hyp.1$schooling = nigeria.hyp.1$schooling + 30
+
+predict.glm(glm.fit, nigeria.hyp.1, type="response")
+
##        802 
+## 0.07410315
+

If everything else remains the same for Nigeria in the period beginning 1965, but the male secondary school enrollment rate increases by 30 percentage points, then the predicted probability of war decreases by more than half to 7.41%.

+
nigeria.hyp.2 <- nigeria.1965
+nigeria.hyp.2$exports = nigeria.hyp.2$exports + .1
+
+predict.glm(glm.fit, nigeria.hyp.2, type="response")
+
##       802 
+## 0.3310044
+

If everything else remains the same for Nigeria in the period beginning 1965, except the ratio of commodity exports to GDP was 0.1 higher, then the predicted probability of war almost doubles to 33.10%.

+
    +
  1. In the parts above, you changed the same predictor variables by the same amounts. If you did your calculations properly, the changes in predicted probabilities are not equal. Explain why not. (The reasons may or may not be the same for the two variables.)
    2. +
+

The primary reason that the changes in predicted probabilities are not equal in magnitude is that the coefficients of a logistic regression model with a logit link measure changes in log odds. To put it in perhaps a simpler light, when the coefficient of a logit is small in magnitude (i.e. less < .1), the coefficient can actually be thought of as an approximation of the percent change for every one unit increase. The percent change in predicted probability is obviously dependent on the baseline and starting input values. India in 1975 (35.04%) and Nigeria in 1965 (17.09%) are starting from two very different places, and the male secondary school enrollment rates are also quite different (36% for India versus 7% for Nigeria). This is in contrast to a linear regression where the coefficients measure the linear effect of a one unit increase in the input (meaning the base and starting value are irrelevant). That being said, even in a regular regression model we would still observe a different magnitude change in predicted probabilites for equivalent increases in the ratio of commodity exports to GDP. The reasoning why is quite similar. If you recall, we modeled the exports variable with an additional quadratic term, meaning the linear interpretation of the coefficients continues to no longer be applicable because once again the starting input values will matter. For example, a .1 increase from .026 (India 1975) will result in a .0152 increase in the quadratic input. Whereas a .1 increase from .123 (Nigeria 1965) will result in a .0346 quadratic input (more than double the magnitude).

+
+
+

3

+

Confusion: Logistic regression predicts a probability of civil war for each country and period. Suppose we want to make a definite prediction of civil war or not, that is, to classify each data point. The probability of misclassification is minimized by predicting war if the probability is ≥ 0.5, and peace otherwise.

+
    +
  1. Build a 2 × 2 confusion matrix (a.k.a. “classification table” or “contigency table”) which counts: the number of outbreaks of civil war correctly predicted by the logistic regression; the number of civil wars not predicted by the model; the number of false predictions of civil wars; and the number of correctly predicted absences of civil wars. (Note that some entries in the table may be zero.)
    2. +
+
pred.labels <- rep("Peace", length(glm.fit$fitted.values))
+pred.labels[glm.fit$fitted.values>=0.5] = "War"
+war$start.labels[war$start==0] = "Peace"
+war$start.labels[war$start==1] = "War"
+(cm <- table(pred.labels, war[complete.cases(war),]$start.labels))
+
##            
+## pred.labels Peace War
+##       Peace   637  43
+##       War       5   3
+
    +
  1. What fraction of the logistic regression’s predictions are incorrect, i.e. what is the misclassification rate? (Note that this is if anything too kind to the model, since it’s looking at predictions to the same training data set).
    2. +
+
mean(pred.labels!=war[complete.cases(war),]$start.labels)
+
## [1] 0.06976744
+

The misclassification rate for this training data set is 6.98%.

+
    +
  1. Consider a foolish (?) pundit who always predicts “no war”. What fraction of the pundit’s predictions are correct on the whole data set? What fraction are correct on data points where the logistic regression model also makes a prediction?
    2. +
+
(naive.1 = table(war$start.labels))
+
## 
+## Peace   War 
+##  1089    78
+
prop.table(naive.1)
+
## 
+##      Peace        War 
+## 0.93316195 0.06683805
+
(naive.2 = table(war[complete.cases(war),]$start.labels))
+
## 
+## Peace   War 
+##   642    46
+
prop.table(naive.2)
+
## 
+##      Peace        War 
+## 0.93313953 0.06686047
+

A naive prediction (always picking “Peace”) has a misclassification rate of about 6.69% for both the whole data set and just the observations with no missing data. Because civil war is a relatively unlikely event to occur, and because even when it does occur it is relatively difficult to predict, the naive approach actually does slightly better than the logistic regression in terms of accuracy. It is always a good idea to calculate the naive prediction accuracy so we have an idea of what the baseline is when evaluating models.

+
+
+

4

+

Comparison: Since this is a classification problem with only two classes, we can compare Logistic Regression right along side Discriminant Analysis. This will require some reading. (see Introduction to Statistical Learning pages 138-149)

+
    +
  1. Fit an Linear Discriminant Analysis (LDA) model using the same predictors that you used for your logistic regression model. What is the training misclassification rate?
    2. +
+
lda.fit <- lda(start ~ ., data=war[,-c(1,2,12)])
+lda.fit
+
## Call:
+## lda(start ~ ., data = war[, -c(1, 2, 12)])
+## 
+## Prior probabilities of groups:
+##          0          1 
+## 0.93313953 0.06686047 
+## 
+## Group means:
+##     exports schooling     growth    peace concentration    lnpop
+## 0 0.1574330  45.64548 1.73095794 357.7850     0.6038349 15.68224
+## 1 0.1668478  28.34783 0.04384783 204.2826     0.5762391 16.58465
+##   fractionalization dominance
+## 0          1764.882 0.4376947
+## 1          2146.696 0.4565217
+## 
+## Coefficients of linear discriminants:
+##                             LD1
+## exports            2.088286e+00
+## schooling         -6.287664e-03
+## growth            -1.380695e-01
+## peace             -4.398539e-03
+## concentration     -1.152895e+00
+## lnpop              3.438987e-01
+## fractionalization -8.559768e-05
+## dominance          3.149086e-01
+
lda.pred <- predict(lda.fit, war[complete.cases(war),])
+
+(cm <- table(lda.pred$class, war[complete.cases(war),]$start))
+
##    
+##       0   1
+##   0 638  40
+##   1   4   6
+
mean(lda.pred$class!=war[complete.cases(war),]$start)
+
## [1] 0.06395349
+

The training misclassification rate for the LDA is 6.40%.

+
    +
  1. Fit a Quadratic Discriminant Analysis (QDA) model using the very same predictors. What is the training misclassification rate?
    2. +
+
qda.fit <- qda(start ~ ., data=war[,-c(1,2,12)])
+qda.fit
+
## Call:
+## qda(start ~ ., data = war[, -c(1, 2, 12)])
+## 
+## Prior probabilities of groups:
+##          0          1 
+## 0.93313953 0.06686047 
+## 
+## Group means:
+##     exports schooling     growth    peace concentration    lnpop
+## 0 0.1574330  45.64548 1.73095794 357.7850     0.6038349 15.68224
+## 1 0.1668478  28.34783 0.04384783 204.2826     0.5762391 16.58465
+##   fractionalization dominance
+## 0          1764.882 0.4376947
+## 1          2146.696 0.4565217
+
qda.pred <- predict(qda.fit, war[complete.cases(war),])
+
+(cm <- table(qda.pred$class, war[complete.cases(war),]$start))
+
##    
+##       0   1
+##   0 628  30
+##   1  14  16
+
mean(qda.pred$class!=war[complete.cases(war),]$start)
+
## [1] 0.06395349
+

The training misclassification rate for the QDA is 6.40%.

+
    +
  1. How does the prediction accuracy of the three models compare? Why do you think this is?
    2. +
+

The prediction accuracy for both the LDA and QDA is exactly the same at 6.40% (44 errors distributed differently with the QDA more generously predicting civil wars). This misclassification rate is lower (albeit modestly) than both the logistic regression (48 errors) and naive approach (46 errors)! LDA and QDA can give more stable estimates when the sample size is small and assumptions can be made about the distributions of the input variables, and when the classes are fairly well-separated (which, given the relative infrequency of civil wars, is somewhat true in this case).

+
+
+

5

+

ROC: Construct an ROC curve for all three of your models. Plot the ROC curves of all three models on the same plot.

+
rocr.pred.glm <- prediction(glm.fit$fitted.values, war[complete.cases(war),]$start)
+performance(rocr.pred.glm, "auc")@y.values[[1]]
+
## [1] 0.8597792
+
rocr.pred.lda <- prediction(lda.pred$posterior[,2], war[complete.cases(war),]$start)
+performance(rocr.pred.lda, "auc")@y.values[[1]]
+
## [1] 0.8315387
+
rocr.pred.qda <- prediction(qda.pred$posterior[,2], war[complete.cases(war),]$start)
+performance(rocr.pred.qda, "auc")@y.values[[1]]
+
## [1] 0.8864621
+
perf.glm <- performance(rocr.pred.glm,"tpr","fpr")
+perf.lda <- performance(rocr.pred.lda,"tpr","fpr")
+perf.qda <- performance(rocr.pred.qda,"tpr","fpr")
+
+plot(perf.glm, col="red", lwd=3, main="ROC Curve for GLM, LDA, and QDA")
+plot(perf.lda, col="blue", lwd=3, add=TRUE)
+plot(perf.qda, col="darkgreen", lwd=3, add=TRUE)
+abline(a=0, b=1, col="black", lwd=2)
+
+legend("bottomright", legend=c("GLM", "LDA", "QDA"),
+col=c("red", "blue", "darkgreen"), lty=1, lwd=3,
+box.lty=0, cex=1.4)
+

+

Looking at the plot of the ROC curve, we can easily observe that the QDA has the largest area under the curve (AUC) of the three methods (0.8864621 versus 0.8597792 for the GLM, and 0.8315387 for the LDA). This result is important in model evaluation because it indicates which method yields the best balance between maximizing the true positive rate (sensitivity) while minimizing the false positive rate (1 - specificity) across different possible thresholds (naturally, any increase in sensitivity will correspond with a decrease in specificity). This area under the curve measures what is referred to as discrimination, which can be loosely defined as the ability of the model to correctly classify randomly drawn pairs from each of the classes. So, while the QDA and LDA both had the same misclassification rate, QDA appears to be the best fit for classification for this specific training set. Another consideration when doing classification is whether or not there is a preference for which type of error matters the most. In this case, I would lean towards saying that for planning purposes it is worse to predict peace when there is actually going to be a civil war than it is to predict civil war when there will actually be peace. With this in mind, it might be beneficial to take that into account when choosing an optimal threshold. When using the .5 threshold we used for the confusion matrices in the previous problem, the QDA had the lowest false negative rate which is perhaps another point in its favor.

+
+
+

6

+

Fit a logistic regression using y as the response with x1 and x2 as indepedent variables. Does anything strange happen? Explain.

+
y<- c(0,0,0,0,1,1,1,1)
+x1<-c(1,2,3,3,5,6,10,11)
+x2<-c(3,2,-1,-1,2,4,1,0)
+
+glm.fit2 <- glm(y ~ x1 + x2, family=binomial(link="logit"))
+
## Warning: glm.fit: fitted probabilities numerically 0 or 1 occurred
+
summary(glm.fit2)
+
## 
+## Call:
+## glm(formula = y ~ x1 + x2, family = binomial(link = "logit"))
+## 
+## Deviance Residuals: 
+##          1           2           3           4           5           6  
+## -2.110e-08  -1.404e-05  -2.522e-06  -2.522e-06   1.564e-05   2.110e-08  
+##          7           8  
+##  2.110e-08   2.110e-08  
+## 
+## Coefficients:
+##               Estimate Std. Error z value Pr(>|z|)
+## (Intercept)    -66.098 183471.722   0.000        1
+## x1              15.288  27362.843   0.001        1
+## x2               6.241  81543.720   0.000        1
+## 
+## (Dispersion parameter for binomial family taken to be 1)
+## 
+##     Null deviance: 1.1090e+01  on 7  degrees of freedom
+## Residual deviance: 4.5454e-10  on 5  degrees of freedom
+## AIC: 6
+## 
+## Number of Fisher Scoring iterations: 24
+

When we run a logistic regression on this sample data, we get the following warning message: “glm.fit: fitted probabilities numerically 0 or 1 occurred”. This warning means that the data is separable, or, in other words, that there is a cut in one or more of the predictors that can perfectly predict the response. Looking at the data, we can identify where this separation occurs. When the independent variable “x1” is less than or equal to 3, the response variable “y” is always 0, and when the independent variable “x1” is greater than 3, the response variable “y” is always 1. This type of separation is called complete separation. When this occurs, there is no maximum likelihood estimate for the “x1” parameter and so the coefficients of the model want to go to infinity since the classification is perfect. In R, this behavior is represented with a large estimate and an enormous standard error. While perfect predictors can certainly provide useful information, that information must be evaluated externally and then those predictors must be taken out before running the logistic regression (or you can use a different method that handles these predictors such as an exact logistic regression or introducing a complexion penalty parameter). For example, let us suppose that we have reason to believe that values less than or equal to 3 in “x1” would not always lead to a 0 response, and this phenomenon in our sample data is simply due to the small sample size. In this case, we might be interested in knowing if “x2” has any predictive power. Dropping out “x1” results in the following:

+
glm.fit3 <- glm(y ~ x2, family=binomial(link="logit"))
+summary(glm.fit3)
+
## 
+## Call:
+## glm(formula = y ~ x2, family = binomial(link = "logit"))
+## 
+## Deviance Residuals: 
+##      Min        1Q    Median        3Q       Max  
+## -1.45339  -0.96788  -0.03171   1.10268   1.37246  
+## 
+## Coefficients:
+##             Estimate Std. Error z value Pr(>|z|)
+## (Intercept)  -0.4477     0.9248  -0.484    0.628
+## x2            0.3588     0.4466   0.803    0.422
+## 
+## (Dispersion parameter for binomial family taken to be 1)
+## 
+##     Null deviance: 11.090  on 7  degrees of freedom
+## Residual deviance: 10.392  on 6  degrees of freedom
+## AIC: 14.392
+## 
+## Number of Fisher Scoring iterations: 4
+

We no longer get a warning message and the coefficients and standard errors look normal, but we also see that “x2” is not a statistically significant predictor. We would now probably have to either gather more data so that we can observe cases that would result in no longer having perfect prediction (i.e. observations with “x1” less than or equal to 3 and a “y” of 1, and “x1” greater than 3 and a “y” of 0), or try to understand why “x1” might in fact be a perfect predictor (in which case, running fancy algorithms for prediction would be unneccessary assuming “x1” is easily obtained!)

+
+
+
+
    +

  1. Based on an exercise of Cosmo Shalizi’s that uses data from Collier, Paul and Anke Hoeffler (2004). Greed and Grievance in Civil War. Oxford Economic Papers, 56: 563–595. URL: http://economics.ouls.ox.ac.uk/12055/1/2002-01text.pdf.↩

    2. +
+
+ + + + +
+ + + + + + + +