-
Notifications
You must be signed in to change notification settings - Fork 0
/
coursework2.Rmd
149 lines (93 loc) · 5 KB
/
coursework2.Rmd
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
---
title: "Ethics4DS: Coursework 2"
author: "[Candidate number here]"
date: "`r Sys.Date()`"
output: html_document
---
```{r setup, include=FALSE}
knitr::opts_chunk$set(echo = TRUE)
library(tidyverse)
theme_set(theme_minimal())
candidate_number <- 1 # change this
set.seed(candidate_number)
```
### Reproducibility and causality
#### Reproducible and causal [20 marks]
- Create a simulation with data generating code that is consistent with a specific causal model structure. Choose the structure to satisfy all these requirements [10 marks]
```{r}
# Data generation
# Structure: e.g. "Z -> Y and Z -> X -> Y"
```
- Show that there is a reproducible effect, i.e. one that can be found fairly consistently (e.g. in more than five percent of experiments) without using p-hacking [5 marks]
```{r}
# Code for simulating multiple experiments
```
```{r}
# Code for showing the significance of the effect
```
- Show, by simulating an intervention, that the above effect is causal [5 marks]
```{r}
# Code for simulating an intervention
# and showing the effect on an outcome
```
#### Reproducible but not causal [20 marks]
- Repeat the above section, but in this case choose the causal model generating your data so that the reproducible effect is not causal, i.e. an intervention on that variable does not change the outcome variable [10 marks]
```{r}
# Data generation
# Structure:
```
- Show that there is a reproducible effect, i.e. one that can be found fairly consistently (e.g. in more than five percent of experiments) without using p-hacking [5 marks]
```{r}
# Code for simulating multiple experiments
```
```{r}
# Code for showing the significance of the effect
```
- Show, by simulating an intervention, that the above effect is not causal [5 marks]
```{r}
# Code for simulating an intervention
# and showing the effect on an outcome
```
### Fairness and causality
- Create a simulation with data generating code that is consistent with a specific causal model structure. Choose the structure to satisfy all these requirements
#### Data generation [10 marks]
- Variable `A` should be a categorical "sensitive attribute," `Y` should be an outcome to be predicted, and `X` some other predictor. If you decide to include an unobserved variable, name it `U` [5 marks]
- In your example, `A` should not have any causal effect on `Y` (including direct or indirect effects), i.e. there should be no directed pathway from `A` to `Y` in the structure graph [5 marks]
```{r}
# Data generation
# Structure:
```
#### Predictive accuracy [20 marks]
- Fit a "full" model predicting `Y` from `A` and `X`, and a separate "unaware" model predicting `Y` from `X` only [10 marks]
```{r}
# Generate training data (if not already)
# Fit models
```
- Generate a second sample of data and compare the predictive accuracy of these models when predicting on the new sample. If the full model is not significantly more accurate, change the data generating code until it is
- Use this test data for all remaining parts of the coursework below this point
```{r}
# Generate test data
```
- Compare predictive accuracy of the two models on test data [10 marks]
- Hint: You may wish to read about the `newdata` argument in `?predict.lm` or `?predict.glm`
- Choose any accuracy measure you wish, e.g. if `Y` is numeric you could use mean squared error, sqrt(MSE), median absolute error, etc. If it's binary you could use misclassification rate, or false positive rate, or false negative rate, etc.
- Note: if `Y` is binary and you're using logistic regression, you may want to see `?predict.glm` and read about the `response` argument
```{r}
# Compare predictive accuracy on test data
```
#### Disparate predictions? [20 marks]
- For each of the two predictive models, compare the average predicted outcomes for two subsamples with different values of `A` (e.g. if `A` is a binary, 0/1 variable, compare average predictions for the `A == 0` group and the `A == 1` group) [10 marks]
```{r}
# Hint: use `subset()` or `dplyr::filter()` with A
```
- For each of the two predictive models, compare the predictive accuracy for the same two subsamples as above [10 marks]
```{r}
# Predictive accuracy in each group
```
#### Story time [10 marks]
(Delete from here to the beginning of your own writing before the final knit)
- Describe a (reasonably plausible) real world scenario that could fit with your answers to this section.
- What do the variables represent? Who would be fitting/using the predictive models, and what would they use the predictions for? How could a disparity in the predictions of the models potentially affect people and make ethics relevant for the example? [5 marks]
- Give an explanation for why the variables, with the real world meanings you have given them, could possibly not have any causal relationship between `A` and `Y`, even though using `A` results in more accurate predictions [5 marks]
Remember to replace `candidate_number` and "[Candidate number here]" at the top of the document and knit one last time before submitting
Write here and delete this text