NBER-methods-post.Rmd

---
title: "NBER-methods-post"
author: "JJayes"
date: "03/11/2021"
output: html_document
---

```{r setup, include=FALSE}
knitr::opts_chunk$set(echo = TRUE)

library(here)
library(tidyverse)
library(tidyquant)
theme_set(theme_light())
```

## Purpose

Visualize trends in NBER paper abstract terms over time.

### Reading in data

We have nearly 30,000 papers.

```{r}
df <- read_rds(here("data", "abstracts_df.rds")) %>% 
    unnest(abstract)

df %>% 
    skimr::skim()
```

### Function

Large function to plot the term frequency over time. It has two inputs, a list of terms, and a smoothing factor.

```{r}
nber_method_plot <- function(terms, smoother) {
  get_term_share <- function(method) {
    df %>%
      mutate(
        time_lump = year - year %% {{ smoother }},
        abstract = str_to_lower(abstract)
      ) %>%
      count(time_lump, rd = str_detect(abstract, method)) %>%
      pivot_wider(names_from = rd, values_from = n, values_fill = 0) %>%
      mutate(share = `TRUE` / (`FALSE` + `TRUE`)) %>%
      select(time_lump, share)
  }

  terms_df <- terms %>%
    as_tibble() %>%
    rename(method = value) %>%
    mutate(
      method = str_to_lower(method),
      share = map(method, get_term_share)
    ) %>%
    unnest(share) %>%
    mutate(method = str_to_title(method))

  terms_df_labs <- terms_df %>%
    filter(time_lump == max(time_lump))
  
  n_terms <- terms_df %>%
    distinct(method) %>% 
    count() %>% pull()
  
  n_rows <- round((n_terms + 0) / 2)

  terms_df %>%
    ggplot(aes(time_lump, share, colour = method)) +
    geom_point() +
    geom_line(cex = 1) +
    geom_text(aes(label = method),
      data = terms_df_labs %>%
        filter(time_lump == max(time_lump)),
      hjust = -.1,
      show.legend = F,
      check_overlap = T
    ) +
    expand_limits(x = 2040) +
    scale_y_continuous(labels = scales::percent_format()) +
    scale_colour_tq(theme = "dark") +
    labs(
      x = "Year of most recent revision as NBER wp",
      colour = "Term",
      y = "Share of NBER working papers with term in abstract",
      title = "Popular terms in NBER working paper abstracts"
    ) +
    theme(legend.position = "bottom") +
    guides(colour = guide_legend(nrow = n_rows, byrow = TRUE))
}
```

### Term frequency over time

Let us begin with popular methods, to replicate figure from the economist.

```{r}
jpeg(filename = here("images", "Methods_2021.jpeg"),
     height = 6,
     width = 8,
     res = 1000,
     units = "in")

nber_method_plot(c("difference-in-difference", 
                   "regression discontinuity", 
                   "randomized controlled trial", 
                   "dynamic stochastic", 
                   "machine learning|big data"), 2) +
    theme(legend.position = "none")

dev.off()
```

Why do we observe a drop off towards the end of the series??

```{r}
jpeg(filename = here("images", "Covid.jpeg"),
     height = 6,
     width = 8,
     res = 1000,
     units = "in")

nber_method_plot(c("covid-19", 
                   "difference-in-difference", 
                   "regression discontinuity", 
                   "randomized controlled trial", 
                   "dynamic stochastic", 
                   "machine learning|big data"), 2) +
    theme(legend.position = "none")

dev.off()
```

Covid-19! Perhaps not such a surprise, if you're active on twitter.

```{r}
jpeg(filename = here("images", "Money_interest_unemployment.jpeg"),
     height = 6,
     width = 8,
     res = 1000,
     units = "in") 

nber_method_plot(c("money", "unemployment", "interest rates"), 2) +
    theme(legend.position = "none")

dev.off()
```

What about other trends? A slight downward trend in the focus on growth - good news for [Kate Raworth](https://twitter.com/kateraworth).

```{r}
jpeg(filename = here("images", "Growth_development_poverty.jpeg"),
     height = 6,
     width = 8,
     res = 1000,
     units = "in") 

nber_method_plot(c("growth", "development", "poverty"), 2) +
    scale_colour_tq(theme = "green") +
    theme(legend.position = "none")

dev.off()
```