diff --git a/_freeze/chapters/aggregation/execute-results/html.json b/_freeze/chapters/aggregation/execute-results/html.json index 82a702f..8381503 100644 --- a/_freeze/chapters/aggregation/execute-results/html.json +++ b/_freeze/chapters/aggregation/execute-results/html.json @@ -1,8 +1,10 @@ { - "hash": "7f66fb53fad0da9a8e9d43a42b42b3de", + "hash": "68440e6d131ccee0354bc095732ca0e2", "result": { - "markdown": "# Aggregation of evaluators judgments (modeling)\n\n\n\n\n\n\n## Notes on sources and approaches\n\n\n::: {.callout-note collapse=\"true\"}\n\n## Hanea et al {-}\n(Consult, e.g., repliCATS/Hanea and others work; meta-science and meta-analysis approaches)\n\n`aggrecat` package\n\n> Although the accuracy, calibration, and informativeness of the majority of methods are very similar, a couple of the aggregation methods consistently distinguish themselves as among the best or worst. Moreover, the majority of methods outperform the usual benchmarks provided by the simple average or the median of estimates.\n\n[Hanea et al, 2021](https://journals.plos.org/plosone/article?id=10.1371/journal.pone.0256919#sec007)\n\n However, these are in a different context. Most of those measures are designed to deal with probablistic forecasts for binary outcomes, where the predictor also gives a 'lower bound' and 'upper bound' for that probability. We could roughly compare that to our continuous metrics with 90% CI's (or imputations for these).\n\nFurthermore, many (all their successful measures?) use 'performance-based weights', accessing metrics from prior prediction performance of the same forecasters We do not have these, nor do we have a sensible proxy for this. \n:::\n\n\n::: {.callout-note collapse=\"true\"}\n## D Veen et al (2017)\n\n[link](https://www.researchgate.net/profile/Duco-Veen/publication/319662351_Using_the_Data_Agreement_Criterion_to_Rank_Experts'_Beliefs/links/5b73e2dc299bf14c6da6c663/Using-the-Data-Agreement-Criterion-to-Rank-Experts-Beliefs.pdf)\n\n... we show how experts can be ranked based on their knowledge and their level of (un)certainty. By letting experts specify their knowledge in the form of a probability distribution, we can assess how accurately they can predict new data, and how appropriate their level of (un)certainty is. The expert’s specified probability distribution can be seen as a prior in a Bayesian statistical setting. We evaluate these priors by extending an existing prior-data (dis)agreement measure, the Data Agreement Criterion, and compare this approach to using Bayes factors to assess prior specification. We compare experts with each other and the data to evaluate their appropriateness. Using this method, new research questions can be asked and answered, for instance: Which expert predicts the new data best? Is there agreement between my experts and the data? Which experts’ representation is more valid or useful? Can we reach convergence between expert judgement and data? We provided an empirical example ranking (regional) directors of a large financial institution based on their predictions of turnover. \n\nBe sure to consult the [correction made here](https://www.semanticscholar.org/paper/Correction%3A-Veen%2C-D.%3B-Stoel%2C-D.%3B-Schalken%2C-N.%3B-K.%3B-Veen-Stoel/a2882e0e8606ef876133f25a901771259e7033b1)\n\n::: \n\n\n::: {.callout-note collapse=\"true\"}\n## Also seems relevant:\n\nSee [Gsheet HERE](https://docs.google.com/spreadsheets/d/14japw6eLGpGjEWy1MjHNJXU1skZY_GAIc2uC2HIUalM/edit#gid=0), generated from an Elicit.org inquiry.\n\n\n::: \n\n\n\nIn spite of the caveats in the fold above, we construct some measures of aggregate beliefs using the `aggrecat` package. We will make (and explain) some ad-hoc choices here. We present these:\n\n1. For each paper\n2. For categories of papers and cross-paper categories of evaluations\n3. For the overall set of papers and evaluations\n\nWe can also hold onto these aggregated metrics for later use in modeling.\n\n\n- Simple averaging\n\n- Bayesian approaches \n\n- Best-performing approaches from elsewhere \n\n- Assumptions over unit-level random terms \n\n\n### Simple rating aggregation {-}\n\nBelow, we are preparing the data for the aggreCATS package.\n\n\n::: {.cell}\n\n:::\n\n\n\n\n\n### Explicit modeling of 'research quality' (for use in prizes, etc.) {-}\n\n- Use the above aggregation as the outcome of interest, or weight towards categories of greater interest?\n\n- Model with controls -- look for greatest positive residual? \n\n\n## Inter-rater reliability\n\n## Decomposing variation, dimension reduction, simple linear models\n\n\n## Later possiblities\n\n- Relation to evaluation text content (NLP?)\n\n- Relation/prediction of later outcomes (traditional publication, citations, replication)\n", - "supporting": [], + "markdown": "# Aggregation of evaluators judgments (modeling)\n\n\n\n\n\n\n## Notes on sources and approaches\n\n\n::: {.callout-note collapse=\"true\"}\n\n## Hanea et al {-}\n(Consult, e.g., repliCATS/Hanea and others work; meta-science and meta-analysis approaches)\n\n`aggrecat` package\n\n> Although the accuracy, calibration, and informativeness of the majority of methods are very similar, a couple of the aggregation methods consistently distinguish themselves as among the best or worst. Moreover, the majority of methods outperform the usual benchmarks provided by the simple average or the median of estimates.\n\n[Hanea et al, 2021](https://journals.plos.org/plosone/article?id=10.1371/journal.pone.0256919#sec007)\n\n However, these are in a different context. Most of those measures are designed to deal with probablistic forecasts for binary outcomes, where the predictor also gives a 'lower bound' and 'upper bound' for that probability. We could roughly compare that to our continuous metrics with 90% CI's (or imputations for these).\n\nFurthermore, many (all their successful measures?) use 'performance-based weights', accessing metrics from prior prediction performance of the same forecasters We do not have these, nor do we have a sensible proxy for this. \n:::\n\n\n::: {.callout-note collapse=\"true\"}\n## D Veen et al (2017)\n\n[link](https://www.researchgate.net/profile/Duco-Veen/publication/319662351_Using_the_Data_Agreement_Criterion_to_Rank_Experts'_Beliefs/links/5b73e2dc299bf14c6da6c663/Using-the-Data-Agreement-Criterion-to-Rank-Experts-Beliefs.pdf)\n\n... we show how experts can be ranked based on their knowledge and their level of (un)certainty. By letting experts specify their knowledge in the form of a probability distribution, we can assess how accurately they can predict new data, and how appropriate their level of (un)certainty is. The expert’s specified probability distribution can be seen as a prior in a Bayesian statistical setting. We evaluate these priors by extending an existing prior-data (dis)agreement measure, the Data Agreement Criterion, and compare this approach to using Bayes factors to assess prior specification. We compare experts with each other and the data to evaluate their appropriateness. Using this method, new research questions can be asked and answered, for instance: Which expert predicts the new data best? Is there agreement between my experts and the data? Which experts’ representation is more valid or useful? Can we reach convergence between expert judgement and data? We provided an empirical example ranking (regional) directors of a large financial institution based on their predictions of turnover. \n\nBe sure to consult the [correction made here](https://www.semanticscholar.org/paper/Correction%3A-Veen%2C-D.%3B-Stoel%2C-D.%3B-Schalken%2C-N.%3B-K.%3B-Veen-Stoel/a2882e0e8606ef876133f25a901771259e7033b1)\n\n::: \n\n\n::: {.callout-note collapse=\"true\"}\n## Also seems relevant:\n\nSee [Gsheet HERE](https://docs.google.com/spreadsheets/d/14japw6eLGpGjEWy1MjHNJXU1skZY_GAIc2uC2HIUalM/edit#gid=0), generated from an Elicit.org inquiry.\n\n\n::: \n\n\n\nIn spite of the caveats in the fold above, we construct some measures of aggregate beliefs using the `aggrecat` package. We will make (and explain) some ad-hoc choices here. We present these:\n\n1. For each paper\n2. For categories of papers and cross-paper categories of evaluations\n3. For the overall set of papers and evaluations\n\nWe can also hold onto these aggregated metrics for later use in modeling.\n\n\n- Simple averaging\n\n- Bayesian approaches \n\n- Best-performing approaches from elsewhere \n\n- Assumptions over unit-level random terms \n\n\n### Simple rating aggregation {-}\n\nBelow, we are preparing the data for the aggreCATS package.\n\n\n::: {.cell}\n\n```{.r .cell-code}\n# JB: This section is a work in progress, please do not edit\n\n# paper_ratings: one row per rating category and 'type' (score, upper, lower bound.)\nevals_pub %>% \n select(id, eval_name, paper_abbrev, \n overall, overall_lb_imp, overall_ub_imp,\n adv_knowledge, adv_knowledge_lb_imp, adv_knowledge_ub_imp,\n methods, methods_lb_imp, methods_ub_imp,\n logic_comms, logic_comms_lb_imp, logic_comms_ub_imp,\n real_world, real_world_lb_imp, real_world_ub_imp,\n gp_relevance, gp_relevance_lb_imp, gp_relevance_ub_imp,\n open_sci, open_sci_lb_imp, open_sci_ub_imp) %>% \n rename_with(function(x) paste0(x,\"_score\"), all_of(rating_cats)) %>%\n pivot_longer(cols = c(-id, -eval_name, -paper_abbrev),\n names_pattern = \"(.+)_(score|[ul]b_imp)\",\n names_to = c(\"criterion\",\"element\"),\n values_to = \"value\") -> paper_ratings\n\n# renaming to conform with aggreCATS expectations\npaper_ratings <- paper_ratings %>% \n rename(paper_id = paper_abbrev,\n user_name = eval_name) %>% \n mutate(round = \"round_1\",\n element = case_when(element == \"lb_imp\" ~ \"three_point_lower\",\n element == \"ub_imp\" ~ \"three_point_upper\",\n element == \"score\" ~ \"three_point_best\"))\n\n# filter only overall for now\npaper_ratings %>% \n filter(criterion == \"overall\") %>% \n group_by(user_name, paper_id) %>% \n filter(sum(is.na(value))==0) %>% \n ungroup() -> temp\n \n\nAverageWAgg(expert_judgements = temp, round_2_filter = FALSE, type = \"ArMean\")\n\nIntervalWAgg(expert_judgements = temp, round_2_filter = FALSE, type = \"IntWAgg\")\n\naggreCAT::DistributionWAgg(expert_judgements = temp, round_2_filter = FALSE, type = \"DistribArMean\", percent_toggle = T)\n\n# EXAMPLE CODE ===============================\n# data(data_ratings)\n# set.seed(1234)\n# \n# participant_subset <- data_ratings %>%\n# distinct(user_name) %>%\n# sample_n(5) %>%\n# mutate(participant_name = paste(\"participant\", rep(1:n())))\n# \n# single_claim <- data_ratings %>%\n# filter(paper_id == \"28\") %>%\n# right_join(participant_subset, by = \"user_name\") %>%\n# filter(grepl(x = element, pattern = \"three_.+\")) %>%\n# select(-group, -participant_name, -question)\n# \n# DistributionWAgg(expert_judgements = single_claim,\n# type = \"DistribArMean\", percent_toggle = T)\n# \n```\n:::\n\n\n\n\n\n### Explicit modeling of 'research quality' (for use in prizes, etc.) {-}\n\n- Use the above aggregation as the outcome of interest, or weight towards categories of greater interest?\n\n- Model with controls -- look for greatest positive residual? \n\n\n## Inter-rater reliability\n\n\n::: {.cell}\n::: {.cell-output-display}\n![](aggregation_files/figure-html/unnamed-chunk-1-1.png){width=672}\n:::\n:::\n\n\n\n\n## Decomposing variation, dimension reduction, simple linear models\n\n\n## Later possiblities\n\n- Relation to evaluation text content (NLP?)\n\n- Relation/prediction of later outcomes (traditional publication, citations, replication)\n", + "supporting": [ + "aggregation_files" + ], "filters": [ "rmarkdown/pagebreak.lua" ], diff --git a/_freeze/chapters/aggregation/figure-html/unnamed-chunk-1-1.png b/_freeze/chapters/aggregation/figure-html/unnamed-chunk-1-1.png new file mode 100644 index 0000000..d9a6f75 Binary files /dev/null and b/_freeze/chapters/aggregation/figure-html/unnamed-chunk-1-1.png differ diff --git a/_freeze/chapters/evaluation_data_analysis/execute-results/html.json b/_freeze/chapters/evaluation_data_analysis/execute-results/html.json index 04cb54a..8ab47bb 100644 --- a/_freeze/chapters/evaluation_data_analysis/execute-results/html.json +++ b/_freeze/chapters/evaluation_data_analysis/execute-results/html.json @@ -1,7 +1,7 @@ { - "hash": "8ed745b0ad61f701c79bdc54ec8c065a", + "hash": "248c42795c0e7b2fbe512822cb705803", "result": { - "markdown": "# Evaluation data: description, exploration, checks\n\n## Data input, cleaning, feature construction and imputation \n\n\n::: {.cell}\n\n```{.r .cell-code code-summary=\"load packages\"}\nlibrary(tidyverse) \n\n# markdown et al. ----\nlibrary(knitr)\nlibrary(bookdown)\nlibrary(rmarkdown)\nlibrary(shiny)\nlibrary(quarto)\nlibrary(formattable) # Create 'Formattable' Data Structures\nlibrary(DT) # R interface to DataTables library (JavaScript)\n\n# dataviz ----\nlibrary(ggrepel)\nlibrary(plotly) # Create Interactive Web Graphics via 'plotly.js'\n\n# others ----\nlibrary(here) # A Simpler Way to Find Your Files\n# renv::install(packages = \"metamelb-repliCATS/aggreCAT\")\n#library(aggreCAT)\n\n# Make sure select is always the dplyr version\nselect <- dplyr::select \n\n# options\noptions(knitr.duplicate.label = \"allow\")\noptions(mc.cores = parallel::detectCores())\n```\n:::\n\n\n\n::: {.callout-note collapse=\"true\"}\n## Note on data input (10-Aug-23)\n\nBelow, the evaluation data is input from an Airtable, which itself was largely hand-input from evaluators' reports. As PubPub builds (target: end of Sept. 2023), this will allow us to include the ratings and predictions as structured data objects. We then plan to access and input this data *directly* from the PubPub (API?) into the present analysis. This will improve automation and limit the potential for data entry errors.\n\n::: \n\n\n::: {.cell}\n\n```{.r .cell-code code-summary=\"Input evaluation data\"}\nevals_pub <- readRDS(file = here(\"data\", \"evals.Rdata\"))\nall_papers_p <- readRDS(file = here(\"data\", \"all_papers_p.Rdata\"))\n```\n:::\n\n::: {.cell}\n\n```{.r .cell-code code-summary=\"Define lists of columns to use later\"}\n# Lists of categories\nrating_cats <- c(\"overall\", \"adv_knowledge\", \"methods\", \"logic_comms\", \"real_world\", \"gp_relevance\", \"open_sci\")\n\n#... 'predictions' are currently 1-5 (0-5?)\npred_cats <- c(\"journal_predict\", \"merits_journal\")\n```\n:::\n\n\n\n \n## Basic presentation\n\n### What sorts of papers/projects are we considering and evaluating? {-}\n\nIn this section, we give some simple data summaries and visualizations, for a broad description of The Unjournal's coverage. \n\nIn the interactive table below we give some key attributes of the papers and the evaluators.\n\n\n::: column-body-outset \n\n\n::: {.cell}\n\n```{.r .cell-code}\nevals_pub_df_overview <- evals_pub %>%\n arrange(paper_abbrev, eval_name) %>%\n dplyr::select(paper_abbrev, crucial_rsx, eval_name, cat_1, cat_2, source_main, author_agreement) %>%\n dplyr::select(-matches(\"ub_|lb_|conf\")) \n\nevals_pub_df_overview %>% \n rename(\n \"Paper Abbreviation\" = paper_abbrev,\n \"Paper name\" = crucial_rsx,\n \"Evaluator Name\" = eval_name,\n \"Main category\" = cat_1,\n \"Category 2\" = cat_2,\n \"Main source\" = source_main,\n \"Author contact\" = author_agreement,\n ) %>% \n DT::datatable(\n caption = \"Evaluations (confidence bounds not shown)\", \n filter = 'top',\n rownames= FALSE,\n options = list(pageLength = 5,\n columnDefs = list(list(width = '150px', targets = 1)))) %>% \n formatStyle(columns = 2:ncol(evals_pub_df_overview), \n textAlign = 'center') %>% \nformatStyle(\n \"Paper name\",\n fontSize = '10px'\n )\n```\n\n::: {.cell-output-display}\n```{=html}\n
\n\n```\n:::\n\n```{.r .cell-code}\nrm(evals_pub_df_overview)\n```\n:::\n\n\n:::\n\n\n\n#### Evaluation metrics (ratings) {-}\n\n\n\n::: {.cell}\n\n```{.r .cell-code}\nrename_dtstuff <- function(df){\n df %>% \n rename(\n \"Paper Abbreviation\" = paper_abbrev,\n \"Evaluator Name\" = eval_name,\n \"Advancing knowledge\" = adv_knowledge,\n \"Methods\" = methods,\n \"Logic & comm.\" = logic_comms,\n \"Real world engagement\" = real_world,\n \"Global priorities relevance\" = gp_relevance,\n \"Open Science\" = open_sci\n )\n}\n```\n:::\n\n\n\n::: column-body-outset \n\n\n::: {.cell}\n\n```{.r .cell-code}\n# Need to find a way to control column width but it seems to be a problem with DT\n# https://github.com/rstudio/DT/issues/29\n\n# we didn't seem to be using all_evals_dt so I removed it to increase readability\n\n\n\nevals_pub_df <- evals_pub %>%\n # Arrange data\n arrange(paper_abbrev, eval_name, overall) %>%\n \n # Select and rename columns\n dplyr::select(paper_abbrev, eval_name, all_of(rating_cats)) %>%\n rename_dtstuff \n\n\n(\n evals_pub_dt <- evals_pub_df %>% \n # Convert to a datatable and apply styling\n datatable(\n caption = \"Evaluations and predictions (confidence bounds not shown)\", \n filter = 'top',\n rownames = FALSE,\n options = list(pageLength = 5, \n columnDefs = list(list(width = '150px', targets = 0)))) %>% \n formatStyle(columns = 2:ncol(evals_pub_df), \n textAlign = 'center')\n)\n```\n\n::: {.cell-output-display}\n```{=html}\n
\n\n```\n:::\n:::\n\n:::\n\n\\\n\nNext, a preview of the evaluations, focusing on the 'middle ratings and predictions':\n\n::: column-body-outset \n\n\n::: {.cell}\n\n```{.r .cell-code code-summary=\"Data datable (all shareable relevant data)\"}\n# we didn't seem to be using all_evals_dt so I removed it to increase readability\n\n\nevals_pub %>%\n arrange(paper_abbrev, eval_name, overall) %>%\n dplyr::select(paper_abbrev, eval_name, all_of(rating_cats)) %>%\n rename_dtstuff %>% \n DT::datatable(\n caption = \"Evaluations and predictions (confidence bounds not shown)\", \n filter = 'top',\n rownames= FALSE,\n options = list(pageLength = 5,\n columnDefs = list(list(width = '150px', targets = 0))) \n\n )\n```\n\n::: {.cell-output-display}\n```{=html}\n
\n\n```\n:::\n:::\n\n:::\n\n\\ \n\n\n\n\n::: {.cell}\n\n```{.r .cell-code}\n# we did not seem to be using all_evals_dt_ci so I removed it to improve readability\nevals_pub %>%\n arrange(paper_abbrev, eval_name) %>%\n dplyr::select(paper_abbrev, eval_name, conf_overall, all_of(rating_cats), matches(\"ub_imp|lb_imp\")) %>%\n rename_dtstuff %>% \n DT::datatable(\n caption = \"Evaluations and (imputed*) confidence bounds)\", \n filter = 'top',\n rownames= FALSE,\n options = list(pageLength = 5)\n )\n```\n:::\n\n:::\n\n\n\n\n\n::: {.callout-note collapse=\"true\"}\n##### Next consider...\n\n- Composition of research evaluated\n - By field (economics, psychology, etc.)\n - By subfield of economics \n - By topic/cause area (Global health, economic development, impact of technology, global catastrophic risks, etc. )\n - By source (submitted, identified with author permission, direct evaluation)\n \n- Timing of intake and evaluation^[Consider: timing might be its own section or chapter; this is a major thing journals track, and we want to keep track of ourselves]\n\n:::\n\n#### Paper selection {-}\n\nThe Sankey diagram below starts with the papers we prioritized for likely *Unjournal* evaluation:^[Those marked as 'considering' in the Airtable].\n\n\n::: {.cell}\n\n```{.r .cell-code}\n#Add in the 3 different evaluation input sources\n#update to be automated rather than hard-coded - to look at David's work here\n\npapers_considered <- all_papers_p %>% \n nrow()\n\npapers_deprio <- all_papers_p %>% \n filter(`stage of process/todo` == \"de-prioritized\") %>% \n nrow()\n\npapers_evaluated <- all_papers_p %>% \n filter(`stage of process/todo` %in% c(\"published\",\n \"contacting/awaiting_authors_response_to_evaluation\",\n \"awaiting_publication_ME_comments\",\"awaiting_evaluations\")) %>% \n nrow()\n\npapers_complete <- all_papers_p %>% \n filter(`stage of process/todo` == \"published\") %>%\n nrow()\n\npapers_in_progress <- papers_evaluated - papers_complete\n\npapers_still_in_consideration <- all_papers_p %>% filter(`stage of process/todo` == \"considering\") %>% nrow()\n\n\n#todo: adjust wording of hover notes ('source, target...etc')\n\nfig <- plot_ly(\n type = \"sankey\",\n orientation = \"h\",\n \n node = list(\n label = c(\"Prioritized\", \"Evaluating\", \"Complete\", \"In progress\", \"Still in consideration\", \"De-prioritized\"),\n color = c(\"orange\", \"green\", \"green\", \"orange\", \"orange\", \"red\"),\n #Todo: adjust 'location' to group these left to right\n pad = 15,\n thickness = 20,\n line = list(\n color = \"black\",\n width = 0.5\n )\n ),\n \n link = list(\n source = c(0,1,1,0,0),\n target = c(1,2,3,4,5),\n value = c(\n papers_evaluated,\n papers_complete,\n papers_in_progress,\n papers_still_in_consideration,\n papers_deprio\n ))\n)\nfig <- fig %>% layout(\n title = \"Unjournal paper funnel\",\n font = list(\n size = 10\n )\n)\n\nfig \n```\n\n::: {.cell-output-display}\n```{=html}\n
\n\n```\n:::\n:::\n\n\n\nTodo: ^[Make interactive/dashboards of the elements below]\n\n#### Paper categories {-}\n\n\n\n::: {.cell}\n\n```{.r .cell-code}\nevals_pub %>% \n select(paper_abbrev, starts_with(\"cat_\")) %>% \n pivot_longer(cols = starts_with(\"cat_\"), names_to = \"CatNum\", values_to = \"Category\") %>% \n group_by(CatNum, Category) %>% \n count() %>% \n filter(!is.na(Category)) %>% \n mutate(Category = str_to_title(Category),\n CatNum = ordered(CatNum, \n levels = c(\"cat_1\", \"cat_2\", \"cat_3\"),\n labels = c(\"Primary\", \"Secondary\", \"Tertiary\"))) %>%\n ggplot(aes(x = reorder(Category, -n), y = n)) +\n geom_bar(aes(fill = CatNum), stat = \"identity\", color = \"grey30\") + \n labs(x = \"Paper category\", y = \"Count\", fill = \"Cat Level\",\n title = \"Paper categories represented in pilot data\") +\n theme_bw() +\n facet_grid(~CatNum, scales=\"free_x\", space=\"free_x\") +\n theme(axis.text.x=element_text(angle=45,hjust=1)) +\n theme(legend.position = \"none\")\n```\n\n::: {.cell-output-display}\n![](evaluation_data_analysis_files/figure-html/all_categories-1.png){width=672}\n:::\n:::\n\n\n#### Paper source {-}\n\n\n::: {.cell}\n\n```{.r .cell-code}\n# Bar plot\nevals_pub %>% \n rowwise() %>% \n mutate(source_main = str_replace_all(string = source_main, \n pattern = \"-\", \n replace = \" \") %>% str_to_title()) %>% \n ggplot(aes(x = source_main)) + \n geom_bar(position = \"stack\", stat = \"count\", color = \"grey30\", fill = \"grey80\") +\n labs(x = \"Source\", y = \"Count\") +\n labs(title = \"Pool of research/evaluations by paper source\") +\n theme_bw() +\n theme(text = element_text(size = 15)) +\n scale_x_discrete(labels = function(x) str_wrap(x, width = 20))\n```\n\n::: {.cell-output-display}\n![](evaluation_data_analysis_files/figure-html/paper_source-1.png){width=672}\n:::\n:::\n\n::: {.cell}\n\n```{.r .cell-code}\n# JB: Most of these should probably be cleaned in data storage\n\nlibrary(RColorBrewer) # for color palettes\n\n# paper statuses that are considered \"being evaluated\"\neval_true = c(\"published\", \n \"contacting/awaiting_authors_response_to_evaluation\",\n \"awaiting_publication_ME_comments\",\n \"awaiting_evaluations\")\n\n# Is the paper being evaluated? \nall_papers_p <- all_papers_p %>% \n mutate(is_evaluated = if_else(`stage of process/todo` %in% eval_true, TRUE, FALSE))\n\n# main source clean\nall_papers_p <- all_papers_p %>% \n mutate(source_main = case_when(source_main == \"NA\" ~ \"Not applicable\",\n source_main == \"internal-from-syllabus-agenda-policy-database\" ~ \"Internal: syllabus, agenda, etc.\",\n is.na(source_main) ~ \"Unknown\",\n TRUE ~ source_main))\n\nall_papers_p %>% \nggplot(aes(x = fct_infreq(source_main), fill = is_evaluated)) + \n geom_bar(position = \"stack\", stat = \"count\") +\n labs(x = \"Source\", y = \"Count\", fill = \"Selected for\\nevaluation?\") +\n coord_flip() + # flipping the coordinates to have categories on y-axis (on the left)\n labs(title = \"Evaluations by source of the paper\") +\n theme_bw() +\n theme(text = element_text(size = 15)) +\n scale_fill_brewer(palette = \"Set1\") +\n scale_x_discrete(labels = function(x) str_wrap(x, width = 20))\n```\n\n::: {.cell-output-display}\n![](evaluation_data_analysis_files/figure-html/data_clean_-1.png){width=672}\n:::\n:::\n\n\n\n## The distribution of ratings and predictions {-}\n\nNext, we present the ratings and predictions along with 'uncertainty measures'.^[We use \"ub imp\" (and \"lb imp\") to denote the upper and lower bounds given by evaluators.] Where evaluators gave only a 1-5 confidence level^[More or less, the ones who report a level for 'conf overall', although some people did this for some but not others], we use the imputations discussed and coded above. \n\n\n- For each category and prediction (overall and by paper)\n\n::: {.cell}\n\n```{.r .cell-code}\n# evals_pub %>% \n# select(matches(\"overall\")) %>% \n# view()\n```\n:::\n\n\n\n::: column-body-outset\n\n\n\n::: {.cell}\n\n```{.r .cell-code}\n# Generate a color palette with more colors\ncolor_count <- length(unique(evals_pub$paper_abbrev))\ncolor_palette <- colorRampPalette(brewer.pal(8, \"Set1\"))(color_count)\n\n# set one \"set\" of dodge width values across layers\npd = position_dodge(width = 0.8)\n\n# Dot plot\ng1 <- evals_pub %>% \n ggplot(aes(x = paper_abbrev, y = overall, \n text = paste0('Evaluator: ', eval_name, # tooltip data\n '
Rating [CI]: ', overall, \" [\", overall_lb_imp, \", \", overall_ub_imp, \"]\"))) +\n geom_point(aes(color = paper_abbrev), \n stat = \"identity\", size = 2, shape = 18, stroke = 1, \n position = pd) +\n geom_linerange(aes(ymin = overall_lb_imp, ymax = overall_ub_imp, color = paper_abbrev), \n position = pd) +\n geom_text(data = subset(evals_pub, str_detect(eval_name, \"Anonymous\")),\n aes(label = \"anon.\"), size=3) +\n coord_flip() + # flipping the coordinates to have categories on y-axis (on the left)\n labs(x = \"Paper\", y = \"Overall score\",\n title = \"Overall scores of evaluated papers\") +\n theme_bw() +\n theme(text = element_text(size = 15)) +\n theme(legend.position = \"none\") +\n scale_x_discrete(labels = function(x) str_wrap(x, width = 20)) + \n scale_color_manual(values = color_palette)\n\n\nggplotly(g1, tooltip = c(\"text\"))\n```\n\n::: {.cell-output-display}\n```{=html}\n
\n\n```\n:::\n:::\n\n:::\n\nIn future, we aim to build a dashboard allowing people to use the complete set of ratings and predictions, and choose their own weightings. (Also incorporating the evaluator uncertainty in reasonable ways.)\n\n### Shiny dashboard {-}\n\n::: column-body-outset\n\n\n```{=html}\n\n\n\n```\n\n\n:::\n\n::: {.callout-note collapse=\"true\"}\n## Future vis\n\nSpider or radial chart \n\nEach rating is a dimension or attribute (potentially normalized)\npotentially superimpose a 'circle' for the suggested weighting or overall. \n\nEach paper gets its own spider, with all others (or the average) in faded color behind it as a comparator. \n\nIdeally user can switch on/off \n\nBeware -- people infer things from the shape's size\n\n::: \n\n\n::: column-body-outset\n\n\n::: {.cell}\n\n```{.r .cell-code}\n# JB: what is the purpose of this table? It's very large and I'm not totally\n# sure what it's doing so I'm just turning it off for now\nunit.scale = function(x) (x*100 - min(x*100)) / (max(x*100) - min(x*100))\n\nevaluations_table <- evals_pub %>%\n select(paper_abbrev, eval_name, cat_1, \n source_main, overall, adv_knowledge,\n methods, logic_comms, journal_predict) %>%\n arrange(desc(paper_abbrev))\n\nformattable(\n evaluations_table,\n list(\n #area(col = 5:8) ~ function(x) percent(x / 100, digits = 0),\n area(col = 5:8) ~ color_tile(\"#FA614B66\",\"#3E7DCC\"),\n `journal_predict` = proportion_bar(\"#DeF7E9\", unit.scale)\n )\n)\n```\n:::\n\n:::\n\n\n### Sources of variation {-}\n\nNext, look for systematic variation in the ratings \n\n- By field and topic area of paper\n\n- By submission/selection route\n\n- By evaluation manager (or their seniority, or whether they are US/Commonwealth/Other)^[DR: My theory is that people in commonwealth countries target a 70+ as 'strong' (because of their marking system) and that may drive a bias.]\n\n... perhaps building a model of this. We are looking for systematic 'biases and trends', loosely speaking, to help us better understand how our evaluation system is working.\n\n\\\n\n\n\n\n### Relationship among the ratings (and predictions) {-} \n\n::: {.callout-note collapse=\"true\"}\n## Next steps (suggested analyses)\n\n- Correlation matrix\n\n- ANOVA\n\n- PCA (Principle components)\n\n- With other 'control' factors?\n\n- How do the specific measures predict the aggregate ones (overall rating, merited publication)\n - CF 'our suggested weighting'\n\n::: \n\n\nNext chapter (analysis): *aggregation of evaluator judgment*\n\n\n::: {.callout-note collapse=\"true\"}\n## Scoping our future coverage\n\nWe have funding to evaluate roughly 50-70 papers/projects per year, given our proposed incentives.\n\nConsider:\n\n- How many relevant NBER papers come out per year?\n\n- How much relevant work in other prestige archives?\n\n- What quotas do we want (by cause, etc.) and how feasible are these?\n\n::: \n\n", + "markdown": "# Evaluation data: description, exploration, checks\n\n## Data input, cleaning, feature construction and imputation \n\n\n::: {.cell}\n\n```{.r .cell-code code-summary=\"load packages\"}\nlibrary(tidyverse) \n\n# markdown et al. ----\nlibrary(knitr)\nlibrary(bookdown)\nlibrary(rmarkdown)\nlibrary(shiny)\nlibrary(quarto)\nlibrary(formattable) # Create 'Formattable' Data Structures\nlibrary(DT) # R interface to DataTables library (JavaScript)\n\n# dataviz ----\nlibrary(ggrepel)\nlibrary(plotly) # Create Interactive Web Graphics via 'plotly.js'\n\n# others ----\nlibrary(here) # A Simpler Way to Find Your Files\n# renv::install(packages = \"metamelb-repliCATS/aggreCAT\")\n#library(aggreCAT)\n\n# Make sure select is always the dplyr version\nselect <- dplyr::select \n\n# options\noptions(knitr.duplicate.label = \"allow\")\noptions(mc.cores = parallel::detectCores())\n```\n:::\n\n\n\n::: {.callout-note collapse=\"true\"}\n## Note on data input (10-Aug-23)\n\nBelow, the evaluation data is input from an Airtable, which itself was largely hand-input from evaluators' reports. As PubPub builds (target: end of Sept. 2023), this will allow us to include the ratings and predictions as structured data objects. We then plan to access and input this data *directly* from the PubPub (API?) into the present analysis. This will improve automation and limit the potential for data entry errors.\n\n::: \n\n\n::: {.cell}\n\n```{.r .cell-code code-summary=\"Input evaluation data\"}\nevals_pub <- readRDS(file = here(\"data\", \"evals.Rdata\"))\nall_papers_p <- readRDS(file = here(\"data\", \"all_papers_p.Rdata\"))\n```\n:::\n\n::: {.cell}\n\n```{.r .cell-code code-summary=\"Define lists of columns to use later\"}\n# Lists of categories\nrating_cats <- c(\"overall\", \"adv_knowledge\", \"methods\", \"logic_comms\", \"real_world\", \"gp_relevance\", \"open_sci\")\n\n#... 'predictions' are currently 1-5 (0-5?)\npred_cats <- c(\"journal_predict\", \"merits_journal\")\n```\n:::\n\n\n\n \n## Basic presentation\n\n### What sorts of papers/projects are we considering and evaluating? {-}\n\nIn this section, we give some simple data summaries and visualizations, for a broad description of The Unjournal's coverage. \n\nIn the interactive table below we give some key attributes of the papers and the evaluators.\n\n\n::: column-body-outset \n\n\n::: {.cell}\n\n```{.r .cell-code}\nevals_pub_df_overview <- evals_pub %>%\n arrange(paper_abbrev, eval_name) %>%\n dplyr::select(paper_abbrev, crucial_rsx, eval_name, cat_1, cat_2, source_main, author_agreement) %>%\n dplyr::select(-matches(\"ub_|lb_|conf\")) \n\nevals_pub_df_overview %>% \n rename(\n \"Paper Abbreviation\" = paper_abbrev,\n \"Paper name\" = crucial_rsx,\n \"Evaluator Name\" = eval_name,\n \"Main category\" = cat_1,\n \"Category 2\" = cat_2,\n \"Main source\" = source_main,\n \"Author contact\" = author_agreement,\n ) %>% \n DT::datatable(\n caption = \"Evaluations (confidence bounds not shown)\", \n filter = 'top',\n rownames= FALSE,\n options = list(pageLength = 5,\n columnDefs = list(list(width = '150px', targets = 1)))) %>% \n formatStyle(columns = 2:ncol(evals_pub_df_overview), \n textAlign = 'center') %>% \nformatStyle(\n \"Paper name\",\n fontSize = '10px'\n )\n```\n\n::: {.cell-output-display}\n```{=html}\n
\n\n```\n:::\n\n```{.r .cell-code}\nrm(evals_pub_df_overview)\n```\n:::\n\n\n:::\n\n\n\n#### Evaluation metrics (ratings) {-}\n\n\n\n::: {.cell}\n\n```{.r .cell-code}\nrename_dtstuff <- function(df){\n df %>% \n rename(\n \"Paper Abbreviation\" = paper_abbrev,\n \"Evaluator Name\" = eval_name,\n \"Advancing knowledge\" = adv_knowledge,\n \"Methods\" = methods,\n \"Logic & comm.\" = logic_comms,\n \"Real world engagement\" = real_world,\n \"Global priorities relevance\" = gp_relevance,\n \"Open Science\" = open_sci\n )\n}\n```\n:::\n\n\n\n::: column-body-outset \n\n\n::: {.cell}\n\n```{.r .cell-code}\n# Need to find a way to control column width but it seems to be a problem with DT\n# https://github.com/rstudio/DT/issues/29\n\n# we didn't seem to be using all_evals_dt so I removed it to increase readability\n\n\n\nevals_pub_df <- evals_pub %>%\n # Arrange data\n arrange(paper_abbrev, eval_name, overall) %>%\n \n # Select and rename columns\n dplyr::select(paper_abbrev, eval_name, all_of(rating_cats)) %>%\n rename_dtstuff \n\n\n(\n evals_pub_dt <- evals_pub_df %>% \n # Convert to a datatable and apply styling\n datatable(\n caption = \"Evaluations and predictions (confidence bounds not shown)\", \n filter = 'top',\n rownames = FALSE,\n options = list(pageLength = 5, \n columnDefs = list(list(width = '150px', targets = 0)))) %>% \n formatStyle(columns = 2:ncol(evals_pub_df), \n textAlign = 'center')\n)\n```\n\n::: {.cell-output-display}\n```{=html}\n
\n\n```\n:::\n:::\n\n:::\n\n\\\n\nNext, a preview of the evaluations, focusing on the 'middle ratings and predictions':\n\n::: column-body-outset \n\n\n::: {.cell}\n\n```{.r .cell-code code-summary=\"Data datable (all shareable relevant data)\"}\n# we didn't seem to be using all_evals_dt so I removed it to increase readability\n\n\nevals_pub %>%\n arrange(paper_abbrev, eval_name, overall) %>%\n dplyr::select(paper_abbrev, eval_name, all_of(rating_cats)) %>%\n rename_dtstuff %>% \n DT::datatable(\n caption = \"Evaluations and predictions (confidence bounds not shown)\", \n filter = 'top',\n rownames= FALSE,\n options = list(pageLength = 5,\n columnDefs = list(list(width = '150px', targets = 0))) \n\n )\n```\n\n::: {.cell-output-display}\n```{=html}\n
\n\n```\n:::\n:::\n\n:::\n\n\\ \n\n\n\n\n::: {.cell}\n\n```{.r .cell-code}\n# we did not seem to be using all_evals_dt_ci so I removed it to improve readability\nevals_pub %>%\n arrange(paper_abbrev, eval_name) %>%\n dplyr::select(paper_abbrev, eval_name, conf_overall, all_of(rating_cats), matches(\"ub_imp|lb_imp\")) %>%\n rename_dtstuff %>% \n DT::datatable(\n caption = \"Evaluations and (imputed*) confidence bounds)\", \n filter = 'top',\n rownames= FALSE,\n options = list(pageLength = 5)\n )\n```\n:::\n\n:::\n\n\n\n\n\n::: {.callout-note collapse=\"true\"}\n##### Next consider...\n\n- Composition of research evaluated\n - By field (economics, psychology, etc.)\n - By subfield of economics \n - By topic/cause area (Global health, economic development, impact of technology, global catastrophic risks, etc. )\n - By source (submitted, identified with author permission, direct evaluation)\n \n- Timing of intake and evaluation^[Consider: timing might be its own section or chapter; this is a major thing journals track, and we want to keep track of ourselves]\n\n:::\n\n#### Paper selection {-}\n\nThe Sankey diagram below starts with the papers we prioritized for likely *Unjournal* evaluation:^[Those marked as 'considering' in the Airtable].\n\n\n::: {.cell}\n\n```{.r .cell-code}\n#Add in the 3 different evaluation input sources\n#update to be automated rather than hard-coded - to look at David's work here\n\npapers_considered <- all_papers_p %>% \n nrow()\n\npapers_deprio <- all_papers_p %>% \n filter(`stage of process/todo` == \"de-prioritized\") %>% \n nrow()\n\npapers_evaluated <- all_papers_p %>% \n filter(`stage of process/todo` %in% c(\"published\",\n \"contacting/awaiting_authors_response_to_evaluation\",\n \"awaiting_publication_ME_comments\",\"awaiting_evaluations\")) %>% \n nrow()\n\npapers_complete <- all_papers_p %>% \n filter(`stage of process/todo` == \"published\") %>%\n nrow()\n\npapers_in_progress <- papers_evaluated - papers_complete\n\npapers_still_in_consideration <- all_papers_p %>% filter(`stage of process/todo` == \"considering\") %>% nrow()\n\n\n#todo: adjust wording of hover notes ('source, target...etc')\n\nfig <- plot_ly(\n type = \"sankey\",\n orientation = \"h\",\n \n node = list(\n label = c(\"Prioritized\", \"Evaluating\", \"Complete\", \"In progress\", \"Still in consideration\", \"De-prioritized\"),\n color = c(\"orange\", \"green\", \"green\", \"orange\", \"orange\", \"red\"),\n #Todo: adjust 'location' to group these left to right\n pad = 15,\n thickness = 20,\n line = list(\n color = \"black\",\n width = 0.5\n )\n ),\n \n link = list(\n source = c(0,1,1,0,0),\n target = c(1,2,3,4,5),\n value = c(\n papers_evaluated,\n papers_complete,\n papers_in_progress,\n papers_still_in_consideration,\n papers_deprio\n ))\n)\nfig <- fig %>% layout(\n title = \"Unjournal paper funnel\",\n font = list(\n size = 10\n )\n)\n\nfig \n```\n\n::: {.cell-output-display}\n```{=html}\n
\n\n```\n:::\n:::\n\n\n\nTodo: ^[Make interactive/dashboards of the elements below]\n\n#### Paper categories {-}\n\n\n\n::: {.cell}\n\n```{.r .cell-code}\nevals_pub %>% \n select(paper_abbrev, starts_with(\"cat_\")) %>%\n distinct() %>% \n pivot_longer(cols = starts_with(\"cat_\"), names_to = \"CatNum\", values_to = \"Category\") %>% \n group_by(CatNum, Category) %>% \n count() %>% \n filter(!is.na(Category)) %>% \n mutate(Category = str_to_title(Category),\n CatNum = ordered(CatNum, \n levels = c(\"cat_1\", \"cat_2\", \"cat_3\"),\n labels = c(\"Primary\", \"Secondary\", \"Tertiary\"))) %>%\n ggplot(aes(x = reorder(Category, -n), y = n)) +\n geom_bar(aes(fill = CatNum), stat = \"identity\", color = \"grey30\") + \n labs(x = \"Paper category\", y = \"Count\", fill = \"Cat Level\",\n title = \"Paper categories represented in pilot data\") +\n theme_bw() +\n facet_grid(~CatNum, scales=\"free_x\", space=\"free_x\") +\n theme(axis.text.x=element_text(angle=45,hjust=1)) +\n theme(legend.position = \"none\")\n```\n\n::: {.cell-output-display}\n![](evaluation_data_analysis_files/figure-html/all_categories-1.png){width=672}\n:::\n:::\n\n\n#### Paper source {-}\n\n\n::: {.cell}\n\n```{.r .cell-code}\n# Bar plot\nevals_pub %>% \n rowwise() %>% \n mutate(source_main = str_replace_all(string = source_main, \n pattern = \"-\", \n replace = \" \") %>% str_to_title()) %>%\n select(paper_abbrev, source_main) %>% \n distinct() %>%\n ggplot(aes(x = source_main)) + \n geom_bar(position = \"stack\", stat = \"count\", color = \"grey30\", fill = \"grey80\") +\n labs(x = \"Source\", y = \"Count\") +\n labs(title = \"Pool of research/evaluations by paper source\") +\n theme_bw() +\n theme(text = element_text(size = 15)) +\n scale_x_discrete(labels = function(x) str_wrap(x, width = 20))\n```\n\n::: {.cell-output-display}\n![](evaluation_data_analysis_files/figure-html/paper_source-1.png){width=672}\n:::\n:::\n\n::: {.cell}\n\n```{.r .cell-code}\n# JB: Most of these should probably be cleaned in data storage\n\nlibrary(RColorBrewer) # for color palettes\n\n# paper statuses that are considered \"being evaluated\"\neval_true = c(\"published\", \n \"contacting/awaiting_authors_response_to_evaluation\",\n \"awaiting_publication_ME_comments\",\n \"awaiting_evaluations\")\n\n# Is the paper being evaluated? \nall_papers_p <- all_papers_p %>% \n mutate(is_evaluated = if_else(`stage of process/todo` %in% eval_true, TRUE, FALSE))\n\n# main source clean\nall_papers_p <- all_papers_p %>% \n mutate(source_main = case_when(source_main == \"NA\" ~ \"Not applicable\",\n source_main == \"internal-from-syllabus-agenda-policy-database\" ~ \"Internal: syllabus, agenda, etc.\",\n is.na(source_main) ~ \"Unknown\",\n TRUE ~ source_main))\n\nall_papers_p %>% \nggplot(aes(x = fct_infreq(source_main), fill = is_evaluated)) + \n geom_bar(position = \"stack\", stat = \"count\") +\n labs(x = \"Source\", y = \"Count\", fill = \"Selected for\\nevaluation?\") +\n coord_flip() + # flipping the coordinates to have categories on y-axis (on the left)\n labs(title = \"Evaluations by source of the paper\") +\n theme_bw() +\n theme(text = element_text(size = 15)) +\n scale_fill_brewer(palette = \"Set1\") +\n scale_x_discrete(labels = function(x) str_wrap(x, width = 20))\n```\n\n::: {.cell-output-display}\n![](evaluation_data_analysis_files/figure-html/data clean-1.png){width=672}\n:::\n:::\n\n\n\n## The distribution of ratings and predictions {-}\n\nNext, we present the ratings and predictions along with 'uncertainty measures'.^[We use \"ub imp\" (and \"lb imp\") to denote the upper and lower bounds given by evaluators.] Where evaluators gave only a 1-5 confidence level^[More or less, the ones who report a level for 'conf overall', although some people did this for some but not others], we use the imputations discussed and coded above. \n\n\n- For each category and prediction (overall and by paper)\n\n::: {.cell}\n\n```{.r .cell-code}\n# evals_pub %>% \n# select(matches(\"overall\")) %>% \n# view()\n```\n:::\n\n\n\n::: column-body-outset\n\n\n\n::: {.cell}\n\n```{.r .cell-code}\n# Generate a color palette with more colors\ncolor_count <- length(unique(evals_pub$paper_abbrev))\ncolor_palette <- colorRampPalette(brewer.pal(8, \"Set1\"))(color_count)\n\n# set one \"set\" of dodge width values across layers\npd = position_dodge(width = 0.8)\n\n# Dot plot\ng1 <- evals_pub %>% \n ggplot(aes(x = paper_abbrev, y = overall, \n text = paste0('Evaluator: ', eval_name, # tooltip data\n '
Rating [CI]: ', overall, \" [\", overall_lb_imp, \", \", overall_ub_imp, \"]\"))) +\n geom_point(aes(color = paper_abbrev), \n stat = \"identity\", size = 2, shape = 18, stroke = 1, \n position = pd) +\n geom_linerange(aes(ymin = overall_lb_imp, ymax = overall_ub_imp, color = paper_abbrev), \n position = pd) +\n geom_text(data = subset(evals_pub, str_detect(eval_name, \"Anonymous\")),\n aes(label = \"anon.\"), size=3) +\n coord_flip() + # flipping the coordinates to have categories on y-axis (on the left)\n labs(x = \"Paper\", y = \"Overall score\",\n title = \"Overall scores of evaluated papers\") +\n theme_bw() +\n theme(text = element_text(size = 15)) +\n theme(legend.position = \"none\") +\n scale_x_discrete(labels = function(x) str_wrap(x, width = 20)) + \n scale_color_manual(values = color_palette)\n\n\nggplotly(g1, tooltip = c(\"text\"))\n```\n\n::: {.cell-output-display}\n```{=html}\n
\n\n```\n:::\n:::\n\n:::\n\nIn future, we aim to build a dashboard allowing people to use the complete set of ratings and predictions, and choose their own weightings. (Also incorporating the evaluator uncertainty in reasonable ways.)\n\n### Shiny dashboard {-}\n\n::: column-body-outset\n\n\n```{=html}\n\n\n\n```\n\n\n:::\n\n::: {.callout-note collapse=\"true\"}\n## Future vis\n\nSpider or radial chart \n\nEach rating is a dimension or attribute (potentially normalized)\npotentially superimpose a 'circle' for the suggested weighting or overall. \n\nEach paper gets its own spider, with all others (or the average) in faded color behind it as a comparator. \n\nIdeally user can switch on/off \n\nBeware -- people infer things from the shape's size\n\n::: \n\n\n::: column-body-outset\n\n\n::: {.cell}\n\n```{.r .cell-code}\n# JB: what is the purpose of this table? It's very large and I'm not totally\n# sure what it's doing so I'm just turning it off for now\nunit.scale = function(x) (x*100 - min(x*100)) / (max(x*100) - min(x*100))\n\nevaluations_table <- evals_pub %>%\n select(paper_abbrev, eval_name, cat_1, \n source_main, overall, adv_knowledge,\n methods, logic_comms, journal_predict) %>%\n arrange(desc(paper_abbrev))\n\nformattable(\n evaluations_table,\n list(\n #area(col = 5:8) ~ function(x) percent(x / 100, digits = 0),\n area(col = 5:8) ~ color_tile(\"#FA614B66\",\"#3E7DCC\"),\n `journal_predict` = proportion_bar(\"#DeF7E9\", unit.scale)\n )\n)\n```\n:::\n\n:::\n\n\n### Sources of variation {-}\n\nNext, look for systematic variation in the ratings \n\n- By field and topic area of paper\n\n- By submission/selection route\n\n- By evaluation manager (or their seniority, or whether they are US/Commonwealth/Other)^[DR: My theory is that people in commonwealth countries target a 70+ as 'strong' (because of their marking system) and that may drive a bias.]\n\n... perhaps building a model of this. We are looking for systematic 'biases and trends', loosely speaking, to help us better understand how our evaluation system is working.\n\n\\\n\n\n### Relationship among the ratings (and predictions) {-} \n\n::: {.callout-note collapse=\"true\"}\n## Next steps (suggested analyses)\n\n- Correlation matrix\n\n- ANOVA\n\n- PCA (Principle components)\n\n- With other 'control' factors?\n\n- How do the specific measures predict the aggregate ones (overall rating, merited publication)\n - CF 'our suggested weighting'\n\n::: \n\n\nNext chapter (analysis): *aggregation of evaluator judgment*\n\n\n::: {.callout-note collapse=\"true\"}\n## Scoping our future coverage\n\nWe have funding to evaluate roughly 50-70 papers/projects per year, given our proposed incentives.\n\nConsider:\n\n- How many relevant NBER papers come out per year?\n\n- How much relevant work in other prestige archives?\n\n- What quotas do we want (by cause, etc.) and how feasible are these?\n\n::: \n\n", "supporting": [ "evaluation_data_analysis_files" ], diff --git a/_freeze/chapters/evaluation_data_analysis/figure-html/all_categories-1.png b/_freeze/chapters/evaluation_data_analysis/figure-html/all_categories-1.png index c30dcce..c215cbe 100644 Binary files a/_freeze/chapters/evaluation_data_analysis/figure-html/all_categories-1.png and b/_freeze/chapters/evaluation_data_analysis/figure-html/all_categories-1.png differ diff --git a/_freeze/chapters/evaluation_data_analysis/figure-html/data clean-1.png b/_freeze/chapters/evaluation_data_analysis/figure-html/data clean-1.png new file mode 100644 index 0000000..4df59c7 Binary files /dev/null and b/_freeze/chapters/evaluation_data_analysis/figure-html/data clean-1.png differ diff --git a/_freeze/chapters/evaluation_data_analysis/figure-html/paper_source-1.png b/_freeze/chapters/evaluation_data_analysis/figure-html/paper_source-1.png index 15cac0c..b097648 100644 Binary files a/_freeze/chapters/evaluation_data_analysis/figure-html/paper_source-1.png and b/_freeze/chapters/evaluation_data_analysis/figure-html/paper_source-1.png differ diff --git a/_freeze/chapters/evaluation_data_input/execute-results/html.json b/_freeze/chapters/evaluation_data_input/execute-results/html.json index 7df1f81..3696a88 100644 --- a/_freeze/chapters/evaluation_data_input/execute-results/html.json +++ b/_freeze/chapters/evaluation_data_input/execute-results/html.json @@ -1,10 +1,8 @@ { - "hash": "607c24d4924dc8c955e476a7ab08bb97", + "hash": "c76fbe826325ec2c8e8ad614a78c1b21", "result": { - "markdown": "# Evaluation data: input/features\n\n\n::: {.cell}\n\n```{.r .cell-code code-summary=\"load packages\"}\nlibrary(tidyverse) \n\n# data acquisition ----\n#devtools::install_github(\"bergant/airtabler\")\nlibrary(airtabler)\n\n# data cleaning & shaping ----\n\n# data analysis ----\n# library(lme4)\n# library(lmtest) # Testing Linear Regression Models\n\n# markdown et al. ----\nlibrary(knitr)\nlibrary(bookdown)\nlibrary(quarto)\nlibrary(formattable) # Create 'Formattable' Data Structures\n\n# others ----\nlibrary(here) # A Simpler Way to Find Your Files\n#devtools::install_github(\"metamelb-repliCATS/aggreCAT\")\n#library(aggrecat)\n\n# Make sure select is always the dplyr version\nselect <- dplyr::select \n\n# options\noptions(knitr.duplicate.label = \"allow\")\n```\n:::\n\n\n\n::: {.callout-note collapse=\"true\"}\n## Note on data input (10-Aug-23)\n\nBelow, the evaluation data is input from an Airtable, which itself was largely hand-input from evaluators' reports. As PubPub builds (target: end of Sept. 2023), this will allow us to include the ratings and predictions as structured data objects. We then plan to access and input this data *directly* from the PubPub (API?) into the present analysis. This will improve automation and limit the potential for data entry errors.\n\n::: \n\n\n::: {.cell}\n\n```{.r .cell-code code-summary=\"input from airtable\"}\nbase_id <- \"appbPYEw9nURln7Qg\"\n\n\n# Set your Airtable API key \nSys.setenv(AIRTABLE_API_KEY = Sys.getenv(\"AIRTABLE_API_KEY\"))\n#this should be set in my .Renviron file\n\n# Read data from a specific view\n\nevals <- air_get(base = base_id, \"output_eval\") \n\nall_pub_records <- data.frame()\npub_records <- air_select(base = base_id, table = \"crucial_research\")\n\n# Append the records to the list\nall_pub_records <- bind_rows(all_pub_records, pub_records)\n\n# While the length of the records list is 100 (the maximum), fetch more records\nwhile(nrow(pub_records) == 100) {\n # Get the ID of the last record in the list\n offset <- get_offset(pub_records)\n \n # Fetch the next 100 records, starting after the last ID\n pub_records <- air_select(base = base_id, table = \"crucial_research\", offset = offset)\n \n # Append the records to the df\n all_pub_records <- bind_rows(all_pub_records, pub_records)\n}\n\n# housekeeping\nrm(pub_records)\n```\n:::\n\n::: {.cell}\n\n```{.r .cell-code code-summary=\"just the useful and publish-able data, clean a bit\"}\n# clean evals names to snakecase\ncolnames(evals) <- snakecase::to_snake_case(colnames(evals))\n\nevals_pub <- evals %>% \n dplyr::rename(stage_of_process = stage_of_process_todo_from_crucial_research_2) %>% \n mutate(stage_of_process = unlist(stage_of_process)) %>% \n dplyr::filter(stage_of_process == \"published\") %>% \n select(id, \n crucial_research, \n paper_abbrev, \n evaluator_name, \n category, \n source_main, \n author_agreement, \n overall, \n lb_overall, \n ub_overall, \n conf_index_overall, \n advancing_knowledge_and_practice, \n lb_advancing_knowledge_and_practice, \n ub_advancing_knowledge_and_practice, \n conf_index_advancing_knowledge_and_practice,\n methods_justification_reasonableness_validity_robustness,\n lb_methods_justification_reasonableness_validity_robustness,\n ub_methods_justification_reasonableness_validity_robustness,\n conf_index_methods_justification_reasonableness_validity_robustness, \n logic_communication, lb_logic_communication, ub_logic_communication, \n conf_index_logic_communication,\n engaging_with_real_world_impact_quantification_practice_realism_and_relevance,\n lb_engaging_with_real_world_impact_quantification_practice_realism_and_relevance,\n ub_engaging_with_real_world_impact_quantification_practice_realism_and_relevance,\n conf_index_engaging_with_real_world_impact_quantification_practice_realism_and_relevance,\n relevance_to_global_priorities, \n lb_relevance_to_global_priorities, \n ub_relevance_to_global_priorities, \n conf_index_relevance_to_global_priorities, \n journal_quality_predict, \n lb_journal_quality_predict, \n ub_journal_quality_predict,\n conf_index_journal_quality_predict, \n open_collaborative_replicable, \n conf_index_open_collaborative_replicable, \n lb_open_collaborative_replicable, \n ub_open_collaborative_replicable, \n merits_journal, \n lb_merits_journal, \n ub_merits_journal, \n conf_index_merits_journal)\n\n# shorten names (before you expand into columns)\nnew_names <- c(\n \"eval_name\" = \"evaluator_name\",\n \"cat\" = \"category\",\n \"crucial_rsx\" = \"crucial_research\",\n \"conf_overall\" = \"conf_index_overall\",\n \"adv_knowledge\" = \"advancing_knowledge_and_practice\",\n \"lb_adv_knowledge\" = \"lb_advancing_knowledge_and_practice\",\n \"ub_adv_knowledge\" = \"ub_advancing_knowledge_and_practice\",\n \"conf_adv_knowledge\" = \"conf_index_advancing_knowledge_and_practice\",\n \"methods\" = \"methods_justification_reasonableness_validity_robustness\",\n \"lb_methods\" = \"lb_methods_justification_reasonableness_validity_robustness\",\n \"ub_methods\" = \"ub_methods_justification_reasonableness_validity_robustness\",\n \"conf_methods\" = \"conf_index_methods_justification_reasonableness_validity_robustness\",\n \"logic_comms\" = \"logic_communication\",\n \"lb_logic_comms\" = \"lb_logic_communication\",\n \"ub_logic_comms\" = \"ub_logic_communication\",\n \"conf_logic_comms\" = \"conf_index_logic_communication\",\n \"real_world\" = \"engaging_with_real_world_impact_quantification_practice_realism_and_relevance\",\n \"lb_real_world\" = \"lb_engaging_with_real_world_impact_quantification_practice_realism_and_relevance\",\n \"ub_real_world\" = \"ub_engaging_with_real_world_impact_quantification_practice_realism_and_relevance\",\n \"conf_real_world\" = \"conf_index_engaging_with_real_world_impact_quantification_practice_realism_and_relevance\",\n \"gp_relevance\" = \"relevance_to_global_priorities\",\n \"lb_gp_relevance\" = \"lb_relevance_to_global_priorities\",\n \"ub_gp_relevance\" = \"ub_relevance_to_global_priorities\",\n \"conf_gp_relevance\" = \"conf_index_relevance_to_global_priorities\",\n \"journal_predict\" = \"journal_quality_predict\",\n \"lb_journal_predict\" = \"lb_journal_quality_predict\",\n \"ub_journal_predict\" = \"ub_journal_quality_predict\",\n \"conf_journal_predict\" = \"conf_index_journal_quality_predict\",\n \"open_sci\" = \"open_collaborative_replicable\",\n \"conf_open_sci\" = \"conf_index_open_collaborative_replicable\",\n \"lb_open_sci\" = \"lb_open_collaborative_replicable\",\n \"ub_open_sci\" = \"ub_open_collaborative_replicable\",\n \"conf_merits_journal\" = \"conf_index_merits_journal\"\n)\n\nevals_pub <- evals_pub %>%\n rename(!!!new_names)\n\n# Create a list of labels with the old, longer names\nlabels <- str_replace_all(new_names, \"_\", \" \") %>% str_to_title()\n\n# Assign labels to the dataframe / tibble\n# (maybe this can be done as an attribute, not currently working)\n# for(i in seq_along(labels)) {\n# col_name <- new_names[names(new_names)[i]]\n# label <- labels[i]\n# attr(evals_pub[[col_name]], \"label\") <- label\n# }\n\n\n# expand categories into columns, unlist everything\nevals_pub %<>%\n tidyr::unnest_wider(cat, names_sep = \"_\") %>% # give each of these its own col\n mutate(across(everything(), unlist)) # maybe check why some of these are lists in the first place\n \n\n# clean the Anonymous names\nevals_pub$eval_name <- ifelse(\n grepl(\"^\\\\b\\\\w+\\\\b$|\\\\bAnonymous\\\\b\", evals_pub$eval_name),\n paste0(\"Anonymous_\", seq_along(evals_pub$eval_name)),\n evals_pub$eval_name\n)\n\n#housekeeping\nrm(evals)\n\n#Todo -- check the unlist is not propagating the entry\n#Note: category, topic_subfield, and source have multiple meaningful categories. These will need care \n```\n:::\n\n\n\n\n\n\n\n### Reconcile uncertainty ratings and CIs {-}\n\nWhere people gave only confidence level 'dots', we impute CIs (confidence/credible intervals). We follow the correspondence described [here](https://effective-giving-marketing.gitbook.io/unjournal-x-ea-and-global-priorities-research/policies-projects-evaluation-workflow/evaluation/guidelines-for-evaluators#1-5-dots-explanation-and-relation-to-cis). (Otherwise, where they gave actual CIs, we use these.)^[Note this is only a first-pass; a more sophisticated approach may be warranted in future.]\n\n::: {.callout-note collapse=\"true\"}\n## Dots to interval choices\n\n> 5 = Extremely confident, i.e., 90% confidence interval spans +/- 4 points or less)\n\nFor 0-100 ratings, code the LB as $max(R - 4,0)$ and the UB as $min(R + 4,100)$, where R is the stated (middle) rating. This 'scales' the CI, as interpreted, to be proportional to the rating, with a maximum 'interval' of about 8, with the rating is about 96.\n\n> 4 = Very*confident: 90% confidence interval +/- 8 points or less\n\nFor 0-100 ratings, code the LB as $max(R - 8,0)$ and the UB as $min(R + 8,100)$, where R is the stated (middle) rating. \n\n> 3 = Somewhat** confident: 90% confidence interval +/- 15 points or less \n\n> 2 = Not very** confident: 90% confidence interval, +/- 25 points or less\n\nComparable scaling for the 2-3 ratings as for the 4 and 5 rating.\n\n> 1 = Not** confident: (90% confidence interval +/- more than 25 points)\n \nCode LB as $max(R - 37.5,0)$ and the UB as $min(R + 37.5,100)$. \n \nThis is just a first-pass. There might be a more information-theoretic way of doing this. On the other hand, we might be switching the evaluations to use a different tool soon, perhaps getting rid of the 1-5 confidence ratings altogether.\n\n::: \n\n\n\n::: {.cell}\n\n```{.r .cell-code code-summary=\"reconcile explicit bounds and stated confidence level\"}\n# Define the baseline widths for each confidence rating\n# JB: it would be good to have some more backing of whether this\n# is a valid way to translate these confidence levels into CIs\n\n# baseline_widths <- c(4, 8, 15, 25, 37.5)\n\n# Lists of categories\nrating_cats <- c(\"overall\", \"adv_knowledge\", \"methods\", \"logic_comms\", \"real_world\", \"gp_relevance\", \"open_sci\")\n\n#... 'predictions' are currently 1-5 (0-5?)\npred_cats <- c(\"journal_predict\", \"merits_journal\")\n\n#DR: Note that I use these objects in future chapters, but they are not connected to the data frame. Either one saves and reinputs the whole environment (messy, replicability issue), or you save this somewhere and re-input it or connect it to the data frame that gets saved (not sure how to do it), or you hard-code reinput it in the next chapter. \n\n#I do the latter for now, but I'm not happy about it, because the idea is 'input definitions in a single place to use later'\n\n#JB: I don't really understand why not just hard code it in the next chapter. These are very short strings. Do you expect them to change often? If so, we can derive them from a dataframe somewhere in the future or save as a separate object. \n\n\n# JB: Rewritten functions for adding imputed ub and lb\n\n# calculate the lower and upper bounds, \n# rating: given a rating, \n# conf: a confidence (1-5) score,\n# type: a bound type (lower, upper),\n# scale: a scale (100 is 0-100, 5 is 1-5)\n# This function is not vectorized\ncalc_bounds <- function(rating, conf, type, scale) {\n \n if(scale == 5){ #for the 'journal tier prediction case'\n baseline_width = case_match(conf, \n 5 ~ .2, #4*5/100\n 4 ~ .4, #8*5/100\n 3 ~ .75, #15*5/100\n 2 ~ 1.25, #25*5/100\n 1 ~ 1.875, #37.5*5/100\n .default = NA_real_) \n\n upper = min(rating + baseline_width, 5)\n lower = max(rating - baseline_width, 1)\n }#/if(scale == 5)\n \n if(scale == 100){ #for the 'ratings case'\n \n baseline_width = case_match(conf, \n 5 ~ 4, \n 4 ~ 8, \n 3 ~ 15, \n 2 ~ 25, \n 1 ~ 37.5,\n .default = NA_real_)\n \n upper = min(rating + baseline_width, 100)\n lower = max(rating - baseline_width, 0)\n } #/if(scale == 100)\n \n if(type == \"lower\") return(lower)\n if(type == \"upper\") return(upper)\n}\n\n\n# calculate or find correct lower or upper bound\n# based on rating type, and lb, ub, and conf values\nimpute_bounds <- function(var_name, est, lb, ub, conf, bound_type) {\n \n # get scale of each variable\n scale = if_else(var_name %in% c(\"journal_predict\", \"merits_journal\"), # if variable is a prediction\n 5, 100) #scale is 5, else scale is 100\n # if calculating lower bound variable\n if(bound_type == \"lower\") { #we are calculating a lower bound imputation\n # \n calculated_bound = map_dbl(.x = est, .f = calc_bounds, conf = conf, type = bound_type, scale = scale)\n \n imp_bound = if_else(is.na(lb), calculated_bound, lb)\n }\n \n # if calculating upper bound variable\n if(bound_type == \"upper\") { #we are calculating an upper bound imputation\n # \n calculated_bound = map_dbl(.x = est, .f = calc_bounds, conf = conf, type = bound_type, scale = scale)\n imp_bound = if_else(is.na(ub), calculated_bound, ub)\n }\n \n return(imp_bound)\n}\n\n# apply functions to evals_pub\n# JB: I had to do a lot of gymnastics to make this data frame\n# right for applying the functions, then turn it back into\n# the data structure we are currently using I didn't want to deal\n# with how changing the data input would break the rest of the\n# code But we should prioritize saving this data frame in a long format\n# so there isn't data in the variable names (i.e., each row should at least \n# be one type of rating, not one article). \n\nevals_pub %>% \n pivot_longer(cols = -c(id, crucial_rsx, paper_abbrev, eval_name, \n cat_1,cat_2, cat_3,source_main,author_agreement),\n names_pattern = \"(lb_|ub_|conf_)?(.+)\",\n names_to = c(\"value_type\", \"rating_type\")) %>% # one line per rating type\n mutate(value_type = if_else(value_type == \"\", \"est_\", value_type)) %>% #add main rating id\n pivot_wider(names_from = value_type, \n values_from = value) %>%\n rowwise() %>% # apply function to each row\n mutate(lb_imp_ = impute_bounds(var_name = rating_type,\n est = est_,\n lb = lb_, ub = ub_, conf = conf_,\n bound_type = \"lower\")) %>% \n mutate(ub_imp_ = impute_bounds(var_name = rating_type,\n est = est_,\n lb = lb_, ub = ub_, conf = conf_,\n bound_type = \"upper\")) %>% \n pivot_wider(names_from = rating_type, # take the dataframe back to old format\n values_from = c(est_, ub_, lb_, conf_, lb_imp_, ub_imp_),\n names_sep = \"\") %>% \n dplyr::rename_with(.cols = matches(\"^[ul]b_imp\"),\n .fn = gsub,\n pattern = \"(ub_imp|lb_imp)_(.+)\", \n replacement = \"\\\\2_\\\\1\") %>% \n dplyr::rename_with(.cols = starts_with(\"est_\"),\n .fn = gsub,\n pattern = \"est_(.+)\",\n replacement = \"\\\\1\") -> evals_pub\n```\n:::\n\n\n\nWe cannot publicly share the 'papers under consideration', but we can share some of the statistics on these papers. Let's generate an ID (or later, salted hash) for each such paper, and keep only the shareable features of interest\n\n\n::: {.cell}\n\n```{.r .cell-code}\nall_papers_p <- all_pub_records %>% \n dplyr::select(\n id,\n category,\n cfdc_DR,\n 'confidence -- user entered',\n cfdc_assessor,\n avg_cfdc,\n category,\n cause_cat_1_text,\n cause_cat_2_text,\n topic_subfield_text,\n eval_manager_text,\n 'publication status',\n 'Contacted author?',\n 'stage of process/todo',\n 'source_main', \n 'author permission?',\n'Direct Kotahi Prize Submission?',\n 'createdTime' \n )\n```\n:::\n\n::: {.cell}\n\n```{.r .cell-code}\n# Create a dataset to be used as the input to a shiny app\n\n#| include: false\n#| \npaper_ratings <- evals_pub %>% \n select(paper_abbrev, eval_name, one_of(rating_cats), one_of(pred_cats), ends_with(\"_imp\")) %>% # rating vars\n dplyr::rename_with(.cols = c(one_of(rating_cats), one_of(pred_cats)),\n .fn = gsub,\n pattern = \"(.+)\", \n replacement = \"\\\\1_best\") %>% \n pivot_longer(cols = -c(paper_abbrev, eval_name),\n names_pattern = \"(overall|adv_knowledge|methods|logic_comms|real_world|gp_relevance|open_sci|journal_predict|merits_journal)_(.+)\",\n names_to = c(\"rating_type\", \".value\"))# one line per rating type\n\nwrite_rds(paper_ratings, file = here(\"shinyapp/DataExplorer\", \"shiny_explorer.rds\"))\n```\n:::\n\n::: {.cell}\n\n```{.r .cell-code code-summary=\"save data for others' use\"}\nall_papers_p %>% saveRDS(file = here(\"data\", \"all_papers_p.Rdata\"))\nall_papers_p %>% write_csv(file = here(\"data\", \"all_papers_p.csv\"))\n\nevals_pub %>% saveRDS(file = here(\"data\", \"evals.Rdata\"))\nevals_pub %>% write_csv(file = here(\"data\", \"evals.csv\"))\n\n#evals_pub %>% readRDS(file = here(\"data\", \"evals.Rdata\"))\n```\n:::\n", - "supporting": [ - "evaluation_data_input_files" - ], + "markdown": "# Evaluation data: input/features\n\n\n::: {.cell}\n\n```{.r .cell-code code-summary=\"load packages\"}\nlibrary(tidyverse) \n\n# data acquisition ----\n#devtools::install_github(\"bergant/airtabler\")\nlibrary(airtabler)\n\n# data cleaning & shaping ----\n\n# data analysis ----\n# library(lme4)\n# library(lmtest) # Testing Linear Regression Models\n\n# markdown et al. ----\nlibrary(knitr)\nlibrary(bookdown)\nlibrary(quarto)\nlibrary(formattable) # Create 'Formattable' Data Structures\n\n# others ----\nlibrary(here) # A Simpler Way to Find Your Files\n#devtools::install_github(\"metamelb-repliCATS/aggreCAT\")\n#library(aggrecat)\n\n# Make sure select is always the dplyr version\nselect <- dplyr::select \n\n# options\noptions(knitr.duplicate.label = \"allow\")\n```\n:::\n\n\n\n::: {.callout-note collapse=\"true\"}\n## Note on data input (10-Aug-23)\n\nBelow, the evaluation data is input from an Airtable, which itself was largely hand-input from evaluators' reports. As PubPub builds (target: end of Sept. 2023), this will allow us to include the ratings and predictions as structured data objects. We then plan to access and input this data *directly* from the PubPub (API?) into the present analysis. This will improve automation and limit the potential for data entry errors.\n\n::: \n\n\n::: {.cell}\n\n```{.r .cell-code code-summary=\"input from airtable\"}\nbase_id <- \"appbPYEw9nURln7Qg\"\n\n\n# Set your Airtable API key \nSys.setenv(AIRTABLE_API_KEY = Sys.getenv(\"AIRTABLE_API_KEY\"))\n#this should be set in my .Renviron file\n\n# Read data from a specific view\n\nevals <- air_get(base = base_id, \"output_eval\") \n\nall_pub_records <- data.frame()\npub_records <- air_select(base = base_id, table = \"crucial_research\")\n\n# Append the records to the list\nall_pub_records <- bind_rows(all_pub_records, pub_records)\n\n# While the length of the records list is 100 (the maximum), fetch more records\nwhile(nrow(pub_records) == 100) {\n # Get the ID of the last record in the list\n offset <- get_offset(pub_records)\n \n # Fetch the next 100 records, starting after the last ID\n pub_records <- air_select(base = base_id, table = \"crucial_research\", offset = offset)\n \n # Append the records to the df\n all_pub_records <- bind_rows(all_pub_records, pub_records)\n}\n\n# housekeeping\nrm(pub_records)\n```\n:::\n\n::: {.cell}\n\n```{.r .cell-code code-summary=\"just the useful and publish-able data, clean a bit\"}\n# clean evals names to snakecase\ncolnames(evals) <- snakecase::to_snake_case(colnames(evals))\n\nevals_pub <- evals %>% \n dplyr::rename(stage_of_process = stage_of_process_todo_from_crucial_research_2) %>% \n mutate(stage_of_process = unlist(stage_of_process)) %>% \n dplyr::filter(stage_of_process == \"published\") %>% \n select(id, \n crucial_research, \n paper_abbrev, \n evaluator_name, \n category, \n source_main, \n author_agreement, \n overall, \n lb_overall, \n ub_overall, \n conf_index_overall, \n advancing_knowledge_and_practice, \n lb_advancing_knowledge_and_practice, \n ub_advancing_knowledge_and_practice, \n conf_index_advancing_knowledge_and_practice,\n methods_justification_reasonableness_validity_robustness,\n lb_methods_justification_reasonableness_validity_robustness,\n ub_methods_justification_reasonableness_validity_robustness,\n conf_index_methods_justification_reasonableness_validity_robustness, \n logic_communication, lb_logic_communication, ub_logic_communication, \n conf_index_logic_communication,\n engaging_with_real_world_impact_quantification_practice_realism_and_relevance,\n lb_engaging_with_real_world_impact_quantification_practice_realism_and_relevance,\n ub_engaging_with_real_world_impact_quantification_practice_realism_and_relevance,\n conf_index_engaging_with_real_world_impact_quantification_practice_realism_and_relevance,\n relevance_to_global_priorities, \n lb_relevance_to_global_priorities, \n ub_relevance_to_global_priorities, \n conf_index_relevance_to_global_priorities, \n journal_quality_predict, \n lb_journal_quality_predict, \n ub_journal_quality_predict,\n conf_index_journal_quality_predict, \n open_collaborative_replicable, \n conf_index_open_collaborative_replicable, \n lb_open_collaborative_replicable, \n ub_open_collaborative_replicable, \n merits_journal, \n lb_merits_journal, \n ub_merits_journal, \n conf_index_merits_journal)\n\n# shorten names (before you expand into columns)\nnew_names <- c(\n \"eval_name\" = \"evaluator_name\",\n \"cat\" = \"category\",\n \"crucial_rsx\" = \"crucial_research\",\n \"conf_overall\" = \"conf_index_overall\",\n \"adv_knowledge\" = \"advancing_knowledge_and_practice\",\n \"lb_adv_knowledge\" = \"lb_advancing_knowledge_and_practice\",\n \"ub_adv_knowledge\" = \"ub_advancing_knowledge_and_practice\",\n \"conf_adv_knowledge\" = \"conf_index_advancing_knowledge_and_practice\",\n \"methods\" = \"methods_justification_reasonableness_validity_robustness\",\n \"lb_methods\" = \"lb_methods_justification_reasonableness_validity_robustness\",\n \"ub_methods\" = \"ub_methods_justification_reasonableness_validity_robustness\",\n \"conf_methods\" = \"conf_index_methods_justification_reasonableness_validity_robustness\",\n \"logic_comms\" = \"logic_communication\",\n \"lb_logic_comms\" = \"lb_logic_communication\",\n \"ub_logic_comms\" = \"ub_logic_communication\",\n \"conf_logic_comms\" = \"conf_index_logic_communication\",\n \"real_world\" = \"engaging_with_real_world_impact_quantification_practice_realism_and_relevance\",\n \"lb_real_world\" = \"lb_engaging_with_real_world_impact_quantification_practice_realism_and_relevance\",\n \"ub_real_world\" = \"ub_engaging_with_real_world_impact_quantification_practice_realism_and_relevance\",\n \"conf_real_world\" = \"conf_index_engaging_with_real_world_impact_quantification_practice_realism_and_relevance\",\n \"gp_relevance\" = \"relevance_to_global_priorities\",\n \"lb_gp_relevance\" = \"lb_relevance_to_global_priorities\",\n \"ub_gp_relevance\" = \"ub_relevance_to_global_priorities\",\n \"conf_gp_relevance\" = \"conf_index_relevance_to_global_priorities\",\n \"journal_predict\" = \"journal_quality_predict\",\n \"lb_journal_predict\" = \"lb_journal_quality_predict\",\n \"ub_journal_predict\" = \"ub_journal_quality_predict\",\n \"conf_journal_predict\" = \"conf_index_journal_quality_predict\",\n \"open_sci\" = \"open_collaborative_replicable\",\n \"conf_open_sci\" = \"conf_index_open_collaborative_replicable\",\n \"lb_open_sci\" = \"lb_open_collaborative_replicable\",\n \"ub_open_sci\" = \"ub_open_collaborative_replicable\",\n \"conf_merits_journal\" = \"conf_index_merits_journal\"\n)\n\nevals_pub <- evals_pub %>%\n rename(!!!new_names)\n\n# Create a list of labels with the old, longer names\nlabels <- str_replace_all(new_names, \"_\", \" \") %>% str_to_title()\n\n# Assign labels to the dataframe / tibble\n# (maybe this can be done as an attribute, not currently working)\n# for(i in seq_along(labels)) {\n# col_name <- new_names[names(new_names)[i]]\n# label <- labels[i]\n# attr(evals_pub[[col_name]], \"label\") <- label\n# }\n\n\n# expand categories into columns, unlist everything\nevals_pub %<>%\n tidyr::unnest_wider(cat, names_sep = \"_\") %>% # give each of these its own col\n mutate(across(everything(), unlist)) # maybe check why some of these are lists in the first place\n \n\n# clean the Anonymous names\nevals_pub$eval_name <- ifelse(\n grepl(\"^\\\\b\\\\w+\\\\b$|\\\\bAnonymous\\\\b\", evals_pub$eval_name),\n paste0(\"Anonymous_\", seq_along(evals_pub$eval_name)),\n evals_pub$eval_name\n)\n\n#housekeeping\nrm(evals)\n\n#Todo -- check the unlist is not propagating the entry\n#Note: category, topic_subfield, and source have multiple meaningful categories. These will need care \n```\n:::\n\n::: {.cell}\n\n```{.r .cell-code}\nevals_pub_long <- evals_pub %>% \n pivot_longer(cols = -c(id, crucial_rsx, paper_abbrev, eval_name, \n cat_1,cat_2, cat_3,source_main,author_agreement),\n names_pattern = \"(lb_|ub_|conf_)?(.+)\",\n names_to = c(\"value_type\", \"rating_type\")) %>% # one line per rating type\n mutate(value_type = if_else(value_type == \"\", \"est_\", value_type)) %>% #add main rating id\n pivot_wider(names_from = value_type, \n values_from = value)\n```\n:::\n\n\n\n\n\n### Reconcile uncertainty ratings and CIs {-}\n\nWhere people gave only confidence level 'dots', we impute CIs (confidence/credible intervals). We follow the correspondence described [here](https://effective-giving-marketing.gitbook.io/unjournal-x-ea-and-global-priorities-research/policies-projects-evaluation-workflow/evaluation/guidelines-for-evaluators#1-5-dots-explanation-and-relation-to-cis). (Otherwise, where they gave actual CIs, we use these.)^[Note this is only a first-pass; a more sophisticated approach may be warranted in future.]\n\n::: {.callout-note collapse=\"true\"}\n## Dots to interval choices\n\n> 5 = Extremely confident, i.e., 90% confidence interval spans +/- 4 points or less)\n\nFor 0-100 ratings, code the LB as $max(R - 4,0)$ and the UB as $min(R + 4,100)$, where R is the stated (middle) rating. This 'scales' the CI, as interpreted, to be proportional to the rating, with a maximum 'interval' of about 8, with the rating is about 96.\n\n> 4 = Very*confident: 90% confidence interval +/- 8 points or less\n\nFor 0-100 ratings, code the LB as $max(R - 8,0)$ and the UB as $min(R + 8,100)$, where R is the stated (middle) rating. \n\n> 3 = Somewhat** confident: 90% confidence interval +/- 15 points or less \n\n> 2 = Not very** confident: 90% confidence interval, +/- 25 points or less\n\nComparable scaling for the 2-3 ratings as for the 4 and 5 rating.\n\n> 1 = Not** confident: (90% confidence interval +/- more than 25 points)\n \nCode LB as $max(R - 37.5,0)$ and the UB as $min(R + 37.5,100)$. \n \nThis is just a first-pass. There might be a more information-theoretic way of doing this. On the other hand, we might be switching the evaluations to use a different tool soon, perhaps getting rid of the 1-5 confidence ratings altogether.\n\n::: \n\n\n\n::: {.cell}\n\n```{.r .cell-code code-summary=\"reconcile explicit bounds and stated confidence level\"}\n# Define the baseline widths for each confidence rating\n# JB: it would be good to have some more backing of whether this\n# is a valid way to translate these confidence levels into CIs\n\n# baseline_widths <- c(4, 8, 15, 25, 37.5)\n\n# Lists of categories\nrating_cats <- c(\"overall\", \"adv_knowledge\", \"methods\", \"logic_comms\", \"real_world\", \"gp_relevance\", \"open_sci\")\n\n#... 'predictions' are currently 1-5 (0-5?)\npred_cats <- c(\"journal_predict\", \"merits_journal\")\n\n#DR: Note that I use these objects in future chapters, but they are not connected to the data frame. Either one saves and reinputs the whole environment (messy, replicability issue), or you save this somewhere and re-input it or connect it to the data frame that gets saved (not sure how to do it), or you hard-code reinput it in the next chapter. \n\n#I do the latter for now, but I'm not happy about it, because the idea is 'input definitions in a single place to use later'\n\n#JB: I don't really understand why not just hard code it in the next chapter. These are very short strings. Do you expect them to change often? If so, we can derive them from a dataframe somewhere in the future or save as a separate object. \n\n\n# JB: Rewritten functions for adding imputed ub and lb\n\n# calculate the lower and upper bounds, \n# rating: given a rating, \n# conf: a confidence (1-5) score,\n# type: a bound type (lower, upper),\n# scale: a scale (100 is 0-100, 5 is 1-5)\n# This function is not vectorized\ncalc_bounds <- function(rating, conf, type, scale) {\n \n if(scale == 5){ #for the 'journal tier prediction case'\n baseline_width = case_match(conf, \n 5 ~ .2, #4*5/100\n 4 ~ .4, #8*5/100\n 3 ~ .75, #15*5/100\n 2 ~ 1.25, #25*5/100\n 1 ~ 1.875, #37.5*5/100\n .default = NA_real_) \n\n upper = min(rating + baseline_width, 5)\n lower = max(rating - baseline_width, 1)\n }#/if(scale == 5)\n \n if(scale == 100){ #for the 'ratings case'\n \n baseline_width = case_match(conf, \n 5 ~ 4, \n 4 ~ 8, \n 3 ~ 15, \n 2 ~ 25, \n 1 ~ 37.5,\n .default = NA_real_)\n \n upper = min(rating + baseline_width, 100)\n lower = max(rating - baseline_width, 0)\n } #/if(scale == 100)\n \n if(type == \"lower\") return(lower)\n if(type == \"upper\") return(upper)\n}\n\n\n# calculate or find correct lower or upper bound\n# based on rating type, and lb, ub, and conf values\nimpute_bounds <- function(var_name, est, lb, ub, conf, bound_type) {\n \n # get scale of each variable\n scale = if_else(var_name %in% c(\"journal_predict\", \"merits_journal\"), # if variable is a prediction\n 5, 100) #scale is 5, else scale is 100\n # if calculating lower bound variable\n if(bound_type == \"lower\") { #we are calculating a lower bound imputation\n # \n calculated_bound = map_dbl(.x = est, .f = calc_bounds, conf = conf, type = bound_type, scale = scale)\n \n imp_bound = if_else(is.na(lb), calculated_bound, lb)\n }\n \n # if calculating upper bound variable\n if(bound_type == \"upper\") { #we are calculating an upper bound imputation\n # \n calculated_bound = map_dbl(.x = est, .f = calc_bounds, conf = conf, type = bound_type, scale = scale)\n imp_bound = if_else(is.na(ub), calculated_bound, ub)\n }\n \n return(imp_bound)\n}\n\n# apply functions to evals_pub_long\n# where each row is one type of rating\n# so each evaluation is 9 rows long\nevals_pub_long <- evals_pub_long %>% \n rowwise() %>% # apply function to each row\n mutate(lb_imp_ = impute_bounds(var_name = rating_type,\n est = est_,\n lb = lb_, ub = ub_, conf = conf_,\n bound_type = \"lower\")) %>% \n mutate(ub_imp_ = impute_bounds(var_name = rating_type,\n est = est_,\n lb = lb_, ub = ub_, conf = conf_,\n bound_type = \"upper\"))\n\n# Reshape evals_pub_long into evals_pub to add imputed bounds\nevals_pub <- evals_pub_long %>% \n pivot_wider(names_from = rating_type, # take the dataframe back to old format\n values_from = c(est_, ub_, lb_, conf_, lb_imp_, ub_imp_),\n names_sep = \"\") %>% \n dplyr::rename_with(.cols = matches(\"^[ul]b_imp\"),\n .fn = gsub,\n pattern = \"(ub_imp|lb_imp)_(.+)\", \n replacement = \"\\\\2_\\\\1\") %>% \n dplyr::rename_with(.cols = starts_with(\"est_\"),\n .fn = gsub,\n pattern = \"est_(.+)\",\n replacement = \"\\\\1\")\n\n# Clean evals_pub_long names (remove _ at end)\nevals_pub_long <- evals_pub_long %>% \n rename_with(.cols = ends_with(\"_\"),\n .fn = str_remove,\n pattern = \"_$\")\n```\n:::\n\n\n\nWe cannot publicly share the 'papers under consideration', but we can share some of the statistics on these papers. Let's generate an ID (or later, salted hash) for each such paper, and keep only the shareable features of interest\n\n\n::: {.cell}\n\n```{.r .cell-code}\nall_papers_p <- all_pub_records %>% \n dplyr::select(\n id,\n category,\n cfdc_DR,\n 'confidence -- user entered',\n cfdc_assessor,\n avg_cfdc,\n category,\n cause_cat_1_text,\n cause_cat_2_text,\n topic_subfield_text,\n eval_manager_text,\n 'publication status',\n 'Contacted author?',\n 'stage of process/todo',\n 'source_main', \n 'author permission?',\n'Direct Kotahi Prize Submission?',\n 'createdTime' \n )\n```\n:::\n\n::: {.cell}\n\n```{.r .cell-code}\n# Create a dataset to be used as the input to a shiny app\n\n#| include: false\n#| \nevals_pub_long %>% \n write_rds(file = here(\"shinyapp/DataExplorer\", \"shiny_explorer.rds\"))\n```\n:::\n\n::: {.cell}\n\n```{.r .cell-code code-summary=\"save data for others' use\"}\nall_papers_p %>% saveRDS(file = here(\"data\", \"all_papers_p.Rdata\"))\nall_papers_p %>% write_csv(file = here(\"data\", \"all_papers_p.csv\"))\n\nevals_pub %>% saveRDS(file = here(\"data\", \"evals.Rdata\"))\nevals_pub %>% write_csv(file = here(\"data\", \"evals.csv\"))\n\nevals_pub_long %>% write_rds(file = here(\"data\", \"evals_long.rds\"))\nevals_pub_long %>% write_csv(file = here(\"data\", \"evals_long.csv\"))\n\n#evals_pub %>% readRDS(file = here(\"data\", \"evals.Rdata\"))\n```\n:::\n", + "supporting": [], "filters": [ "rmarkdown/pagebreak.lua" ], diff --git a/chapters/aggregation.qmd b/chapters/aggregation.qmd index 482de5c..5b68852 100644 --- a/chapters/aggregation.qmd +++ b/chapters/aggregation.qmd @@ -8,9 +8,14 @@ library(tidyverse) library(aggreCAT) library(here) +library(irr) evals_pub <- read_rds(file = here("data", "evals.Rdata")) +# Lists of categories +rating_cats <- c("overall", "adv_knowledge", "methods", "logic_comms", "real_world", "gp_relevance", "open_sci") +pred_cats <- c("journal_predict", "merits_journal") + ``` @@ -154,6 +159,52 @@ aggreCAT::DistributionWAgg(expert_judgements = temp, round_2_filter = FALSE, typ ## Inter-rater reliability +```{r} +#| echo: false +#| fig-height: 8 + +# function that returns kripp.alpha +# value. It checks if there is more than +# one rater first to avoid errors +# and converts the nested data into +# a matrix to allow +mod_kripp_alpha <- function(dat) { + dat = as.matrix.POSIXlt(dat) + if(nrow(dat)>1) { + a = kripp.alpha(dat, method = "ratio") + res = a$value + } else { + res = NA_integer_ + } + return(res) +} + +# plot +evals_pub %>% + group_by(paper_abbrev) %>% + select(paper_abbrev, all_of(rating_cats)) %>% + nest(data = -paper_abbrev) %>% + mutate(KrippAlpha = map_dbl(.x = data, .f = mod_kripp_alpha)) %>% + unnest(data) %>% + group_by(KrippAlpha, add = T) %>% + summarize(Raters = n()) %>% + ungroup() %>% + ggplot(aes(x = reorder(paper_abbrev, KrippAlpha), y = KrippAlpha)) + + geom_point(aes(color = paper_abbrev, size = Raters), + stat = "identity", shape = 16, stroke = 1) + + coord_flip() + + labs(x = "Paper", y = "Krippendorf's Alpha") + + theme_bw() + + theme(text = element_text(size = 15)) + + scale_x_discrete(labels = function(x) str_wrap(x, width = 20)) + + scale_size(breaks = c(2,3)) + + scale_y_continuous(limits = c(-1,1)) + + guides(color=F) + +``` + + + ## Decomposing variation, dimension reduction, simple linear models diff --git a/chapters/evaluation_data_analysis.qmd b/chapters/evaluation_data_analysis.qmd index b795c13..8f5de56 100644 --- a/chapters/evaluation_data_analysis.qmd +++ b/chapters/evaluation_data_analysis.qmd @@ -145,9 +145,6 @@ rename_dtstuff <- function(df){ # Need to find a way to control column width but it seems to be a problem with DT # https://github.com/rstudio/DT/issues/29 -# we didn't seem to be using all_evals_dt so I removed it to increase readability - - evals_pub_df <- evals_pub %>% # Arrange data @@ -323,7 +320,8 @@ Todo: ^[Make interactive/dashboards of the elements below] ```{r all_categories} evals_pub %>% - select(paper_abbrev, starts_with("cat_")) %>% + select(paper_abbrev, starts_with("cat_")) %>% + distinct() %>% pivot_longer(cols = starts_with("cat_"), names_to = "CatNum", values_to = "Category") %>% group_by(CatNum, Category) %>% count() %>% @@ -352,7 +350,9 @@ evals_pub %>% rowwise() %>% mutate(source_main = str_replace_all(string = source_main, pattern = "-", - replace = " ") %>% str_to_title()) %>% + replace = " ") %>% str_to_title()) %>% + select(paper_abbrev, source_main) %>% + distinct() %>% ggplot(aes(x = source_main)) + geom_bar(position = "stack", stat = "count", color = "grey30", fill = "grey80") + labs(x = "Source", y = "Count") + @@ -363,7 +363,7 @@ evals_pub %>% ``` -```{r data_clean_} +```{r data clean} # JB: Most of these should probably be cleaned in data storage library(RColorBrewer) # for color palettes @@ -460,7 +460,7 @@ In future, we aim to build a dashboard allowing people to use the complete set o ```{=html} - + ``` @@ -525,8 +525,6 @@ Next, look for systematic variation in the ratings \ - - ### Relationship among the ratings (and predictions) {-} ::: {.callout-note collapse="true"} diff --git a/chapters/evaluation_data_input.qmd b/chapters/evaluation_data_input.qmd index 4ccbbe6..2de334a 100644 --- a/chapters/evaluation_data_input.qmd +++ b/chapters/evaluation_data_input.qmd @@ -207,6 +207,16 @@ rm(evals) ``` +```{r evals_pub to longer format} +evals_pub_long <- evals_pub %>% + pivot_longer(cols = -c(id, crucial_rsx, paper_abbrev, eval_name, + cat_1,cat_2, cat_3,source_main,author_agreement), + names_pattern = "(lb_|ub_|conf_)?(.+)", + names_to = c("value_type", "rating_type")) %>% # one line per rating type + mutate(value_type = if_else(value_type == "", "est_", value_type)) %>% #add main rating id + pivot_wider(names_from = value_type, + values_from = value) +``` @@ -331,23 +341,10 @@ impute_bounds <- function(var_name, est, lb, ub, conf, bound_type) { return(imp_bound) } -# apply functions to evals_pub -# JB: I had to do a lot of gymnastics to make this data frame -# right for applying the functions, then turn it back into -# the data structure we are currently using I didn't want to deal -# with how changing the data input would break the rest of the -# code But we should prioritize saving this data frame in a long format -# so there isn't data in the variable names (i.e., each row should at least -# be one type of rating, not one article). - -evals_pub %>% - pivot_longer(cols = -c(id, crucial_rsx, paper_abbrev, eval_name, - cat_1,cat_2, cat_3,source_main,author_agreement), - names_pattern = "(lb_|ub_|conf_)?(.+)", - names_to = c("value_type", "rating_type")) %>% # one line per rating type - mutate(value_type = if_else(value_type == "", "est_", value_type)) %>% #add main rating id - pivot_wider(names_from = value_type, - values_from = value) %>% +# apply functions to evals_pub_long +# where each row is one type of rating +# so each evaluation is 9 rows long +evals_pub_long <- evals_pub_long %>% rowwise() %>% # apply function to each row mutate(lb_imp_ = impute_bounds(var_name = rating_type, est = est_, @@ -356,7 +353,10 @@ evals_pub %>% mutate(ub_imp_ = impute_bounds(var_name = rating_type, est = est_, lb = lb_, ub = ub_, conf = conf_, - bound_type = "upper")) %>% + bound_type = "upper")) + +# Reshape evals_pub_long into evals_pub to add imputed bounds +evals_pub <- evals_pub_long %>% pivot_wider(names_from = rating_type, # take the dataframe back to old format values_from = c(est_, ub_, lb_, conf_, lb_imp_, ub_imp_), names_sep = "") %>% @@ -367,7 +367,13 @@ evals_pub %>% dplyr::rename_with(.cols = starts_with("est_"), .fn = gsub, pattern = "est_(.+)", - replacement = "\\1") -> evals_pub + replacement = "\\1") + +# Clean evals_pub_long names (remove _ at end) +evals_pub_long <- evals_pub_long %>% + rename_with(.cols = ends_with("_"), + .fn = str_remove, + pattern = "_$") ``` @@ -405,22 +411,13 @@ all_papers_p <- all_pub_records %>% #| include: false #| -paper_ratings <- evals_pub %>% - select(paper_abbrev, eval_name, one_of(rating_cats), one_of(pred_cats), ends_with("_imp")) %>% # rating vars - dplyr::rename_with(.cols = c(one_of(rating_cats), one_of(pred_cats)), - .fn = gsub, - pattern = "(.+)", - replacement = "\\1_best") %>% - pivot_longer(cols = -c(paper_abbrev, eval_name), - names_pattern = "(overall|adv_knowledge|methods|logic_comms|real_world|gp_relevance|open_sci|journal_predict|merits_journal)_(.+)", - names_to = c("rating_type", ".value"))# one line per rating type - -write_rds(paper_ratings, file = here("shinyapp/DataExplorer", "shiny_explorer.rds")) +evals_pub_long %>% + write_rds(file = here("shinyapp/DataExplorer", "shiny_explorer.rds")) ``` -```{r} +```{r save data} #| label: savedata #| code-summary: "save data for others' use" @@ -431,6 +428,9 @@ all_papers_p %>% write_csv(file = here("data", "all_papers_p.csv")) evals_pub %>% saveRDS(file = here("data", "evals.Rdata")) evals_pub %>% write_csv(file = here("data", "evals.csv")) +evals_pub_long %>% write_rds(file = here("data", "evals_long.rds")) +evals_pub_long %>% write_csv(file = here("data", "evals_long.csv")) + #evals_pub %>% readRDS(file = here("data", "evals.Rdata")) ``` diff --git a/data/evals_long.csv b/data/evals_long.csv new file mode 100644 index 0000000..b3d71a0 --- /dev/null +++ b/data/evals_long.csv @@ -0,0 +1,208 @@ +id,crucial_rsx,paper_abbrev,eval_name,cat_1,cat_2,cat_3,source_main,author_agreement,rating_type,est,lb,ub,conf,lb_imp,ub_imp +rec1QrAkko0DAPQ8v,"When Celebrities Speak: A Nationwide Twitter Experiment Promoting Vaccination In Indonesia (Alatas et al, 2019/2021)","Celeb. Twitter promo, Indonesia vacc.",Anonymous_1,GH&D,biorisk,NA,internal-NBER,Emailed,overall,62,NA,NA,3,47,77 +rec1QrAkko0DAPQ8v,"When Celebrities Speak: A Nationwide Twitter Experiment Promoting Vaccination In Indonesia (Alatas et al, 2019/2021)","Celeb. Twitter promo, Indonesia vacc.",Anonymous_1,GH&D,biorisk,NA,internal-NBER,Emailed,adv_knowledge,55,NA,NA,3,40,70 +rec1QrAkko0DAPQ8v,"When Celebrities Speak: A Nationwide Twitter Experiment Promoting Vaccination In Indonesia (Alatas et al, 2019/2021)","Celeb. Twitter promo, Indonesia vacc.",Anonymous_1,GH&D,biorisk,NA,internal-NBER,Emailed,methods,55,NA,NA,2,30,80 +rec1QrAkko0DAPQ8v,"When Celebrities Speak: A Nationwide Twitter Experiment Promoting Vaccination In Indonesia (Alatas et al, 2019/2021)","Celeb. Twitter promo, Indonesia vacc.",Anonymous_1,GH&D,biorisk,NA,internal-NBER,Emailed,logic_comms,70,NA,NA,3,55,85 +rec1QrAkko0DAPQ8v,"When Celebrities Speak: A Nationwide Twitter Experiment Promoting Vaccination In Indonesia (Alatas et al, 2019/2021)","Celeb. Twitter promo, Indonesia vacc.",Anonymous_1,GH&D,biorisk,NA,internal-NBER,Emailed,real_world,55,NA,NA,3,40,70 +rec1QrAkko0DAPQ8v,"When Celebrities Speak: A Nationwide Twitter Experiment Promoting Vaccination In Indonesia (Alatas et al, 2019/2021)","Celeb. Twitter promo, Indonesia vacc.",Anonymous_1,GH&D,biorisk,NA,internal-NBER,Emailed,gp_relevance,70,NA,NA,3,55,85 +rec1QrAkko0DAPQ8v,"When Celebrities Speak: A Nationwide Twitter Experiment Promoting Vaccination In Indonesia (Alatas et al, 2019/2021)","Celeb. Twitter promo, Indonesia vacc.",Anonymous_1,GH&D,biorisk,NA,internal-NBER,Emailed,journal_predict,3,NA,NA,2,1.75,4.25 +rec1QrAkko0DAPQ8v,"When Celebrities Speak: A Nationwide Twitter Experiment Promoting Vaccination In Indonesia (Alatas et al, 2019/2021)","Celeb. Twitter promo, Indonesia vacc.",Anonymous_1,GH&D,biorisk,NA,internal-NBER,Emailed,open_sci,45,NA,NA,2,20,70 +rec1QrAkko0DAPQ8v,"When Celebrities Speak: A Nationwide Twitter Experiment Promoting Vaccination In Indonesia (Alatas et al, 2019/2021)","Celeb. Twitter promo, Indonesia vacc.",Anonymous_1,GH&D,biorisk,NA,internal-NBER,Emailed,merits_journal,3,NA,NA,2,1.75,4.25 +rec2YXifnKtaC6NN7,Long term cost-effectiveness of resilient foods for global catastrophes compared to artificial general intelligence safety (Denkenberger et al),LT CEA: Resilient foods vs. AGI safety,Scott Janzwood,long-term-relevant ,quantitative,NA,submitted,Agreed,overall,65,NA,NA,3,50,80 +rec2YXifnKtaC6NN7,Long term cost-effectiveness of resilient foods for global catastrophes compared to artificial general intelligence safety (Denkenberger et al),LT CEA: Resilient foods vs. AGI safety,Scott Janzwood,long-term-relevant ,quantitative,NA,submitted,Agreed,adv_knowledge,70,NA,NA,3,55,85 +rec2YXifnKtaC6NN7,Long term cost-effectiveness of resilient foods for global catastrophes compared to artificial general intelligence safety (Denkenberger et al),LT CEA: Resilient foods vs. AGI safety,Scott Janzwood,long-term-relevant ,quantitative,NA,submitted,Agreed,methods,NA,NA,NA,NA,NA,NA +rec2YXifnKtaC6NN7,Long term cost-effectiveness of resilient foods for global catastrophes compared to artificial general intelligence safety (Denkenberger et al),LT CEA: Resilient foods vs. AGI safety,Scott Janzwood,long-term-relevant ,quantitative,NA,submitted,Agreed,logic_comms,80,NA,NA,4,72,88 +rec2YXifnKtaC6NN7,Long term cost-effectiveness of resilient foods for global catastrophes compared to artificial general intelligence safety (Denkenberger et al),LT CEA: Resilient foods vs. AGI safety,Scott Janzwood,long-term-relevant ,quantitative,NA,submitted,Agreed,real_world,NA,NA,NA,NA,NA,NA +rec2YXifnKtaC6NN7,Long term cost-effectiveness of resilient foods for global catastrophes compared to artificial general intelligence safety (Denkenberger et al),LT CEA: Resilient foods vs. AGI safety,Scott Janzwood,long-term-relevant ,quantitative,NA,submitted,Agreed,gp_relevance,80,NA,NA,5,76,84 +rec2YXifnKtaC6NN7,Long term cost-effectiveness of resilient foods for global catastrophes compared to artificial general intelligence safety (Denkenberger et al),LT CEA: Resilient foods vs. AGI safety,Scott Janzwood,long-term-relevant ,quantitative,NA,submitted,Agreed,journal_predict,3.5,NA,NA,3,2.75,4.25 +rec2YXifnKtaC6NN7,Long term cost-effectiveness of resilient foods for global catastrophes compared to artificial general intelligence safety (Denkenberger et al),LT CEA: Resilient foods vs. AGI safety,Scott Janzwood,long-term-relevant ,quantitative,NA,submitted,Agreed,open_sci,NA,NA,NA,NA,NA,NA +rec2YXifnKtaC6NN7,Long term cost-effectiveness of resilient foods for global catastrophes compared to artificial general intelligence safety (Denkenberger et al),LT CEA: Resilient foods vs. AGI safety,Scott Janzwood,long-term-relevant ,quantitative,NA,submitted,Agreed,merits_journal,3.5,NA,NA,5,3.3,3.7 +recA4IOZhdq2THdVx,"Kremer, M., Levin, J. and Snyder, C.M., 2020, May. Advance Market Commitments: Insights from Theory and Experience. In AEA Papers and Proceedings (Vol. 110, pp. 269-73).",Advance market commit. (vaccines),David Manheim,policy,economics,biorisk,internal-from-syllabus-agenda-policy-database,Agreed,overall,80,70,90,NA,70,90 +recA4IOZhdq2THdVx,"Kremer, M., Levin, J. and Snyder, C.M., 2020, May. Advance Market Commitments: Insights from Theory and Experience. In AEA Papers and Proceedings (Vol. 110, pp. 269-73).",Advance market commit. (vaccines),David Manheim,policy,economics,biorisk,internal-from-syllabus-agenda-policy-database,Agreed,adv_knowledge,25,20,40,NA,20,40 +recA4IOZhdq2THdVx,"Kremer, M., Levin, J. and Snyder, C.M., 2020, May. Advance Market Commitments: Insights from Theory and Experience. In AEA Papers and Proceedings (Vol. 110, pp. 269-73).",Advance market commit. (vaccines),David Manheim,policy,economics,biorisk,internal-from-syllabus-agenda-policy-database,Agreed,methods,95,85,97.5,NA,85,97.5 +recA4IOZhdq2THdVx,"Kremer, M., Levin, J. and Snyder, C.M., 2020, May. Advance Market Commitments: Insights from Theory and Experience. In AEA Papers and Proceedings (Vol. 110, pp. 269-73).",Advance market commit. (vaccines),David Manheim,policy,economics,biorisk,internal-from-syllabus-agenda-policy-database,Agreed,logic_comms,75,60,90,NA,60,90 +recA4IOZhdq2THdVx,"Kremer, M., Levin, J. and Snyder, C.M., 2020, May. Advance Market Commitments: Insights from Theory and Experience. In AEA Papers and Proceedings (Vol. 110, pp. 269-73).",Advance market commit. (vaccines),David Manheim,policy,economics,biorisk,internal-from-syllabus-agenda-policy-database,Agreed,real_world,NA,NA,NA,NA,NA,NA +recA4IOZhdq2THdVx,"Kremer, M., Levin, J. and Snyder, C.M., 2020, May. Advance Market Commitments: Insights from Theory and Experience. In AEA Papers and Proceedings (Vol. 110, pp. 269-73).",Advance market commit. (vaccines),David Manheim,policy,economics,biorisk,internal-from-syllabus-agenda-policy-database,Agreed,gp_relevance,60,40,75,NA,40,75 +recA4IOZhdq2THdVx,"Kremer, M., Levin, J. and Snyder, C.M., 2020, May. Advance Market Commitments: Insights from Theory and Experience. In AEA Papers and Proceedings (Vol. 110, pp. 269-73).",Advance market commit. (vaccines),David Manheim,policy,economics,biorisk,internal-from-syllabus-agenda-policy-database,Agreed,journal_predict,3,2.5,4.5,NA,2.5,4.5 +recA4IOZhdq2THdVx,"Kremer, M., Levin, J. and Snyder, C.M., 2020, May. Advance Market Commitments: Insights from Theory and Experience. In AEA Papers and Proceedings (Vol. 110, pp. 269-73).",Advance market commit. (vaccines),David Manheim,policy,economics,biorisk,internal-from-syllabus-agenda-policy-database,Agreed,open_sci,NA,NA,NA,NA,NA,NA +recA4IOZhdq2THdVx,"Kremer, M., Levin, J. and Snyder, C.M., 2020, May. Advance Market Commitments: Insights from Theory and Experience. In AEA Papers and Proceedings (Vol. 110, pp. 269-73).",Advance market commit. (vaccines),David Manheim,policy,economics,biorisk,internal-from-syllabus-agenda-policy-database,Agreed,merits_journal,NA,NA,NA,NA,NA,NA +recAX4gXhbfSH01vA,"Does the Squeaky Wheel Get More Grease? The Direct and Indirect Effects of Citizen Participation on Environmental Governance in China, The Comparative Impact of Cash Transfers and a Psychotherapy Program on Psychological and Economic Well-being",Citizen participation in envt. gov. China,Robert Kubinec,Environment,NA,NA,suggested - externally,Acknowledged,overall,90,85,95,NA,85,95 +recAX4gXhbfSH01vA,"Does the Squeaky Wheel Get More Grease? The Direct and Indirect Effects of Citizen Participation on Environmental Governance in China, The Comparative Impact of Cash Transfers and a Psychotherapy Program on Psychological and Economic Well-being",Citizen participation in envt. gov. China,Robert Kubinec,Environment,NA,NA,suggested - externally,Acknowledged,adv_knowledge,90,85,95,NA,85,95 +recAX4gXhbfSH01vA,"Does the Squeaky Wheel Get More Grease? The Direct and Indirect Effects of Citizen Participation on Environmental Governance in China, The Comparative Impact of Cash Transfers and a Psychotherapy Program on Psychological and Economic Well-being",Citizen participation in envt. gov. China,Robert Kubinec,Environment,NA,NA,suggested - externally,Acknowledged,methods,80,70,90,NA,70,90 +recAX4gXhbfSH01vA,"Does the Squeaky Wheel Get More Grease? The Direct and Indirect Effects of Citizen Participation on Environmental Governance in China, The Comparative Impact of Cash Transfers and a Psychotherapy Program on Psychological and Economic Well-being",Citizen participation in envt. gov. China,Robert Kubinec,Environment,NA,NA,suggested - externally,Acknowledged,logic_comms,80,75,85,NA,75,85 +recAX4gXhbfSH01vA,"Does the Squeaky Wheel Get More Grease? The Direct and Indirect Effects of Citizen Participation on Environmental Governance in China, The Comparative Impact of Cash Transfers and a Psychotherapy Program on Psychological and Economic Well-being",Citizen participation in envt. gov. China,Robert Kubinec,Environment,NA,NA,suggested - externally,Acknowledged,real_world,90,85,95,NA,85,95 +recAX4gXhbfSH01vA,"Does the Squeaky Wheel Get More Grease? The Direct and Indirect Effects of Citizen Participation on Environmental Governance in China, The Comparative Impact of Cash Transfers and a Psychotherapy Program on Psychological and Economic Well-being",Citizen participation in envt. gov. China,Robert Kubinec,Environment,NA,NA,suggested - externally,Acknowledged,gp_relevance,90,85,95,NA,85,95 +recAX4gXhbfSH01vA,"Does the Squeaky Wheel Get More Grease? The Direct and Indirect Effects of Citizen Participation on Environmental Governance in China, The Comparative Impact of Cash Transfers and a Psychotherapy Program on Psychological and Economic Well-being",Citizen participation in envt. gov. China,Robert Kubinec,Environment,NA,NA,suggested - externally,Acknowledged,journal_predict,4.8,4,5,NA,4,5 +recAX4gXhbfSH01vA,"Does the Squeaky Wheel Get More Grease? The Direct and Indirect Effects of Citizen Participation on Environmental Governance in China, The Comparative Impact of Cash Transfers and a Psychotherapy Program on Psychological and Economic Well-being",Citizen participation in envt. gov. China,Robert Kubinec,Environment,NA,NA,suggested - externally,Acknowledged,open_sci,70,60,80,NA,60,80 +recAX4gXhbfSH01vA,"Does the Squeaky Wheel Get More Grease? The Direct and Indirect Effects of Citizen Participation on Environmental Governance in China, The Comparative Impact of Cash Transfers and a Psychotherapy Program on Psychological and Economic Well-being",Citizen participation in envt. gov. China,Robert Kubinec,Environment,NA,NA,suggested - externally,Acknowledged,merits_journal,4.8,4.5,5,NA,4.5,5 +recBDa2jHqagZHFa5,Banning wildlife trade can boost demand for unregulated threatened species,Banning wildlife trade can boost demand,Anonymous_5,conservation,biodiversity,NA,submitted,Agreed,overall,75,NA,NA,NA,NA,NA +recBDa2jHqagZHFa5,Banning wildlife trade can boost demand for unregulated threatened species,Banning wildlife trade can boost demand,Anonymous_5,conservation,biodiversity,NA,submitted,Agreed,adv_knowledge,70,NA,NA,4,62,78 +recBDa2jHqagZHFa5,Banning wildlife trade can boost demand for unregulated threatened species,Banning wildlife trade can boost demand,Anonymous_5,conservation,biodiversity,NA,submitted,Agreed,methods,80,NA,NA,3,65,95 +recBDa2jHqagZHFa5,Banning wildlife trade can boost demand for unregulated threatened species,Banning wildlife trade can boost demand,Anonymous_5,conservation,biodiversity,NA,submitted,Agreed,logic_comms,70,NA,NA,4,62,78 +recBDa2jHqagZHFa5,Banning wildlife trade can boost demand for unregulated threatened species,Banning wildlife trade can boost demand,Anonymous_5,conservation,biodiversity,NA,submitted,Agreed,real_world,90,NA,NA,5,86,94 +recBDa2jHqagZHFa5,Banning wildlife trade can boost demand for unregulated threatened species,Banning wildlife trade can boost demand,Anonymous_5,conservation,biodiversity,NA,submitted,Agreed,gp_relevance,80,NA,NA,3,65,95 +recBDa2jHqagZHFa5,Banning wildlife trade can boost demand for unregulated threatened species,Banning wildlife trade can boost demand,Anonymous_5,conservation,biodiversity,NA,submitted,Agreed,journal_predict,3,NA,NA,4,2.6,3.4 +recBDa2jHqagZHFa5,Banning wildlife trade can boost demand for unregulated threatened species,Banning wildlife trade can boost demand,Anonymous_5,conservation,biodiversity,NA,submitted,Agreed,open_sci,70,NA,NA,4,62,78 +recBDa2jHqagZHFa5,Banning wildlife trade can boost demand for unregulated threatened species,Banning wildlife trade can boost demand,Anonymous_5,conservation,biodiversity,NA,submitted,Agreed,merits_journal,3,NA,NA,4,2.6,3.4 +recFWeZjd74NTWvdG,"Kremer, M., Levin, J. and Snyder, C.M., 2020, May. Advance Market Commitments: Insights from Theory and Experience. In AEA Papers and Proceedings (Vol. 110, pp. 269-73).",Advance market commit. (vaccines),Joel Tan,policy,economics,biorisk,internal-from-syllabus-agenda-policy-database,Agreed,overall,79,59,94,NA,59,94 +recFWeZjd74NTWvdG,"Kremer, M., Levin, J. and Snyder, C.M., 2020, May. Advance Market Commitments: Insights from Theory and Experience. In AEA Papers and Proceedings (Vol. 110, pp. 269-73).",Advance market commit. (vaccines),Joel Tan,policy,economics,biorisk,internal-from-syllabus-agenda-policy-database,Agreed,adv_knowledge,90,70,100,NA,70,100 +recFWeZjd74NTWvdG,"Kremer, M., Levin, J. and Snyder, C.M., 2020, May. Advance Market Commitments: Insights from Theory and Experience. In AEA Papers and Proceedings (Vol. 110, pp. 269-73).",Advance market commit. (vaccines),Joel Tan,policy,economics,biorisk,internal-from-syllabus-agenda-policy-database,Agreed,methods,70,50,90,NA,50,90 +recFWeZjd74NTWvdG,"Kremer, M., Levin, J. and Snyder, C.M., 2020, May. Advance Market Commitments: Insights from Theory and Experience. In AEA Papers and Proceedings (Vol. 110, pp. 269-73).",Advance market commit. (vaccines),Joel Tan,policy,economics,biorisk,internal-from-syllabus-agenda-policy-database,Agreed,logic_comms,70,50,90,NA,50,90 +recFWeZjd74NTWvdG,"Kremer, M., Levin, J. and Snyder, C.M., 2020, May. Advance Market Commitments: Insights from Theory and Experience. In AEA Papers and Proceedings (Vol. 110, pp. 269-73).",Advance market commit. (vaccines),Joel Tan,policy,economics,biorisk,internal-from-syllabus-agenda-policy-database,Agreed,real_world,90,70,100,NA,70,100 +recFWeZjd74NTWvdG,"Kremer, M., Levin, J. and Snyder, C.M., 2020, May. Advance Market Commitments: Insights from Theory and Experience. In AEA Papers and Proceedings (Vol. 110, pp. 269-73).",Advance market commit. (vaccines),Joel Tan,policy,economics,biorisk,internal-from-syllabus-agenda-policy-database,Agreed,gp_relevance,90,70,100,NA,70,100 +recFWeZjd74NTWvdG,"Kremer, M., Levin, J. and Snyder, C.M., 2020, May. Advance Market Commitments: Insights from Theory and Experience. In AEA Papers and Proceedings (Vol. 110, pp. 269-73).",Advance market commit. (vaccines),Joel Tan,policy,economics,biorisk,internal-from-syllabus-agenda-policy-database,Agreed,journal_predict,5,NA,NA,5,4.8,5 +recFWeZjd74NTWvdG,"Kremer, M., Levin, J. and Snyder, C.M., 2020, May. Advance Market Commitments: Insights from Theory and Experience. In AEA Papers and Proceedings (Vol. 110, pp. 269-73).",Advance market commit. (vaccines),Joel Tan,policy,economics,biorisk,internal-from-syllabus-agenda-policy-database,Agreed,open_sci,50,30,70,NA,30,70 +recFWeZjd74NTWvdG,"Kremer, M., Levin, J. and Snyder, C.M., 2020, May. Advance Market Commitments: Insights from Theory and Experience. In AEA Papers and Proceedings (Vol. 110, pp. 269-73).",Advance market commit. (vaccines),Joel Tan,policy,economics,biorisk,internal-from-syllabus-agenda-policy-database,Agreed,merits_journal,5,NA,NA,5,4.8,5 +recLDQlgTfBrqhVA2,"Does the Squeaky Wheel Get More Grease? The Direct and Indirect Effects of Citizen Participation on Environmental Governance in China, The Comparative Impact of Cash Transfers and a Psychotherapy Program on Psychological and Economic Well-being",Citizen participation in envt. gov. China,Anonymous_7,Environment,NA,NA,suggested - externally,Acknowledged,overall,80,71,86,NA,71,86 +recLDQlgTfBrqhVA2,"Does the Squeaky Wheel Get More Grease? The Direct and Indirect Effects of Citizen Participation on Environmental Governance in China, The Comparative Impact of Cash Transfers and a Psychotherapy Program on Psychological and Economic Well-being",Citizen participation in envt. gov. China,Anonymous_7,Environment,NA,NA,suggested - externally,Acknowledged,adv_knowledge,76,68,83,NA,68,83 +recLDQlgTfBrqhVA2,"Does the Squeaky Wheel Get More Grease? The Direct and Indirect Effects of Citizen Participation on Environmental Governance in China, The Comparative Impact of Cash Transfers and a Psychotherapy Program on Psychological and Economic Well-being",Citizen participation in envt. gov. China,Anonymous_7,Environment,NA,NA,suggested - externally,Acknowledged,methods,72,63,81,NA,63,81 +recLDQlgTfBrqhVA2,"Does the Squeaky Wheel Get More Grease? The Direct and Indirect Effects of Citizen Participation on Environmental Governance in China, The Comparative Impact of Cash Transfers and a Psychotherapy Program on Psychological and Economic Well-being",Citizen participation in envt. gov. China,Anonymous_7,Environment,NA,NA,suggested - externally,Acknowledged,logic_comms,67,62,78,NA,62,78 +recLDQlgTfBrqhVA2,"Does the Squeaky Wheel Get More Grease? The Direct and Indirect Effects of Citizen Participation on Environmental Governance in China, The Comparative Impact of Cash Transfers and a Psychotherapy Program on Psychological and Economic Well-being",Citizen participation in envt. gov. China,Anonymous_7,Environment,NA,NA,suggested - externally,Acknowledged,real_world,91,78,94,NA,78,94 +recLDQlgTfBrqhVA2,"Does the Squeaky Wheel Get More Grease? The Direct and Indirect Effects of Citizen Participation on Environmental Governance in China, The Comparative Impact of Cash Transfers and a Psychotherapy Program on Psychological and Economic Well-being",Citizen participation in envt. gov. China,Anonymous_7,Environment,NA,NA,suggested - externally,Acknowledged,gp_relevance,92,82,95,NA,82,95 +recLDQlgTfBrqhVA2,"Does the Squeaky Wheel Get More Grease? The Direct and Indirect Effects of Citizen Participation on Environmental Governance in China, The Comparative Impact of Cash Transfers and a Psychotherapy Program on Psychological and Economic Well-being",Citizen participation in envt. gov. China,Anonymous_7,Environment,NA,NA,suggested - externally,Acknowledged,journal_predict,4.6,3.8,5,NA,3.8,5 +recLDQlgTfBrqhVA2,"Does the Squeaky Wheel Get More Grease? The Direct and Indirect Effects of Citizen Participation on Environmental Governance in China, The Comparative Impact of Cash Transfers and a Psychotherapy Program on Psychological and Economic Well-being",Citizen participation in envt. gov. China,Anonymous_7,Environment,NA,NA,suggested - externally,Acknowledged,open_sci,48,42,68,NA,42,68 +recLDQlgTfBrqhVA2,"Does the Squeaky Wheel Get More Grease? The Direct and Indirect Effects of Citizen Participation on Environmental Governance in China, The Comparative Impact of Cash Transfers and a Psychotherapy Program on Psychological and Economic Well-being",Citizen participation in envt. gov. China,Anonymous_7,Environment,NA,NA,suggested - externally,Acknowledged,merits_journal,4.3,3.7,4.8,NA,3.7,4.8 +recMAQaYQFRL0DQJB,"""The Environmental Effects of Economic Production: Evidence from Ecological Observations + (previous title: Economic Production and Biodiversity in the United States)""",Env. fx of prod.: ecological obs,Elias Cisneros,Environment,NA,NA,internal-NBER,Agreed,overall,88,NA,NA,5,84,92 +recMAQaYQFRL0DQJB,"""The Environmental Effects of Economic Production: Evidence from Ecological Observations + (previous title: Economic Production and Biodiversity in the United States)""",Env. fx of prod.: ecological obs,Elias Cisneros,Environment,NA,NA,internal-NBER,Agreed,adv_knowledge,90,NA,NA,4,82,98 +recMAQaYQFRL0DQJB,"""The Environmental Effects of Economic Production: Evidence from Ecological Observations + (previous title: Economic Production and Biodiversity in the United States)""",Env. fx of prod.: ecological obs,Elias Cisneros,Environment,NA,NA,internal-NBER,Agreed,methods,75,NA,NA,3,60,90 +recMAQaYQFRL0DQJB,"""The Environmental Effects of Economic Production: Evidence from Ecological Observations + (previous title: Economic Production and Biodiversity in the United States)""",Env. fx of prod.: ecological obs,Elias Cisneros,Environment,NA,NA,internal-NBER,Agreed,logic_comms,80,NA,NA,4,72,88 +recMAQaYQFRL0DQJB,"""The Environmental Effects of Economic Production: Evidence from Ecological Observations + (previous title: Economic Production and Biodiversity in the United States)""",Env. fx of prod.: ecological obs,Elias Cisneros,Environment,NA,NA,internal-NBER,Agreed,real_world,95,NA,NA,5,91,99 +recMAQaYQFRL0DQJB,"""The Environmental Effects of Economic Production: Evidence from Ecological Observations + (previous title: Economic Production and Biodiversity in the United States)""",Env. fx of prod.: ecological obs,Elias Cisneros,Environment,NA,NA,internal-NBER,Agreed,gp_relevance,95,NA,NA,4,87,100 +recMAQaYQFRL0DQJB,"""The Environmental Effects of Economic Production: Evidence from Ecological Observations + (previous title: Economic Production and Biodiversity in the United States)""",Env. fx of prod.: ecological obs,Elias Cisneros,Environment,NA,NA,internal-NBER,Agreed,journal_predict,4,NA,NA,5,3.8,4.2 +recMAQaYQFRL0DQJB,"""The Environmental Effects of Economic Production: Evidence from Ecological Observations + (previous title: Economic Production and Biodiversity in the United States)""",Env. fx of prod.: ecological obs,Elias Cisneros,Environment,NA,NA,internal-NBER,Agreed,open_sci,90,NA,NA,5,86,94 +recMAQaYQFRL0DQJB,"""The Environmental Effects of Economic Production: Evidence from Ecological Observations + (previous title: Economic Production and Biodiversity in the United States)""",Env. fx of prod.: ecological obs,Elias Cisneros,Environment,NA,NA,internal-NBER,Agreed,merits_journal,4,NA,NA,5,3.8,4.2 +recNMQY75RCZdIcyG,"Kremer, M., Levin, J. and Snyder, C.M., 2020, May. Advance Market Commitments: Insights from Theory and Experience. In AEA Papers and Proceedings (Vol. 110, pp. 269-73).",Advance market commit. (vaccines),Dan Tortorice,policy,economics,biorisk,internal-from-syllabus-agenda-policy-database,Agreed,overall,80,NA,NA,4,72,88 +recNMQY75RCZdIcyG,"Kremer, M., Levin, J. and Snyder, C.M., 2020, May. Advance Market Commitments: Insights from Theory and Experience. In AEA Papers and Proceedings (Vol. 110, pp. 269-73).",Advance market commit. (vaccines),Dan Tortorice,policy,economics,biorisk,internal-from-syllabus-agenda-policy-database,Agreed,adv_knowledge,90,NA,NA,5,86,94 +recNMQY75RCZdIcyG,"Kremer, M., Levin, J. and Snyder, C.M., 2020, May. Advance Market Commitments: Insights from Theory and Experience. In AEA Papers and Proceedings (Vol. 110, pp. 269-73).",Advance market commit. (vaccines),Dan Tortorice,policy,economics,biorisk,internal-from-syllabus-agenda-policy-database,Agreed,methods,80,NA,NA,4,72,88 +recNMQY75RCZdIcyG,"Kremer, M., Levin, J. and Snyder, C.M., 2020, May. Advance Market Commitments: Insights from Theory and Experience. In AEA Papers and Proceedings (Vol. 110, pp. 269-73).",Advance market commit. (vaccines),Dan Tortorice,policy,economics,biorisk,internal-from-syllabus-agenda-policy-database,Agreed,logic_comms,80,NA,NA,4,72,88 +recNMQY75RCZdIcyG,"Kremer, M., Levin, J. and Snyder, C.M., 2020, May. Advance Market Commitments: Insights from Theory and Experience. In AEA Papers and Proceedings (Vol. 110, pp. 269-73).",Advance market commit. (vaccines),Dan Tortorice,policy,economics,biorisk,internal-from-syllabus-agenda-policy-database,Agreed,real_world,NA,NA,NA,NA,NA,NA +recNMQY75RCZdIcyG,"Kremer, M., Levin, J. and Snyder, C.M., 2020, May. Advance Market Commitments: Insights from Theory and Experience. In AEA Papers and Proceedings (Vol. 110, pp. 269-73).",Advance market commit. (vaccines),Dan Tortorice,policy,economics,biorisk,internal-from-syllabus-agenda-policy-database,Agreed,gp_relevance,95,NA,NA,5,91,99 +recNMQY75RCZdIcyG,"Kremer, M., Levin, J. and Snyder, C.M., 2020, May. Advance Market Commitments: Insights from Theory and Experience. In AEA Papers and Proceedings (Vol. 110, pp. 269-73).",Advance market commit. (vaccines),Dan Tortorice,policy,economics,biorisk,internal-from-syllabus-agenda-policy-database,Agreed,journal_predict,4,NA,NA,5,3.8,4.2 +recNMQY75RCZdIcyG,"Kremer, M., Levin, J. and Snyder, C.M., 2020, May. Advance Market Commitments: Insights from Theory and Experience. In AEA Papers and Proceedings (Vol. 110, pp. 269-73).",Advance market commit. (vaccines),Dan Tortorice,policy,economics,biorisk,internal-from-syllabus-agenda-policy-database,Agreed,open_sci,90,NA,NA,3,75,100 +recNMQY75RCZdIcyG,"Kremer, M., Levin, J. and Snyder, C.M., 2020, May. Advance Market Commitments: Insights from Theory and Experience. In AEA Papers and Proceedings (Vol. 110, pp. 269-73).",Advance market commit. (vaccines),Dan Tortorice,policy,economics,biorisk,internal-from-syllabus-agenda-policy-database,Agreed,merits_journal,4,NA,NA,5,3.8,4.2 +recOV0UgplEXwJf81,"Aghion, P., Jones, B.F., and Jones, C.I., 2017. Artificial Intelligence and Economic Growth",AI and econ. growth,Seth Benzell,macroeconomics,Artificial intelligence,prominent,internal-from-syllabus-agenda-policy-database,Agreed,overall,80,70,90,NA,70,90 +recOV0UgplEXwJf81,"Aghion, P., Jones, B.F., and Jones, C.I., 2017. Artificial Intelligence and Economic Growth",AI and econ. growth,Seth Benzell,macroeconomics,Artificial intelligence,prominent,internal-from-syllabus-agenda-policy-database,Agreed,adv_knowledge,75,65,85,NA,65,85 +recOV0UgplEXwJf81,"Aghion, P., Jones, B.F., and Jones, C.I., 2017. Artificial Intelligence and Economic Growth",AI and econ. growth,Seth Benzell,macroeconomics,Artificial intelligence,prominent,internal-from-syllabus-agenda-policy-database,Agreed,methods,80,75,85,NA,75,85 +recOV0UgplEXwJf81,"Aghion, P., Jones, B.F., and Jones, C.I., 2017. Artificial Intelligence and Economic Growth",AI and econ. growth,Seth Benzell,macroeconomics,Artificial intelligence,prominent,internal-from-syllabus-agenda-policy-database,Agreed,logic_comms,70,60,80,NA,60,80 +recOV0UgplEXwJf81,"Aghion, P., Jones, B.F., and Jones, C.I., 2017. Artificial Intelligence and Economic Growth",AI and econ. growth,Seth Benzell,macroeconomics,Artificial intelligence,prominent,internal-from-syllabus-agenda-policy-database,Agreed,real_world,NA,NA,NA,NA,NA,NA +recOV0UgplEXwJf81,"Aghion, P., Jones, B.F., and Jones, C.I., 2017. Artificial Intelligence and Economic Growth",AI and econ. growth,Seth Benzell,macroeconomics,Artificial intelligence,prominent,internal-from-syllabus-agenda-policy-database,Agreed,gp_relevance,90,85,100,NA,85,100 +recOV0UgplEXwJf81,"Aghion, P., Jones, B.F., and Jones, C.I., 2017. Artificial Intelligence and Economic Growth",AI and econ. growth,Seth Benzell,macroeconomics,Artificial intelligence,prominent,internal-from-syllabus-agenda-policy-database,Agreed,journal_predict,NA,NA,NA,NA,NA,NA +recOV0UgplEXwJf81,"Aghion, P., Jones, B.F., and Jones, C.I., 2017. Artificial Intelligence and Economic Growth",AI and econ. growth,Seth Benzell,macroeconomics,Artificial intelligence,prominent,internal-from-syllabus-agenda-policy-database,Agreed,open_sci,95,90,100,NA,90,100 +recOV0UgplEXwJf81,"Aghion, P., Jones, B.F., and Jones, C.I., 2017. Artificial Intelligence and Economic Growth",AI and econ. growth,Seth Benzell,macroeconomics,Artificial intelligence,prominent,internal-from-syllabus-agenda-policy-database,Agreed,merits_journal,4,3.5,5,NA,3.5,5 +recXaqPPgA911nuoY,Banning wildlife trade can boost demand for unregulated threatened species,Banning wildlife trade can boost demand,Liew Jia Huan,conservation,biodiversity,NA,submitted,Agreed,overall,75,NA,NA,4,67,83 +recXaqPPgA911nuoY,Banning wildlife trade can boost demand for unregulated threatened species,Banning wildlife trade can boost demand,Liew Jia Huan,conservation,biodiversity,NA,submitted,Agreed,adv_knowledge,80,NA,NA,4,72,88 +recXaqPPgA911nuoY,Banning wildlife trade can boost demand for unregulated threatened species,Banning wildlife trade can boost demand,Liew Jia Huan,conservation,biodiversity,NA,submitted,Agreed,methods,50,NA,NA,2,25,75 +recXaqPPgA911nuoY,Banning wildlife trade can boost demand for unregulated threatened species,Banning wildlife trade can boost demand,Liew Jia Huan,conservation,biodiversity,NA,submitted,Agreed,logic_comms,70,NA,NA,5,66,74 +recXaqPPgA911nuoY,Banning wildlife trade can boost demand for unregulated threatened species,Banning wildlife trade can boost demand,Liew Jia Huan,conservation,biodiversity,NA,submitted,Agreed,real_world,90,NA,NA,4,82,98 +recXaqPPgA911nuoY,Banning wildlife trade can boost demand for unregulated threatened species,Banning wildlife trade can boost demand,Liew Jia Huan,conservation,biodiversity,NA,submitted,Agreed,gp_relevance,65,NA,NA,3,50,80 +recXaqPPgA911nuoY,Banning wildlife trade can boost demand for unregulated threatened species,Banning wildlife trade can boost demand,Liew Jia Huan,conservation,biodiversity,NA,submitted,Agreed,journal_predict,2.5,NA,NA,5,2.3,2.7 +recXaqPPgA911nuoY,Banning wildlife trade can boost demand for unregulated threatened species,Banning wildlife trade can boost demand,Liew Jia Huan,conservation,biodiversity,NA,submitted,Agreed,open_sci,50,NA,NA,5,46,54 +recXaqPPgA911nuoY,Banning wildlife trade can boost demand for unregulated threatened species,Banning wildlife trade can boost demand,Liew Jia Huan,conservation,biodiversity,NA,submitted,Agreed,merits_journal,3,NA,NA,5,2.8,3.2 +recYb2JcJlGrHoI2H,The Governance Of Non-Profits And Their Social Impact: Evidence From A Randomized Program In Healthcare In DRC,Nonprofit Govc.: Randomized healthcare DRC,Wayne Aaron Sandholtz,GH&D,NA,NA,internal-NBER,Emailed,overall,65,55,74,NA,55,74 +recYb2JcJlGrHoI2H,The Governance Of Non-Profits And Their Social Impact: Evidence From A Randomized Program In Healthcare In DRC,Nonprofit Govc.: Randomized healthcare DRC,Wayne Aaron Sandholtz,GH&D,NA,NA,internal-NBER,Emailed,adv_knowledge,70,55,75,NA,55,75 +recYb2JcJlGrHoI2H,The Governance Of Non-Profits And Their Social Impact: Evidence From A Randomized Program In Healthcare In DRC,Nonprofit Govc.: Randomized healthcare DRC,Wayne Aaron Sandholtz,GH&D,NA,NA,internal-NBER,Emailed,methods,60,55,70,NA,55,70 +recYb2JcJlGrHoI2H,The Governance Of Non-Profits And Their Social Impact: Evidence From A Randomized Program In Healthcare In DRC,Nonprofit Govc.: Randomized healthcare DRC,Wayne Aaron Sandholtz,GH&D,NA,NA,internal-NBER,Emailed,logic_comms,55,50,65,NA,50,65 +recYb2JcJlGrHoI2H,The Governance Of Non-Profits And Their Social Impact: Evidence From A Randomized Program In Healthcare In DRC,Nonprofit Govc.: Randomized healthcare DRC,Wayne Aaron Sandholtz,GH&D,NA,NA,internal-NBER,Emailed,real_world,55,45,75,NA,45,75 +recYb2JcJlGrHoI2H,The Governance Of Non-Profits And Their Social Impact: Evidence From A Randomized Program In Healthcare In DRC,Nonprofit Govc.: Randomized healthcare DRC,Wayne Aaron Sandholtz,GH&D,NA,NA,internal-NBER,Emailed,gp_relevance,80,70,90,NA,70,90 +recYb2JcJlGrHoI2H,The Governance Of Non-Profits And Their Social Impact: Evidence From A Randomized Program In Healthcare In DRC,Nonprofit Govc.: Randomized healthcare DRC,Wayne Aaron Sandholtz,GH&D,NA,NA,internal-NBER,Emailed,journal_predict,3.6,2.8,4,NA,2.8,4 +recYb2JcJlGrHoI2H,The Governance Of Non-Profits And Their Social Impact: Evidence From A Randomized Program In Healthcare In DRC,Nonprofit Govc.: Randomized healthcare DRC,Wayne Aaron Sandholtz,GH&D,NA,NA,internal-NBER,Emailed,open_sci,45,30,60,NA,30,60 +recYb2JcJlGrHoI2H,The Governance Of Non-Profits And Their Social Impact: Evidence From A Randomized Program In Healthcare In DRC,Nonprofit Govc.: Randomized healthcare DRC,Wayne Aaron Sandholtz,GH&D,NA,NA,internal-NBER,Emailed,merits_journal,3.8,3,4.1,NA,3,4.1 +recbXm55IKEWH4DAM,"Aghion, P., Jones, B.F., and Jones, C.I., 2017. Artificial Intelligence and Economic Growth ",AI and econ. growth,Phil Trammel,macroeconomics,Artificial intelligence,prominent,internal-from-syllabus-agenda-policy-database,Agreed,overall,92,80,100,NA,80,100 +recbXm55IKEWH4DAM,"Aghion, P., Jones, B.F., and Jones, C.I., 2017. Artificial Intelligence and Economic Growth ",AI and econ. growth,Phil Trammel,macroeconomics,Artificial intelligence,prominent,internal-from-syllabus-agenda-policy-database,Agreed,adv_knowledge,97,80,100,NA,80,100 +recbXm55IKEWH4DAM,"Aghion, P., Jones, B.F., and Jones, C.I., 2017. Artificial Intelligence and Economic Growth ",AI and econ. growth,Phil Trammel,macroeconomics,Artificial intelligence,prominent,internal-from-syllabus-agenda-policy-database,Agreed,methods,70,40,90,NA,40,90 +recbXm55IKEWH4DAM,"Aghion, P., Jones, B.F., and Jones, C.I., 2017. Artificial Intelligence and Economic Growth ",AI and econ. growth,Phil Trammel,macroeconomics,Artificial intelligence,prominent,internal-from-syllabus-agenda-policy-database,Agreed,logic_comms,45,30,70,NA,30,70 +recbXm55IKEWH4DAM,"Aghion, P., Jones, B.F., and Jones, C.I., 2017. Artificial Intelligence and Economic Growth ",AI and econ. growth,Phil Trammel,macroeconomics,Artificial intelligence,prominent,internal-from-syllabus-agenda-policy-database,Agreed,real_world,NA,NA,NA,NA,NA,NA +recbXm55IKEWH4DAM,"Aghion, P., Jones, B.F., and Jones, C.I., 2017. Artificial Intelligence and Economic Growth ",AI and econ. growth,Phil Trammel,macroeconomics,Artificial intelligence,prominent,internal-from-syllabus-agenda-policy-database,Agreed,gp_relevance,92,80,100,NA,80,100 +recbXm55IKEWH4DAM,"Aghion, P., Jones, B.F., and Jones, C.I., 2017. Artificial Intelligence and Economic Growth ",AI and econ. growth,Phil Trammel,macroeconomics,Artificial intelligence,prominent,internal-from-syllabus-agenda-policy-database,Agreed,journal_predict,3.5,NA,NA,NA,NA,NA +recbXm55IKEWH4DAM,"Aghion, P., Jones, B.F., and Jones, C.I., 2017. Artificial Intelligence and Economic Growth ",AI and econ. growth,Phil Trammel,macroeconomics,Artificial intelligence,prominent,internal-from-syllabus-agenda-policy-database,Agreed,open_sci,80,NA,NA,1,42.5,100 +recbXm55IKEWH4DAM,"Aghion, P., Jones, B.F., and Jones, C.I., 2017. Artificial Intelligence and Economic Growth ",AI and econ. growth,Phil Trammel,macroeconomics,Artificial intelligence,prominent,internal-from-syllabus-agenda-policy-database,Agreed,merits_journal,5,NA,NA,4,4.6,5 +recf9O8DFGO98TPWk,"Mental Health Therapy as a Core Strategy for Increasing Human Capital: Evidence from Ghana (renamed ""Cognitive Behavioral Therapy among Ghana's Rural Poor Is Effective Regardless of Baseline Mental Distress"")","CBT Human K, Ghana",Anonymous_14,GH&D,NA,NA,internal-NBER,Emailed,overall,75,70,84,NA,70,84 +recf9O8DFGO98TPWk,"Mental Health Therapy as a Core Strategy for Increasing Human Capital: Evidence from Ghana (renamed ""Cognitive Behavioral Therapy among Ghana's Rural Poor Is Effective Regardless of Baseline Mental Distress"")","CBT Human K, Ghana",Anonymous_14,GH&D,NA,NA,internal-NBER,Emailed,adv_knowledge,60,55,65,NA,55,65 +recf9O8DFGO98TPWk,"Mental Health Therapy as a Core Strategy for Increasing Human Capital: Evidence from Ghana (renamed ""Cognitive Behavioral Therapy among Ghana's Rural Poor Is Effective Regardless of Baseline Mental Distress"")","CBT Human K, Ghana",Anonymous_14,GH&D,NA,NA,internal-NBER,Emailed,methods,90,82,94,NA,82,94 +recf9O8DFGO98TPWk,"Mental Health Therapy as a Core Strategy for Increasing Human Capital: Evidence from Ghana (renamed ""Cognitive Behavioral Therapy among Ghana's Rural Poor Is Effective Regardless of Baseline Mental Distress"")","CBT Human K, Ghana",Anonymous_14,GH&D,NA,NA,internal-NBER,Emailed,logic_comms,70,62,82,NA,62,82 +recf9O8DFGO98TPWk,"Mental Health Therapy as a Core Strategy for Increasing Human Capital: Evidence from Ghana (renamed ""Cognitive Behavioral Therapy among Ghana's Rural Poor Is Effective Regardless of Baseline Mental Distress"")","CBT Human K, Ghana",Anonymous_14,GH&D,NA,NA,internal-NBER,Emailed,real_world,50,48,52,NA,48,52 +recf9O8DFGO98TPWk,"Mental Health Therapy as a Core Strategy for Increasing Human Capital: Evidence from Ghana (renamed ""Cognitive Behavioral Therapy among Ghana's Rural Poor Is Effective Regardless of Baseline Mental Distress"")","CBT Human K, Ghana",Anonymous_14,GH&D,NA,NA,internal-NBER,Emailed,gp_relevance,50,40,60,NA,40,60 +recf9O8DFGO98TPWk,"Mental Health Therapy as a Core Strategy for Increasing Human Capital: Evidence from Ghana (renamed ""Cognitive Behavioral Therapy among Ghana's Rural Poor Is Effective Regardless of Baseline Mental Distress"")","CBT Human K, Ghana",Anonymous_14,GH&D,NA,NA,internal-NBER,Emailed,journal_predict,4,NA,NA,4,3.6,4.4 +recf9O8DFGO98TPWk,"Mental Health Therapy as a Core Strategy for Increasing Human Capital: Evidence from Ghana (renamed ""Cognitive Behavioral Therapy among Ghana's Rural Poor Is Effective Regardless of Baseline Mental Distress"")","CBT Human K, Ghana",Anonymous_14,GH&D,NA,NA,internal-NBER,Emailed,open_sci,90,80,95,NA,80,95 +recf9O8DFGO98TPWk,"Mental Health Therapy as a Core Strategy for Increasing Human Capital: Evidence from Ghana (renamed ""Cognitive Behavioral Therapy among Ghana's Rural Poor Is Effective Regardless of Baseline Mental Distress"")","CBT Human K, Ghana",Anonymous_14,GH&D,NA,NA,internal-NBER,Emailed,merits_journal,4,NA,NA,4,3.6,4.4 +rechZ3KqBeCecnHOU,"When Celebrities Speak: A Nationwide Twitter Experiment Promoting Vaccination In Indonesia (Alatas et al, 2019/2021)","Celeb. Twitter promo, Indonesia vacc.",Anirugh Tagat,GH&D,biorisk,NA,internal-NBER,Emailed,overall,85,NA,NA,4,77,93 +rechZ3KqBeCecnHOU,"When Celebrities Speak: A Nationwide Twitter Experiment Promoting Vaccination In Indonesia (Alatas et al, 2019/2021)","Celeb. Twitter promo, Indonesia vacc.",Anirugh Tagat,GH&D,biorisk,NA,internal-NBER,Emailed,adv_knowledge,90,NA,NA,4,82,98 +rechZ3KqBeCecnHOU,"When Celebrities Speak: A Nationwide Twitter Experiment Promoting Vaccination In Indonesia (Alatas et al, 2019/2021)","Celeb. Twitter promo, Indonesia vacc.",Anirugh Tagat,GH&D,biorisk,NA,internal-NBER,Emailed,methods,80,NA,NA,3,65,95 +rechZ3KqBeCecnHOU,"When Celebrities Speak: A Nationwide Twitter Experiment Promoting Vaccination In Indonesia (Alatas et al, 2019/2021)","Celeb. Twitter promo, Indonesia vacc.",Anirugh Tagat,GH&D,biorisk,NA,internal-NBER,Emailed,logic_comms,85,NA,NA,4,77,93 +rechZ3KqBeCecnHOU,"When Celebrities Speak: A Nationwide Twitter Experiment Promoting Vaccination In Indonesia (Alatas et al, 2019/2021)","Celeb. Twitter promo, Indonesia vacc.",Anirugh Tagat,GH&D,biorisk,NA,internal-NBER,Emailed,real_world,100,NA,NA,4,92,100 +rechZ3KqBeCecnHOU,"When Celebrities Speak: A Nationwide Twitter Experiment Promoting Vaccination In Indonesia (Alatas et al, 2019/2021)","Celeb. Twitter promo, Indonesia vacc.",Anirugh Tagat,GH&D,biorisk,NA,internal-NBER,Emailed,gp_relevance,100,NA,NA,5,96,100 +rechZ3KqBeCecnHOU,"When Celebrities Speak: A Nationwide Twitter Experiment Promoting Vaccination In Indonesia (Alatas et al, 2019/2021)","Celeb. Twitter promo, Indonesia vacc.",Anirugh Tagat,GH&D,biorisk,NA,internal-NBER,Emailed,journal_predict,4,NA,NA,5,3.8,4.2 +rechZ3KqBeCecnHOU,"When Celebrities Speak: A Nationwide Twitter Experiment Promoting Vaccination In Indonesia (Alatas et al, 2019/2021)","Celeb. Twitter promo, Indonesia vacc.",Anirugh Tagat,GH&D,biorisk,NA,internal-NBER,Emailed,open_sci,80,NA,NA,3,65,95 +rechZ3KqBeCecnHOU,"When Celebrities Speak: A Nationwide Twitter Experiment Promoting Vaccination In Indonesia (Alatas et al, 2019/2021)","Celeb. Twitter promo, Indonesia vacc.",Anirugh Tagat,GH&D,biorisk,NA,internal-NBER,Emailed,merits_journal,5,NA,NA,5,4.8,5 +recifgZ3CxEQaz3m4,"""The Environmental Effects of Economic Production: Evidence from Ecological Observations + (previous title: Economic Production and Biodiversity in the United States)""`",Env. fx of prod.: ecological obs,Anonymous_16,Environment,NA,NA,internal-NBER,Agreed,overall,70,NA,NA,3,55,85 +recifgZ3CxEQaz3m4,"""The Environmental Effects of Economic Production: Evidence from Ecological Observations + (previous title: Economic Production and Biodiversity in the United States)""`",Env. fx of prod.: ecological obs,Anonymous_16,Environment,NA,NA,internal-NBER,Agreed,adv_knowledge,70,NA,NA,3,55,85 +recifgZ3CxEQaz3m4,"""The Environmental Effects of Economic Production: Evidence from Ecological Observations + (previous title: Economic Production and Biodiversity in the United States)""`",Env. fx of prod.: ecological obs,Anonymous_16,Environment,NA,NA,internal-NBER,Agreed,methods,70,NA,NA,3,55,85 +recifgZ3CxEQaz3m4,"""The Environmental Effects of Economic Production: Evidence from Ecological Observations + (previous title: Economic Production and Biodiversity in the United States)""`",Env. fx of prod.: ecological obs,Anonymous_16,Environment,NA,NA,internal-NBER,Agreed,logic_comms,75,NA,NA,4,67,83 +recifgZ3CxEQaz3m4,"""The Environmental Effects of Economic Production: Evidence from Ecological Observations + (previous title: Economic Production and Biodiversity in the United States)""`",Env. fx of prod.: ecological obs,Anonymous_16,Environment,NA,NA,internal-NBER,Agreed,real_world,60,NA,NA,3,45,75 +recifgZ3CxEQaz3m4,"""The Environmental Effects of Economic Production: Evidence from Ecological Observations + (previous title: Economic Production and Biodiversity in the United States)""`",Env. fx of prod.: ecological obs,Anonymous_16,Environment,NA,NA,internal-NBER,Agreed,gp_relevance,80,NA,NA,4,72,88 +recifgZ3CxEQaz3m4,"""The Environmental Effects of Economic Production: Evidence from Ecological Observations + (previous title: Economic Production and Biodiversity in the United States)""`",Env. fx of prod.: ecological obs,Anonymous_16,Environment,NA,NA,internal-NBER,Agreed,journal_predict,4,NA,NA,3,3.25,4.75 +recifgZ3CxEQaz3m4,"""The Environmental Effects of Economic Production: Evidence from Ecological Observations + (previous title: Economic Production and Biodiversity in the United States)""`",Env. fx of prod.: ecological obs,Anonymous_16,Environment,NA,NA,internal-NBER,Agreed,open_sci,65,NA,NA,2,40,90 +recifgZ3CxEQaz3m4,"""The Environmental Effects of Economic Production: Evidence from Ecological Observations + (previous title: Economic Production and Biodiversity in the United States)""`",Env. fx of prod.: ecological obs,Anonymous_16,Environment,NA,NA,internal-NBER,Agreed,merits_journal,4,NA,NA,5,3.8,4.2 +reck0WYhoffiyWRGH,The Comparative Impact of Cash Transfers and a Psychotherapy Program on Psychological and Economic Well-being,Well-being: Cash vs. psychotherapy,Anonymous_17,GH&D,NA,NA,internal-NBER,Acknowledged,overall,90,NA,NA,3,75,100 +reck0WYhoffiyWRGH,The Comparative Impact of Cash Transfers and a Psychotherapy Program on Psychological and Economic Well-being,Well-being: Cash vs. psychotherapy,Anonymous_17,GH&D,NA,NA,internal-NBER,Acknowledged,adv_knowledge,90,NA,NA,2,65,100 +reck0WYhoffiyWRGH,The Comparative Impact of Cash Transfers and a Psychotherapy Program on Psychological and Economic Well-being,Well-being: Cash vs. psychotherapy,Anonymous_17,GH&D,NA,NA,internal-NBER,Acknowledged,methods,90,NA,NA,3,75,100 +reck0WYhoffiyWRGH,The Comparative Impact of Cash Transfers and a Psychotherapy Program on Psychological and Economic Well-being,Well-being: Cash vs. psychotherapy,Anonymous_17,GH&D,NA,NA,internal-NBER,Acknowledged,logic_comms,80,NA,NA,4,72,88 +reck0WYhoffiyWRGH,The Comparative Impact of Cash Transfers and a Psychotherapy Program on Psychological and Economic Well-being,Well-being: Cash vs. psychotherapy,Anonymous_17,GH&D,NA,NA,internal-NBER,Acknowledged,real_world,100,NA,NA,5,96,100 +reck0WYhoffiyWRGH,The Comparative Impact of Cash Transfers and a Psychotherapy Program on Psychological and Economic Well-being,Well-being: Cash vs. psychotherapy,Anonymous_17,GH&D,NA,NA,internal-NBER,Acknowledged,gp_relevance,100,NA,NA,5,96,100 +reck0WYhoffiyWRGH,The Comparative Impact of Cash Transfers and a Psychotherapy Program on Psychological and Economic Well-being,Well-being: Cash vs. psychotherapy,Anonymous_17,GH&D,NA,NA,internal-NBER,Acknowledged,journal_predict,4,NA,NA,3,3.25,4.75 +reck0WYhoffiyWRGH,The Comparative Impact of Cash Transfers and a Psychotherapy Program on Psychological and Economic Well-being,Well-being: Cash vs. psychotherapy,Anonymous_17,GH&D,NA,NA,internal-NBER,Acknowledged,open_sci,70,NA,NA,5,66,74 +reck0WYhoffiyWRGH,The Comparative Impact of Cash Transfers and a Psychotherapy Program on Psychological and Economic Well-being,Well-being: Cash vs. psychotherapy,Anonymous_17,GH&D,NA,NA,internal-NBER,Acknowledged,merits_journal,5,NA,NA,5,4.8,5 +recmEzlRNxLNEWD8e,The Comparative Impact of Cash Transfers and a Psychotherapy Program on Psychological and Economic Well-being,Well-being: Cash vs. psychotherapy,Hannah Metzler,GH&D,NA,NA,internal-NBER,Acknowledged,overall,75,65,85,NA,65,85 +recmEzlRNxLNEWD8e,The Comparative Impact of Cash Transfers and a Psychotherapy Program on Psychological and Economic Well-being,Well-being: Cash vs. psychotherapy,Hannah Metzler,GH&D,NA,NA,internal-NBER,Acknowledged,adv_knowledge,70,60,90,NA,60,90 +recmEzlRNxLNEWD8e,The Comparative Impact of Cash Transfers and a Psychotherapy Program on Psychological and Economic Well-being,Well-being: Cash vs. psychotherapy,Hannah Metzler,GH&D,NA,NA,internal-NBER,Acknowledged,methods,90,85,95,NA,85,95 +recmEzlRNxLNEWD8e,The Comparative Impact of Cash Transfers and a Psychotherapy Program on Psychological and Economic Well-being,Well-being: Cash vs. psychotherapy,Hannah Metzler,GH&D,NA,NA,internal-NBER,Acknowledged,logic_comms,75,70,90,NA,70,90 +recmEzlRNxLNEWD8e,The Comparative Impact of Cash Transfers and a Psychotherapy Program on Psychological and Economic Well-being,Well-being: Cash vs. psychotherapy,Hannah Metzler,GH&D,NA,NA,internal-NBER,Acknowledged,real_world,75,60,90,NA,60,90 +recmEzlRNxLNEWD8e,The Comparative Impact of Cash Transfers and a Psychotherapy Program on Psychological and Economic Well-being,Well-being: Cash vs. psychotherapy,Hannah Metzler,GH&D,NA,NA,internal-NBER,Acknowledged,gp_relevance,90,80,100,NA,80,100 +recmEzlRNxLNEWD8e,The Comparative Impact of Cash Transfers and a Psychotherapy Program on Psychological and Economic Well-being,Well-being: Cash vs. psychotherapy,Hannah Metzler,GH&D,NA,NA,internal-NBER,Acknowledged,journal_predict,3,2,4,NA,2,4 +recmEzlRNxLNEWD8e,The Comparative Impact of Cash Transfers and a Psychotherapy Program on Psychological and Economic Well-being,Well-being: Cash vs. psychotherapy,Hannah Metzler,GH&D,NA,NA,internal-NBER,Acknowledged,open_sci,50,40,80,NA,40,80 +recmEzlRNxLNEWD8e,The Comparative Impact of Cash Transfers and a Psychotherapy Program on Psychological and Economic Well-being,Well-being: Cash vs. psychotherapy,Hannah Metzler,GH&D,NA,NA,internal-NBER,Acknowledged,merits_journal,4,3,4,NA,3,4 +recrdCChKWw5gZr4e,Long term cost-effectiveness of resilient foods for global catastrophes compared to artificial general intelligence safety (Denkenberger et al),LT CEA: Resilient foods vs. AGI safety,Anca Hanea,long-term-relevant ,quantitative,NA,submitted,Agreed,overall,80,60,90,NA,60,90 +recrdCChKWw5gZr4e,Long term cost-effectiveness of resilient foods for global catastrophes compared to artificial general intelligence safety (Denkenberger et al),LT CEA: Resilient foods vs. AGI safety,Anca Hanea,long-term-relevant ,quantitative,NA,submitted,Agreed,adv_knowledge,80,70,90,NA,70,90 +recrdCChKWw5gZr4e,Long term cost-effectiveness of resilient foods for global catastrophes compared to artificial general intelligence safety (Denkenberger et al),LT CEA: Resilient foods vs. AGI safety,Anca Hanea,long-term-relevant ,quantitative,NA,submitted,Agreed,methods,70,50,90,NA,50,90 +recrdCChKWw5gZr4e,Long term cost-effectiveness of resilient foods for global catastrophes compared to artificial general intelligence safety (Denkenberger et al),LT CEA: Resilient foods vs. AGI safety,Anca Hanea,long-term-relevant ,quantitative,NA,submitted,Agreed,logic_comms,85,65,95,NA,65,95 +recrdCChKWw5gZr4e,Long term cost-effectiveness of resilient foods for global catastrophes compared to artificial general intelligence safety (Denkenberger et al),LT CEA: Resilient foods vs. AGI safety,Anca Hanea,long-term-relevant ,quantitative,NA,submitted,Agreed,real_world,NA,NA,NA,NA,NA,NA +recrdCChKWw5gZr4e,Long term cost-effectiveness of resilient foods for global catastrophes compared to artificial general intelligence safety (Denkenberger et al),LT CEA: Resilient foods vs. AGI safety,Anca Hanea,long-term-relevant ,quantitative,NA,submitted,Agreed,gp_relevance,85,70,90,NA,70,90 +recrdCChKWw5gZr4e,Long term cost-effectiveness of resilient foods for global catastrophes compared to artificial general intelligence safety (Denkenberger et al),LT CEA: Resilient foods vs. AGI safety,Anca Hanea,long-term-relevant ,quantitative,NA,submitted,Agreed,journal_predict,3.5,3,5,NA,3,5 +recrdCChKWw5gZr4e,Long term cost-effectiveness of resilient foods for global catastrophes compared to artificial general intelligence safety (Denkenberger et al),LT CEA: Resilient foods vs. AGI safety,Anca Hanea,long-term-relevant ,quantitative,NA,submitted,Agreed,open_sci,73,50,95,NA,50,95 +recrdCChKWw5gZr4e,Long term cost-effectiveness of resilient foods for global catastrophes compared to artificial general intelligence safety (Denkenberger et al),LT CEA: Resilient foods vs. AGI safety,Anca Hanea,long-term-relevant ,quantitative,NA,submitted,Agreed,merits_journal,4,3,5,NA,3,5 +recscTNMRp1wYdD26,"Mental Health Therapy as a Core Strategy for Increasing Human Capital: Evidence from Ghana (renamed ""Cognitive Behavioral Therapy among Ghana's Rural Poor Is Effective Regardless of Baseline Mental Distress"")","CBT Human K, Ghana",Anonymous_20,GH&D,NA,NA,internal-NBER,Emailed,overall,75,NA,NA,4,67,83 +recscTNMRp1wYdD26,"Mental Health Therapy as a Core Strategy for Increasing Human Capital: Evidence from Ghana (renamed ""Cognitive Behavioral Therapy among Ghana's Rural Poor Is Effective Regardless of Baseline Mental Distress"")","CBT Human K, Ghana",Anonymous_20,GH&D,NA,NA,internal-NBER,Emailed,adv_knowledge,65,NA,NA,4,57,73 +recscTNMRp1wYdD26,"Mental Health Therapy as a Core Strategy for Increasing Human Capital: Evidence from Ghana (renamed ""Cognitive Behavioral Therapy among Ghana's Rural Poor Is Effective Regardless of Baseline Mental Distress"")","CBT Human K, Ghana",Anonymous_20,GH&D,NA,NA,internal-NBER,Emailed,methods,60,NA,NA,3,45,75 +recscTNMRp1wYdD26,"Mental Health Therapy as a Core Strategy for Increasing Human Capital: Evidence from Ghana (renamed ""Cognitive Behavioral Therapy among Ghana's Rural Poor Is Effective Regardless of Baseline Mental Distress"")","CBT Human K, Ghana",Anonymous_20,GH&D,NA,NA,internal-NBER,Emailed,logic_comms,75,NA,NA,3,60,90 +recscTNMRp1wYdD26,"Mental Health Therapy as a Core Strategy for Increasing Human Capital: Evidence from Ghana (renamed ""Cognitive Behavioral Therapy among Ghana's Rural Poor Is Effective Regardless of Baseline Mental Distress"")","CBT Human K, Ghana",Anonymous_20,GH&D,NA,NA,internal-NBER,Emailed,real_world,75,NA,NA,4,67,83 +recscTNMRp1wYdD26,"Mental Health Therapy as a Core Strategy for Increasing Human Capital: Evidence from Ghana (renamed ""Cognitive Behavioral Therapy among Ghana's Rural Poor Is Effective Regardless of Baseline Mental Distress"")","CBT Human K, Ghana",Anonymous_20,GH&D,NA,NA,internal-NBER,Emailed,gp_relevance,75,NA,NA,3,60,90 +recscTNMRp1wYdD26,"Mental Health Therapy as a Core Strategy for Increasing Human Capital: Evidence from Ghana (renamed ""Cognitive Behavioral Therapy among Ghana's Rural Poor Is Effective Regardless of Baseline Mental Distress"")","CBT Human K, Ghana",Anonymous_20,GH&D,NA,NA,internal-NBER,Emailed,journal_predict,NA,NA,NA,NA,NA,NA +recscTNMRp1wYdD26,"Mental Health Therapy as a Core Strategy for Increasing Human Capital: Evidence from Ghana (renamed ""Cognitive Behavioral Therapy among Ghana's Rural Poor Is Effective Regardless of Baseline Mental Distress"")","CBT Human K, Ghana",Anonymous_20,GH&D,NA,NA,internal-NBER,Emailed,open_sci,50,NA,NA,3,35,65 +recscTNMRp1wYdD26,"Mental Health Therapy as a Core Strategy for Increasing Human Capital: Evidence from Ghana (renamed ""Cognitive Behavioral Therapy among Ghana's Rural Poor Is Effective Regardless of Baseline Mental Distress"")","CBT Human K, Ghana",Anonymous_20,GH&D,NA,NA,internal-NBER,Emailed,merits_journal,NA,NA,NA,NA,NA,NA +recsivZraxfYLKEqJ,Long term cost-effectiveness of resilient foods for global catastrophes compared to artificial general intelligence safety (Denkenberger et al),LT CEA: Resilient foods vs. AGI safety,Alex Bates,long-term-relevant ,quantitative,NA,submitted,Agreed,overall,40,20,60,NA,20,60 +recsivZraxfYLKEqJ,Long term cost-effectiveness of resilient foods for global catastrophes compared to artificial general intelligence safety (Denkenberger et al),LT CEA: Resilient foods vs. AGI safety,Alex Bates,long-term-relevant ,quantitative,NA,submitted,Agreed,adv_knowledge,30,20,60,NA,20,60 +recsivZraxfYLKEqJ,Long term cost-effectiveness of resilient foods for global catastrophes compared to artificial general intelligence safety (Denkenberger et al),LT CEA: Resilient foods vs. AGI safety,Alex Bates,long-term-relevant ,quantitative,NA,submitted,Agreed,methods,50,40,60,NA,40,60 +recsivZraxfYLKEqJ,Long term cost-effectiveness of resilient foods for global catastrophes compared to artificial general intelligence safety (Denkenberger et al),LT CEA: Resilient foods vs. AGI safety,Alex Bates,long-term-relevant ,quantitative,NA,submitted,Agreed,logic_comms,60,40,75,NA,40,75 +recsivZraxfYLKEqJ,Long term cost-effectiveness of resilient foods for global catastrophes compared to artificial general intelligence safety (Denkenberger et al),LT CEA: Resilient foods vs. AGI safety,Alex Bates,long-term-relevant ,quantitative,NA,submitted,Agreed,real_world,NA,NA,NA,NA,NA,NA +recsivZraxfYLKEqJ,Long term cost-effectiveness of resilient foods for global catastrophes compared to artificial general intelligence safety (Denkenberger et al),LT CEA: Resilient foods vs. AGI safety,Alex Bates,long-term-relevant ,quantitative,NA,submitted,Agreed,gp_relevance,90,60,95,NA,60,95 +recsivZraxfYLKEqJ,Long term cost-effectiveness of resilient foods for global catastrophes compared to artificial general intelligence safety (Denkenberger et al),LT CEA: Resilient foods vs. AGI safety,Alex Bates,long-term-relevant ,quantitative,NA,submitted,Agreed,journal_predict,2,1,2,NA,1,2 +recsivZraxfYLKEqJ,Long term cost-effectiveness of resilient foods for global catastrophes compared to artificial general intelligence safety (Denkenberger et al),LT CEA: Resilient foods vs. AGI safety,Alex Bates,long-term-relevant ,quantitative,NA,submitted,Agreed,open_sci,70,40,75,NA,40,75 +recsivZraxfYLKEqJ,Long term cost-effectiveness of resilient foods for global catastrophes compared to artificial general intelligence safety (Denkenberger et al),LT CEA: Resilient foods vs. AGI safety,Alex Bates,long-term-relevant ,quantitative,NA,submitted,Agreed,merits_journal,2,1,2,NA,1,2 diff --git a/data/evals_long.rds b/data/evals_long.rds new file mode 100644 index 0000000..f76e724 Binary files /dev/null and b/data/evals_long.rds differ diff --git a/docs/chapters/aggregation.html b/docs/chapters/aggregation.html index 9e815e8..eeed737 100644 --- a/docs/chapters/aggregation.html +++ b/docs/chapters/aggregation.html @@ -173,7 +173,7 @@

3  Aggregation of evaluators judgments (modeling) -

+
@@ -260,12 +260,79 @@

Simple rating aggregation

Below, we are preparing the data for the aggreCATS package.

+
+
Code
# JB: This section is a work in progress, please do not edit
+
+# paper_ratings: one row per rating category and 'type' (score, upper, lower bound.)
+evals_pub %>% 
+  select(id, eval_name, paper_abbrev, 
+         overall, overall_lb_imp, overall_ub_imp,
+         adv_knowledge, adv_knowledge_lb_imp, adv_knowledge_ub_imp,
+         methods, methods_lb_imp, methods_ub_imp,
+         logic_comms, logic_comms_lb_imp, logic_comms_ub_imp,
+         real_world, real_world_lb_imp, real_world_ub_imp,
+         gp_relevance, gp_relevance_lb_imp, gp_relevance_ub_imp,
+         open_sci, open_sci_lb_imp, open_sci_ub_imp) %>% 
+  rename_with(function(x) paste0(x,"_score"), all_of(rating_cats)) %>%
+  pivot_longer(cols = c(-id, -eval_name, -paper_abbrev),
+              names_pattern = "(.+)_(score|[ul]b_imp)",
+              names_to = c("criterion","element"),
+              values_to = "value") -> paper_ratings
+
+# renaming to conform with aggreCATS expectations
+paper_ratings <- paper_ratings %>% 
+  rename(paper_id = paper_abbrev,
+         user_name = eval_name) %>% 
+  mutate(round = "round_1",
+         element = case_when(element == "lb_imp" ~ "three_point_lower",
+                             element == "ub_imp" ~ "three_point_upper",
+                             element == "score" ~ "three_point_best"))
+
+# filter only overall for now
+paper_ratings %>% 
+  filter(criterion == "overall") %>% 
+  group_by(user_name, paper_id) %>% 
+  filter(sum(is.na(value))==0) %>% 
+  ungroup() -> temp
+  
+
+AverageWAgg(expert_judgements = temp, round_2_filter = FALSE, type = "ArMean")
+
+IntervalWAgg(expert_judgements = temp, round_2_filter = FALSE, type = "IntWAgg")
+
+aggreCAT::DistributionWAgg(expert_judgements = temp, round_2_filter = FALSE, type = "DistribArMean", percent_toggle = T)
+
+# EXAMPLE CODE ===============================
+# data(data_ratings)
+# set.seed(1234)
+# 
+# participant_subset <- data_ratings %>%
+#   distinct(user_name) %>%
+#   sample_n(5) %>%
+#   mutate(participant_name = paste("participant", rep(1:n())))
+# 
+# single_claim <- data_ratings %>%
+#   filter(paper_id == "28") %>%
+#   right_join(participant_subset, by = "user_name") %>%
+#   filter(grepl(x = element, pattern = "three_.+")) %>%
+#   select(-group, -participant_name, -question)
+# 
+# DistributionWAgg(expert_judgements = single_claim,
+#             type = "DistribArMean", percent_toggle = T)
+#
+
+

Explicit modeling of ‘research quality’ (for use in prizes, etc.)

  • Use the above aggregation as the outcome of interest, or weight towards categories of greater interest?

  • Model with controls – look for greatest positive residual?

3.2 Inter-rater reliability

+
+
+

+
+

3.3 Decomposing variation, dimension reduction, simple linear models

@@ -569,170 +636,222 @@
diff --git a/docs/chapters/aggregation_files/figure-html/unnamed-chunk-1-1.png b/docs/chapters/aggregation_files/figure-html/unnamed-chunk-1-1.png new file mode 100644 index 0000000..d9a6f75 Binary files /dev/null and b/docs/chapters/aggregation_files/figure-html/unnamed-chunk-1-1.png differ diff --git a/docs/chapters/evaluation_data_analysis.html b/docs/chapters/evaluation_data_analysis.html index 0dad59a..4f5cab7 100644 --- a/docs/chapters/evaluation_data_analysis.html +++ b/docs/chapters/evaluation_data_analysis.html @@ -295,8 +295,8 @@ )
-
- +
+
Code
rm(evals_pub_df_overview)
@@ -352,8 +352,8 @@ )
-
- +
+
@@ -378,8 +378,8 @@ )
-
- +
+
@@ -493,15 +493,16 @@ fig
-
- +
+

Todo: 3

Paper categories

Code
evals_pub %>% 
-  select(paper_abbrev, starts_with("cat_")) %>% 
+  select(paper_abbrev, starts_with("cat_")) %>%
+  distinct() %>% 
   pivot_longer(cols = starts_with("cat_"), names_to = "CatNum", values_to = "Category") %>% 
   group_by(CatNum, Category) %>% 
   count() %>% 
@@ -529,7 +530,9 @@
   rowwise() %>% 
   mutate(source_main = str_replace_all(string = source_main, 
                                        pattern = "-", 
-                                       replace = " ") %>% str_to_title()) %>% 
+                                       replace = " ") %>% str_to_title()) %>%
+  select(paper_abbrev, source_main) %>% 
+  distinct() %>%
   ggplot(aes(x = source_main)) + 
   geom_bar(position = "stack", stat = "count", color = "grey30", fill = "grey80") +
   labs(x = "Source", y = "Count") +
@@ -574,7 +577,7 @@
   scale_fill_brewer(palette = "Set1") +
   scale_x_discrete(labels = function(x) str_wrap(x, width = 20))
-

+

The distribution of ratings and predictions

@@ -621,8 +624,8 @@ ggplotly(g1, tooltip = c("text"))
-
- +
+
@@ -630,7 +633,7 @@

Shiny dashboard

- +
@@ -1366,246 +1369,247 @@ ```{r all_categories} evals_pub %>% - select(paper_abbrev, starts_with("cat_")) %>% - pivot_longer(cols = starts_with("cat_"), names_to = "CatNum", values_to = "Category") %>% - group_by(CatNum, Category) %>% - count() %>% - filter(!is.na(Category)) %>% - mutate(Category = str_to_title(Category), - CatNum = ordered(CatNum, - levels = c("cat_1", "cat_2", "cat_3"), - labels = c("Primary", "Secondary", "Tertiary"))) %>% - ggplot(aes(x = reorder(Category, -n), y = n)) + - geom_bar(aes(fill = CatNum), stat = "identity", color = "grey30") + - labs(x = "Paper category", y = "Count", fill = "Cat Level", - title = "Paper categories represented in pilot data") + - theme_bw() + - facet_grid(~CatNum, scales="free_x", space="free_x") + - theme(axis.text.x=element_text(angle=45,hjust=1)) + - theme(legend.position = "none") - -``` - -#### Paper source {-} - -```{r paper_source} - -# Bar plot -evals_pub %>% - rowwise() %>% - mutate(source_main = str_replace_all(string = source_main, - pattern = "-", - replace = " ") %>% str_to_title()) %>% - ggplot(aes(x = source_main)) + - geom_bar(position = "stack", stat = "count", color = "grey30", fill = "grey80") + - labs(x = "Source", y = "Count") + - labs(title = "Pool of research/evaluations by paper source") + - theme_bw() + - theme(text = element_text(size = 15)) + - scale_x_discrete(labels = function(x) str_wrap(x, width = 20)) - -``` - -```{r data_clean_} -# JB: Most of these should probably be cleaned in data storage + select(paper_abbrev, starts_with("cat_")) %>% + distinct() %>% + pivot_longer(cols = starts_with("cat_"), names_to = "CatNum", values_to = "Category") %>% + group_by(CatNum, Category) %>% + count() %>% + filter(!is.na(Category)) %>% + mutate(Category = str_to_title(Category), + CatNum = ordered(CatNum, + levels = c("cat_1", "cat_2", "cat_3"), + labels = c("Primary", "Secondary", "Tertiary"))) %>% + ggplot(aes(x = reorder(Category, -n), y = n)) + + geom_bar(aes(fill = CatNum), stat = "identity", color = "grey30") + + labs(x = "Paper category", y = "Count", fill = "Cat Level", + title = "Paper categories represented in pilot data") + + theme_bw() + + facet_grid(~CatNum, scales="free_x", space="free_x") + + theme(axis.text.x=element_text(angle=45,hjust=1)) + + theme(legend.position = "none") + +``` + +#### Paper source {-} + +```{r paper_source} + +# Bar plot +evals_pub %>% + rowwise() %>% + mutate(source_main = str_replace_all(string = source_main, + pattern = "-", + replace = " ") %>% str_to_title()) %>% + select(paper_abbrev, source_main) %>% + distinct() %>% + ggplot(aes(x = source_main)) + + geom_bar(position = "stack", stat = "count", color = "grey30", fill = "grey80") + + labs(x = "Source", y = "Count") + + labs(title = "Pool of research/evaluations by paper source") + + theme_bw() + + theme(text = element_text(size = 15)) + + scale_x_discrete(labels = function(x) str_wrap(x, width = 20)) + +``` -library(RColorBrewer) # for color palettes - -# paper statuses that are considered "being evaluated" -eval_true = c("published", - "contacting/awaiting_authors_response_to_evaluation", - "awaiting_publication_ME_comments", - "awaiting_evaluations") - -# Is the paper being evaluated? -all_papers_p <- all_papers_p %>% - mutate(is_evaluated = if_else(`stage of process/todo` %in% eval_true, TRUE, FALSE)) - -# main source clean -all_papers_p <- all_papers_p %>% - mutate(source_main = case_when(source_main == "NA" ~ "Not applicable", - source_main == "internal-from-syllabus-agenda-policy-database" ~ "Internal: syllabus, agenda, etc.", - is.na(source_main) ~ "Unknown", - TRUE ~ source_main)) - -all_papers_p %>% -ggplot(aes(x = fct_infreq(source_main), fill = is_evaluated)) + - geom_bar(position = "stack", stat = "count") + - labs(x = "Source", y = "Count", fill = "Selected for\nevaluation?") + - coord_flip() + # flipping the coordinates to have categories on y-axis (on the left) - labs(title = "Evaluations by source of the paper") + - theme_bw() + - theme(text = element_text(size = 15)) + - scale_fill_brewer(palette = "Set1") + - scale_x_discrete(labels = function(x) str_wrap(x, width = 20)) -``` - - -## The distribution of ratings and predictions {-} +```{r data clean} +# JB: Most of these should probably be cleaned in data storage + +library(RColorBrewer) # for color palettes + +# paper statuses that are considered "being evaluated" +eval_true = c("published", + "contacting/awaiting_authors_response_to_evaluation", + "awaiting_publication_ME_comments", + "awaiting_evaluations") + +# Is the paper being evaluated? +all_papers_p <- all_papers_p %>% + mutate(is_evaluated = if_else(`stage of process/todo` %in% eval_true, TRUE, FALSE)) + +# main source clean +all_papers_p <- all_papers_p %>% + mutate(source_main = case_when(source_main == "NA" ~ "Not applicable", + source_main == "internal-from-syllabus-agenda-policy-database" ~ "Internal: syllabus, agenda, etc.", + is.na(source_main) ~ "Unknown", + TRUE ~ source_main)) + +all_papers_p %>% +ggplot(aes(x = fct_infreq(source_main), fill = is_evaluated)) + + geom_bar(position = "stack", stat = "count") + + labs(x = "Source", y = "Count", fill = "Selected for\nevaluation?") + + coord_flip() + # flipping the coordinates to have categories on y-axis (on the left) + labs(title = "Evaluations by source of the paper") + + theme_bw() + + theme(text = element_text(size = 15)) + + scale_fill_brewer(palette = "Set1") + + scale_x_discrete(labels = function(x) str_wrap(x, width = 20)) +``` -Next, we present the ratings and predictions along with 'uncertainty measures'.^[We use "ub imp" (and "lb imp") to denote the upper and lower bounds given by evaluators.] Where evaluators gave only a 1-5 confidence level^[More or less, the ones who report a level for 'conf overall', although some people did this for some but not others], we use the imputations discussed and coded above. - + +## The distribution of ratings and predictions {-} -- For each category and prediction (overall and by paper) -```{r CI_bounds} +Next, we present the ratings and predictions along with 'uncertainty measures'.^[We use "ub imp" (and "lb imp") to denote the upper and lower bounds given by evaluators.] Where evaluators gave only a 1-5 confidence level^[More or less, the ones who report a level for 'conf overall', although some people did this for some but not others], we use the imputations discussed and coded above. + -# evals_pub %>% -# select(matches("overall")) %>% -# view() - -``` - +- For each category and prediction (overall and by paper) +```{r CI_bounds} + +# evals_pub %>% +# select(matches("overall")) %>% +# view() -::: column-body-outset +``` -```{r} -#| fig-height: 9 -#| -# Generate a color palette with more colors -color_count <- length(unique(evals_pub$paper_abbrev)) -color_palette <- colorRampPalette(brewer.pal(8, "Set1"))(color_count) - -# set one "set" of dodge width values across layers -pd = position_dodge(width = 0.8) +::: column-body-outset + + +```{r} +#| fig-height: 9 +#| +# Generate a color palette with more colors +color_count <- length(unique(evals_pub$paper_abbrev)) +color_palette <- colorRampPalette(brewer.pal(8, "Set1"))(color_count) -# Dot plot -g1 <- evals_pub %>% - ggplot(aes(x = paper_abbrev, y = overall, - text = paste0('Evaluator: ', eval_name, # tooltip data - '<br>Rating [CI]: ', overall, " [", overall_lb_imp, ", ", overall_ub_imp, "]"))) + - geom_point(aes(color = paper_abbrev), - stat = "identity", size = 2, shape = 18, stroke = 1, - position = pd) + - geom_linerange(aes(ymin = overall_lb_imp, ymax = overall_ub_imp, color = paper_abbrev), - position = pd) + - geom_text(data = subset(evals_pub, str_detect(eval_name, "Anonymous")), - aes(label = "anon."), size=3) + - coord_flip() + # flipping the coordinates to have categories on y-axis (on the left) - labs(x = "Paper", y = "Overall score", - title = "Overall scores of evaluated papers") + - theme_bw() + - theme(text = element_text(size = 15)) + - theme(legend.position = "none") + - scale_x_discrete(labels = function(x) str_wrap(x, width = 20)) + - scale_color_manual(values = color_palette) - - -ggplotly(g1, tooltip = c("text")) -``` -::: - -In future, we aim to build a dashboard allowing people to use the complete set of ratings and predictions, and choose their own weightings. (Also incorporating the evaluator uncertainty in reasonable ways.) - -### Shiny dashboard {-} - -::: column-body-outset - -```{=html} -<!-- Currently this is a random Shiny app --> -<iframe height="800" width="120%" frameborder="no" src="https://juliagb.shinyapps.io/DataExplorer/"> </iframe> - -``` - -::: - -::: {.callout-note collapse="true"} -## Future vis +# set one "set" of dodge width values across layers +pd = position_dodge(width = 0.8) + +# Dot plot +g1 <- evals_pub %>% + ggplot(aes(x = paper_abbrev, y = overall, + text = paste0('Evaluator: ', eval_name, # tooltip data + '<br>Rating [CI]: ', overall, " [", overall_lb_imp, ", ", overall_ub_imp, "]"))) + + geom_point(aes(color = paper_abbrev), + stat = "identity", size = 2, shape = 18, stroke = 1, + position = pd) + + geom_linerange(aes(ymin = overall_lb_imp, ymax = overall_ub_imp, color = paper_abbrev), + position = pd) + + geom_text(data = subset(evals_pub, str_detect(eval_name, "Anonymous")), + aes(label = "anon."), size=3) + + coord_flip() + # flipping the coordinates to have categories on y-axis (on the left) + labs(x = "Paper", y = "Overall score", + title = "Overall scores of evaluated papers") + + theme_bw() + + theme(text = element_text(size = 15)) + + theme(legend.position = "none") + + scale_x_discrete(labels = function(x) str_wrap(x, width = 20)) + + scale_color_manual(values = color_palette) + + +ggplotly(g1, tooltip = c("text")) +``` +::: + +In future, we aim to build a dashboard allowing people to use the complete set of ratings and predictions, and choose their own weightings. (Also incorporating the evaluator uncertainty in reasonable ways.) + +### Shiny dashboard {-} + +::: column-body-outset + +```{=html} +<!-- Currently this is a random Shiny app --> +<iframe height="800" width="120%" frameborder="no" src="https://unjournal.shinyapps.io/DataExplorer/"> </iframe> + +``` + +::: -Spider or radial chart - -Each rating is a dimension or attribute (potentially normalized) -potentially superimpose a 'circle' for the suggested weighting or overall. +::: {.callout-note collapse="true"} +## Future vis + +Spider or radial chart -Each paper gets its own spider, with all others (or the average) in faded color behind it as a comparator. - -Ideally user can switch on/off - -Beware -- people infer things from the shape's size - -::: - +Each rating is a dimension or attribute (potentially normalized) +potentially superimpose a 'circle' for the suggested weighting or overall. + +Each paper gets its own spider, with all others (or the average) in faded color behind it as a comparator. + +Ideally user can switch on/off + +Beware -- people infer things from the shape's size -::: column-body-outset +::: -```{r, eval=FALSE} -# JB: what is the purpose of this table? It's very large and I'm not totally -# sure what it's doing so I'm just turning it off for now -unit.scale = function(x) (x*100 - min(x*100)) / (max(x*100) - min(x*100)) - -evaluations_table <- evals_pub %>% - select(paper_abbrev, eval_name, cat_1, - source_main, overall, adv_knowledge, - methods, logic_comms, journal_predict) %>% - arrange(desc(paper_abbrev)) - -formattable( - evaluations_table, - list( - #area(col = 5:8) ~ function(x) percent(x / 100, digits = 0), - area(col = 5:8) ~ color_tile("#FA614B66","#3E7DCC"), - `journal_predict` = proportion_bar("#DeF7E9", unit.scale) - ) -) - - -``` -::: + +::: column-body-outset + +```{r, eval=FALSE} +# JB: what is the purpose of this table? It's very large and I'm not totally +# sure what it's doing so I'm just turning it off for now +unit.scale = function(x) (x*100 - min(x*100)) / (max(x*100) - min(x*100)) + +evaluations_table <- evals_pub %>% + select(paper_abbrev, eval_name, cat_1, + source_main, overall, adv_knowledge, + methods, logic_comms, journal_predict) %>% + arrange(desc(paper_abbrev)) + +formattable( + evaluations_table, + list( + #area(col = 5:8) ~ function(x) percent(x / 100, digits = 0), + area(col = 5:8) ~ color_tile("#FA614B66","#3E7DCC"), + `journal_predict` = proportion_bar("#DeF7E9", unit.scale) + ) +) + - -### Sources of variation {-} +``` +::: -Next, look for systematic variation in the ratings - -- By field and topic area of paper - -- By submission/selection route - -- By evaluation manager (or their seniority, or whether they are US/Commonwealth/Other)^[DR: My theory is that people in commonwealth countries target a 70+ as 'strong' (because of their marking system) and that may drive a bias.] - -... perhaps building a model of this. We are looking for systematic 'biases and trends', loosely speaking, to help us better understand how our evaluation system is working. - -\ - + +### Sources of variation {-} + +Next, look for systematic variation in the ratings + +- By field and topic area of paper + +- By submission/selection route + +- By evaluation manager (or their seniority, or whether they are US/Commonwealth/Other)^[DR: My theory is that people in commonwealth countries target a 70+ as 'strong' (because of their marking system) and that may drive a bias.] + +... perhaps building a model of this. We are looking for systematic 'biases and trends', loosely speaking, to help us better understand how our evaluation system is working. - +\ -### Relationship among the ratings (and predictions) {-} - -::: {.callout-note collapse="true"} -## Next steps (suggested analyses) - -- Correlation matrix - -- ANOVA - -- PCA (Principle components) - -- With other 'control' factors? - -- How do the specific measures predict the aggregate ones (overall rating, merited publication) - - CF 'our suggested weighting' - -::: - + +### Relationship among the ratings (and predictions) {-} + +::: {.callout-note collapse="true"} +## Next steps (suggested analyses) + +- Correlation matrix + +- ANOVA + +- PCA (Principle components) + +- With other 'control' factors? + +- How do the specific measures predict the aggregate ones (overall rating, merited publication) + - CF 'our suggested weighting' + +::: -Next chapter (analysis): *aggregation of evaluator judgment* - + +Next chapter (analysis): *aggregation of evaluator judgment* -::: {.callout-note collapse="true"} -## Scoping our future coverage - -We have funding to evaluate roughly 50-70 papers/projects per year, given our proposed incentives. - -Consider: - -- How many relevant NBER papers come out per year? - -- How much relevant work in other prestige archives? - -- What quotas do we want (by cause, etc.) and how feasible are these? - -::: + +::: {.callout-note collapse="true"} +## Scoping our future coverage + +We have funding to evaluate roughly 50-70 papers/projects per year, given our proposed incentives. + +Consider: + +- How many relevant NBER papers come out per year? + +- How much relevant work in other prestige archives? + +- What quotas do we want (by cause, etc.) and how feasible are these? + +:::
diff --git a/docs/chapters/evaluation_data_analysis_files/figure-html/all_categories-1.png b/docs/chapters/evaluation_data_analysis_files/figure-html/all_categories-1.png index c30dcce..c215cbe 100644 Binary files a/docs/chapters/evaluation_data_analysis_files/figure-html/all_categories-1.png and b/docs/chapters/evaluation_data_analysis_files/figure-html/all_categories-1.png differ diff --git a/docs/chapters/evaluation_data_analysis_files/figure-html/data clean-1.png b/docs/chapters/evaluation_data_analysis_files/figure-html/data clean-1.png new file mode 100644 index 0000000..4df59c7 Binary files /dev/null and b/docs/chapters/evaluation_data_analysis_files/figure-html/data clean-1.png differ diff --git a/docs/chapters/evaluation_data_analysis_files/figure-html/paper_source-1.png b/docs/chapters/evaluation_data_analysis_files/figure-html/paper_source-1.png index 15cac0c..b097648 100644 Binary files a/docs/chapters/evaluation_data_analysis_files/figure-html/paper_source-1.png and b/docs/chapters/evaluation_data_analysis_files/figure-html/paper_source-1.png differ diff --git a/docs/chapters/evaluation_data_input.html b/docs/chapters/evaluation_data_input.html index 03007fe..739bd82 100644 --- a/docs/chapters/evaluation_data_input.html +++ b/docs/chapters/evaluation_data_input.html @@ -381,6 +381,17 @@ #Note: category, topic_subfield, and source have multiple meaningful categories. These will need care +
+
Code
evals_pub_long <- evals_pub %>% 
+  pivot_longer(cols = -c(id, crucial_rsx, paper_abbrev, eval_name, 
+                         cat_1,cat_2, cat_3,source_main,author_agreement),
+               names_pattern = "(lb_|ub_|conf_)?(.+)",
+               names_to = c("value_type", "rating_type")) %>% # one line per rating type
+  mutate(value_type = if_else(value_type == "", "est_", value_type)) %>% #add main rating id
+  pivot_wider(names_from = value_type, 
+              values_from = value)
+
+

Reconcile uncertainty ratings and CIs

@@ -421,7 +432,7 @@
-
reconcile explicit bounds and stated confidence level
# Define the baseline widths for each confidence rating
+
reconcile explicit bounds and stated confidence level
# Define the baseline widths for each confidence rating
 # JB: it would be good to have some more backing of whether this
 # is a valid way to translate these confidence levels into CIs
 
@@ -507,23 +518,10 @@
   return(imp_bound)
 }
 
-# apply functions to evals_pub
-# JB: I had to do a lot of gymnastics to make this data frame
-# right for applying the functions, then turn it back into
-# the data structure we are currently using I didn't want to deal
-# with how changing the data input would break the rest of the
-# code But we should prioritize saving this data frame in a long format
-# so there isn't data in the variable names (i.e., each row should at least 
-# be one type of rating, not one article). 
-
-evals_pub %>% 
-  pivot_longer(cols = -c(id, crucial_rsx, paper_abbrev, eval_name, 
-                         cat_1,cat_2, cat_3,source_main,author_agreement),
-               names_pattern = "(lb_|ub_|conf_)?(.+)",
-               names_to = c("value_type", "rating_type")) %>% # one line per rating type
-  mutate(value_type = if_else(value_type == "", "est_", value_type)) %>% #add main rating id
-  pivot_wider(names_from = value_type, 
-              values_from = value) %>%
+# apply functions to evals_pub_long
+# where each row is one type of rating
+# so each evaluation is 9 rows long
+evals_pub_long <- evals_pub_long %>% 
   rowwise() %>% # apply function to each row
   mutate(lb_imp_ = impute_bounds(var_name = rating_type,
                                    est = est_,
@@ -532,7 +530,10 @@
   mutate(ub_imp_ = impute_bounds(var_name = rating_type,
                                    est = est_,
                                    lb = lb_, ub = ub_, conf = conf_,
-                                   bound_type = "upper")) %>% 
+                                   bound_type = "upper"))
+
+# Reshape evals_pub_long into evals_pub to add imputed bounds
+evals_pub <- evals_pub_long %>% 
   pivot_wider(names_from = rating_type, # take the dataframe back to old format
               values_from = c(est_, ub_, lb_, conf_, lb_imp_, ub_imp_),
               names_sep = "") %>% 
@@ -543,12 +544,18 @@
   dplyr::rename_with(.cols = starts_with("est_"),
                      .fn = gsub,
                      pattern = "est_(.+)",
-                     replacement = "\\1") -> evals_pub

+                     replacement = "\\1")
+
+# Clean evals_pub_long names (remove _ at end)
+evals_pub_long <- evals_pub_long %>% 
+  rename_with(.cols = ends_with("_"),
+              .fn = str_remove,
+              pattern = "_$")

We cannot publicly share the ‘papers under consideration’, but we can share some of the statistics on these papers. Let’s generate an ID (or later, salted hash) for each such paper, and keep only the shareable features of interest

-
Code
all_papers_p <- all_pub_records %>% 
+
Code
all_papers_p <- all_pub_records %>% 
   dplyr::select(
     id,
     category,
@@ -572,30 +579,24 @@
-
Code
# Create a dataset to be used as the input to a shiny app
+
Code
# Create a dataset to be used as the input to a shiny app
 
 #| include: false
 #| 
-paper_ratings <- evals_pub %>% 
-  select(paper_abbrev, eval_name, one_of(rating_cats), one_of(pred_cats), ends_with("_imp")) %>% # rating vars
-  dplyr::rename_with(.cols = c(one_of(rating_cats), one_of(pred_cats)),
-                     .fn = gsub,
-                     pattern = "(.+)", 
-                     replacement = "\\1_best") %>% 
-  pivot_longer(cols = -c(paper_abbrev, eval_name),
-               names_pattern = "(overall|adv_knowledge|methods|logic_comms|real_world|gp_relevance|open_sci|journal_predict|merits_journal)_(.+)",
-               names_to = c("rating_type", ".value"))# one line per rating type
-
-write_rds(paper_ratings, file = here("shinyapp/DataExplorer", "shiny_explorer.rds"))

+evals_pub_long %>% 
+  write_rds(file = here("shinyapp/DataExplorer", "shiny_explorer.rds"))
-
save data for others’ use
all_papers_p %>% saveRDS(file = here("data", "all_papers_p.Rdata"))
+
save data for others’ use
all_papers_p %>% saveRDS(file = here("data", "all_papers_p.Rdata"))
 all_papers_p %>% write_csv(file = here("data", "all_papers_p.csv"))
 
 evals_pub %>% saveRDS(file = here("data", "evals.Rdata"))
 evals_pub %>% write_csv(file = here("data", "evals.csv"))
 
+evals_pub_long %>% write_rds(file = here("data", "evals_long.rds"))
+evals_pub_long %>% write_csv(file = here("data", "evals_long.csv"))
+
 #evals_pub %>% readRDS(file = here("data", "evals.Rdata"))
@@ -900,441 +901,441 @@
diff --git a/docs/index.html b/docs/index.html index 2e43117..4327e91 100644 --- a/docs/index.html +++ b/docs/index.html @@ -5,7 +5,7 @@ - + The Unjournal evaluations: data and analysis