From aa9866a522dd1e9e1265cec3d0820302c4e5df41 Mon Sep 17 00:00:00 2001 From: "@daaronr" <8229168+daaronr@users.noreply.github.com> Date: Tue, 12 Sep 2023 20:29:56 -0400 Subject: [PATCH] temp --- .../aggregation/execute-results/html.json | 4 +- .../execute-results/html.json | 4 +- chapters/aggregation.qmd | 10 +- data/all_papers_p.Rdata | Bin 7481 -> 7478 bytes data/all_papers_p.csv | 3 +- data/evals.Rdata | Bin 3869 -> 3856 bytes data/evals.csv | 2 +- data/evals_long.csv | 18 +- data/evals_long.rds | Bin 77498 -> 77417 bytes docs/chapters/aggregation.html | 305 ++-- docs/chapters/evaluation_data_analysis.html | 1438 ++++++++--------- docs/chapters/evaluation_data_input.html | 260 +-- docs/index.html | 252 +-- docs/search.json | 42 +- docs/site_libs/bootstrap/bootstrap-icons.css | 316 +--- docs/site_libs/bootstrap/bootstrap-icons.woff | Bin 164168 -> 137124 bytes docs/site_libs/bootstrap/bootstrap.min.css | 4 +- .../quarto-syntax-highlighting.css | 32 - docs/site_libs/quarto-html/quarto.js | 378 ++--- docs/site_libs/quarto-nav/quarto-nav.js | 62 +- .../quarto-search/autocomplete.umd.js | 4 +- docs/site_libs/quarto-search/fuse.min.js | 6 +- docs/site_libs/quarto-search/quarto-search.js | 33 +- renv.lock | 225 +-- shinyapp/DataExplorer/shiny_explorer.rds | Bin 77498 -> 77417 bytes 25 files changed, 1083 insertions(+), 2315 deletions(-) diff --git a/_freeze/chapters/aggregation/execute-results/html.json b/_freeze/chapters/aggregation/execute-results/html.json index 8381503..673406f 100644 --- a/_freeze/chapters/aggregation/execute-results/html.json +++ b/_freeze/chapters/aggregation/execute-results/html.json @@ -1,7 +1,7 @@ { - "hash": "68440e6d131ccee0354bc095732ca0e2", + "hash": "2726ad363da560877f7c0b4345be5343", "result": { - "markdown": "# Aggregation of evaluators judgments (modeling)\n\n\n\n\n\n\n## Notes on sources and approaches\n\n\n::: {.callout-note collapse=\"true\"}\n\n## Hanea et al {-}\n(Consult, e.g., repliCATS/Hanea and others work; meta-science and meta-analysis approaches)\n\n`aggrecat` package\n\n> Although the accuracy, calibration, and informativeness of the majority of methods are very similar, a couple of the aggregation methods consistently distinguish themselves as among the best or worst. Moreover, the majority of methods outperform the usual benchmarks provided by the simple average or the median of estimates.\n\n[Hanea et al, 2021](https://journals.plos.org/plosone/article?id=10.1371/journal.pone.0256919#sec007)\n\n However, these are in a different context. Most of those measures are designed to deal with probablistic forecasts for binary outcomes, where the predictor also gives a 'lower bound' and 'upper bound' for that probability. We could roughly compare that to our continuous metrics with 90% CI's (or imputations for these).\n\nFurthermore, many (all their successful measures?) use 'performance-based weights', accessing metrics from prior prediction performance of the same forecasters We do not have these, nor do we have a sensible proxy for this. \n:::\n\n\n::: {.callout-note collapse=\"true\"}\n## D Veen et al (2017)\n\n[link](https://www.researchgate.net/profile/Duco-Veen/publication/319662351_Using_the_Data_Agreement_Criterion_to_Rank_Experts'_Beliefs/links/5b73e2dc299bf14c6da6c663/Using-the-Data-Agreement-Criterion-to-Rank-Experts-Beliefs.pdf)\n\n... we show how experts can be ranked based on their knowledge and their level of (un)certainty. By letting experts specify their knowledge in the form of a probability distribution, we can assess how accurately they can predict new data, and how appropriate their level of (un)certainty is. The expert’s specified probability distribution can be seen as a prior in a Bayesian statistical setting. We evaluate these priors by extending an existing prior-data (dis)agreement measure, the Data Agreement Criterion, and compare this approach to using Bayes factors to assess prior specification. We compare experts with each other and the data to evaluate their appropriateness. Using this method, new research questions can be asked and answered, for instance: Which expert predicts the new data best? Is there agreement between my experts and the data? Which experts’ representation is more valid or useful? Can we reach convergence between expert judgement and data? We provided an empirical example ranking (regional) directors of a large financial institution based on their predictions of turnover. \n\nBe sure to consult the [correction made here](https://www.semanticscholar.org/paper/Correction%3A-Veen%2C-D.%3B-Stoel%2C-D.%3B-Schalken%2C-N.%3B-K.%3B-Veen-Stoel/a2882e0e8606ef876133f25a901771259e7033b1)\n\n::: \n\n\n::: {.callout-note collapse=\"true\"}\n## Also seems relevant:\n\nSee [Gsheet HERE](https://docs.google.com/spreadsheets/d/14japw6eLGpGjEWy1MjHNJXU1skZY_GAIc2uC2HIUalM/edit#gid=0), generated from an Elicit.org inquiry.\n\n\n::: \n\n\n\nIn spite of the caveats in the fold above, we construct some measures of aggregate beliefs using the `aggrecat` package. We will make (and explain) some ad-hoc choices here. We present these:\n\n1. For each paper\n2. For categories of papers and cross-paper categories of evaluations\n3. For the overall set of papers and evaluations\n\nWe can also hold onto these aggregated metrics for later use in modeling.\n\n\n- Simple averaging\n\n- Bayesian approaches \n\n- Best-performing approaches from elsewhere \n\n- Assumptions over unit-level random terms \n\n\n### Simple rating aggregation {-}\n\nBelow, we are preparing the data for the aggreCATS package.\n\n\n::: {.cell}\n\n```{.r .cell-code}\n# JB: This section is a work in progress, please do not edit\n\n# paper_ratings: one row per rating category and 'type' (score, upper, lower bound.)\nevals_pub %>% \n select(id, eval_name, paper_abbrev, \n overall, overall_lb_imp, overall_ub_imp,\n adv_knowledge, adv_knowledge_lb_imp, adv_knowledge_ub_imp,\n methods, methods_lb_imp, methods_ub_imp,\n logic_comms, logic_comms_lb_imp, logic_comms_ub_imp,\n real_world, real_world_lb_imp, real_world_ub_imp,\n gp_relevance, gp_relevance_lb_imp, gp_relevance_ub_imp,\n open_sci, open_sci_lb_imp, open_sci_ub_imp) %>% \n rename_with(function(x) paste0(x,\"_score\"), all_of(rating_cats)) %>%\n pivot_longer(cols = c(-id, -eval_name, -paper_abbrev),\n names_pattern = \"(.+)_(score|[ul]b_imp)\",\n names_to = c(\"criterion\",\"element\"),\n values_to = \"value\") -> paper_ratings\n\n# renaming to conform with aggreCATS expectations\npaper_ratings <- paper_ratings %>% \n rename(paper_id = paper_abbrev,\n user_name = eval_name) %>% \n mutate(round = \"round_1\",\n element = case_when(element == \"lb_imp\" ~ \"three_point_lower\",\n element == \"ub_imp\" ~ \"three_point_upper\",\n element == \"score\" ~ \"three_point_best\"))\n\n# filter only overall for now\npaper_ratings %>% \n filter(criterion == \"overall\") %>% \n group_by(user_name, paper_id) %>% \n filter(sum(is.na(value))==0) %>% \n ungroup() -> temp\n \n\nAverageWAgg(expert_judgements = temp, round_2_filter = FALSE, type = \"ArMean\")\n\nIntervalWAgg(expert_judgements = temp, round_2_filter = FALSE, type = \"IntWAgg\")\n\naggreCAT::DistributionWAgg(expert_judgements = temp, round_2_filter = FALSE, type = \"DistribArMean\", percent_toggle = T)\n\n# EXAMPLE CODE ===============================\n# data(data_ratings)\n# set.seed(1234)\n# \n# participant_subset <- data_ratings %>%\n# distinct(user_name) %>%\n# sample_n(5) %>%\n# mutate(participant_name = paste(\"participant\", rep(1:n())))\n# \n# single_claim <- data_ratings %>%\n# filter(paper_id == \"28\") %>%\n# right_join(participant_subset, by = \"user_name\") %>%\n# filter(grepl(x = element, pattern = \"three_.+\")) %>%\n# select(-group, -participant_name, -question)\n# \n# DistributionWAgg(expert_judgements = single_claim,\n# type = \"DistribArMean\", percent_toggle = T)\n# \n```\n:::\n\n\n\n\n\n### Explicit modeling of 'research quality' (for use in prizes, etc.) {-}\n\n- Use the above aggregation as the outcome of interest, or weight towards categories of greater interest?\n\n- Model with controls -- look for greatest positive residual? \n\n\n## Inter-rater reliability\n\n\n::: {.cell}\n::: {.cell-output-display}\n![](aggregation_files/figure-html/unnamed-chunk-1-1.png){width=672}\n:::\n:::\n\n\n\n\n## Decomposing variation, dimension reduction, simple linear models\n\n\n## Later possiblities\n\n- Relation to evaluation text content (NLP?)\n\n- Relation/prediction of later outcomes (traditional publication, citations, replication)\n", + "markdown": "# Aggregation of evaluators judgments (modeling)\n\n\n\n\n\n\n## Notes on sources and approaches\n\n\n::: {.callout-note collapse=\"true\"}\n\n## Hanea et al {-}\n(Consult, e.g., repliCATS/Hanea and others work; meta-science and meta-analysis approaches)\n\n`aggrecat` package\n\n> Although the accuracy, calibration, and informativeness of the majority of methods are very similar, a couple of the aggregation methods consistently distinguish themselves as among the best or worst. Moreover, the majority of methods outperform the usual benchmarks provided by the simple average or the median of estimates.\n\n[Hanea et al, 2021](https://journals.plos.org/plosone/article?id=10.1371/journal.pone.0256919#sec007)\n\n However, these are in a different context. Most of those measures are designed to deal with probablistic forecasts for binary outcomes, where the predictor also gives a 'lower bound' and 'upper bound' for that probability. We could roughly compare that to our continuous metrics with 90% CI's (or imputations for these).\n\nFurthermore, many (all their successful measures?) use 'performance-based weights', accessing metrics from prior prediction performance of the same forecasters We do not have these, nor do we have a sensible proxy for this. \n:::\n\n\n::: {.callout-note collapse=\"true\"}\n## D Veen et al (2017)\n\n[link](https://www.researchgate.net/profile/Duco-Veen/publication/319662351_Using_the_Data_Agreement_Criterion_to_Rank_Experts'_Beliefs/links/5b73e2dc299bf14c6da6c663/Using-the-Data-Agreement-Criterion-to-Rank-Experts-Beliefs.pdf)\n\n... we show how experts can be ranked based on their knowledge and their level of (un)certainty. By letting experts specify their knowledge in the form of a probability distribution, we can assess how accurately they can predict new data, and how appropriate their level of (un)certainty is. The expert’s specified probability distribution can be seen as a prior in a Bayesian statistical setting. We evaluate these priors by extending an existing prior-data (dis)agreement measure, the Data Agreement Criterion, and compare this approach to using Bayes factors to assess prior specification. We compare experts with each other and the data to evaluate their appropriateness. Using this method, new research questions can be asked and answered, for instance: Which expert predicts the new data best? Is there agreement between my experts and the data? Which experts’ representation is more valid or useful? Can we reach convergence between expert judgement and data? We provided an empirical example ranking (regional) directors of a large financial institution based on their predictions of turnover. \n\nBe sure to consult the [correction made here](https://www.semanticscholar.org/paper/Correction%3A-Veen%2C-D.%3B-Stoel%2C-D.%3B-Schalken%2C-N.%3B-K.%3B-Veen-Stoel/a2882e0e8606ef876133f25a901771259e7033b1)\n\n::: \n\n\n::: {.callout-note collapse=\"true\"}\n## Also seems relevant:\n\nSee [Gsheet HERE](https://docs.google.com/spreadsheets/d/14japw6eLGpGjEWy1MjHNJXU1skZY_GAIc2uC2HIUalM/edit#gid=0), generated from an Elicit.org inquiry.\n\n\n::: \n\n\n\nIn spite of the caveats in the fold above, we construct some measures of aggregate beliefs using the `aggrecat` package. We will make (and explain) some ad-hoc choices here. We present these:\n\n1. For each paper\n2. For categories of papers and cross-paper categories of evaluations\n3. For the overall set of papers and evaluations\n\nWe can also hold onto these aggregated metrics for later use in modeling.\n\n\n- Simple averaging\n\n- Bayesian approaches \n\n- Best-performing approaches from elsewhere \n\n- Assumptions over unit-level random terms \n\n\n### Simple rating aggregation {-}\n\nBelow, we are preparing the data for the aggreCATS package.\n\n\n::: {.cell}\n\n```{.r .cell-code}\n# JB: This section is a work in progress, please do not edit\n\n# paper_ratings: one row per rating category and 'type' (score, upper, lower bound.)\nevals_pub %>% \n select(id, eval_name, paper_abbrev, \n overall, overall_lb_imp, overall_ub_imp,\n adv_knowledge, adv_knowledge_lb_imp, adv_knowledge_ub_imp,\n methods, methods_lb_imp, methods_ub_imp,\n logic_comms, logic_comms_lb_imp, logic_comms_ub_imp,\n real_world, real_world_lb_imp, real_world_ub_imp,\n gp_relevance, gp_relevance_lb_imp, gp_relevance_ub_imp,\n open_sci, open_sci_lb_imp, open_sci_ub_imp) %>% \n rename_with(function(x) paste0(x,\"_score\"), all_of(rating_cats)) %>%\n pivot_longer(cols = c(-id, -eval_name, -paper_abbrev),\n names_pattern = \"(.+)_(score|[ul]b_imp)\",\n names_to = c(\"criterion\",\"element\"),\n values_to = \"value\") -> paper_ratings\n\n# renaming to conform with aggreCATS expectations\npaper_ratings <- paper_ratings %>% \n rename(paper_id = paper_abbrev,\n user_name = eval_name) %>% \n mutate(round = \"round_1\",\n element = case_when(element == \"lb_imp\" ~ \"three_point_lower\",\n element == \"ub_imp\" ~ \"three_point_upper\",\n element == \"score\" ~ \"three_point_best\"))\n\n# filter only overall for now\npaper_ratings %>% \n filter(criterion == \"overall\") %>% \n group_by(user_name, paper_id) %>% \n filter(sum(is.na(value))==0) %>% \n ungroup() -> temp\n \n\nAverageWAgg(expert_judgements = temp, round_2_filter = FALSE, type = \"ArMean\")\n\nIntervalWAgg(expert_judgements = temp, round_2_filter = FALSE, type = \"IntWAgg\")\n\naggreCAT::DistributionWAgg(expert_judgements = temp, round_2_filter = FALSE, type = \"DistribArMean\", percent_toggle = T)\n\n# EXAMPLE CODE ===============================\n# data(data_ratings)\n# set.seed(1234)\n# \n# participant_subset <- data_ratings %>%\n# distinct(user_name) %>%\n# sample_n(5) %>%\n# mutate(participant_name = paste(\"participant\", rep(1:n())))\n# \n# single_claim <- data_ratings %>%\n# filter(paper_id == \"28\") %>%\n# right_join(participant_subset, by = \"user_name\") %>%\n# filter(grepl(x = element, pattern = \"three_.+\")) %>%\n# select(-group, -participant_name, -question)\n# \n# DistributionWAgg(expert_judgements = single_claim,\n# type = \"DistribArMean\", percent_toggle = T)\n# \n```\n:::\n\n\n\n\n\n### Explicit modeling of 'research quality' (for use in prizes, etc.) {-}\n\n- Use the above aggregation as the outcome of interest, or weight towards categories of greater interest?\n\n- Model with controls -- look for greatest positive residual? \n\n\n## Inter-rater reliability\n\n\n::: {.cell}\n::: {.cell-output-display}\n![](aggregation_files/figure-html/unnamed-chunk-1-1.png){width=672}\n:::\n:::\n\n\n\n\n\n\n## Decomposing variation, dimension reduction, simple linear models\n\n\n## Later possiblities\n\n- Relation to evaluation text content (NLP?)\n\n- Relation/prediction of later outcomes (traditional publication, citations, replication)\n", "supporting": [ "aggregation_files" ], diff --git a/_freeze/chapters/evaluation_data_analysis/execute-results/html.json b/_freeze/chapters/evaluation_data_analysis/execute-results/html.json index 8ab47bb..367f547 100644 --- a/_freeze/chapters/evaluation_data_analysis/execute-results/html.json +++ b/_freeze/chapters/evaluation_data_analysis/execute-results/html.json @@ -1,7 +1,7 @@ { - "hash": "248c42795c0e7b2fbe512822cb705803", + "hash": "2782bd0646e3cbf4457779179f0ba134", "result": { - "markdown": "# Evaluation data: description, exploration, checks\n\n## Data input, cleaning, feature construction and imputation \n\n\n::: {.cell}\n\n```{.r .cell-code code-summary=\"load packages\"}\nlibrary(tidyverse) \n\n# markdown et al. ----\nlibrary(knitr)\nlibrary(bookdown)\nlibrary(rmarkdown)\nlibrary(shiny)\nlibrary(quarto)\nlibrary(formattable) # Create 'Formattable' Data Structures\nlibrary(DT) # R interface to DataTables library (JavaScript)\n\n# dataviz ----\nlibrary(ggrepel)\nlibrary(plotly) # Create Interactive Web Graphics via 'plotly.js'\n\n# others ----\nlibrary(here) # A Simpler Way to Find Your Files\n# renv::install(packages = \"metamelb-repliCATS/aggreCAT\")\n#library(aggreCAT)\n\n# Make sure select is always the dplyr version\nselect <- dplyr::select \n\n# options\noptions(knitr.duplicate.label = \"allow\")\noptions(mc.cores = parallel::detectCores())\n```\n:::\n\n\n\n::: {.callout-note collapse=\"true\"}\n## Note on data input (10-Aug-23)\n\nBelow, the evaluation data is input from an Airtable, which itself was largely hand-input from evaluators' reports. As PubPub builds (target: end of Sept. 2023), this will allow us to include the ratings and predictions as structured data objects. We then plan to access and input this data *directly* from the PubPub (API?) into the present analysis. This will improve automation and limit the potential for data entry errors.\n\n::: \n\n\n::: {.cell}\n\n```{.r .cell-code code-summary=\"Input evaluation data\"}\nevals_pub <- readRDS(file = here(\"data\", \"evals.Rdata\"))\nall_papers_p <- readRDS(file = here(\"data\", \"all_papers_p.Rdata\"))\n```\n:::\n\n::: {.cell}\n\n```{.r .cell-code code-summary=\"Define lists of columns to use later\"}\n# Lists of categories\nrating_cats <- c(\"overall\", \"adv_knowledge\", \"methods\", \"logic_comms\", \"real_world\", \"gp_relevance\", \"open_sci\")\n\n#... 'predictions' are currently 1-5 (0-5?)\npred_cats <- c(\"journal_predict\", \"merits_journal\")\n```\n:::\n\n\n\n \n## Basic presentation\n\n### What sorts of papers/projects are we considering and evaluating? {-}\n\nIn this section, we give some simple data summaries and visualizations, for a broad description of The Unjournal's coverage. \n\nIn the interactive table below we give some key attributes of the papers and the evaluators.\n\n\n::: column-body-outset \n\n\n::: {.cell}\n\n```{.r .cell-code}\nevals_pub_df_overview <- evals_pub %>%\n arrange(paper_abbrev, eval_name) %>%\n dplyr::select(paper_abbrev, crucial_rsx, eval_name, cat_1, cat_2, source_main, author_agreement) %>%\n dplyr::select(-matches(\"ub_|lb_|conf\")) \n\nevals_pub_df_overview %>% \n rename(\n \"Paper Abbreviation\" = paper_abbrev,\n \"Paper name\" = crucial_rsx,\n \"Evaluator Name\" = eval_name,\n \"Main category\" = cat_1,\n \"Category 2\" = cat_2,\n \"Main source\" = source_main,\n \"Author contact\" = author_agreement,\n ) %>% \n DT::datatable(\n caption = \"Evaluations (confidence bounds not shown)\", \n filter = 'top',\n rownames= FALSE,\n options = list(pageLength = 5,\n columnDefs = list(list(width = '150px', targets = 1)))) %>% \n formatStyle(columns = 2:ncol(evals_pub_df_overview), \n textAlign = 'center') %>% \nformatStyle(\n \"Paper name\",\n fontSize = '10px'\n )\n```\n\n::: {.cell-output-display}\n```{=html}\n
\n\n```\n:::\n\n```{.r .cell-code}\nrm(evals_pub_df_overview)\n```\n:::\n\n\n:::\n\n\n\n#### Evaluation metrics (ratings) {-}\n\n\n\n::: {.cell}\n\n```{.r .cell-code}\nrename_dtstuff <- function(df){\n df %>% \n rename(\n \"Paper Abbreviation\" = paper_abbrev,\n \"Evaluator Name\" = eval_name,\n \"Advancing knowledge\" = adv_knowledge,\n \"Methods\" = methods,\n \"Logic & comm.\" = logic_comms,\n \"Real world engagement\" = real_world,\n \"Global priorities relevance\" = gp_relevance,\n \"Open Science\" = open_sci\n )\n}\n```\n:::\n\n\n\n::: column-body-outset \n\n\n::: {.cell}\n\n```{.r .cell-code}\n# Need to find a way to control column width but it seems to be a problem with DT\n# https://github.com/rstudio/DT/issues/29\n\n# we didn't seem to be using all_evals_dt so I removed it to increase readability\n\n\n\nevals_pub_df <- evals_pub %>%\n # Arrange data\n arrange(paper_abbrev, eval_name, overall) %>%\n \n # Select and rename columns\n dplyr::select(paper_abbrev, eval_name, all_of(rating_cats)) %>%\n rename_dtstuff \n\n\n(\n evals_pub_dt <- evals_pub_df %>% \n # Convert to a datatable and apply styling\n datatable(\n caption = \"Evaluations and predictions (confidence bounds not shown)\", \n filter = 'top',\n rownames = FALSE,\n options = list(pageLength = 5, \n columnDefs = list(list(width = '150px', targets = 0)))) %>% \n formatStyle(columns = 2:ncol(evals_pub_df), \n textAlign = 'center')\n)\n```\n\n::: {.cell-output-display}\n```{=html}\n
\n\n```\n:::\n:::\n\n:::\n\n\\\n\nNext, a preview of the evaluations, focusing on the 'middle ratings and predictions':\n\n::: column-body-outset \n\n\n::: {.cell}\n\n```{.r .cell-code code-summary=\"Data datable (all shareable relevant data)\"}\n# we didn't seem to be using all_evals_dt so I removed it to increase readability\n\n\nevals_pub %>%\n arrange(paper_abbrev, eval_name, overall) %>%\n dplyr::select(paper_abbrev, eval_name, all_of(rating_cats)) %>%\n rename_dtstuff %>% \n DT::datatable(\n caption = \"Evaluations and predictions (confidence bounds not shown)\", \n filter = 'top',\n rownames= FALSE,\n options = list(pageLength = 5,\n columnDefs = list(list(width = '150px', targets = 0))) \n\n )\n```\n\n::: {.cell-output-display}\n```{=html}\n
\n\n```\n:::\n:::\n\n:::\n\n\\ \n\n\n\n\n::: {.cell}\n\n```{.r .cell-code}\n# we did not seem to be using all_evals_dt_ci so I removed it to improve readability\nevals_pub %>%\n arrange(paper_abbrev, eval_name) %>%\n dplyr::select(paper_abbrev, eval_name, conf_overall, all_of(rating_cats), matches(\"ub_imp|lb_imp\")) %>%\n rename_dtstuff %>% \n DT::datatable(\n caption = \"Evaluations and (imputed*) confidence bounds)\", \n filter = 'top',\n rownames= FALSE,\n options = list(pageLength = 5)\n )\n```\n:::\n\n:::\n\n\n\n\n\n::: {.callout-note collapse=\"true\"}\n##### Next consider...\n\n- Composition of research evaluated\n - By field (economics, psychology, etc.)\n - By subfield of economics \n - By topic/cause area (Global health, economic development, impact of technology, global catastrophic risks, etc. )\n - By source (submitted, identified with author permission, direct evaluation)\n \n- Timing of intake and evaluation^[Consider: timing might be its own section or chapter; this is a major thing journals track, and we want to keep track of ourselves]\n\n:::\n\n#### Paper selection {-}\n\nThe Sankey diagram below starts with the papers we prioritized for likely *Unjournal* evaluation:^[Those marked as 'considering' in the Airtable].\n\n\n::: {.cell}\n\n```{.r .cell-code}\n#Add in the 3 different evaluation input sources\n#update to be automated rather than hard-coded - to look at David's work here\n\npapers_considered <- all_papers_p %>% \n nrow()\n\npapers_deprio <- all_papers_p %>% \n filter(`stage of process/todo` == \"de-prioritized\") %>% \n nrow()\n\npapers_evaluated <- all_papers_p %>% \n filter(`stage of process/todo` %in% c(\"published\",\n \"contacting/awaiting_authors_response_to_evaluation\",\n \"awaiting_publication_ME_comments\",\"awaiting_evaluations\")) %>% \n nrow()\n\npapers_complete <- all_papers_p %>% \n filter(`stage of process/todo` == \"published\") %>%\n nrow()\n\npapers_in_progress <- papers_evaluated - papers_complete\n\npapers_still_in_consideration <- all_papers_p %>% filter(`stage of process/todo` == \"considering\") %>% nrow()\n\n\n#todo: adjust wording of hover notes ('source, target...etc')\n\nfig <- plot_ly(\n type = \"sankey\",\n orientation = \"h\",\n \n node = list(\n label = c(\"Prioritized\", \"Evaluating\", \"Complete\", \"In progress\", \"Still in consideration\", \"De-prioritized\"),\n color = c(\"orange\", \"green\", \"green\", \"orange\", \"orange\", \"red\"),\n #Todo: adjust 'location' to group these left to right\n pad = 15,\n thickness = 20,\n line = list(\n color = \"black\",\n width = 0.5\n )\n ),\n \n link = list(\n source = c(0,1,1,0,0),\n target = c(1,2,3,4,5),\n value = c(\n papers_evaluated,\n papers_complete,\n papers_in_progress,\n papers_still_in_consideration,\n papers_deprio\n ))\n)\nfig <- fig %>% layout(\n title = \"Unjournal paper funnel\",\n font = list(\n size = 10\n )\n)\n\nfig \n```\n\n::: {.cell-output-display}\n```{=html}\n
\n\n```\n:::\n:::\n\n\n\nTodo: ^[Make interactive/dashboards of the elements below]\n\n#### Paper categories {-}\n\n\n\n::: {.cell}\n\n```{.r .cell-code}\nevals_pub %>% \n select(paper_abbrev, starts_with(\"cat_\")) %>%\n distinct() %>% \n pivot_longer(cols = starts_with(\"cat_\"), names_to = \"CatNum\", values_to = \"Category\") %>% \n group_by(CatNum, Category) %>% \n count() %>% \n filter(!is.na(Category)) %>% \n mutate(Category = str_to_title(Category),\n CatNum = ordered(CatNum, \n levels = c(\"cat_1\", \"cat_2\", \"cat_3\"),\n labels = c(\"Primary\", \"Secondary\", \"Tertiary\"))) %>%\n ggplot(aes(x = reorder(Category, -n), y = n)) +\n geom_bar(aes(fill = CatNum), stat = \"identity\", color = \"grey30\") + \n labs(x = \"Paper category\", y = \"Count\", fill = \"Cat Level\",\n title = \"Paper categories represented in pilot data\") +\n theme_bw() +\n facet_grid(~CatNum, scales=\"free_x\", space=\"free_x\") +\n theme(axis.text.x=element_text(angle=45,hjust=1)) +\n theme(legend.position = \"none\")\n```\n\n::: {.cell-output-display}\n![](evaluation_data_analysis_files/figure-html/all_categories-1.png){width=672}\n:::\n:::\n\n\n#### Paper source {-}\n\n\n::: {.cell}\n\n```{.r .cell-code}\n# Bar plot\nevals_pub %>% \n rowwise() %>% \n mutate(source_main = str_replace_all(string = source_main, \n pattern = \"-\", \n replace = \" \") %>% str_to_title()) %>%\n select(paper_abbrev, source_main) %>% \n distinct() %>%\n ggplot(aes(x = source_main)) + \n geom_bar(position = \"stack\", stat = \"count\", color = \"grey30\", fill = \"grey80\") +\n labs(x = \"Source\", y = \"Count\") +\n labs(title = \"Pool of research/evaluations by paper source\") +\n theme_bw() +\n theme(text = element_text(size = 15)) +\n scale_x_discrete(labels = function(x) str_wrap(x, width = 20))\n```\n\n::: {.cell-output-display}\n![](evaluation_data_analysis_files/figure-html/paper_source-1.png){width=672}\n:::\n:::\n\n::: {.cell}\n\n```{.r .cell-code}\n# JB: Most of these should probably be cleaned in data storage\n\nlibrary(RColorBrewer) # for color palettes\n\n# paper statuses that are considered \"being evaluated\"\neval_true = c(\"published\", \n \"contacting/awaiting_authors_response_to_evaluation\",\n \"awaiting_publication_ME_comments\",\n \"awaiting_evaluations\")\n\n# Is the paper being evaluated? \nall_papers_p <- all_papers_p %>% \n mutate(is_evaluated = if_else(`stage of process/todo` %in% eval_true, TRUE, FALSE))\n\n# main source clean\nall_papers_p <- all_papers_p %>% \n mutate(source_main = case_when(source_main == \"NA\" ~ \"Not applicable\",\n source_main == \"internal-from-syllabus-agenda-policy-database\" ~ \"Internal: syllabus, agenda, etc.\",\n is.na(source_main) ~ \"Unknown\",\n TRUE ~ source_main))\n\nall_papers_p %>% \nggplot(aes(x = fct_infreq(source_main), fill = is_evaluated)) + \n geom_bar(position = \"stack\", stat = \"count\") +\n labs(x = \"Source\", y = \"Count\", fill = \"Selected for\\nevaluation?\") +\n coord_flip() + # flipping the coordinates to have categories on y-axis (on the left)\n labs(title = \"Evaluations by source of the paper\") +\n theme_bw() +\n theme(text = element_text(size = 15)) +\n scale_fill_brewer(palette = \"Set1\") +\n scale_x_discrete(labels = function(x) str_wrap(x, width = 20))\n```\n\n::: {.cell-output-display}\n![](evaluation_data_analysis_files/figure-html/data clean-1.png){width=672}\n:::\n:::\n\n\n\n## The distribution of ratings and predictions {-}\n\nNext, we present the ratings and predictions along with 'uncertainty measures'.^[We use \"ub imp\" (and \"lb imp\") to denote the upper and lower bounds given by evaluators.] Where evaluators gave only a 1-5 confidence level^[More or less, the ones who report a level for 'conf overall', although some people did this for some but not others], we use the imputations discussed and coded above. \n\n\n- For each category and prediction (overall and by paper)\n\n::: {.cell}\n\n```{.r .cell-code}\n# evals_pub %>% \n# select(matches(\"overall\")) %>% \n# view()\n```\n:::\n\n\n\n::: column-body-outset\n\n\n\n::: {.cell}\n\n```{.r .cell-code}\n# Generate a color palette with more colors\ncolor_count <- length(unique(evals_pub$paper_abbrev))\ncolor_palette <- colorRampPalette(brewer.pal(8, \"Set1\"))(color_count)\n\n# set one \"set\" of dodge width values across layers\npd = position_dodge(width = 0.8)\n\n# Dot plot\ng1 <- evals_pub %>% \n ggplot(aes(x = paper_abbrev, y = overall, \n text = paste0('Evaluator: ', eval_name, # tooltip data\n '
Rating [CI]: ', overall, \" [\", overall_lb_imp, \", \", overall_ub_imp, \"]\"))) +\n geom_point(aes(color = paper_abbrev), \n stat = \"identity\", size = 2, shape = 18, stroke = 1, \n position = pd) +\n geom_linerange(aes(ymin = overall_lb_imp, ymax = overall_ub_imp, color = paper_abbrev), \n position = pd) +\n geom_text(data = subset(evals_pub, str_detect(eval_name, \"Anonymous\")),\n aes(label = \"anon.\"), size=3) +\n coord_flip() + # flipping the coordinates to have categories on y-axis (on the left)\n labs(x = \"Paper\", y = \"Overall score\",\n title = \"Overall scores of evaluated papers\") +\n theme_bw() +\n theme(text = element_text(size = 15)) +\n theme(legend.position = \"none\") +\n scale_x_discrete(labels = function(x) str_wrap(x, width = 20)) + \n scale_color_manual(values = color_palette)\n\n\nggplotly(g1, tooltip = c(\"text\"))\n```\n\n::: {.cell-output-display}\n```{=html}\n
\n\n```\n:::\n:::\n\n:::\n\nIn future, we aim to build a dashboard allowing people to use the complete set of ratings and predictions, and choose their own weightings. (Also incorporating the evaluator uncertainty in reasonable ways.)\n\n### Shiny dashboard {-}\n\n::: column-body-outset\n\n\n```{=html}\n\n\n\n```\n\n\n:::\n\n::: {.callout-note collapse=\"true\"}\n## Future vis\n\nSpider or radial chart \n\nEach rating is a dimension or attribute (potentially normalized)\npotentially superimpose a 'circle' for the suggested weighting or overall. \n\nEach paper gets its own spider, with all others (or the average) in faded color behind it as a comparator. \n\nIdeally user can switch on/off \n\nBeware -- people infer things from the shape's size\n\n::: \n\n\n::: column-body-outset\n\n\n::: {.cell}\n\n```{.r .cell-code}\n# JB: what is the purpose of this table? It's very large and I'm not totally\n# sure what it's doing so I'm just turning it off for now\nunit.scale = function(x) (x*100 - min(x*100)) / (max(x*100) - min(x*100))\n\nevaluations_table <- evals_pub %>%\n select(paper_abbrev, eval_name, cat_1, \n source_main, overall, adv_knowledge,\n methods, logic_comms, journal_predict) %>%\n arrange(desc(paper_abbrev))\n\nformattable(\n evaluations_table,\n list(\n #area(col = 5:8) ~ function(x) percent(x / 100, digits = 0),\n area(col = 5:8) ~ color_tile(\"#FA614B66\",\"#3E7DCC\"),\n `journal_predict` = proportion_bar(\"#DeF7E9\", unit.scale)\n )\n)\n```\n:::\n\n:::\n\n\n### Sources of variation {-}\n\nNext, look for systematic variation in the ratings \n\n- By field and topic area of paper\n\n- By submission/selection route\n\n- By evaluation manager (or their seniority, or whether they are US/Commonwealth/Other)^[DR: My theory is that people in commonwealth countries target a 70+ as 'strong' (because of their marking system) and that may drive a bias.]\n\n... perhaps building a model of this. We are looking for systematic 'biases and trends', loosely speaking, to help us better understand how our evaluation system is working.\n\n\\\n\n\n### Relationship among the ratings (and predictions) {-} \n\n::: {.callout-note collapse=\"true\"}\n## Next steps (suggested analyses)\n\n- Correlation matrix\n\n- ANOVA\n\n- PCA (Principle components)\n\n- With other 'control' factors?\n\n- How do the specific measures predict the aggregate ones (overall rating, merited publication)\n - CF 'our suggested weighting'\n\n::: \n\n\nNext chapter (analysis): *aggregation of evaluator judgment*\n\n\n::: {.callout-note collapse=\"true\"}\n## Scoping our future coverage\n\nWe have funding to evaluate roughly 50-70 papers/projects per year, given our proposed incentives.\n\nConsider:\n\n- How many relevant NBER papers come out per year?\n\n- How much relevant work in other prestige archives?\n\n- What quotas do we want (by cause, etc.) and how feasible are these?\n\n::: \n\n", + "markdown": "# Evaluation data: description, exploration, checks\n\n## Data input, cleaning, feature construction and imputation \n\n\n::: {.cell}\n\n```{.r .cell-code code-summary=\"load packages\"}\nlibrary(tidyverse) \n\n# markdown et al. ----\nlibrary(knitr)\nlibrary(bookdown)\nlibrary(rmarkdown)\nlibrary(shiny)\nlibrary(quarto)\nlibrary(formattable) # Create 'Formattable' Data Structures\nlibrary(DT) # R interface to DataTables library (JavaScript)\n\n# dataviz ----\nlibrary(ggrepel)\nlibrary(plotly) # Create Interactive Web Graphics via 'plotly.js'\n\n# others ----\nlibrary(here) # A Simpler Way to Find Your Files\n# renv::install(packages = \"metamelb-repliCATS/aggreCAT\")\n#library(aggreCAT)\n\n# Make sure select is always the dplyr version\nselect <- dplyr::select \n\n# options\noptions(knitr.duplicate.label = \"allow\")\noptions(mc.cores = parallel::detectCores())\n```\n:::\n\n\n\n::: {.callout-note collapse=\"true\"}\n## Note on data input (10-Aug-23)\n\nBelow, the evaluation data is input from an Airtable, which itself was largely hand-input from evaluators' reports. As PubPub builds (target: end of Sept. 2023), this will allow us to include the ratings and predictions as structured data objects. We then plan to access and input this data *directly* from the PubPub (API?) into the present analysis. This will improve automation and limit the potential for data entry errors.\n\n::: \n\n\n::: {.cell}\n\n```{.r .cell-code code-summary=\"Input evaluation data\"}\nevals_pub <- readRDS(file = here(\"data\", \"evals.Rdata\"))\nall_papers_p <- readRDS(file = here(\"data\", \"all_papers_p.Rdata\"))\n```\n:::\n\n::: {.cell}\n\n```{.r .cell-code code-summary=\"Define lists of columns to use later\"}\n# Lists of categories\nrating_cats <- c(\"overall\", \"adv_knowledge\", \"methods\", \"logic_comms\", \"real_world\", \"gp_relevance\", \"open_sci\")\n\n#... 'predictions' are currently 1-5 (0-5?)\npred_cats <- c(\"journal_predict\", \"merits_journal\")\n```\n:::\n\n\n\n \n## Basic presentation\n\n### What sorts of papers/projects are we considering and evaluating? {-}\n\nIn this section, we give some simple data summaries and visualizations, for a broad description of The Unjournal's coverage. \n\nIn the interactive table below we give some key attributes of the papers and the evaluators.\n\n\n::: column-body-outset \n\n\n::: {.cell}\n\n```{.r .cell-code}\nevals_pub_df_overview <- evals_pub %>%\n arrange(paper_abbrev, eval_name) %>%\n dplyr::select(paper_abbrev, crucial_rsx, eval_name, cat_1, cat_2, source_main, author_agreement) %>%\n dplyr::select(-matches(\"ub_|lb_|conf\")) \n\nevals_pub_df_overview %>% \n rename(\n \"Paper Abbreviation\" = paper_abbrev,\n \"Paper name\" = crucial_rsx,\n \"Evaluator Name\" = eval_name,\n \"Main category\" = cat_1,\n \"Category 2\" = cat_2,\n \"Main source\" = source_main,\n \"Author contact\" = author_agreement,\n ) %>% \n DT::datatable(\n caption = \"Evaluations (confidence bounds not shown)\", \n filter = 'top',\n rownames= FALSE,\n options = list(pageLength = 5,\n columnDefs = list(list(width = '150px', targets = 1)))) %>% \n formatStyle(columns = 2:ncol(evals_pub_df_overview), \n textAlign = 'center') %>% \nformatStyle(\n \"Paper name\",\n fontSize = '10px'\n )\n```\n\n::: {.cell-output-display}\n```{=html}\n
\n\n```\n:::\n\n```{.r .cell-code}\nrm(evals_pub_df_overview)\n```\n:::\n\n\n:::\n\n\n\n#### Evaluation metrics (ratings) {-}\n\n\n\n::: {.cell}\n\n```{.r .cell-code}\nrename_dtstuff <- function(df){\n df %>% \n rename(\n \"Paper Abbreviation\" = paper_abbrev,\n \"Evaluator Name\" = eval_name,\n \"Advancing knowledge\" = adv_knowledge,\n \"Methods\" = methods,\n \"Logic & comm.\" = logic_comms,\n \"Real world engagement\" = real_world,\n \"Global priorities relevance\" = gp_relevance,\n \"Open Science\" = open_sci\n )\n}\n```\n:::\n\n\n\n::: column-body-outset \n\n\n::: {.cell}\n\n```{.r .cell-code}\n# Need to find a way to control column width but it seems to be a problem with DT\n# https://github.com/rstudio/DT/issues/29\n\n\nevals_pub_df <- evals_pub %>%\n # Arrange data\n arrange(paper_abbrev, eval_name, overall) %>%\n \n # Select and rename columns\n dplyr::select(paper_abbrev, eval_name, all_of(rating_cats)) %>%\n rename_dtstuff \n\n\n(\n evals_pub_dt <- evals_pub_df %>% \n # Convert to a datatable and apply styling\n datatable(\n caption = \"Evaluations and predictions (confidence bounds not shown)\", \n filter = 'top',\n rownames = FALSE,\n options = list(pageLength = 5, \n columnDefs = list(list(width = '150px', targets = 0)))) %>% \n formatStyle(columns = 2:ncol(evals_pub_df), \n textAlign = 'center')\n)\n```\n\n::: {.cell-output-display}\n```{=html}\n
\n\n```\n:::\n:::\n\n:::\n\n\\\n\nNext, a preview of the evaluations, focusing on the 'middle ratings and predictions':\n\n::: column-body-outset \n\n\n::: {.cell}\n\n```{.r .cell-code code-summary=\"Data datable (all shareable relevant data)\"}\n# we didn't seem to be using all_evals_dt so I removed it to increase readability\n\n\nevals_pub %>%\n arrange(paper_abbrev, eval_name, overall) %>%\n dplyr::select(paper_abbrev, eval_name, all_of(rating_cats)) %>%\n rename_dtstuff %>% \n DT::datatable(\n caption = \"Evaluations and predictions (confidence bounds not shown)\", \n filter = 'top',\n rownames= FALSE,\n options = list(pageLength = 5,\n columnDefs = list(list(width = '150px', targets = 0))) \n\n )\n```\n\n::: {.cell-output-display}\n```{=html}\n
\n\n```\n:::\n:::\n\n:::\n\n\\ \n\n\n\n\n::: {.cell}\n\n```{.r .cell-code}\n# we did not seem to be using all_evals_dt_ci so I removed it to improve readability\nevals_pub %>%\n arrange(paper_abbrev, eval_name) %>%\n dplyr::select(paper_abbrev, eval_name, conf_overall, all_of(rating_cats), matches(\"ub_imp|lb_imp\")) %>%\n rename_dtstuff %>% \n DT::datatable(\n caption = \"Evaluations and (imputed*) confidence bounds)\", \n filter = 'top',\n rownames= FALSE,\n options = list(pageLength = 5)\n )\n```\n:::\n\n:::\n\n\n\n\n\n::: {.callout-note collapse=\"true\"}\n##### Next consider...\n\n- Composition of research evaluated\n - By field (economics, psychology, etc.)\n - By subfield of economics \n - By topic/cause area (Global health, economic development, impact of technology, global catastrophic risks, etc. )\n - By source (submitted, identified with author permission, direct evaluation)\n \n- Timing of intake and evaluation^[Consider: timing might be its own section or chapter; this is a major thing journals track, and we want to keep track of ourselves]\n\n:::\n\n#### Paper selection {-}\n\nThe Sankey diagram below starts with the papers we prioritized for likely *Unjournal* evaluation:^[Those marked as 'considering' in the Airtable].\n\n\n::: {.cell}\n\n```{.r .cell-code}\n#Add in the 3 different evaluation input sources\n#update to be automated rather than hard-coded - to look at David's work here\n\npapers_considered <- all_papers_p %>% \n nrow()\n\npapers_deprio <- all_papers_p %>% \n filter(`stage of process/todo` == \"de-prioritized\") %>% \n nrow()\n\npapers_evaluated <- all_papers_p %>% \n filter(`stage of process/todo` %in% c(\"published\",\n \"contacting/awaiting_authors_response_to_evaluation\",\n \"awaiting_publication_ME_comments\",\"awaiting_evaluations\")) %>% \n nrow()\n\npapers_complete <- all_papers_p %>% \n filter(`stage of process/todo` == \"published\") %>%\n nrow()\n\npapers_in_progress <- papers_evaluated - papers_complete\n\npapers_still_in_consideration <- all_papers_p %>% filter(`stage of process/todo` == \"considering\") %>% nrow()\n\n\n#todo: adjust wording of hover notes ('source, target...etc')\n\nfig <- plot_ly(\n type = \"sankey\",\n orientation = \"h\",\n \n node = list(\n label = c(\"Prioritized\", \"Evaluating\", \"Complete\", \"In progress\", \"Still in consideration\", \"De-prioritized\"),\n color = c(\"orange\", \"green\", \"green\", \"orange\", \"orange\", \"red\"),\n #Todo: adjust 'location' to group these left to right\n pad = 15,\n thickness = 20,\n line = list(\n color = \"black\",\n width = 0.5\n )\n ),\n \n link = list(\n source = c(0,1,1,0,0),\n target = c(1,2,3,4,5),\n value = c(\n papers_evaluated,\n papers_complete,\n papers_in_progress,\n papers_still_in_consideration,\n papers_deprio\n ))\n)\nfig <- fig %>% layout(\n title = \"Unjournal paper funnel\",\n font = list(\n size = 10\n )\n)\n\nfig \n```\n\n::: {.cell-output-display}\n```{=html}\n
\n\n```\n:::\n:::\n\n\n\nTodo: ^[Make interactive/dashboards of the elements below]\n\n#### Paper categories {-}\n\n\n\n::: {.cell}\n\n```{.r .cell-code}\nevals_pub %>% \n select(paper_abbrev, starts_with(\"cat_\")) %>%\n distinct() %>% \n pivot_longer(cols = starts_with(\"cat_\"), names_to = \"CatNum\", values_to = \"Category\") %>% \n group_by(CatNum, Category) %>% \n count() %>% \n filter(!is.na(Category)) %>% \n mutate(Category = str_to_title(Category),\n CatNum = ordered(CatNum, \n levels = c(\"cat_1\", \"cat_2\", \"cat_3\"),\n labels = c(\"Primary\", \"Secondary\", \"Tertiary\"))) %>%\n ggplot(aes(x = reorder(Category, -n), y = n)) +\n geom_bar(aes(fill = CatNum), stat = \"identity\", color = \"grey30\") + \n labs(x = \"Paper category\", y = \"Count\", fill = \"Cat Level\",\n title = \"Paper categories represented in pilot data\") +\n theme_bw() +\n facet_grid(~CatNum, scales=\"free_x\", space=\"free_x\") +\n theme(axis.text.x=element_text(angle=45,hjust=1)) +\n theme(legend.position = \"none\")\n```\n\n::: {.cell-output-display}\n![](evaluation_data_analysis_files/figure-html/all_categories-1.png){width=672}\n:::\n:::\n\n\n#### Paper source {-}\n\n\n::: {.cell}\n\n```{.r .cell-code}\n# Bar plot\nevals_pub %>% \n rowwise() %>% \n mutate(source_main = str_replace_all(string = source_main, \n pattern = \"-\", \n replace = \" \") %>% str_to_title()) %>%\n select(paper_abbrev, source_main) %>% \n distinct() %>%\n ggplot(aes(x = source_main)) + \n geom_bar(position = \"stack\", stat = \"count\", color = \"grey30\", fill = \"grey80\") +\n labs(x = \"Source\", y = \"Count\") +\n labs(title = \"Pool of research/evaluations by paper source\") +\n theme_bw() +\n theme(text = element_text(size = 15)) +\n scale_x_discrete(labels = function(x) str_wrap(x, width = 20))\n```\n\n::: {.cell-output-display}\n![](evaluation_data_analysis_files/figure-html/paper_source-1.png){width=672}\n:::\n:::\n\n::: {.cell}\n\n```{.r .cell-code}\n# JB: Most of these should probably be cleaned in data storage\n\nlibrary(RColorBrewer) # for color palettes\n\n# paper statuses that are considered \"being evaluated\"\neval_true = c(\"published\", \n \"contacting/awaiting_authors_response_to_evaluation\",\n \"awaiting_publication_ME_comments\",\n \"awaiting_evaluations\")\n\n# Is the paper being evaluated? \nall_papers_p <- all_papers_p %>% \n mutate(is_evaluated = if_else(`stage of process/todo` %in% eval_true, TRUE, FALSE))\n\n# main source clean\nall_papers_p <- all_papers_p %>% \n mutate(source_main = case_when(source_main == \"NA\" ~ \"Not applicable\",\n source_main == \"internal-from-syllabus-agenda-policy-database\" ~ \"Internal: syllabus, agenda, etc.\",\n is.na(source_main) ~ \"Unknown\",\n TRUE ~ source_main))\n\nall_papers_p %>% \nggplot(aes(x = fct_infreq(source_main), fill = is_evaluated)) + \n geom_bar(position = \"stack\", stat = \"count\") +\n labs(x = \"Source\", y = \"Count\", fill = \"Selected for\\nevaluation?\") +\n coord_flip() + # flipping the coordinates to have categories on y-axis (on the left)\n labs(title = \"Evaluations by source of the paper\") +\n theme_bw() +\n theme(text = element_text(size = 15)) +\n scale_fill_brewer(palette = \"Set1\") +\n scale_x_discrete(labels = function(x) str_wrap(x, width = 20))\n```\n\n::: {.cell-output-display}\n![](evaluation_data_analysis_files/figure-html/data clean-1.png){width=672}\n:::\n:::\n\n\n\n## The distribution of ratings and predictions {-}\n\nNext, we present the ratings and predictions along with 'uncertainty measures'.^[We use \"ub imp\" (and \"lb imp\") to denote the upper and lower bounds given by evaluators.] Where evaluators gave only a 1-5 confidence level^[More or less, the ones who report a level for 'conf overall', although some people did this for some but not others], we use the imputations discussed and coded above. \n\n\n- For each category and prediction (overall and by paper)\n\n::: {.cell}\n\n```{.r .cell-code}\n# evals_pub %>% \n# select(matches(\"overall\")) %>% \n# view()\n```\n:::\n\n\n\n::: column-body-outset\n\n\n\n::: {.cell}\n\n```{.r .cell-code}\n# Generate a color palette with more colors\ncolor_count <- length(unique(evals_pub$paper_abbrev))\ncolor_palette <- colorRampPalette(brewer.pal(8, \"Set1\"))(color_count)\n\n# set one \"set\" of dodge width values across layers\npd = position_dodge(width = 0.8)\n\n# Dot plot\ng1 <- evals_pub %>% \n ggplot(aes(x = paper_abbrev, y = overall, \n text = paste0('Evaluator: ', eval_name, # tooltip data\n '
Rating [CI]: ', overall, \" [\", overall_lb_imp, \", \", overall_ub_imp, \"]\"))) +\n geom_point(aes(color = paper_abbrev), \n stat = \"identity\", size = 2, shape = 18, stroke = 1, \n position = pd) +\n geom_linerange(aes(ymin = overall_lb_imp, ymax = overall_ub_imp, color = paper_abbrev), \n position = pd) +\n geom_text(data = subset(evals_pub, str_detect(eval_name, \"Anonymous\")),\n aes(label = \"anon.\"), size=3) +\n coord_flip() + # flipping the coordinates to have categories on y-axis (on the left)\n labs(x = \"Paper\", y = \"Overall score\",\n title = \"Overall scores of evaluated papers\") +\n theme_bw() +\n theme(text = element_text(size = 15)) +\n theme(legend.position = \"none\") +\n scale_x_discrete(labels = function(x) str_wrap(x, width = 20)) + \n scale_color_manual(values = color_palette)\n\n\nggplotly(g1, tooltip = c(\"text\"))\n```\n\n::: {.cell-output-display}\n```{=html}\n
\n\n```\n:::\n:::\n\n:::\n\nIn future, we aim to build a dashboard allowing people to use the complete set of ratings and predictions, and choose their own weightings. (Also incorporating the evaluator uncertainty in reasonable ways.)\n\n### Shiny dashboard {-}\n\n::: column-body-outset\n\n\n```{=html}\n\n\n\n```\n\n\n:::\n\n::: {.callout-note collapse=\"true\"}\n## Future vis\n\nSpider or radial chart \n\nEach rating is a dimension or attribute (potentially normalized)\npotentially superimpose a 'circle' for the suggested weighting or overall. \n\nEach paper gets its own spider, with all others (or the average) in faded color behind it as a comparator. \n\nIdeally user can switch on/off \n\nBeware -- people infer things from the shape's size\n\n::: \n\n\n::: column-body-outset\n\n\n::: {.cell}\n\n```{.r .cell-code}\n# JB: what is the purpose of this table? It's very large and I'm not totally\n# sure what it's doing so I'm just turning it off for now\nunit.scale = function(x) (x*100 - min(x*100)) / (max(x*100) - min(x*100))\n\nevaluations_table <- evals_pub %>%\n select(paper_abbrev, eval_name, cat_1, \n source_main, overall, adv_knowledge,\n methods, logic_comms, journal_predict) %>%\n arrange(desc(paper_abbrev))\n\nformattable(\n evaluations_table,\n list(\n #area(col = 5:8) ~ function(x) percent(x / 100, digits = 0),\n area(col = 5:8) ~ color_tile(\"#FA614B66\",\"#3E7DCC\"),\n `journal_predict` = proportion_bar(\"#DeF7E9\", unit.scale)\n )\n)\n```\n:::\n\n:::\n\n\n### Sources of variation {-}\n\nNext, look for systematic variation in the ratings \n\n- By field and topic area of paper\n\n- By submission/selection route\n\n- By evaluation manager (or their seniority, or whether they are US/Commonwealth/Other)^[DR: My theory is that people in commonwealth countries target a 70+ as 'strong' (because of their marking system) and that may drive a bias.]\n\n... perhaps building a model of this. We are looking for systematic 'biases and trends', loosely speaking, to help us better understand how our evaluation system is working.\n\n\\\n\n\n### Relationship among the ratings (and predictions) {-} \n\n::: {.callout-note collapse=\"true\"}\n## Next steps (suggested analyses)\n\n- Correlation matrix\n\n- ANOVA\n\n- PCA (Principle components)\n\n- With other 'control' factors?\n\n- How do the specific measures predict the aggregate ones (overall rating, merited publication)\n - CF 'our suggested weighting'\n\n::: \n\n\nNext chapter (analysis): *aggregation of evaluator judgment*\n\n\n::: {.callout-note collapse=\"true\"}\n## Scoping our future coverage\n\nWe have funding to evaluate roughly 50-70 papers/projects per year, given our proposed incentives.\n\nConsider:\n\n- How many relevant NBER papers come out per year?\n\n- How much relevant work in other prestige archives?\n\n- What quotas do we want (by cause, etc.) and how feasible are these?\n\n::: \n\n", "supporting": [ "evaluation_data_analysis_files" ], diff --git a/chapters/aggregation.qmd b/chapters/aggregation.qmd index 5b68852..8eae612 100644 --- a/chapters/aggregation.qmd +++ b/chapters/aggregation.qmd @@ -6,7 +6,7 @@ #| include: false library(tidyverse) -library(aggreCAT) +#library(aggreCAT) library(here) library(irr) @@ -204,6 +204,14 @@ evals_pub %>% ``` + + ## Decomposing variation, dimension reduction, simple linear models diff --git a/data/all_papers_p.Rdata b/data/all_papers_p.Rdata index 345e63beeabd9477f83b312c2b71314d9a3e631d..2665dccdb67f56198198e7fcecfe5b42032ea2c7 100644 GIT binary patch literal 7478 zcmV-69m(P!iwFP!000001KnLokmJ^Q?ip-#n+_GG9@g*0ZldE&~C6$sKNwFfsRod;j~F?_VAs$i8@WZ*T8~y%+Dku=m1C;Joi%cSIJou}c=IOs zeE@zx+{=1(nyX!Nh+6@(B8wS*vhl&Y4MljsRTi1ur-;cCG*s{HfRi* zDOB9pWSDASh9lQ4IfbKUkB6lgUe4qEb@`f~p@ejkKF(cK%CYwJoJR@Oqgo|4&E>D~ z)Pg?}iRQ3`Cu2N$JRMOeLC$94#eRnA7t_-MLP&Lv*+jn~b@=GChboxHgEgW<7A|tN zVF?98y3Lg;b>@87yG&6>2q9DM;t9OVRY$3Atc?<;!Dgj(auwmJK@y#jZOL+%3r}+4 zSSf^>B+Fjrm05CFICa>*_d@sy54b9F;0Z%`RJXGgKMzVkL?at|oMw9_5=Y zGDo7C=euIDo0>|vJnkh>sq#WO9M6|K=a=H>qJt0$5mD?g>HOiew!{Vv#FlPA^MJ zcOsoNvk0M7Q&J4U%~F#Ye%V6_A{8yO=SnYFYMh{pmn(QtY)~XV#jC{(La3y}Q7%Fx zgp0E%fht(Zm(Rl0Vpl3oFZ!qvE8J!DEE=cB#gIayQCpGvc%sXbVSzfkOrV5Gtv0(B zGiSZvQ6Y~Is?}aUSWDLN;^}!Jh!W~-F;o+VX*Sxunj(aw^yoU8raE9|qlSjTQ8pvv zboew{O*8#CN;u9isrGPuk(!K~*C=5Y9~O?n^?oN2i=QKe<0RcEUUIFroDI(w2q$S= ztjFT{Qau{EL*i&dVDNgP6`$1cfPBiZkeN^gj=s@JMlr|p6~X`NQ^QBUykIt`@t5S<7qJ++jK8{U}`D&q%MPu-kkqV(!g~+qbVgX&F z(dx9W#ad@NIX@>+BsK;Kwo|%Fb<>m55H;dinw|3a@zJo<=4u2=I1h{2af~Wenu$6J zpJq11gr`@%$cfld(4^n&X(^mv)@-x&CeyaSrU#YGS`n@ z4G_YG+`H(CiQ~a}t3Jpfgv(kfNN~x^k{Bt=Xq;aj)pF5%_c9bBO5zY9v@$rKYEQ;7 zWmfEkPy*fWjD$?0oK5s*XnJi8Qyf7>lho{N5JPd*4z!P!17PKj|Pmc*HO1vNXB91G+ul1*n$^9-s7#T4+ea8 zzsklECndZh4^eRkM|55Z=fzsFEB8_C4w~}OEYDWNN})wH7YLJ_(CtY_!Am)h=Bq)Q zl`gNRZAFgMF3?Op5HEPW?2^Y{ZadjRXhd}R_Jt4c{R^xOqxTrK2 z2yB82pVUusv@}4iD^I!8dhwdL4Al!AGzOJ86=m8?lOJD4Tc}h@_q0l-^JS^oR7Mk& zzz!4TN+~)!3FQS842ncfuWS8GJ4j{P=;EXEaUniU$NS|-T|sR%Ze;pJoF%B4EHv5F3Ee&+vZx0qoj9+Y3~S<`KPjOVXL1!8@NGO-j?Zwq zg%D)QX{7}5$EAbFCT z$*gQ)^pY`XB1i+4XE2czX~~pyD6$eukw&xg`qhZ!6;>e?cI=?{qD)S5F17b-T>*HW zlocR{gAJW*`iegFEl%K>fI>^dfJAfj7)XgN)67Yc*EL0GMVjmKs%Zg(-Te^{hK+&L ztiYRwd7D%Kw^4_d-)qx7a?A=6pxOOpg^s&7mw6jrds)Bs^=vZ0cAz+2mhNB@Ez_h# z^-N9uu*}O!j~1kvH<+RVyP0LVNo)Zlj@le6_vp?Nd#n=?_rO|Qo7P~q3Z~9FEHUnK zpKgW*c-_c)FN*@lQieBd;rB`#NXW_qhxJ~|@?%yK_@O=dy$-m@v@|9mtc`1cQwN|x zl38Wu0g}tC=$tb*!^gS^`w9#Qpoca0-C6F%^}OYn{^5WZ{Y+76!F!L9go)B~0QoYk zzApAy4sh&Af;d|Pq*fR%k6Uj>a|}wW0lh6|jSfd!+S9mewO;V4Nb()p!KZiZv9=`6 zY;4i=v#h##GxZA_gqTZb;N5dBi@_{18qyLbf=-*3U|B%jd3SXzEW+MBkT>wTkvQG6 z)eENM&5GmFt9Z;A*5G+@nZ(ZBm8Au_mju=-=Ze)OAv{p;^N`oF(a$G!3o=D1IO3V;6n$0xj^>sh~{6~NIlapVY^#9PV+w5ZHughuU|8f4uZ(o00 zFzukPXVR_qwCZE{?V}SYSpCT_^!NY$sw&9bk5KH-e`H<9=7&$A`G50o*7eQzpIG!Y z_q?mg^>;UoMU_$7^_FY}c*0 z*X6X0GxRXlG4N=f8&{jY?pgALQ4h`cA1{~7 z8aIX>Pd-|_?mFID2aNef{r-G;JzdwXZu!S1*Jq3Ug|)5N_jR`Z&Ejvne=Ys46(2@_z5)kx+-rL8&CeCy*n_(^?HA^bI|dfa z_d2gRU0c(9|B=D-?pr#a2Yd6K;0raf1~xx;kxMV}d)+tYd);>C`(J9kp{xGZ@mBh$mifQ5jMp&x)44ngJ)XgjaOJP>Wi0mj zz&?J3Kkj_t*#(~(D)`6J>a`t6{ki`vS7Gm07P=n4ev0ss0T&;Au_EF>cSE3$2-6bv z+;Em4vId=Lc>hGOYHf|nkNpe+d)n2vg68{QSeDnf=C0!ErF*r-Jf=0hWbeen{;{+= z$p&SMRrDJrzKF1cWay(|R#Z6~e6)_u6>cURT;OC4ShFhmnE5Os_!^MTR$3KApBuIM zR>4WXn>Gi%kz9TBh1p0tzyH+#!spy4A$_UEH`cz$3ZESWof!nXi({*n`p&}DO7r+;I&fF+TBrh?+kyju|dx{661VN-F^_{vVVAthXN;$xcLsLxXVjL+kw9XW+Ais~1 zBc&%utU^jG$+Pkh+t*hpR!M0<18KVgJPj2(BPjZWReCxbda4S*u?kSw)JZ>pBN>UM zMw~K|Xb6aVUw|L;)lbV^?I(@%VRee3E*P8S6lnxhI>3hT%ow9siQ;Jaz>rZp>lLBR z4luggg^w$aX`YtB5O~t6sAsR{@iQ2sQ13q8-_&LP=zFrC7KXzS4`gd|G}d8dnv|h% zZB_`#G&KUgoT@nCM)cxNrjXZNf3KrmKt(XR#xEChX(Tl3{wCe z%x3|x?g(bF0}Oe%bKiG<{-(x7ZXXzKgYyu_Jz`Y%9smzg+^7RDP1J$2xfdV%9iBdx zm9;^zXqFumIKNEgTri(msq@`uz94dEY324*wB1zNwbSPF&|@u{%I`$~{8_*+uM^ZvT77-9-PcdR)IlZ0#(yvzUBl!LyrQ;`ZE#SUg_$uR8dB zOK{6h(;Cl<=vudw?S6xBo67yo?9bQEHCyz*{`af9wvn^sYun7>@Ok+P|9mXSG7Fn; z3Uo9K$bg41&C=LI^W)q}?HO`lWQ z7@|8kM)xzcZx~;@>F9R0qxP?Jc=k^e-E=ZvFO}@3Bi%gCZ>p$1w<&bv6u9*rJ|9{n z-x`o@!GFVL{kb^S_pJq9(ii%j-gkA2;LbO1{1(9h_PnMfzn*x!J^AX^oMJl*wu8QF z@y<;;uSTQBLW?Q@Eto_yh$$g8uZQit-m91E>KBhW$^QK!-x-1a2O{H0Vt~D_^)^8Z zW$YAKmVzOvGRmENoJCSsfoQL)SmKMfrClE@wi1gj< zMAEtNu3Ws^>;Q8$yb0#h8R!;=)|7hcwumdkDvN%7v7-ldv{^^$RSU=j*y%dLDd*Z3 zx2c&7gqRM&bL0C4SNh>%HDiYXZi*w_3ZL1nPP*)Q2^hA&q`!po8#;s5v6yzc=j9&5NRuYSD2aawz%x29gy zT$-W2ZPXc}vvaHr8>XTxk7!R-&_&42_NcszDgeOh zPDxLdstAii>$J9Mk~b({f4ue#&I?K`z)G3l4UFowO<`Yg(`)W;@1?&%PIbB!BOX`% zMulvH-PDpIe!fv7;%+Hhzojix-M-&(@_`={023ss`NN zvTvE-f4qqYT{c>_Op~F90sxZO(}pG)WZ3CQ6gllW7zSQ8OTCL-=;GDCT5yt0I#)T| zn=RMPw%TkM1qDs9ixPDg%T$I`NRS$|PDF4hQU3JwKXPuVz<=QlxFWB1NX|vZy%E_w z_B5e>;9aPlY?O^#Ps_vOcQU}EP&JkZ-8F096Kca=D@&}5<<_`X@{D*m{ZeM6qQs4-i-$O%~7R0e>D?Z^nUYdVtvyjyEwkguIJS7cC5g zLVdhZ3R%A3{YCh%3J!Zwd9XJ>2GNQz{1YVy*odbb%cpNJ#<{xBPgXz{%tj%!a6*tuFHDrQ+YWrq4eS5C9^{lfZjrdY| zJPdrWH*bz!PBIeh5X-x1o?pohNtW}lobIZez`=*;Mj}STiJ0tu%gQtB9ZUYKURQ|e z>}|aWB&h)}Ou#h8{4;)DadAV|(YJv~^|Mjfgya<&J}4;)vLXsRjVV1=##pdT>askm z(A>-|gZ`1mLdIlrhCSqkhuEX`NWq5WtWB%kgv=^W9=l9-G8Cc9B##(|2HTGg2Kn*8 zc2Ge0sV{gdQ+<|^5(3(XyS$z?-8krOCMS4QZn-d?@V@9UZ*uwLP@8xU_U1|Vo_i1F zO~%*YlZkAHmh3mD9ZAky)(}IlDl$#OO7ip(e4_R7Q+iBtBT|9-uN<&M$~6Sv%v^7i z6Ox6Ww=e{P$Bmnq*m>wXyub%YiqxLbl;Oj@LztRmw#{W`AaPNtd(5Iz`Q4L8?IEiurqR7mT`iyS$rEn&%=kE7 z{H8N=@TQhO+QyG=-njwOdeO;|?cG{(EiBB(FJE$bqVgi{7TDa_8n!jp0^<5QItt+{ z_B>>U<*?l#6#Ywr%NIwywXT{rUTYY^4-?J9Xakq*$iB99T!+g~IotQb;N_y_ z7=pZ+V~F*(qTk{L?am+2{m~h39L8i=32eU~dmD5sB;C!T_q7Gz(>Js`R`I4Fax1)g zE0o;I63?X>HKD(a3T@T`tUc4en{SP8F1ZaRgnxMU!ykfwmi>aCJFCXr#P7CpY&w72 zb)UE2y~XKI0!^^<{xR`_+b#Rz?aZN z!NUmQK*v@2;)fBSU&3uZkUx}go^S9c!1}iNi}oR$>=y}ygR1^GFtH%cyC;x!;gV6sWusHCK1ApBd{;|Ud(BIjk|^}i*k7|BafnBp^UrGk;o(Lw1i=J*$1`Za-9puMVe1Z8mjL)D`e)fQW z0^%R$2b=sV|4;q_5 z1s(7K^KTsHUjTuy4gW}fz}@|$>Q6Y!r{(t`EM6c5xaC)vzoO3hIRYRE@ezg)U6S8H z3bO0(DZl9X3C!Qje;PiCD1d;g{Qy1UY#%T`4&w*l4|lS^7QYF#{<_B>z)#q@zIel5 z5XP6IeZ1x`D8EyE*Wxn{^Dp6~pD~{R^}`9jMt>7=wXc@{2;8y$!~7hB_>P2~`J4GS z><}MleI^|8BO0F|1#eJ4A-)sPzo5Ty#}6!?vBR)){aMaGq4rmamGv2}@6g`?`2Zb& zi%2-fM^vpJnt#@gpEc?mh)=@FzEN1;q5la7|F04MP=D9_0{gd!Bfic47uYAVw*S%o zBIvNbr9X7(j}vPDhA){9h|quT`-g~HKbFQP?y!II=06o4e!@MZf)+@*ig@P(`Wn;su~FW%kTy94i2JpjLN>f`US#@WkPjIUAE z-@ih2JJi#RajNhQ^?}?D{H9YZ5Wq$u)K7$Wl+`QM=31Tn8n+2_NWsIwX0Lp(T{?RrMh)?F$z$ zSs|5?@!0H}X@OTr3f^SY?l3;l3BL=5F|_t2u>u7Ycux^Jf=+x*`+5kll=&&-uWHjV zk(P$6EQ5s%PVZ><3$e1GkUbVVme^|=JB9Z)7lyq?!EdIeI}LV7Yp&V5qm%AQ!sK0^ z9MbSPwKt%TehB{g;)^f-!ZPqBiUZcx9co;B6&`!N+m%Sjckj3V4?nvtc%j$>Q!RK=P}x+M|>w|2=B?mN4i z&E8l}O+fGw1PKCY0NmPDjw@9qXWzOuKs_uSs|cc0sP?gjYYy%)}!`GW*J_ZB>O6Ta@j z*N1!0@4bwUWx#}zt^9fEw3JiJfNJ7|JX^UYYh6I4GTAal2<>44Y$M)ca*MG)RWi=Mu_$6@!@f0mabeVu_#Uu zYNgX=|LpXN4kt3YDe{Ie3~y@lj#M2 zB$MaEGMS1C^zn4W-~=U?O_cgswqMFjix?qQx(a79se0l#O@-qaA>BFVlKrOK5n|IG zu6c$G)u|3$yv)~!WgI1$Heaqb*o$HBD$N{WglwftCdn>e8>PGPHb%&`6s5~YCizIb z9L7zO*+S1oWJu8st$8p8?K6VL{u!g-6% z)41k^u2kx#r!uLGdr4f}f>?NSQE-N2Dp;>+37tFqdi z$YB)?|>fr>5iB-4@wHGQkPw>SnRk9>C8Je7uwNe%%R5OtnAElDw z|LW3)X>*6rO#k$v1jBu10-NZ6X2hMEP@h~{bWfc-c zPGhwU+fU$x<1CwQ56743$@u&RC(IJV;!&j0?x7{SK}6hB&K6y>@-tE-I7OX_Y+130-P> zR%>2Yj^n8!9-OE3sa!a^xK^*rEvkeQIx}z_pBxLdVljt@+9@j+!>uY+;Lb}$e2r$S z)4GxBo$2J_g2s{993;6;`8wUrOv*#th-VpYDv-xV!*ZLiQ#j!wBIU+$rd&NwHgNcy z=fZ4cdfkhjNF5bV`sbZ|OTO;)l2Vet#7%N>J{zX{6?S^w@2j}x7czHV7LOP@-;Z4n zFv6wMyX;EIDtW_C7+ zo>TGQY@e2<)VLZ?k!+%ho1ICGMS2#^WwNIQ7T1Gei)2N-uHA6ca2HQ~jMT`9Y~?6a z%*-xF7@XZw}j#Z5MbrzJLiRcKC! z*NI5&vV|96E=oq_vvK}}I!eUx;N)USj;-}BlG9jmf|on4+T^n>>T1$z=ks_;E2V(vt3jKS zuWqJoRf*Ow@k~9CE(P|iR3KR~QClEPNRF1|M6EL^R?-WEv&cl~aB-oZF3>n(3JNj# zQlNxkBaUl6xPkG_jgwq)HeP6PhUNMU(A8L|9XcE1J|A9Qr*3ZgRE;mUCpdP8v|1Bc zz(!TM+`{90IEY_{2BBhfkhvPyF@iW~w;J*4kSd)e!g%8)4(r*=@Nsi|9A&e(*`@T6 zGMF5R@mf0G0yu#M%@#fFR$^+Th?j}$i_ka>d%qOlR|j}zkaNdul?q=(Nu`+*UH^a;KA%({b%air`&_EQ3~t8=b~4sCo`h019_PcO$Ll>0oeKJzpSj zNj`GYILQNffLm9Y@~4f`4RsZ66gzkfstG2>w%K!Gd=qQoQmL&=A*a$=uFc&X<0etN zr!^*1sL1E%>S%%!xM8wVEyrdj;ev>xM3w33O}(FOhnQ>|muhq|E+(d#M86VksJIcw z&1}C!aun0MZY(tPK#B(lv0@^234tMROlMvLCVO<*ZCuO`8 zO|GK@p-tv1i5Ur67-5#7M(4`7BeyHf6khITtxE-{zB95xDCV!o`QSx@)A)$ zumAt`AyDZ9TA=xv!YMXJFPMWSqCDUPmXK%($d;r-iIX{oHoKtLu1B<>aw@HIV;8*_ zWpa~qsl8wAiqP}4qCz=bZ0KgwmyD@z@uI*Us6ZYb$bbi9C?&B>GdD$E(-i@#JlEwF z%K|35H%0;+HilAjqF@>3ZCZugMjfEM-v&K;%!x9jIsIjaj<+|Lc^jR3(Kz?@TBVeRao9C1s=q$0vIuY?QT#Ii5 z9cHUw>aN2Q<1Y8ZTZ)Se5-mtplasmJkP*%~0V!*F@r`ZJnqP}&XXZ!vpxxZ2X0#yzX`f=?w{ z=l~a=-f_m-k~nj)#naES>gLbX&utK59-V=I&v`6{vdCx%WI}>1IFM0Uz}8|LUB?>Y&qitAl3HQoJg6`zGx zU|o~A#XS~7Z@|}wkN$U0`+M|@52Szm`77Ul^j{I}&wBspxBvRrKX~wef2oao=^w0d zpZpB{{rjU|Kq?$!jW^GKg3kWn;iHfL7XAI*qhD*|{>B<-p0};@=(Dc&_vn9&cY~j` zzRBO32c`Ky`X@uK53F$p|GC_Ya$4t&b`oef;@{-~Q>zZ!G$M*yOU=&pL0& z=ivWw;U{n3d|b5bV611+?e?_mWBTo*6C_yU(JzhT|9(XiWbQ{K_U9kk*KzpaQ)K?% z{+oS$yWQt>gYT!B*{{9qH=l3AgC*b3HGwU;Ke<7c`JpAJX(tc=I_zRQZ}Iun;y7!( zjyKkE-i~`iPRBS?4|5$u&iS#;4u(DsyEyD+izCC|_Vu0iUBs!A-y)tY{5ay$@QX2i z&&U^E>u>t*s+;5hTyY27W#?`j(d$xRG z)<^63`q&W7o%1z7_q2UpzVH+xv%A`dIy>*+=R9h3EJ+`-}Ow#s0#R|9}2eD~w+Kg%7_b+Hvy93Vti{n8PlnoJODLz`f1i-hQ_A!FHUOeftVL%<-@5zc;_<@cJIw z&FL?cL++TEu#OF0bGp8!b^MXZ^X^*)pL=`r9pZC!x(+u*cd=712=oxxx8$!W5CY7rj3`P#){e9_A-gWmG3q{H-o_=wF2U0> z3k;@c1?77LJyLt3%&D}@(E_IoiTlP1r5Y^{0F<^XqWiOAXM{wbaB9zB1E!h)Jf}j1 zEuD~%!pS-#D5jxMWSqcZP3yuYc-g3$ytDS%dxaBWUJPyjQ6zMN^y%%>61&2@>EK}XjNjzcRW)zE!Z#nk5kM3f-vPxL^G zp&XWwWl!uVJM!?(pu4u|iuW_LN-L@?N5%^myD;tW6Ppi>iV z;B0R7he3x`faPFqr!AUg2ZbppQ@Izqr&gi@_gNr_+*um9J(XxT`E%_w_bl{Ai>3-L zF+Zyo2nsAa>S}hxc4s+#NAljDRP$vkF;790U3CG^#J2CtuzL& z%4zBQN6&S8vaFT#OoKro_coN{Z+bPT>LaQcLDA! z0#~DMLa?cQj6ug&kK;uJ<`f7&+Wn)A`vmm(2*Jq;6a^RcKX z9O}6l*vPOj;T@u^3W#^D&z`?as4}2cZ3PHw?5hB@q|b6=1WcszyFl%s-1>e;)Fb79 z=Y#?1ybBv7k%JSyhjuy8z8XRL&+qInG#V8M)=O_zIM8yeS4Xk>_03Coe}JV5JHY* z7TW8^XczR5CQhMc85EitqdW%_+%b?L%!jnrM_~ES09gYl;z)+>#L`M?9^(5%MjUgU zgK*MAr0;JhlEFprZCKwiYK>!M{Cihl)Fm{IUd!QMOYC)Z(x=udtar8!yzMAPqgU z#Wg#^_ty*~V_JN}kl5HIZcblq3w>M?2L~KHm4M6{=`;%zj`a+xpp3TM2Vs91e)0QK zcOsY0=FhdJb)Trx6Kx7Eg9FY%$zC^oxf&9Fmh=h#zqhA)A5igGwheiW8yP&WcSJ^4 z>PJnFWTXrJ9kaBcgo58gaaus1@Pqk;WAtUF4DOz~IXllOs7ER($_V%>d@C~MddW!e zPz3^5+awvOQWIfuXanf`C1rz(^(Sl3;QgS)2CTI8y~4O&+tl(E*SqHa_FwIr^wgl+ zG2(O8Z&c|v+(<2{+ZP-49^RI6v{?Ewwe4FRHy;?m=ZRAfX~sbS+rLJZ6#D9uwOeI> zNxcF0cN1+xW>QyVdUwpcY=;QvC9gAat+t74Q)Uh}7CiR>fgmc~qTc`FRL7SbTY3!} zwW{ZKcIg2n`k!p#Nsm32BjaSjP=tVzdiu~5iw!#+NfNK0hr{68W+`~F8(lp6R}0Q^ zNarbschlv$&(@j_v#{W)c2T14V!6uFDh*SF-jRqdr7D=-{zLcH3jP;egDVMIr{rF2 zyc?3uV^A0BNB)J{$sXCf>a;vOelH6>id5qS*kyChy`etr)r!n1L_xb$>~_k1*0-|| zL9nV283?E@2i2ejg&WXBg_cB7_L%j0lNM$)arAD1mSuW{{P$8kn4$}DGPIFA;mn_h zdm}f5SRVm|tPvrnATLn75~i|VR$yVNwBrLpCb8=F*z z#AS4$x`!qLv{&%C);H~L^Rc$w-pg0;>0%GZEKIgZx0L@zi|do#vD;)b({2Q+?U3`W zl)%ZOP7ik6EUm7LL4QpDbcr5YFR`8|jtu4l0_)^~cM6cd0IfIUVKv<+90|vtm>WWV zg>M%v4TZyfy-_MzzTg{+VX!JloWqk+D#UE+$nLOK~I6!x1CDr<2mo=DfEydal1JZi~22XEgcShfk<{0S!2K7CC z*o2{eZBs{n4TIf}R8cY+?!YZRIWALjLzZu2NNi*$1E3erFpkx z6JQ1MG4S5ryfb<+#mc}Xrgzgpzmyx&9PeW}!)1AqM-S7DBm$s`gtGbqr0*qmB>B^N zUm>Quw~byeo+B5)axVl^D{qHqg2Xg;O6r^qA~sD9VsY8L=#Ydyfu* z_zBQ*qFLXUqd!~^Q5A@G;`MvS<#6kCDa)M9g{0ZX=_lu76Ctom*b%=LwZ=QT# z_U@_t$@nUILXqnL*?FbfmE_K44Ka-Bq5yy@%Hs#<3D<{@!IdG8G>1NtVakYZOKxIy_o43yqHsVnwEp;}f}Z~!qO7HG zZI1VTSS1ieH#4IrBpielBYmv0EqpgHUhr=5XJ!xUZRO}CA+&M)1_Ppv^UITurj?%c zA^pmM#wE=TIFMxYo5w15;|_r!{NqEqdwcUDbinD)X7C4HSsWfHGoGj0Bjtc*fzY83 z^iOmgbZC`s(+b#T((BfwdOFuwcJdGAMfZR(2kcjoY*UO3v!kBj#A^3oHN-gV;LdTK4N&&Mxca(VFk5qAp|Zfp(5nrlIB zeH~qe*eF|Ukpq?kbAzA@F5N9(9P`$?YC3qWVT2$|G!LT6{DTv$(uigqJx3a`DX+~Y>@8CjPHGyc)jPL2&_X3>YQeoHmWVjiQABLib5y$$Fo;V;$w0UQK(X3}O=bzR7BO}dF7=qh9KFs_WQ@Z_oHe#VrJvaQA-{`5(F{ zFpm)f684xJ->%g@q#J9E5D-R6?4zeQ3yeVj|hV3lKc)+ zkW+tO`NhajaQxrUcQoS8-^#xc zm-xWzFXfUS@%V%(c!T zO1j5KOsgNdf7XtlHR>CTPs+`{F;w4?|0x&$uMz)9f6x4a`m?AjzODWj+9$fU|MC7J z z1NF!5{Ylu||4_^JuSh@7{DAP2aK~Rl>whUEU&I}MdVWCp)uaB!k$iCf?c85u@fU&p znVbKUTK$cn_;j0Z)o0j0k<0d%q(^_H z|JnvZQE7E#K5zSGMif+EV2UorugvOdfF zRed@k0eQ$N3S7wK^p1W_k*J6&-Q$R3nY#hRDY_22Fzi(ZeZ?*4G`S(rU9)$`AiXS$ zllKIA2+&h(uOlD*82 Du{r>W diff --git a/data/all_papers_p.csv b/data/all_papers_p.csv index 4ece07b..c13e3e4 100644 --- a/data/all_papers_p.csv +++ b/data/all_papers_p.csv @@ -20,11 +20,11 @@ rec8CVePFXLK7bxWn,,0.53,NA,0.8,,"""Innovation, meta-science, and research""",NA, rec8MONL7xFGq2BaM,,0.7,0.58,0.9,,Economic development & governance (LMICs),"""Other: Economics, growth, policy, global markets and population ""","Development, Development and Growth, Regional Economics, Regional and Urban Economics, Migration",Emmanuel Orkoh,Unpublished working paper,Follow-up email sent,seeking_(more)_evaluators,internal-NBER,not needed (Unjournal Direct),NA,2022-11-05T15:57:00.000Z rec99VmJ2A7naPxuc,,0.95,NA,0.85,,"""Global health; """"Health & well-being in low-income countries""""""",NA,Health,Ryan Briggs ,NA,NA,seeking_(more)_evaluators,suggested - externally - NGO,not needed (Unjournal Direct),NA,2023-06-21T13:57:12.000Z rec9sED8IjDgjICxH,,0.65,NA,0.65,,Empirical methods,"""Catastrophic and existential risks, the long-term future, forecasting""",Econometrics,NA,NA,NA,NA,internal-from-syllabus-agenda-policy-database,NA,NA,2022-04-26T20:23:52.000Z -recAIY1CCAN8PB417,,NA,NA,NA,,NA,NA,NA,NA,NA,NA,Not a paper/project,NA,NA,NA,2023-07-31T16:18:12.000Z recBdPi9jTrdn6xhs,,0.65,NA,0.55,,"""Global health; """"Health & well-being in low-income countries""""""",NA,NA,NA,NA,Emailed,considering,suggested - internally,NA,NA,2022-09-24T15:18:34.000Z recC20N4daHXFNmJJ,,0.6,NA,NA,,"""Communicable diseases, bio-security and pandemic preparedness, biological risks""",NA,"""Health, Education, and Welfare"", Health","David Reinstein, Sam Abbott",Unpublished working paper,NA,considering,internal-NBER,NA,NA,2022-11-05T22:40:56.000Z recCblJLRgWmhYBcO,,0.52,NA,NA,,Economic development & governance (LMICs),NA,NA,NA,NA,NA,NA,internal-NBER,NA,NA,2023-06-08T23:29:52.000Z recDKf292flMuBf7b,,0.57,NA,NA,,"""Catastrophic and existential risks, the long-term future, forecasting""",Economic development & governance (LMICs),NA,NA,NA,NA,NA,internal-NBER,NA,NA,2023-06-08T23:34:05.000Z +recDx0VLZQq5nckAO,,0.6,NA,NA,,"""Catastrophic and existential risks, the long-term future, forecasting""",Empirical methods,meta-analysis,NA,Unpublished,NA,NA,suggested - externally,NA,NA,2023-09-11T19:50:14.000Z recEZssfl3wF37J1T,,NA,NA,NA,,NA,NA,NA,NA,NA,NA,NA,suggested - externally - NGO,NA,NA,2023-07-31T16:18:12.000Z recEiYGtyDewEDl9T,,0.55,NA,0.8,,Emerging technologies: social and economic impacts (focus: AI),NA,"Development and Growth, Innovation and R&D, Economic Systems, Industrial Organization",Kris Gulati,Unpublished working paper,NA,NA,internal-NBER,NA,NA,2022-11-05T22:16:13.000Z recFauZqPDBMVK28J,,NA,NA,NA,,NA,NA,NA,NA,NA,NA,NA,NA,NA,NA,2023-07-31T16:18:12.000Z @@ -119,7 +119,6 @@ recsiQaf3ZTSkkXLV,,0.63,NA,NA,,"""Global health; """"Health & well-being in low- recsii9l3QRQFerkU,,1,0.8,1,,"""Catastrophic and existential risks, the long-term future, forecasting""",Emerging technologies: social and economic impacts (focus: AI),NA,NA,"Published, ? journal",Agreed,published,submitted,not needed (submitted by authors),NA,2022-05-08T03:58:56.000Z recsxlSRIz4Y1RHd3,,NA,NA,NA,,NA,NA,NA,NA,NA,NA,NA,NA,NA,NA,2023-07-31T16:18:12.000Z rect8c6gbgVnvz6Zt,,NA,NA,NA,,NA,NA,NA,NA,NA,NA,NA,NA,NA,NA,2023-08-14T19:32:14.000Z -rectZXnEtaDibizPe,,NA,NA,NA,,NA,NA,NA,NA,NA,NA,NA,NA,NA,NA,2023-08-22T22:03:51.000Z rectfSMcCGKrVVtuw,,0.63,NA,NA,,"""Global health; """"Health & well-being in low-income countries""""""",Economic development & governance (LMICs),"Public Economics, ""Health, Education, and Welfare"", Poverty and Wellbeing, Labor Economics, Demography and Aging, Labor Supply and Demand, Development and Growth, Development","Hansika Kapoor, Anirudh Tagat",NA,Emailed,published,internal-NBER,NA,NA,2022-11-23T01:51:58.000Z rectim9KLJ6yQ1Goa,,0.56,NA,0.56,,"""Catastrophic and existential risks, the long-term future, forecasting""",NA,NA,NA,"Published, ~top journal",NA,NA,internal-from-syllabus-agenda-policy-database,NA,NA,2022-04-15T15:57:47.000Z rectraxzNjDb0cDbU,,NA,NA,NA,,NA,NA,NA,NA,NA,NA,NA,NA,NA,NA,2023-08-22T20:29:40.000Z diff --git a/data/evals.Rdata b/data/evals.Rdata index c20ac1aff5897b32325c5e0a6f10f33ab438a16b..af8b687f71a1104de433d5ad5a0b562804d87bcb 100644 GIT binary patch literal 3856 zcmV+r5AX0FiwFP!000001MOT(bKA%fM*LboWLtjh<~ngzcI}kJNV5DqtVxly-Y<$0 zB^{y#!6Ati2v7h>Nxnq?Am`+mLsIz{{sBo<{zoo*$RUSRoE~s`1T-+BeLMaV+8#jV4g8G4&*#P!;|7%Xd~Qvy z`-S~|Z=zIKTYuS)bxiGS+f8>dG>gwyR$jz93e)qeyY0roRCTVgbrkQ|o^EZo>&=b1 ziOGH;)-h8ur@oy#Fk8EGZ_Q`1j<+?w`)%XJ^h&kXZ_LIzmP+eRtJ<9L587LWsaVHy zVcpzWe_L6aD6KEd#5z`%*LPk#s}y$|^VV@()aus6W~=Lzw+|PZFDGLi+vdUATC4DC za?DtfRi&dG>66v0R&*E)|wz9nDv(FH3J{S6{uXuGRKq z9qrwxiw85j$Srqnbu-psH(R?;i$~>k^WD?VbgW~4qPElanoaw-R+)_}@08y;m6fBV zm2$20lE*syMzPpltQ|gU?fTR4j=-v}ELXadhdYhZ)bn_U-QV@iqvp=iV)YC z1%iAiZ8zkOV?)*AEHIlqJZ6te+}-DHo%=1YD?*5{{`?eAcnuB->{dGj_~0^DZFA2* zj-(B|DNyBv;|k>hTQi|4h^hr(v{(H#- zCxmUzHV%3)c*hI|l{;(}kSu#XXR|&x1O7(nW+fX!B8)V^NDV6G$nlZOix33wAR^X; zA1%8(7*_aKcKf#PxxyxJ>8#i1z8jfrL-@2|{Kq2k#b_u@F^X&+{sielsu{Fd)i>Rs ziRPJXEjYH?9>Dg^?y)ep<(nO$YS7_$E!zTBk*6%r^*Xl2YA}%Fbq-FEwlB<>t}CMO z&~_S*-Q+CvO^8^_bXnblsA3J?5h6tF^jx2}dJvQ%RNCN_5F(_(f-cNZ9{hj!e|*FL zhetwd5?st2#R%dXJxdt#nw9&I(ggHqK!5~Qt=0qX_al>oo9uDdhr#!H;95I$`0FfT z5xN=MYk<9eU_)L3cZiYLbZrr28zF>P@Z`~tHMD+JXtlx2F}4Pv7DQSfV>7w8P+^fI z*;LHUN1!KPLskFrBfC4sJ(`D3wp-iHK$ zi`SP+RaPt)UV{&knEL@NtF!Y`ucZDNF0`G<g0}ZH_(eM>z#%pCo*Od!%k+j|3Bz zyM172dHvjAOC%Y-I_L8C$$^OcmSfE$F^=H5M(#D^r&-IZ2Ny-~pfDdzUy$O0zWADc zL8E^G%X2qcoyD?k0kYWA+df@yPZ!%Wbl)$|ROLil92?A#kBOi2y6e$t(t16}gd)JY z@$7eD%8SLW-r)1fNlvp{1=n+rJFsT$;Ukng8O$CRq+rb+M#)Sa?pLsBi;0ptxYI92Pis^^EEZ(H~b zWMqRwCZ6H$J8&`*F4_E$Ex^Fe^^}^Cn05HDX=JT!J7NXvbhwj2htF(o7hK!#wIJi0 zEi=p@e4f#!jG2QuVYVUphVLBi%hsC(*8+T|%gr?J)P$mP_z~o9*ancnx-vWWWeF#R z1L=EQr0;PbA{}ONh`YvXUC*&CjF?-otqy91iKO-|=Vi)hO7h(4n3nHxxxNQy=!)Bs z%@wDJ|D6FsM|y^XhLR=S{#hNCV>`gde{lzE)`Y zTvHnwD66u*buuU`?+4|^ zd(v+t>sTT^%Bzv~P1(MFlC6izDH-?~;9WdP^1mW@soV|HuhWNo+Y;Z(5V=@Kezf=N zY`nkzPW=At_xT#pOZ8WX{giIrNaCZos61*nwj=*M!AtGZ`3cLlA@*R2;b=hxd{HyTl{CiKfkIFhNk^YVxhliwp zSbdb>Fw`f)tDP@oKILn&ee)#IPWevfi=@Z++%z)z6g|c*l|%AFichX5DZFZ)B-^Pz zI=>a)T0WCcmygbkcl(=cJ{5o2dQ*7mdQdyb&>Q{J^^0e~{>AyakFExI2oQB{Xy1r`Gw-i6wr{X?z zSUg;Ky}}-mXG_|q5i#g!F(n`;rcex%5dZ4vhRP@B7`jdk zUB`y#rE<=TR~yHuH?-dNI*GSrXZd?z~ZXYJ+qIrqp9X5ViKM{Qt zFXqqGd^57`J-2_)>6eT8H#NT=W|!yo@9fEdk33&f`&zIEmcKSS9bH2!L z!HziW72=lb@yZ`K14FywKkm@8o=uFC^Oy z>ff8hANbsl`d6MWQhvsJAn8}Nj^TaMh-RGVJ;^Zt;(SH>>2qVmPwp?&yxAgr^1eWK zf03_AeHibmKc6j!-fz--QXCh2K9Bi;++XCk5jWvS{F?g<#YgcXPAu_Wi{iy|EBcY# zU!?9YtNCOd@u7aisky&U@+e-aM|Xcr{etw^uJY{(sT;`>#|`V-q)z9p?mm8t z)OB$z^(nmfBpsII{vv-%<{P?i9hLb>mdYFVu$hqNrhdJ?` z79s04{9F?5`|#nz->5<)5(?kQ#!EK7#dXW_d*YqdJwL$ziD0BFzSp>C*6TjU|IgqC z?*okxrR*?6^(K2%nTl7QqWHk;`4->nm^QwYJZkpBwg>XX>tnI^i$4?JEi_NxGdCOk zJ@rN!(sVfRHt==bThTWM_QaPdu<54H0qCLUJL<)@R(DUnajahCz56Y|a)Gex^M-99 zgzH|HyL*9Ut9PckZ->F2q(&9J`d$+2D70k36_iQDcc3K=`ZlztaNmW-4CwcvIUD^g zfh%diy$C>-4Nl4iM`MGNvcbu;!O610$+p4K+Tf&Za5OfEH{P4Xiu+MpmK_7?q$Q(K zTiTYpQG3dodr@RFWP$q49Ai-oPDzX;%j;oGij=i)6eUVJ zL<53N5+e|x0FYWP(LcyJIp&a5{)K-)QkDOa%N%mZAr=l%+sT(V zn>*|8%1h&=^@ZtJ$I9~h&dcZJ;%#gK1vmmOHn)8SAhc&E03kqnUN{%)tT)im=+^ zE-P|}*L*v)dB8S0+}wY|3T(v;ZO=Wl>zq{%?J(p%n>p%m-)?g^WNW_H_CnijvMtlH zY&Rli^WtCKb9rE!>`B2fLo;AJWTrF9CdVgU|8jDCa^k7f{hK8Z*gH+-rS<~@qro${DIrP0w3m{uwyJPyi&O(owerPvr%LbYz z2=bw{-Huke4u8W>xPV|8-<3ME|=UAcMGlleJ)87YSMX%kqL*b|34LBsQo2?MwgUeW@#XbKx zlGgX8K$R1YE6fzwnh8xo)C&%_IIn|;1NLOgb8>8A0@QUnIX3y?_1Md2PjeFR-%B1i zA#8EBanOaqJ7zGb++nkTWZCmMoAtRF@V7!YE7=edVWa^@s$ZFj93Q#72tjZH5wRxx zXxW|qu)@C?w`cpFD{KOn&U!uWyOGH@gii~`e>4(bjE2G#qsZprPmnI8nn8 zZKv+o4bDQ}gow3Fm(@IoDpu!hAwtAX*Y$a`3qdJDr3Fq2Awudb=)er+!T*Q<$2a_c zcr3Ig!Ntr`j3B<=wS+Nm*h~*nnt&b+2#}zv)mp&)USx7`lRfG9F!){B5rjbJuKxxsQup6Jl7* z<7wV9VIqi?ROxDql{;ejDq!WV=v(uIwm?ol!C2-^)2};nF`tGUiz z9zXp7Y(@U~elY~!kkJL%*kzm4d zw+9SOub1m@i6p~U=Um=CIS`TGa;$MA#t}SM&%I&%ENgkS;Gzf~7UrYr3sPLr7hlsa zX!I{&dG183vskt*Ko(niTW9O-**>VR>Q57EJf#5Cq89J0-|Ub=aG_u$yWl zY1xB09J0l34T8l=%fx3w*B3lErc_KfO@dFL?#?(iB-NrFK;H0zGqolwuR3? zZf|hN#M9h80VgBjlFbj<0u1b2SE>2DY92!hESRv=vkizLSc}3Fl(N>c9kGNlHam_hg=qfJ>g2b05WK^6{A9PZ23n+4Ybpr*^sH1FiNqH_2V zBy!jfkWsrbJNIP?XNLoMd`#r=F&{!6rgMnf$7>zWu`LXpTd_S4YK0l4HZbR9%4kaR z+-;kd?{T@%2Lp7)?a1beQ^fzy0HGrt;Gm&oNw>a1 z0H}O`4-(g6dpnu}K}cx)sVBI28IO!R!lQ5kb;eevXUeE8=+@d|^}!rErl`aoa4a}>9JAI9#!MK9 zx;fULL}PW>T-9LrCYinny3HmJ#Mxks@uR)~=lH*j$Fe)@P9Pm}zw30^99gY@h4 zA>X#dw=zU7){!6W{W=@(ufG$&Kl^>YO7v3wWnw?2n>Uj9C@w0G+KuhVKTq&dyL5iS za&3q`SmL<7BkdRuy78y|D+Dj`uktgsTa{9d90x5Q9XIs*j*-Bj>gA+7S~qn5rEx~* zAI=|)8!WM(&Re9Xcq^oi{M5g)UE#eh>2J#t>kpqrA60)O%Lk3KqIQFZ1Uc%bp9)(?#x)W5`^x^bcUK01DN{*~*< zF#7yRf4%tB{-ygq)xIp-*R%bEe5#K7NwVM6;>CPI$7N~=UfHkBr@D9`>p=>yX5UKV zCuR3AyySeP+HwD}nyojLzYs63m&5We>YY#egPz~Cc2fNcZ&IFiej)wWgZWb9N5^Rh z-l6NOW_?TXlYJ`gLx;t~h1UxlKOJ7pxa#SpywVW7nt3^N|A6O%k!<<6uN~^&YZuIC z5)|&AvHk6LgYjzP5#w-1V+UPFzb5>ZY`)aEUU0rCx(6CucYZ5Aze}~#^U3vWeyu({ zmkjZ*j&7)Ya*m{SxnMH0KK?kNTIKFVyo`vfnx5GYAeT2mMRV z7iTyL-`Z)Co}4dKJm7ez;whh$NAco5TKYHTXB=;GzQB8o6d(E#&llK^&$;n_0R6X} zv{%uuCFL8`zj&`k&KJ7pcKOZs3EZji)SNFAzA~xPd5Qid=Zm-hl^J^i+my3ZczW;B>uqXe$>D6e39}q-UCU$s&x$SlWuFqiQbb8^DoX+^uKM%Ya_Idgy(h(S!RPas56JyRej9NUe#Ec2zfgP> zFXF@!@3kmiJh!4B$^Av@{<4x!))61-N1U4b3nh=@rFwMt$J8%KkL@boo|3weEHR$S zvR{`6F&=Q7s9p_jgTzBBKOi6ah1_4{x8G;;;dqym{dj*t@3&L;0BYSy@*C8@n?z2P zlyuy%zD??M-sVepJb{0a#iNL5hLNB{C#Z4 zye8#fIg*sG_zd+K91rLxj90p!q;Z1fqAaOD>3FI5lx2m`sr~8uBwq9@K4(I_I{#vR zC;nCaH_(sdz8n2Z$47VmO!@c9CHVCd`PS4`%P|9d-}o}VN)^_ey?R5|E<+7pQ5G-M z%{+9%| zJK_tDduFZXbNr7DZtxz^2vN!oGgPm%N0rHV`7`=(XH(zv5r7X7Fd3a}P9J5`riT(thE;c^L3?1f-R_`< zgh|Q1yV9s+>w}nC$)5XXh9x^5#!X9heL;*%_TEGDlFgqd4J4@1&h7v^yP0-&wRU!O fc6Rl4b_d$o&9<{U$WGa582|b=NAuSKS4IE;xT*uB diff --git a/data/evals.csv b/data/evals.csv index e07883a..3cf0a56 100644 --- a/data/evals.csv +++ b/data/evals.csv @@ -11,7 +11,7 @@ recMAQaYQFRL0DQJB,"""The Environmental Effects of Economic Production: Evidence recNMQY75RCZdIcyG,"Kremer, M., Levin, J. and Snyder, C.M., 2020, May. Advance Market Commitments: Insights from Theory and Experience. In AEA Papers and Proceedings (Vol. 110, pp. 269-73).",Advance market commit. (vaccines),Dan Tortorice,policy,economics,biorisk,internal-from-syllabus-agenda-policy-database,Agreed,80,90,80,80,NA,95,4,90,4,NA,NA,NA,NA,NA,NA,NA,NA,NA,NA,NA,NA,NA,NA,NA,NA,NA,NA,4,5,4,4,NA,5,5,3,5,72,86,72,72,NA,91,3.8,75,3.8,88,94,88,88,NA,99,4.2,100,4.2 recOV0UgplEXwJf81,"Aghion, P., Jones, B.F., and Jones, C.I., 2017. Artificial Intelligence and Economic Growth",AI and econ. growth,Seth Benzell,macroeconomics,Artificial intelligence,prominent,internal-from-syllabus-agenda-policy-database,Agreed,80,75,80,70,NA,90,NA,95,4,90,85,85,80,NA,100,NA,100,5,70,65,75,60,NA,85,NA,90,3.5,NA,NA,NA,NA,NA,NA,NA,NA,NA,70,65,75,60,NA,85,NA,90,3.5,90,85,85,80,NA,100,NA,100,5 recXaqPPgA911nuoY,Banning wildlife trade can boost demand for unregulated threatened species,Banning wildlife trade can boost demand,Liew Jia Huan,conservation,biodiversity,NA,submitted,Agreed,75,80,50,70,90,65,2.5,50,3,NA,NA,NA,NA,NA,NA,NA,NA,NA,NA,NA,NA,NA,NA,NA,NA,NA,NA,4,4,2,5,4,3,5,5,5,67,72,25,66,82,50,2.3,46,2.8,83,88,75,74,98,80,2.7,54,3.2 -recYb2JcJlGrHoI2H,The Governance Of Non-Profits And Their Social Impact: Evidence From A Randomized Program In Healthcare In DRC,Nonprofit Govc.: Randomized healthcare DRC,Wayne Aaron Sandholtz,GH&D,NA,NA,internal-NBER,Emailed,65,70,60,55,55,80,3.6,45,3.8,74,75,70,65,75,90,4,60,4.1,55,55,55,50,45,70,2.8,30,3,NA,NA,NA,NA,NA,NA,NA,NA,NA,55,55,55,50,45,70,2.8,30,3,74,75,70,65,75,90,4,60,4.1 +recYb2JcJlGrHoI2H,The Governance Of Non-Profits And Their Social Impact: Evidence From A Randomized Program In Healthcare In DRC,Nonprofit Govc.: Randomized healthcare DRC,Anonymous_12,GH&D,NA,NA,internal-NBER,Emailed,65,70,60,55,55,80,3.6,45,3.8,74,75,70,65,75,90,4,60,4.1,55,55,55,50,45,70,2.8,30,3,NA,NA,NA,NA,NA,NA,NA,NA,NA,55,55,55,50,45,70,2.8,30,3,74,75,70,65,75,90,4,60,4.1 recbXm55IKEWH4DAM,"Aghion, P., Jones, B.F., and Jones, C.I., 2017. Artificial Intelligence and Economic Growth ",AI and econ. growth,Phil Trammel,macroeconomics,Artificial intelligence,prominent,internal-from-syllabus-agenda-policy-database,Agreed,92,97,70,45,NA,92,3.5,80,5,100,100,90,70,NA,100,NA,NA,NA,80,80,40,30,NA,80,NA,NA,NA,NA,NA,NA,NA,NA,NA,NA,1,4,80,80,40,30,NA,80,NA,42.5,4.6,100,100,90,70,NA,100,NA,100,5 recf9O8DFGO98TPWk,"Mental Health Therapy as a Core Strategy for Increasing Human Capital: Evidence from Ghana (renamed ""Cognitive Behavioral Therapy among Ghana's Rural Poor Is Effective Regardless of Baseline Mental Distress"")","CBT Human K, Ghana",Anonymous_14,GH&D,NA,NA,internal-NBER,Emailed,75,60,90,70,50,50,4,90,4,84,65,94,82,52,60,NA,95,NA,70,55,82,62,48,40,NA,80,NA,NA,NA,NA,NA,NA,NA,4,NA,4,70,55,82,62,48,40,3.6,80,3.6,84,65,94,82,52,60,4.4,95,4.4 rechZ3KqBeCecnHOU,"When Celebrities Speak: A Nationwide Twitter Experiment Promoting Vaccination In Indonesia (Alatas et al, 2019/2021)","Celeb. Twitter promo, Indonesia vacc.",Anirugh Tagat,GH&D,biorisk,NA,internal-NBER,Emailed,85,90,80,85,100,100,4,80,5,NA,NA,NA,NA,NA,NA,NA,NA,NA,NA,NA,NA,NA,NA,NA,NA,NA,NA,4,4,3,4,4,5,5,3,5,77,82,65,77,92,96,3.8,65,4.8,93,98,95,93,100,100,4.2,95,5 diff --git a/data/evals_long.csv b/data/evals_long.csv index b3d71a0..ae5a937 100644 --- a/data/evals_long.csv +++ b/data/evals_long.csv @@ -107,15 +107,15 @@ recXaqPPgA911nuoY,Banning wildlife trade can boost demand for unregulated threat recXaqPPgA911nuoY,Banning wildlife trade can boost demand for unregulated threatened species,Banning wildlife trade can boost demand,Liew Jia Huan,conservation,biodiversity,NA,submitted,Agreed,journal_predict,2.5,NA,NA,5,2.3,2.7 recXaqPPgA911nuoY,Banning wildlife trade can boost demand for unregulated threatened species,Banning wildlife trade can boost demand,Liew Jia Huan,conservation,biodiversity,NA,submitted,Agreed,open_sci,50,NA,NA,5,46,54 recXaqPPgA911nuoY,Banning wildlife trade can boost demand for unregulated threatened species,Banning wildlife trade can boost demand,Liew Jia Huan,conservation,biodiversity,NA,submitted,Agreed,merits_journal,3,NA,NA,5,2.8,3.2 -recYb2JcJlGrHoI2H,The Governance Of Non-Profits And Their Social Impact: Evidence From A Randomized Program In Healthcare In DRC,Nonprofit Govc.: Randomized healthcare DRC,Wayne Aaron Sandholtz,GH&D,NA,NA,internal-NBER,Emailed,overall,65,55,74,NA,55,74 -recYb2JcJlGrHoI2H,The Governance Of Non-Profits And Their Social Impact: Evidence From A Randomized Program In Healthcare In DRC,Nonprofit Govc.: Randomized healthcare DRC,Wayne Aaron Sandholtz,GH&D,NA,NA,internal-NBER,Emailed,adv_knowledge,70,55,75,NA,55,75 -recYb2JcJlGrHoI2H,The Governance Of Non-Profits And Their Social Impact: Evidence From A Randomized Program In Healthcare In DRC,Nonprofit Govc.: Randomized healthcare DRC,Wayne Aaron Sandholtz,GH&D,NA,NA,internal-NBER,Emailed,methods,60,55,70,NA,55,70 -recYb2JcJlGrHoI2H,The Governance Of Non-Profits And Their Social Impact: Evidence From A Randomized Program In Healthcare In DRC,Nonprofit Govc.: Randomized healthcare DRC,Wayne Aaron Sandholtz,GH&D,NA,NA,internal-NBER,Emailed,logic_comms,55,50,65,NA,50,65 -recYb2JcJlGrHoI2H,The Governance Of Non-Profits And Their Social Impact: Evidence From A Randomized Program In Healthcare In DRC,Nonprofit Govc.: Randomized healthcare DRC,Wayne Aaron Sandholtz,GH&D,NA,NA,internal-NBER,Emailed,real_world,55,45,75,NA,45,75 -recYb2JcJlGrHoI2H,The Governance Of Non-Profits And Their Social Impact: Evidence From A Randomized Program In Healthcare In DRC,Nonprofit Govc.: Randomized healthcare DRC,Wayne Aaron Sandholtz,GH&D,NA,NA,internal-NBER,Emailed,gp_relevance,80,70,90,NA,70,90 -recYb2JcJlGrHoI2H,The Governance Of Non-Profits And Their Social Impact: Evidence From A Randomized Program In Healthcare In DRC,Nonprofit Govc.: Randomized healthcare DRC,Wayne Aaron Sandholtz,GH&D,NA,NA,internal-NBER,Emailed,journal_predict,3.6,2.8,4,NA,2.8,4 -recYb2JcJlGrHoI2H,The Governance Of Non-Profits And Their Social Impact: Evidence From A Randomized Program In Healthcare In DRC,Nonprofit Govc.: Randomized healthcare DRC,Wayne Aaron Sandholtz,GH&D,NA,NA,internal-NBER,Emailed,open_sci,45,30,60,NA,30,60 -recYb2JcJlGrHoI2H,The Governance Of Non-Profits And Their Social Impact: Evidence From A Randomized Program In Healthcare In DRC,Nonprofit Govc.: Randomized healthcare DRC,Wayne Aaron Sandholtz,GH&D,NA,NA,internal-NBER,Emailed,merits_journal,3.8,3,4.1,NA,3,4.1 +recYb2JcJlGrHoI2H,The Governance Of Non-Profits And Their Social Impact: Evidence From A Randomized Program In Healthcare In DRC,Nonprofit Govc.: Randomized healthcare DRC,Anonymous_12,GH&D,NA,NA,internal-NBER,Emailed,overall,65,55,74,NA,55,74 +recYb2JcJlGrHoI2H,The Governance Of Non-Profits And Their Social Impact: Evidence From A Randomized Program In Healthcare In DRC,Nonprofit Govc.: Randomized healthcare DRC,Anonymous_12,GH&D,NA,NA,internal-NBER,Emailed,adv_knowledge,70,55,75,NA,55,75 +recYb2JcJlGrHoI2H,The Governance Of Non-Profits And Their Social Impact: Evidence From A Randomized Program In Healthcare In DRC,Nonprofit Govc.: Randomized healthcare DRC,Anonymous_12,GH&D,NA,NA,internal-NBER,Emailed,methods,60,55,70,NA,55,70 +recYb2JcJlGrHoI2H,The Governance Of Non-Profits And Their Social Impact: Evidence From A Randomized Program In Healthcare In DRC,Nonprofit Govc.: Randomized healthcare DRC,Anonymous_12,GH&D,NA,NA,internal-NBER,Emailed,logic_comms,55,50,65,NA,50,65 +recYb2JcJlGrHoI2H,The Governance Of Non-Profits And Their Social Impact: Evidence From A Randomized Program In Healthcare In DRC,Nonprofit Govc.: Randomized healthcare DRC,Anonymous_12,GH&D,NA,NA,internal-NBER,Emailed,real_world,55,45,75,NA,45,75 +recYb2JcJlGrHoI2H,The Governance Of Non-Profits And Their Social Impact: Evidence From A Randomized Program In Healthcare In DRC,Nonprofit Govc.: Randomized healthcare DRC,Anonymous_12,GH&D,NA,NA,internal-NBER,Emailed,gp_relevance,80,70,90,NA,70,90 +recYb2JcJlGrHoI2H,The Governance Of Non-Profits And Their Social Impact: Evidence From A Randomized Program In Healthcare In DRC,Nonprofit Govc.: Randomized healthcare DRC,Anonymous_12,GH&D,NA,NA,internal-NBER,Emailed,journal_predict,3.6,2.8,4,NA,2.8,4 +recYb2JcJlGrHoI2H,The Governance Of Non-Profits And Their Social Impact: Evidence From A Randomized Program In Healthcare In DRC,Nonprofit Govc.: Randomized healthcare DRC,Anonymous_12,GH&D,NA,NA,internal-NBER,Emailed,open_sci,45,30,60,NA,30,60 +recYb2JcJlGrHoI2H,The Governance Of Non-Profits And Their Social Impact: Evidence From A Randomized Program In Healthcare In DRC,Nonprofit Govc.: Randomized healthcare DRC,Anonymous_12,GH&D,NA,NA,internal-NBER,Emailed,merits_journal,3.8,3,4.1,NA,3,4.1 recbXm55IKEWH4DAM,"Aghion, P., Jones, B.F., and Jones, C.I., 2017. Artificial Intelligence and Economic Growth ",AI and econ. growth,Phil Trammel,macroeconomics,Artificial intelligence,prominent,internal-from-syllabus-agenda-policy-database,Agreed,overall,92,80,100,NA,80,100 recbXm55IKEWH4DAM,"Aghion, P., Jones, B.F., and Jones, C.I., 2017. Artificial Intelligence and Economic Growth ",AI and econ. growth,Phil Trammel,macroeconomics,Artificial intelligence,prominent,internal-from-syllabus-agenda-policy-database,Agreed,adv_knowledge,97,80,100,NA,80,100 recbXm55IKEWH4DAM,"Aghion, P., Jones, B.F., and Jones, C.I., 2017. Artificial Intelligence and Economic Growth ",AI and econ. growth,Phil Trammel,macroeconomics,Artificial intelligence,prominent,internal-from-syllabus-agenda-policy-database,Agreed,methods,70,40,90,NA,40,90 diff --git a/data/evals_long.rds b/data/evals_long.rds index f76e72460388f9090ac4254ae8970ed5d24dd6a5..101447298146737a8aec1bc6d4017c6f0f695ead 100644 GIT binary patch delta 197 zcmdmWm*wRhmJMrH)$=&!<>yuA=9d=78yYdNFmN(3Fwjv GHvs@1;WtqL delta 278 zcmaEPhh^7YmJMrH)r*EFR_3KDI3^b5=P3jy=A~rh=af`2urP2kFfh^QBRcQbK diff --git a/docs/chapters/aggregation.html b/docs/chapters/aggregation.html index eeed737..160c3a7 100644 --- a/docs/chapters/aggregation.html +++ b/docs/chapters/aggregation.html @@ -2,22 +2,21 @@ - + The Unjournal evaluations: data and analysis - 3  Aggregation of evaluators judgments (modeling) @@ -92,25 +121,22 @@
-
-
-

+

3  Aggregation of evaluators judgments (modeling)

@@ -181,19 +201,18 @@
-

3.1 Notes on sources and approaches

-
+
-
+
-
+
+

3.3 Decomposing variation, dimension reduction, simple linear models

@@ -372,23 +398,9 @@ icon: icon }; anchorJS.add('.anchored'); - const isCodeAnnotation = (el) => { - for (const clz of el.classList) { - if (clz.startsWith('code-annotation-')) { - return true; - } - } - return false; - } const clipboard = new window.ClipboardJS('.code-copy-button', { - text: function(trigger) { - const codeEl = trigger.previousElementSibling.cloneNode(true); - for (const childEl of codeEl.children) { - if (isCodeAnnotation(childEl)) { - childEl.remove(); - } - } - return codeEl.innerText; + target: function(trigger) { + return trigger.previousElementSibling; } }); clipboard.on('success', function(e) { @@ -400,24 +412,7 @@ button.classList.add('code-copy-button-checked'); var currentTitle = button.getAttribute("title"); button.setAttribute("title", "Copied!"); - let tooltip; - if (window.bootstrap) { - button.setAttribute("data-bs-toggle", "tooltip"); - button.setAttribute("data-bs-placement", "left"); - button.setAttribute("data-bs-title", "Copied!"); - tooltip = new bootstrap.Tooltip(button, - { trigger: "manual", - customClass: "code-copy-button-tooltip", - offset: [0, -8]}); - tooltip.show(); - } setTimeout(function() { - if (tooltip) { - tooltip.hide(); - button.removeAttribute("data-bs-title"); - button.removeAttribute("data-bs-toggle"); - button.removeAttribute("data-bs-placement"); - } button.setAttribute("title", currentTitle); button.classList.remove('code-copy-button-checked'); }, 1000); @@ -504,128 +499,24 @@ return note.innerHTML; }); } - let selectedAnnoteEl; - const selectorForAnnotation = ( cell, annotation) => { - let cellAttr = 'data-code-cell="' + cell + '"'; - let lineAttr = 'data-code-annotation="' + annotation + '"'; - const selector = 'span[' + cellAttr + '][' + lineAttr + ']'; - return selector; - } - const selectCodeLines = (annoteEl) => { - const doc = window.document; - const targetCell = annoteEl.getAttribute("data-target-cell"); - const targetAnnotation = annoteEl.getAttribute("data-target-annotation"); - const annoteSpan = window.document.querySelector(selectorForAnnotation(targetCell, targetAnnotation)); - const lines = annoteSpan.getAttribute("data-code-lines").split(","); - const lineIds = lines.map((line) => { - return targetCell + "-" + line; - }) - let top = null; - let height = null; - let parent = null; - if (lineIds.length > 0) { - //compute the position of the single el (top and bottom and make a div) - const el = window.document.getElementById(lineIds[0]); - top = el.offsetTop; - height = el.offsetHeight; - parent = el.parentElement.parentElement; - if (lineIds.length > 1) { - const lastEl = window.document.getElementById(lineIds[lineIds.length - 1]); - const bottom = lastEl.offsetTop + lastEl.offsetHeight; - height = bottom - top; - } - if (top !== null && height !== null && parent !== null) { - // cook up a div (if necessary) and position it - let div = window.document.getElementById("code-annotation-line-highlight"); - if (div === null) { - div = window.document.createElement("div"); - div.setAttribute("id", "code-annotation-line-highlight"); - div.style.position = 'absolute'; - parent.appendChild(div); - } - div.style.top = top - 2 + "px"; - div.style.height = height + 4 + "px"; - let gutterDiv = window.document.getElementById("code-annotation-line-highlight-gutter"); - if (gutterDiv === null) { - gutterDiv = window.document.createElement("div"); - gutterDiv.setAttribute("id", "code-annotation-line-highlight-gutter"); - gutterDiv.style.position = 'absolute'; - const codeCell = window.document.getElementById(targetCell); - const gutter = codeCell.querySelector('.code-annotation-gutter'); - gutter.appendChild(gutterDiv); - } - gutterDiv.style.top = top - 2 + "px"; - gutterDiv.style.height = height + 4 + "px"; - } - selectedAnnoteEl = annoteEl; - } - }; - const unselectCodeLines = () => { - const elementsIds = ["code-annotation-line-highlight", "code-annotation-line-highlight-gutter"]; - elementsIds.forEach((elId) => { - const div = window.document.getElementById(elId); - if (div) { - div.remove(); - } - }); - selectedAnnoteEl = undefined; - }; - // Attach click handler to the DT - const annoteDls = window.document.querySelectorAll('dt[data-target-cell]'); - for (const annoteDlNode of annoteDls) { - annoteDlNode.addEventListener('click', (event) => { - const clickedEl = event.target; - if (clickedEl !== selectedAnnoteEl) { - unselectCodeLines(); - const activeEl = window.document.querySelector('dt[data-target-cell].code-annotation-active'); - if (activeEl) { - activeEl.classList.remove('code-annotation-active'); - } - selectCodeLines(clickedEl); - clickedEl.classList.add('code-annotation-active'); - } else { - // Unselect the line - unselectCodeLines(); - clickedEl.classList.remove('code-annotation-active'); - } - }); - } - const findCites = (el) => { - const parentEl = el.parentElement; - if (parentEl) { - const cites = parentEl.dataset.cites; - if (cites) { - return { - el, - cites: cites.split(' ') - }; - } else { - return findCites(el.parentElement) - } - } else { - return undefined; - } - }; var bibliorefs = window.document.querySelectorAll('a[role="doc-biblioref"]'); for (var i=0; i

diff --git a/docs/chapters/evaluation_data_analysis.html b/docs/chapters/evaluation_data_analysis.html index 4f5cab7..4f21ca3 100644 --- a/docs/chapters/evaluation_data_analysis.html +++ b/docs/chapters/evaluation_data_analysis.html @@ -2,22 +2,21 @@ - + The Unjournal evaluations: data and analysis - 2  Evaluation data: description, exploration, checks @@ -100,25 +129,22 @@
-
-
-

+

2  Evaluation data: description, exploration, checks

@@ -194,7 +214,6 @@
-
@@ -230,12 +249,12 @@ options(mc.cores = parallel::detectCores())
-
+
@@ -378,8 +394,8 @@ )
-
- +
+
@@ -400,12 +416,12 @@

:::

-
+

Todo: 3

@@ -577,7 +593,7 @@ scale_fill_brewer(palette = "Set1") + scale_x_discrete(labels = function(x) str_wrap(x, width = 20))
-

+

The distribution of ratings and predictions

@@ -624,8 +640,8 @@ ggplotly(g1, tooltip = c("text"))
-
- +
+
@@ -636,12 +652,12 @@ -
+

Relationship among the ratings (and predictions)

-
+

Next chapter (analysis): aggregation of evaluator judgment

-
+