[1/2]: Read in case data

[zsusswein]

Important

Do not merge. Using a stacked PR workflow.

Read the case data from a local parquet file in our standard ETL output schema. The various errors & warnings are linked to the tests via classes.

This approach preserves our existing handling of the data with respect to exclusions: it aggregates before there have been any exclusions. Applying exclusions to the aggregated data will be handled in the second stack in this PR.

I also add in here the data from Gostic, 2020. I use it in the unit tests to practice reading in data from tests/testthat/data/ but I also add it as package data (with documentation). We use this dataset in our end-to-end testing, so its addition here is also meant with an eye to that future use.

[codecov]

Codecov Report

All modified and coverable lines are covered by tests ✅

Additional details and impacted files

📢 Thoughts on this report? Let us know!

[natemcintosh]

So calling /document causes it to run the documentation CI for you?

[zsusswein]

Yes!

[athowes]

Just superficial things! (I think reading this is helping me get up to speed nonetheless)

[athowes]

This approach preserves our existing handling of the data with respect to exclusions: it aggregates before there have been any exclusions. Applying exclusions to the aggregated data will be handled in the second stack in this PR.

Just to note without context here that this sounds odd to me. If you aggregate data it'd seem to be harder to then retroactively go in and remove data points from it, rather than do the removing data first then to aggregate. I guess you must have a reason to be doing this. Since I guess now you'd have to be tracking somehow in the code all of the entries that went into a particular aggregate so that if you remove data from them you can remove data from the aggregate. (As in this seems more challenging than it need be.)

[zsusswein]

Yeah this is a good flag. I don't know if this repo is the right place to document it, but here's the gist of our ongoing discussion around data exclusions:

• We receive and store facility-level data. These facilities are tagged with their corresponding state.
• We want to produce state- and national-level estimates, so on-the-fly aggregate our stored data up to the desired level (this PR!)
• We include right-truncation in reporting in the model. We provide a right-truncation PMF based on historical geographic-aggregate-level right-truncation.
• Sometimes one geographic aggregate or another will have a one-off week where right-truncation empirically deviates from historical trend (e.g. 0 cases reported when we normally see ~some)
• In those cases, we consult with the dataset expert and sometimes drop the single day with anomalous reporting for that particular geographic aggregate by converting the aggregate point to NA.
• Sometimes the anomalous reporting is widespread enough that we judge that the modeling is not reflective of disease trends and exclude the whole state's model fit for the week.
• This exclusion does not propagate to other geographic aggregates (e.g. the US overall aggregate) or future report dates. All data are included, even if a different aggregate using a portion of the same data is excluded.

With the key question being, should the NA propagate? When we exclude a state's point, should we also exclude it from the US overall? What about when we exclude a whole state's time series?

We've landed on "No, we should make judgement calls about each aggregated time series individually." But this discussion is still live and we're trying to get to a better place from both the data review/cleaning side and the modeling side.

[kgostic]

Overall looks good!