Config reader with expected schema validation #7
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -33,6 +33,7 @@ Imports: | |
| duckdb, | ||
| EpiNow2 (>= 1.4.0), | ||
| jsonlite, | ||
| jsonvalidate, | ||
| rlang, | ||
| rstan, | ||
| tidybayes | ||
|
|
||
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,100 @@ | ||
| #' Fetch the config from an external resource | ||
| #' | ||
| #' This step is the first part of the modeling pipeline. It looks to Azure Blob | ||
| #' and downloads the Rt model run's config to the local config (if | ||
| #' `blob_storage_container` is specified), reads the config in from the | ||
| #' filesystem, and validates that it matches expectations. If any of these steps | ||
| #' fail, the pipeline fails with an informative error message. Note, however, | ||
| #' that a failure in this initial step suggests that something fundamental is | ||
| #' misspecified and the logs will likely not be preserved in a Blob Container if | ||
| #' running in Azure. | ||
| #' | ||
| #' The validation relies on `inst/data/config_schema.json` for validation. This | ||
| #' file is in `json-schema` notation and generated programatically via | ||
| #' https://www.jsonschema.net/. | ||
| #' | ||
|
There was a problem hiding this comment. General comment, I think these are all character strings but I find it helpful when reading documentation when the type is explicitly specified. |
||
| #' @param config_path The path to the config file, either in the local | ||
| #' filesystem or with an Azure Blob Storage container. If | ||
| #' `blob_storage_container` is specified, the path is assumed to be within | ||
| #' the specified container otherwise it is assumed to be in the local | ||
| #' filesystem. | ||
| #' @param local_dest The local directory to write the config to when downloading | ||
|
There was a problem hiding this comment. What happens if |
||
| #' it from `blob_storage_container`. This argument is ignored unless | ||
| #' `blob_storage_container` is specified. | ||
| #' @param blob_storage_container The storage container holding the config at | ||
| #' `config_path` | ||
| #' @param config_schema_path The path to the file holding the schema for the | ||
| #' config json for the validator to use. | ||
| #' | ||
| #' @return A list of lists, the config for the run. | ||
| #' @export | ||
| fetch_config <- function( | ||
| config_path, | ||
| local_dest, | ||
| blob_storage_container, | ||
| config_schema_path = system.file("extdata/config_schema.json", | ||
natemcintosh marked this conversation as resolved.
Show resolved
Hide resolved
|
||
| package = "CFAEpiNow2Pipeline" | ||
| )) { | ||
| if (!rlang::is_null(blob_storage_container)) { | ||
| download_from_azure_blob( | ||
| config_path, | ||
| local_dest, | ||
| container_name = blob_storage_container | ||
| ) | ||
| } else { | ||
| cli::cli_alert( | ||
| "No blob storage container provided. Reading from local path." | ||
| ) | ||
| } | ||
|
|
||
| cli::cli_alert_info("Loading config from {.path {config_path}}") | ||
| validate_config(config_path, config_schema_path) | ||
|
|
||
| config <- rlang::try_fetch( | ||
| jsonlite::read_json(config_path), | ||
| error = function(con) { | ||
| cli::cli_abort( | ||
| "Error loading config from {.path {config_path}}", | ||
| parent = con, | ||
| class = "CFA_Rt" | ||
| ) | ||
| } | ||
| ) | ||
|
|
||
| return(config) | ||
| } | ||
|
|
||
| #' Compare loaded JSON against expectation in `inst/data/config-schema.json` | ||
| #' | ||
| #' @inheritParams fetch_config | ||
| #' @return NULL, invisibly | ||
| #' @export | ||
| validate_config <- function( | ||
| config_path, | ||
| config_schema_path = system.file("extdata/config_schema.json", | ||
| package = "CFAEpiNow2Pipeline" | ||
| )) { | ||
| is_config_valid <- rlang::try_fetch( | ||
| jsonvalidate::json_validate( | ||
| json = config_path, | ||
| schema = config_schema_path, | ||
| engine = "ajv", | ||
| verbose = TRUE, | ||
| greedy = TRUE, | ||
| error = TRUE | ||
| ), | ||
| error = function(con) { | ||
| cli::cli_abort( | ||
| c( | ||
| "Error while validating config", | ||
| "!" = "Config path: {.path {config_path}}", | ||
| "!" = "Schema path: {.path {config_schema_path}}" | ||
| ), | ||
| parent = con, | ||
| class = "CFA_Rt" | ||
| ) | ||
| } | ||
| ) | ||
|
|
||
| invisible(is_config_valid) | ||
| } | ||
Empty file.
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
"fundamental" is quite vague