Working azure download flow

[zsusswein]

This one turned into a bit of a mad dash to get everything ready for the demo. Everyone was able to run the pipeline locally and in Batch (success!) and I was able to run this branch in a shadow production run today. But now the branch has entirely too much content.

 I'd like to get this PR open and the functionality onto main rather than polishing a long-lived feature branch. If people are on board, that would mean liberally turning feedback into issues and circling back in a series of future PRs.

Thanks for bearing with me on this one folks!

[github-actions]

Thank you for your contribution @zsusswein 🚀! Your pkgdown-site is ready for download 👉 here 👈!
_{(The artifact expires on 2024-12-18T18:05:14Z. You can re-generate it by re-running the workflow here.)}

[codecov]

Codecov Report

Attention: Patch coverage is 46.34146% with 44 lines in your changes missing coverage. Please review.

Files with missing lines	Patch %	Lines
R/azure.R	14.28%	36 Missing ⚠️
R/pipeline.R	80.00%	8 Missing ⚠️

Additional details and impacted files

📢 Thoughts on this report? Let us know!

[kgostic]

Great job @zsusswein! Some notes from the demo. Some of these may be separate issues that we deal with later:

• Roll back .env and config workarounds in demo instructions (after these are fixed)
• Would be nice to add a mermaid diagram of all the relevant repos, environments, and microservices and how they connect
• Would be nice to document in each step - from where is this step reading inputs and writing outputs (this would help with troubleshooting)

[zsusswein]

FYI @amondal2 I tagged you in case you want to take a look, but the azure stuff is kind of a mess. I'm planning on cleaning it up once this merges and tagging you and Nate to take a look.

[natemcintosh]

Suggested change

	task_configs = []
	blobs = container_client.list_blobs()
	for blob in blobs:
	if blob.creation_time > two_mins_ago:
	task_configs.append(blob.name)
	task_configs: list[str] = [
	b.name
	for b in container_client.list_blobs()
	if b.creation_time > two_mins_ago
	]

Feel free to ignore. Just adding bc I like list comprehensions.

[zsusswein]

Some notes from the demo. Some of these may be separate issues that we deal with later:

Tracking in #116

[natemcintosh]

Will this break if we try and run it on a different branch?

[natemcintosh]

Suggested change

	Rscript -e "CFAEpiNow2Pipeline::orchestrate_pipeline('$(CONFIG)', config_container = 'rt-epinow2-config', input_dir = '/cfa-epinow2-pipeline/input', output_dir = '/cfa-epinow2-pipeline', output_container = 'zs-test-pipeline-update')"
	Rscript -e "CFAEpiNow2Pipeline::orchestrate_pipeline('$(CONFIG)', config_container = 'rt-epinow2-config', input_dir = '/cfa-epinow2-pipeline/input', output_dir = '/', output_container = 'zs-test-pipeline-update')"

Use the same output dir as for the run-batch command. This way, the file paths in the metadata will always be the same as the those created by the batch command, which is desirable for maintaining consistency in output folder structure. However, this would require changing the bind mount above to mount to the container's root, instead of /cfa-epinow2-pipeline.

What do you think?

[zsusswein]

I like the point about consistency, but I think we should follow the pattern of this one, not run-batch. We want the local run to write into the bind mount, which this edit would prevent.

[natemcintosh]

But what if we change the bind mount to be / inside the container for this make command?