Linking to container image via --template when creating AZ Batch pool

[gvegayon]

This pull request includes significant updates to the Azure Batch Pool setup process, including adding a new Python script for pool creation and configuration and updates to the workflow and dependencies.

Azure Batch Pool Setup Enhancements:

• Added a new Python script azure/pool.py to handle Azure Batch Pool creation and configuration using a TOML configuration file. This script includes functions for creating containers, reading autoscale formulas, and setting up the batch pool with the necessary credentials and configurations.
• Updated the workflow file .github/workflows/containers-and-az-pool.yaml to run the new Python script, replacing the previous direct Azure CLI commands. This includes steps for writing the configuration file, ensuring the Azure CLI is installed, and running the Python script. [1] [2] [3]

Documentation and Dependency Updates:

• Updated NEWS.md to include a note about the new script for setting up the Azure Batch Pool.
• Added new dependencies to azure/requirements.txt required by the new Python script, including azure-common, azure-core, azure-identity, azure-keyvault, azure-mgmt-batch, and toml.

[codecov]

Codecov Report

All modified and coverable lines are covered by tests ✅

Additional details and impacted files

📢 Thoughts on this report? Let us know!

---

🚨 Try these New Features:

• Flaky Tests Detection - Detect and resolve failed and flaky tests

[github-actions]

Thank you for your contribution @gvegayon 🚀! Your pkgdown-site is ready for download 👉 here 👈!
_{(The artifact expires on 2024-11-29T20:59:37Z. You can re-generate it by re-running the workflow here.)}

[gvegayon]

So we are getting closer @jkislin and @zsusswein. There's an issue with the generated toml file that is makes it fail during the read process:

    config = toml.load(config_file)
             ^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/toml/decoder.py", line 134, in load
    return loads(ffile.read(), _dict, decoder)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/toml/decoder.py", line 514, in loads
    raise TomlDecodeError(str(err), original, pos)
toml.decoder.TomlDecodeError: Found tokens after a closed string. Invalid TOML. (line 10 column 1 char 754)

I'll try to debug it tomorrow, but I think we are almost there!

[gvegayon]

I am getting errors on testthat b/c of duckdb. The issue is intermittent, so either we are missing a seed, or there's something else going on. https://github.com/CDCgov/cfa-epinow2-pipeline/actions/runs/11943535928/job/33293099360#step:3:309. I have had to restart the job many times in the past, with the second try working. Unsure what's the issue

attn @natemcintosh @zsusswein

[zsusswein]

That's weird. I've seen an intermittently flaky test from DuckDB but it's rare and I thought I had resolved it.

Can you open an issue for it and I'll look into it?

EDIT: #97

[jkislin]

@gvegayon @zsusswein do we want to merge this now?

[zsusswein]

Is the PR ready to go? I had been assuming it was waiting for something because it was still draft.

[gvegayon]

Given that the tests pass, this is ready for review @zsusswein and @jkislin!

[zsusswein]

Great work George! Thanks for pushing this through.

Marking to-dos as I understand them here. I'll make these into issues for future fixes:

• Clean up the config toml and distribute differently (and encrypted)
• Move the python action into it's own docker container with its own dependencies than a requirements in this repo
• Move this action over to the cfa actions repo
• take out the ABS stuff
• clean up the python code

If there's anything else can you make it into a new issue?