Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Enable convenient, shorter ingest definitions for single-asset items #87

Merged
merged 4 commits into from
Aug 16, 2023

Conversation

moradology
Copy link
Collaborator

Summary: Summary of changes

This PR is aimed at simplifying the definition of single-asset item ingests. If no assets field is provided to discovery, it will now assume that the older cog_default strategy is appropriate for registering item assets.

Changes

  • Fail faster and under more scenarios (in particular: failure to discover anything is now a DAG run failure)
  • Single asset ingests now look much more like they did in prior versions of the discovery DAG

PR Checklist

  • Update CHANGELOG
  • Ad-hoc testing - Deploy changes and test manually

items_with_assets = group_by_item(file_uris, id_regex, assets)

if len(file_uris) == 0:
raise ValueError(f"No files discovered at bucket: {bucket}, prefix: {prefix}")
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice!

@moradology moradology merged commit 4677d9c into dev Aug 16, 2023
12 checks passed
@ividito ividito deleted the feature/simple-single-asset-ingests branch May 29, 2024 17:13
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants