Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

PI objectives for the transformation pipeline #236

Open
wants to merge 36 commits into
base: dev
Choose a base branch
from
Open

PI objectives for the transformation pipeline #236

wants to merge 36 commits into from

Conversation

SwordSaintLancelot
Copy link

@SwordSaintLancelot SwordSaintLancelot commented Oct 16, 2024 •
edited
Loading

Summary: Summary of changes

Addresses (https://github.com/US-GHG-Center/ghgc-architecture/issues/398)

Changes

  • Added the code snipped to calculate statistics for netCDF file along with its corresponding COG
  • Create a JSON file for the statistics and add it to the S3 bucket where the COGs are stored.

@SwordSaintLancelot SwordSaintLancelot self-assigned this Oct 17, 2024
@SwordSaintLancelot SwordSaintLancelot marked this pull request as ready for review October 18, 2024 17:57
@SwordSaintLancelot SwordSaintLancelot changed the title Add statistics to the transformation pipeline Add PI objectives in the transformation pipeline Oct 28, 2024
@SwordSaintLancelot
Copy link
Author

Till this point, the statistics are being calculated and pushed as JSON file with the name of the file in the S3 bucket which resolved this issue.
Next commits will follow this mentioned in the PI objective.

ividito and others added 20 commits October 28, 2024 14:23
@ividito
feat:port event-driven vector automation to terraform
188f96e
@ividito
fix: feature flag for vector automation lambda
a2e3bad
@ividito
fix: adjust log group count
12ba68c
@smohiudd
fix tf vector automation vars
f532a6f
@smohiudd
update vars in event bridge tf
89d5251
@smohiudd
add new line
9c636e0
@smohiudd
fix tag filter
47b0707
@smohiudd
update vector subnet ref
8169ab6
@smohiudd
fix vector subnet ids
66cfb79
@smohiudd
conditional vector subnets
c449cb9
@ividito
WIP: dynamic ingest mapping for continuous tracing
0df795e
@ividito
feat: use taskflow for concurrent task runs
bfce3d5
@ividito
clean up debug code
31a3e84
@ividito
Bugfix vector ingest
a811fe2
@smohiudd
add generic vector pipeline
3c034ae
@smohiudd
remove process generic vector pipeline
4a5dc02
@ividito
Update dags/veda_data_pipeline/veda_generic_vector_pipeline.py
de8b69d
@ividito
Restructure dataset get_files output, add retries to some tasks to av…
d28432d 
…oid SIGKILL errors
@ividito
Changes to make vector ingest work in shared VPC environment
26feccf
@botanical
feat: disable default api gateway endpoint for workflows api
18470d0
@SwordSaintLancelot SwordSaintLancelot changed the title Add PI objectives in the transformation pipeline PI objectives for the transformation pipeline Oct 29, 2024
@SwordSaintLancelot
Copy link
Author

I tested the new dataset GOSAT-CH4 using the pipeline. The dataset was delivered in the delivery bucket on ghgc-smce AWS. The pipeline read the data from the AWS account, transformed it into COGs, and pushed the transformed COGs into the S3 bucket.

@SwordSaintLancelot SwordSaintLancelot mentioned this pull request Oct 30, 2024
Merged
4 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@amarouane-ABDELHAK amarouane-ABDELHAK Awaiting requested review from amarouane-ABDELHAK

At least 1 approving review is required to merge this pull request.

Assignees

@SwordSaintLancelot SwordSaintLancelot

Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
5 participants
@SwordSaintLancelot @ividito @smohiudd @botanical @amarouane-ABDELHAK