Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add preprocessing child workflow #861

Conversation

jraddaoui
Copy link
Collaborator

@jraddaoui jraddaoui commented Feb 8, 2024

Allows to configure and trigger a custom preprocessing child workflow
to be handled by a different worker. This initial implementation requires
both workers to have access to the same filesystem to share the package.

Refs #886.

@jraddaoui jraddaoui force-pushed the dev/preprocessing-child-workflow branch from 7e71df8 to 6132a18 Compare March 18, 2024 12:10
@jraddaoui jraddaoui marked this pull request as draft March 18, 2024 12:15
@jraddaoui jraddaoui self-assigned this Mar 18, 2024
@jraddaoui jraddaoui force-pushed the dev/preprocessing-child-workflow branch 2 times, most recently from 1009691 to 11ee5a6 Compare April 18, 2024 10:04
Copy link

codecov bot commented Apr 18, 2024

Codecov Report

Attention: Patch coverage is 86.66667% with 6 lines in your changes are missing coverage. Please review.

Project coverage is 48.96%. Comparing base (f657aaf) to head (7d4f4dd).
Report is 1 commits behind head on main.

Files Patch % Lines
internal/workflow/processing.go 80.00% 3 Missing and 3 partials ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##             main     #861      +/-   ##
==========================================
+ Coverage   48.74%   48.96%   +0.22%     
==========================================
  Files         103      104       +1     
  Lines        5588     5620      +32     
==========================================
+ Hits         2724     2752      +28     
- Misses       2615     2617       +2     
- Partials      249      251       +2     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

@jraddaoui jraddaoui force-pushed the dev/preprocessing-child-workflow branch from 11ee5a6 to 9377948 Compare April 19, 2024 19:47
@jraddaoui jraddaoui changed the title WIP: Add preprocessing child workflow Add preprocessing child workflow Apr 19, 2024
@jraddaoui jraddaoui marked this pull request as ready for review April 19, 2024 23:41
Copy link
Collaborator Author

@jraddaoui jraddaoui left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@djjuhasz @sevein this is ready for a first review.

I was planning to include a remote option in this PR, but I think it's better to do that incrementally in another one, after we talk a little about #927.

Comment on lines +81 to +82
volume = {"name": "shared-dir", "persistentVolumeClaim": {"claimName": "preprocessing-pvc"}}
volume_mount = {"name": "shared-dir", "mountPath": "/home/enduro/preprocessing"}
Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I was planning to make the claimName and mountPath configurable through the environment, but I don't like this solution in general. I'll try to find something better while working on the remote option, maybe just working with the k8s manifests, ideas are welcome.

Comment on lines +195 to +202
### PREPROCESSING_PATH

Relative path to a preprocessing child workflow repository. It loads a Tiltfile
called `Tiltfile.enduro` from that repository and mounts a presistent volume
claim (PVC) in the preservation system pod. That PVC must be defined in the
preprocessing and be called `preprocessing-pvc`. Check the [Preprocessing child
workflow] docs to configure the child workflow execution.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This will be better documented (if it's still needed) in a preprocessing-base repository.

WatcherName string
Key string
WatcherName string
DestinationPath string
Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Check #926 for more info.

@jraddaoui jraddaoui force-pushed the dev/preprocessing-child-workflow branch from 9377948 to cddeb0c Compare April 20, 2024 00:04
Copy link
Collaborator

@djjuhasz djjuhasz left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks really good to me @jraddaoui. I had a few notes, but it's all pretty minor stuff.

docs/src/dev-manual/preprocessing.md Show resolved Hide resolved
internal/config/config.go Outdated Show resolved Hide resolved
internal/config/config.go Outdated Show resolved Hide resolved
internal/workflow/processing.go Outdated Show resolved Hide resolved
internal/workflow/processing_test.go Outdated Show resolved Hide resolved
@jraddaoui jraddaoui requested a review from djjuhasz April 22, 2024 20:34
Copy link
Collaborator

@djjuhasz djjuhasz left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

👍

Allows to configure and trigger a custom preprocessing child workflow
to be handled by a different worker. This initial implementation requires
both workers to have access to the same filesystem to share the package.
@jraddaoui jraddaoui force-pushed the dev/preprocessing-child-workflow branch from cc95b17 to 7d4f4dd Compare April 23, 2024 00:03
@jraddaoui jraddaoui merged commit db1a339 into artefactual-sdps:main Apr 23, 2024
14 checks passed
@jraddaoui jraddaoui deleted the dev/preprocessing-child-workflow branch April 23, 2024 00:10
@jraddaoui
Copy link
Collaborator Author

Thanks @djjuhasz!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants