Add preprocessing child workflow #861
Conversation
jraddaoui
force-pushed
the
dev/preprocessing-child-workflow
branch
from
7e71df8
to
6132a18
Compare
jraddaoui
force-pushed
the
dev/preprocessing-child-workflow
branch
2 times, most recently
from
1009691
to
11ee5a6
Compare
Codecov ReportAttention: Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## main #861 +/- ##
==========================================
+ Coverage 48.74% 48.96% +0.22%
==========================================
Files 103 104 +1
Lines 5588 5620 +32
==========================================
+ Hits 2724 2752 +28
- Misses 2615 2617 +2
- Partials 249 251 +2 ☔ View full report in Codecov by Sentry. |
jraddaoui
force-pushed
the
dev/preprocessing-child-workflow
branch
from
11ee5a6
to
9377948
Compare
jraddaoui
changed the title
| volume = {"name": "shared-dir", "persistentVolumeClaim": {"claimName": "preprocessing-pvc"}} | ||
| volume_mount = {"name": "shared-dir", "mountPath": "/home/enduro/preprocessing"} |
There was a problem hiding this comment.
I was planning to make the claimName and mountPath configurable through the environment, but I don't like this solution in general. I'll try to find something better while working on the remote option, maybe just working with the k8s manifests, ideas are welcome.
| ### PREPROCESSING_PATH | ||
|
|
||
| Relative path to a preprocessing child workflow repository. It loads a Tiltfile | ||
| called `Tiltfile.enduro` from that repository and mounts a presistent volume | ||
| claim (PVC) in the preservation system pod. That PVC must be defined in the | ||
| preprocessing and be called `preprocessing-pvc`. Check the [Preprocessing child | ||
| workflow] docs to configure the child workflow execution. | ||
|
|
There was a problem hiding this comment.
This will be better documented (if it's still needed) in a preprocessing-base repository.
| WatcherName string | ||
| Key string | ||
| WatcherName string | ||
| DestinationPath string |
There was a problem hiding this comment.
Check #926 for more info.
jraddaoui
force-pushed
the
dev/preprocessing-child-workflow
branch
from
9377948
to
cddeb0c
Compare
There was a problem hiding this comment.
Looks really good to me @jraddaoui. I had a few notes, but it's all pretty minor stuff.
Allows to configure and trigger a custom preprocessing child workflow to be handled by a different worker. This initial implementation requires both workers to have access to the same filesystem to share the package.
jraddaoui
force-pushed
the
dev/preprocessing-child-workflow
branch
from
cc95b17
to
7d4f4dd
Compare
|
Thanks @djjuhasz! |
Allows to configure and trigger a custom preprocessing child workflow
to be handled by a different worker. This initial implementation requires
both workers to have access to the same filesystem to share the package.
Refs #886.