Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add PythonOperator #35

Merged
merged 3 commits into from
Oct 15, 2024
Merged

Add PythonOperator #35

merged 3 commits into from
Oct 15, 2024

Conversation

kreczko
Copy link
Contributor

@kreczko kreczko commented Oct 15, 2024

Fixes #12

  • add fasthep_flow.operators.PythonOperator
  • add ResultType for validating output

ResultType currently does nothing, need to see how chaining functions will interact here before further implementation

@kreczko kreczko merged commit 4dd18d4 into main Oct 15, 2024
9 checks passed
@kreczko kreczko deleted the kreczko-issue-12 branch October 15, 2024 16:09
@YeungOnion
Copy link

Rust has a pattern for handling results that are mutually exclusive success or failure, stderr poses some distinction here.

I got stuck in the, "how do I do this in a way that handles module aliases?" and then never got back to it. Can I ask how you test the example? I'm expecting to test with hatch

@kreczko
Copy link
Contributor Author

kreczko commented Oct 21, 2024

how do I do this in a way that handles module aliases?
Also got stuck here, not quite sure atm. Currently trying some things to get numpy work as np.

Can I ask how you test the example?
Currently running

fasthep-flow execute docs/examples/hello_world.yaml

in a conda env (python 3.12).
As this projects matures I will add tests to run with

pipx run nox -s tests

including the examples (hello world + CMS open data)

The result dict will likely change as I am looking at https://hamilton.dagworks.io/en/latest/reference/result-builders/Generic/ and the other variants. I would like something that is not dependent on hamilton, but can pass on enough metadata to convert correctly if hamilton is picked. Since awkward array has integrations with Dask, a custom ResultBuilder would be needed at some point.

Long story short, this project is still in alpha - trying to find its sweet-spot for implementation. Expect occasionally some incomplete functionality.
That said, I finally have some time to push towards a beta that matches the current fast-flow release in terms of features.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Python Operator
2 participants