Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[FEATURE] TaskDependencySensor flexibility Airflow sensor #154

Open
1 of 4 tasks
IMC07 opened this issue Sep 12, 2024 · 1 comment
Open
1 of 4 tasks

[FEATURE] TaskDependencySensor flexibility Airflow sensor #154

IMC07 opened this issue Sep 12, 2024 · 1 comment
Labels
enhancement New feature or request
Milestone

Comments

@IMC07
Copy link
Contributor

IMC07 commented Sep 12, 2024

Is your feature request related to a problem? Please describe.
Given the fact that in the taskdependencysensor we can only filter the API response on greater then or equal to timestamp-x, the workflow we make is non-deterministic, because when we would rerun a pipeline, the window of time changes because the end-time is 'now'.
We would like to be able to make this deterministic, to really be able to see if there was a succes within the timeframe we allow for our pipeline.

Cloud Information

  • AWS
  • Azure
  • GCP
  • Other

Describe the solution you'd like
I have created a simple, slightly ugly solution in order to stay backwards compatible, see PR. However, if you agree, I would like to refactor the sensor to allow for more flexibility on filtering results from the API.

Describe alternatives you've considered
Create a more flexible API filter as said above.

Additional context
Add any other context or screenshots about the feature request here.

@IMC07 IMC07 added the enhancement New feature or request label Sep 12, 2024
@asingamaneni
Copy link
Collaborator

asingamaneni commented Sep 16, 2024

Let's go with the patch right now and we can go for breaking change in the future releases.

@asingamaneni asingamaneni added this to the v1.3.0 milestone Oct 3, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

No branches or pull requests

2 participants