Skip to content

Commit

Permalink
fix: remove job flag to prevent multiple mlflow runs (#82)
Browse files Browse the repository at this point in the history
## 📥 Pull Request Description

This pull request fixes a bug, which created two experiment runs out of
one dagster pipeline run. This is a bug, caused by the dagster
itegration of MLFlow. The implementation in niceML was correct. When the
bug of this package is fixed in the future, niceML may has to be updated
as well.
With this fix, niceML, dagster and MLFlow work and only one MLFlow
experiment run is created per dagster pipeline run.

## 👀 Affected Areas

Dagster Pipeline
MLFlow integration

## 📝 Checklist

Please make sure you've completed the following tasks before submitting
this pull request:

- [x] Pre-commit hooks were executed
- [x] Changes have been reviewed by at least one other developer
- [ ] Tests have been added or updated to cover the changes (only
necessary if the changes affect the executable code)
- [x] All tests ran successfully
- [x] All merge conflicts are resolved
- [ ] Documentation has been updated to reflect the changes
- [ ] Any necessary migrations have been run
  • Loading branch information
dstalzjohn authored Oct 4, 2023
2 parents 2945f40 + 8c8b78f commit 31b3875
Showing 1 changed file with 1 addition and 3 deletions.
4 changes: 1 addition & 3 deletions niceml/dagster/jobs/jobs.py
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
"""Module containing all dagster jobs"""
from dagster_mlflow import mlflow_tracking, end_mlflow_on_run_finished
from dagster_mlflow import mlflow_tracking

from niceml.config.hydra import hydra_conf_mapping_factory
from niceml.dagster.ops.analysis import analysis
Expand Down Expand Up @@ -35,7 +35,6 @@ def job_data_generation():
df_normalization(current_data_location)


@end_mlflow_on_run_finished
@job(config=hydra_conf_mapping_factory(), resource_defs={"mlflow": mlflow_tracking})
def job_train():
"""Job for training an experiment"""
Expand All @@ -54,7 +53,6 @@ def job_train():
exptests(exp_context) # pylint: disable=no-value-for-parameter


@end_mlflow_on_run_finished
@job(config=hydra_conf_mapping_factory(), resource_defs={"mlflow": mlflow_tracking})
def job_eval():
"""Job for evaluating experiment"""
Expand Down

0 comments on commit 31b3875

Please sign in to comment.