Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[BUG] Outcome is applied for all types of tasks, but should be only for IF_ELSE_CONDTION task type #172

Open
mikita-sakalouski opened this issue Oct 18, 2024 · 0 comments
Labels
bug Something isn't working

Comments

@mikita-sakalouski
Copy link
Contributor

Describe the bug
When you are putting the if_else_condition_task type in the depends_on together with normal task, the outcome will be applied for all tasks and it will raise terraform error
Error: cannot update job: The dependency on a non-if/else condition "not_condition_task" specifies an outcome "false". Outcomes can only be specified for if/else condition dependencies.

  @property
  def depends_on_names(self) -> Iterator[Dict[str, Optional[str]]]:
      for i in self.depends_on:
          # Here is the issue, the check for task type is required
          if self.if_else_outcome:
              outcome = list(self.if_else_outcome.values())[0]
          else:
              outcome = None

          if callable(i) and hasattr(i, "__name__"):
              yield {i.__name__: outcome}
          else:
              yield {str(i): outcome}

To Reproduce
Steps to reproduce the behavior:

  1. Create workflow
wf = Workflow(
    "test",
    default_cluster=Cluster.from_existing_cluster("existing_cluster_id"),
    schedule_quartz_expression="* * * * *",
    permissions=WorkflowPermissions(
        owner=User("[email protected]"),
        can_manage_run=[User("[email protected]")],
        can_view=[User("[email protected]")],
        can_manage=[User("[email protected]")],
    ),
    run_as_user="[email protected]",
    tags={"test": "test2"},
    common_task_parameters={"all_tasks1": "test", "all_tasks3": "123"},  # type: ignore
    health={
        "rules": [
            {"metric": "RUN_DURATION_SECONDS", "op": "GREATER_THAN", "value": 7200.0}
        ]
    },  # type: ignore
    trigger={
        "file_arrival": {"url": "<my_url>"},
        "pause_status": "UNPAUSED",
    },  # type: ignore
    parameters=[
        JobsParameters(
            default="value1",
            name="wf_param1",
        )
    ],
)
  1. Create several if_condition_task
@workflow.if_else_condition_task()
def is_local():
    return IfElseConditionTask(left="{{job.parameters.environment}}", op="==", right="local")
  1. Create several normal tasks and put dependencies
@workflow.spark_python_task(
)
def not_condition_task():
    return build_spark_python_task()


@workflow.spark_python_task(
    depends_on=[not_condition_task, is_local], if_else_outcome={"is_local": "false"}
)
def sync():
    return build_spark_python_task()
  1. Run brickflow deploy
  2. As result you will get error
Error: terraform apply: exit status 1

Error: cannot update job: The dependency on a non-if/else condition "not_condition_task" specifies an outcome "false". Outcomes can only be specified for if/else condition dependencies.

Expected behavior
Outcome should be applied correctly only for IF_ELSE_CONDITION tasks.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
None yet
Development

No branches or pull requests

1 participant