Real time deck support

[Yicheng-Lu-llll]

TL;DR

This PR adds real-time deck support.

After this PR, User can use flytekit.Deck.persist() to generate the html based on current collected metrics/information. So that user can see the real time deck even the task is running or has already failed. See the below example:

import flytekit
from flytekit import Resources, task, workflow
from flytekit.core.utils import timeit

@task(
    disable_deck=False,
    limits=Resources(mem="4Gi", cpu="1"),
)
def t1():
    import time

    for i in range(2):
        # timeit measure the time used in the block and shown in time line deck. See https://github.com/flyteorg/flytekit/pull/1581. 
        # Or you can add information to your own deck. See https://docs.flyte.org/projects/cookbook/en/latest/auto/core/flyte_basics/deck.html.
        with timeit(f"iteration {i}"):
            time.sleep(50)
        flytekit.Deck.persist()


@workflow
def wf():
    t1()


if __name__ == "__main__":
    wf()

real time deck support includes:

• flytekit: Real time deck support #1704
• flyteadmin: real time deck support flyteadmin#579
• flytepropeller: real time deck support flytepropeller#579
• flyteconsole: feat: real time deck support flyteconsole#781

Type

• Bug Fix
• Feature
• Plugin

Are all requirements met?

• Code completed
• Smoke tested
• Unit tests added
• Code documentation added
• Any pending items have an associated Issue

[fg91]

Just realizing we could use this feature to solve this problem: provide the ability to display links to e.g. wandb runs which are generated at runtime before the task ends 🎉

[cosmicBboy]

This also solves the problem with eager mode, i.e. we can re-render the Flyte deck of subtasks being kicked-off by the eager workflow in real-time.

[cosmicBboy]

I think self should be cls

[Yicheng-Lu-llll]

oh, I see. Thank you for pointing out!

[fg91]

How does this effort here relate to the discussion in flyteorg/flyte#3838 btw?
(cc @hamersaw)

[hamersaw]

@fg91 yeah, this is one of the issues that is effectively blocked, I would rather merge this with a re-usable flytekit -> flytepropeller communication mechanism.

[ketian-indeed]

Sorry to bump up on this PR, but this seems like a pretty useful feature to us. Is there any update on it? Thx!

[kumare3]

@ketian-indeed would love to understand the type of visualization you want

[ketian-indeed]

@kumare3, thanks for the quick response. I'm thinking of a use case like dumping a link (e.g. Datadog/WandB/MLflow links that are valuable to check while a model training is ongoing) to the Deck when a task starts and being able to see the link on the Deck while the task is running. I believe the Deck is currently shown once the task is completed - please correct me if I'm wrong.

[kumare3]

Hi @ketian-indeed we are infact adding support for W&B, Comet and neptune natively into flyte, Check this out. #2449

Also Datadog you should be able to add already using,
https://docs.flyte.org/en/latest/user_guide/productionizing/configuring_logging_links_in_the_ui.html#configure-logging

Please let me know what you think?

But, i also like the idea of Realtime decks - we will be adding support for this, but not like this PR. slightly different.

[ketian-indeed]

Thanks, @kumare3! Configuring logging links in the UI seems to be a better solution for some of our use cases. I'll test that out.
Meanwhile, I'm also looking forward to the real-time deck support, which I believe will provide even more flexibility for visualization.

[kumare3]

I think that is fair. We will prioritize this work in summer. But please add ideas / use cases here that way it is very easy to ensure we address. If you are open to contributing that would be awesome too.

[fg91]

But, i also like the idea of Realtime decks - we will be adding support for this, but not like this PR. slightly different.

I think that is fair. We will prioritize this work in summer. But please add ideas / use cases here that way it is very easy to ensure we address. If you are open to contributing that would be awesome too.

This, this, and this discussion are relevant in this context.

TL;DR multiple use cases are currently blocked by not having a real-time communication channel between a task pod and propeller, e.g. live updates in the eager mode deck or adding a log link dynamically at runtime:

@task
def train():
    run_url = my_experiment_tracking_api.init_run()
    flytekit.current_context().add_log_link(run_url)