[Core feature] Flytekit should support unsafe mode for types

[Mecoli1219]

Tracking issue

Related to flyteorg/flyte#5319

Why are the changes needed?

Use Case: When the user wants to apply Flytekit to their Python codebase, but annotating the type is too hard or time-consuming, we should support unsafe mode for easy usage.

What changes were proposed in this pull request?

1. I add unsafe kwarg to both task and workflow, and the default value of unsafe is False
2. If the unsafe kwarg is True and the types of inputs or outputs are not specified (which is None), replace the type from None to Optional[Any]. This will trigger FlytePickleTransformer to handle each unspecified type.

How was this patch tested?

Two tests are created to examine the ability of unsafe kwarg.

1. test_unsafe_input_wf_and_task: This test mainly examines the unspecified input types. If the unsafe is False, the workflow or task should throw errors. Otherwise, it should succeed.
2. test_unsafe_wf_and_task: This test mimics the use case mentioned above, where the user might not specify all the types in their codebase. The workflow should succeed if the unsafe kwarg of all tasks and workflow are set to True.

Setup process

Screenshots

Check all the applicable boxes

• I updated the documentation accordingly.
• All new and existing tests passed.
• All commits are signed-off.

Related PRs

flyteorg/flyte#5319

Docs link

[pingsutw]

Did you test this workflow in the sandbox cluster?

[Mecoli1219]

No, I only test it on my local environment. Can I ask how to try it?

[Future-Outlier]

This PR can help you, you can either use imageSpec or Dockerfile to create your own image.
#1870

[Mecoli1219]

from flytekit import task, Secret, workflow, ImageSpec

flytekit_dev_version = "https://github.com/Mecoli1219/flytekit.git@c7c9a2ad2bdf6799ec324955a023281cca030038"
image = ImageSpec(
    registry="localhost:30000",
    platform="linux/amd64",
    apt_packages=["git"],
    packages=[
        f"git+{flytekit_dev_version}",
    ],
)


@task(unsafe=True, container_image=image)
def t1(a) -> int:
    if type(a) == int:
        return a + 1
    return 0


@workflow(unsafe=True)
def wf1_with_unsafe(a) -> int:
    return t1(a=a)

[Mecoli1219]

The above code will fail. The reason is detailed in flyteorg/flyte#5261.

[Mecoli1219]

from flytekit import task, Secret, workflow, ImageSpec
from typing import Tuple

flytekit_dev_version = "https://github.com/Mecoli1219/flytekit.git@9885189b2af95c0c0dfa94ff92b71e865d95cf80"
image = ImageSpec(
    registry="localhost:30000",
    platform="linux/amd64",
    builder="fast-builder",
    apt_packages=["git"],
    packages=[
        f"git+{flytekit_dev_version}",
    ],
)


@task(unsafe=True, container_image=image)
def t1(a) -> int:
    if type(a) == int:
        return a + 1
    return 0


@workflow
def wf1_with_unsafe() -> Tuple[int, int, int]:
    a1 = t1(a=1)
    a2 = t1(a="1")
    a3 = t1(a=None)    #! This will fail
    return a1, a2, a3

Currently, the input with a value None will fail, but the issue is solved with this PR (flyteorg/flyte#5408).

[Mecoli1219]

@pingsutw I created 2 new unit tests, and both of them passed.

[codecov]

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 74.99%. Comparing base (4bef161) to head (7db0869).
Report is 5 commits behind head on master.

Additional details and impacted files

@@             Coverage Diff             @@
##           master    #2419       +/-   ##
===========================================
+ Coverage   46.48%   74.99%   +28.51%     
===========================================
  Files         199      279       +80     
  Lines       20707    24265     +3558     
  Branches     2665     2679       +14     
===========================================
+ Hits         9625    18197     +8572     
+ Misses      10610     5341     -5269     
- Partials      472      727      +255     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[thomasjpfan]

I prefer to have a more explicit name. Something like:

Suggested change

	unsafe: bool = ...,
	pickle_untyped: bool = ...,

[thomasjpfan]

@kumare3 Since you proposed the safe parameter name, are you okay with using pickle_untyped?

My concerned is that safe=False could mean so many unsafe behavior. pickle_untyped is explicit about the behavior.

[thomasjpfan]

Can this be removed?

[Mecoli1219]

yes sure. I'll remove that recently

[Mecoli1219]

NVM, it is still the same problem as mentioned in the comment above 6 months ago.

@task(pickle_untyped=True, container_image=image)
def t1(a) -> int:
    if type(a) == int:
        return a + 1
    return 0

@workflow
def wf1_with_unsafe() -> Tuple[int, int, int]:
    a1 = t1(a=1)
    a2 = t1(a="1")
    a3 = t1(a=None)    #! This will fail
    return a1, a2, a3

Currently, the input with a value None will fail

[Mecoli1219]

First example

@task(pickle_untyped=True, container_image=image)
def t1(a) -> int:
    if type(a) == int:
        return a + 1
    return 0


@workflow
def wf1_with_unsafe() -> Tuple[int, int, int]:
    a1 = t1(a=1)
    a2 = t1(a="1")
    # a3 = t1(a=None)  #! This will fail
    return a1, a2, a2

Second example

@task(pickle_untyped=True, container_image=image)
def t1(a):
    if type(a) == int:
        return a + 1
    return 1


@workflow
def wf1_with_unsafe() -> Tuple[Any, Any, Any]:
    a1 = t1(a=1)
    a2 = t1(a="1")
    # a3 = t1(a=None)  #! This will fail
    return a1, a2, a2

[Mecoli1219]

I revert the support for None type in pickle. I think we could discuss that later and solve that in another PR if finally decided to do that.