Adds comet-ml plugin example

[thomasjpfan]

This PR adds a comet-ml example.

To fully run the example, you need to use this ImageSpec, because the plugin is in a PR:

comet_ml_package = (
    "git+https://github.com/thomasjpfan/flytekit.git@"
    "d04d0129f99eb7e7edcffeae04d4f401f49be6d6#subdirectory=plugins/flytekit-comet-ml"
)

image = ImageSpec(
    builder="fast-builder",
    name="unionai",
    apt_packages=["git"],
    packages=[
        "torch==2.3.1",
        "comet-ml==3.43.2",
        "lightning==2.3.0",
        comet_ml_package,
        "unionai==0.1.43",
        "torchvision",
    ],
    registry="ghcr.io/thomasjpfan",
)

flyteorg/flytekit#2550

[Future-Outlier]

We don't need the unionai dependency, right?
Should we add this in Union's doc instead of Open Source's doc?

[thomasjpfan]

This was a mistake. unionai is not needed.

[Future-Outlier]

same as below, we should use ghcr.io/flyteorg

[thomasjpfan]

For ImageSpec, most users will not have push access to ghcr.io/flyteorg, so they will need to change the registry to run it.

Using localhost:30000 means I am optimizing for the sandbox use case. An alternative is to use os.getenv("REGISTRY", "localhost:30000"), and let it be an environment variable.

[Future-Outlier]

No problem, I like your idea

[Future-Outlier]

LGTM

[Future-Outlier]

No problem, I like your idea

[Future-Outlier]

We should add double quote here.

[Future-Outlier]

same thing

[neverett]

Left a few suggestions, otherwise LGTM

[neverett]

Suggested change

	Comet’s machine learning platform integrates with your existing infrastructure and tools so you can manage, visualize, and optimize models—from training runs to production monitoring. This plugin integrates Flyte with Comet by configuring links between the two platforms.
	Comet’s machine learning platform integrates with your existing infrastructure and tools so you can manage, visualize, and optimize models from training runs to production monitoring. This plugin integrates Flyte with Comet by configuring links between the two platforms.

[neverett]

Suggested change

	# tools so you can manage, visualize, and optimize models—from training runs to
	# tools so you can manage, visualize, and optimize models from training runs to

[neverett]

Suggested change

	# Please update `PROJECT_NAME` and `WORKSPACE` to the value associated with your account.
	# Please update `PROJECT_NAME` and `WORKSPACE` to the values associated with your account.

[thomasjpfan]

@neverett The comet-ml check is passing now: https://github.com/flyteorg/flytesnacks/actions/runs/10182511543/job/28165251450?pr=1702

There are a bunch other tests that are failing tho.

[neverett]

@thomasjpfan can you try merging master in and rerunning?

[fiedlerNr9]

Running the example given in this PR was not working for me with ImageSpec.
MisconfigurationException: CometLogger requires either api_key or save_dir during initialization.
Execution on Dogfood

I had to add the following command to create a home dir & assign permissions for the flytekit user ->

image = ImageSpec(
    name="comet-ml",
    packages=[
        "torch==2.3.1",
        "comet-ml==3.43.2",
        "lightning==2.3.0",
        "flytekitplugins-comet-ml",
        "torchvision",
    ],
    commands=["mkdir -p /home/flytekit", "chown -R flytekit /home/flytekit"],
    registry=REGISTRY,
)

Anyone experiencing the same?

[thomasjpfan]

For this example to work, it needs ImageSpec(builder="default", ...). The envd builder does not create a home directory for it's user.