Introduce image Container Registry use

[tarilabs]

Ref KF community meeting 2024-02-27

We note the KF community feedback suggesting to make use of GitHub Container Registry (part of "GitHub Packages") for publishing Container Images using GitHub Actions. Eventually, GH CR images can be synced to DockerHub https://hub.docker.com/u/kubeflow by raising request to KF community.

This issue tracks steps required to enable Container Image production and publication, similarly to what was available in ODH+Quay.

Acceptance criteria:

• ensure this GHA works as expected: https://github.com/kubeflow/model-registry/blob/main/.github/workflows/build-image-pr.yml
• modify this GHA and related script to publish with _ dockerhub Container Registry as needed: https://github.com/kubeflow/model-registry/blob/main/.github/workflows/build-image-pr.yml
  (this GHA and related script at the time of writing were used to publish to Quay)
• re-enable this workflow: https://github.com/kubeflow/model-registry/actions/workflows/build-and-push-image.yml
• verify Container Image get published in Container Registry

Stretch goals:

• modify Docker file, makefile, etc to account for cross-platform build (see comment for examples)
• ask KF community leads to enable sync from GH CR to DockerHub for model-registry

[thesuperzapper]

@tarilabs since we are starting out fresh, can you please ensure you cross-compile for ARM64 in addition to AMD64?

Depending on how many container images you plan to have, you may or may not need something as complex as what we do for Kubeflow Notebooks. But either way, here are some links you might find useful that show using QEMU emulation and Docker BuildX:

• GitHub Actions:
  • GHA Template (workflow_call):
    • https://github.com/kubeflow/kubeflow/blob/master/.github/workflows/example_notebook_servers_publish_TEMPLATE.yaml
  • GHA Root:
    • https://github.com/kubeflow/kubeflow/blob/master/.github/workflows/example_notebook_servers_publish.yaml
• Makefile:
  • Makefile Template:
    • https://github.com/kubeflow/kubeflow/blob/master/components/example-notebook-servers/common.mk
  • Makefile Root:
    • https://github.com/kubeflow/kubeflow/blob/master/components/example-notebook-servers/Makefile

[tarilabs]

@thesuperzapper that's a very good point Mathew thanks especially for sharing examples of cross-platforms container image build!

While we have an extensive Integration Test suite, e2e testing on arm will be limited atm due to the MLMD image unavailability (for arm). That accounted for, we can start early (as you pointed out) to account for cross-platform build so we can be prepared when all components are available on arm imho.

[thesuperzapper]

@tarilabs well... I have a Christmas present for you regarding ARM64 support in the MLMD image.

I made a PR upstream to introduce ARM64 support, and also made a fork and published an ARM image (built under GHA emulation) for my own project (deployKF) until it gets merged upstream.

For more information, please see:

• My ml-metadata PR:
  • allow building ml_metadata_store_server image on ARM64 google/ml-metadata#188
• The associated KFP Issue:
  • build and publish ml_metadata_store_server container image for ARM64 pipelines#10308
• The deployKF fork of the ml-metadata repo:
  • https://github.com/deployKF/ml-metadata
• Forked gcr.io/tfx-oss-public/ml_metadata_store_server image (with ARM/AMD):
  • ghcr.io/deploykf/ci/ml_metadata_store_server:sha-cad0c56

This is all assuming you are talking about ghcr.io/deploykf/ci/ml_metadata_store_serve image needing ARM64 support, and not some other part of ml-metadata (which may actually already support it).

[tarilabs]

@thesuperzapper I well recall your Xmas present :) we kind-of crossed path because on other hand I've been trying to look into the Python side of the library, ie: https://pypi.org/project/ml-metadata

For more information:

• this is upstream contributors work to make Python lib available on Macs:
  • Add support for M1 macs google/ml-metadata#143 (comment)
  • this sounds promising but until merged I couldn't make use of those contributors' fix
• this is a workaround which works fine for development on Mac, for as long as you only need grpc-remote connection to the MLMD server via the MLMD Python client:
  • https://github.com/kubeflow/model-registry/blob/main/CONTRIBUTING.md#remote-only-mlmd-python-library

  • model-registry/clients/python/pyproject.toml

    Lines 17 to 18 in 247c23c

    	# you might consider using locally the following alternative, when developing on Apple-silicon/ARM-based computers:
    	# ml-metadata = { url = "https://github.com/opendatahub-io/ml-metadata/releases/download/v1.14.0%2Bremote.1/ml_metadata-1.14.0+remote.1-py3-none-any.whl" }

• this is an alternative work-around which again only works if you just need grpc-remote connection to the MLMD server via the MLMD Python client:
  • https://github.com/tarilabs/ml-metadata-remote

[rimolive]

Although I'd like to see ARM64 support for all components, I prefer to treat it as a P2 since this component is currently in alpha for Kubeflow 1.9. We still have open issues for 1.9 that need some attention, so unless we can handle everything for 1.9 we can prioritize ARM64 support for future releases.

[tarilabs]

dockerhub token should be setup for this repo, as:

DOCKERHUB_USERNAME
DOCKERHUB_TOKEN

[rchincha]

Hi All, not sure if folks here are aware of the work going on over at Open Containers Initiative (OCI) wrt standardizing registry interfaces and general-purpose "artifacts".

https://github.com/opencontainers/distribution-spec

Apart from specific registry implementations, maybe also include a "OCI conformant registry"

[tarilabs]

@rchincha thank you for reaching out, I invite you to comment the above on the Model Registry proposal for Kubeflow, here: kubeflow/community#682

There is recurrent community call for Kubeflow: https://www.kubeflow.org/docs/about/community/#kubeflow-community-call and I strongly encourage you to engage at that level to spread best awareness of your initiative.

There is also bi-weekly call for Model Registry, as you can find in the above linked calendar.

Hope this helps! :)

[tarilabs]

• ensure this GHA works as expected: https://github.com/kubeflow/model-registry/blob/main/.github/workflows/build-image-pr.yml
• verify Container Image get published in Container Registry

demonstrated by: https://hub.docker.com/layers/kubeflow/model-registry/main-dc872c1/images/sha256-24e934a1ab01d07862fb5c8b98154281ea2c981cbc257dbd9b0b469661dab093?context=explore
and up the tree

• modify this GHA and related script to publish with _ dockerhub Container Registry as needed: https://github.com/kubeflow/model-registry/blob/main/.github/workflows/build-image-pr.yml
  (this GHA and related script at the time of writing were used to publish to Quay)

was done with:

• c9bdfc2
• 784c9fb#diff-c7cbc4a36f236b91dd212d03b3a15dd7ea80de2a647ac34aeec2ba9bf70f8fc4

• re-enable this workflow: https://github.com/kubeflow/model-registry/actions/workflows/build-and-push-image.yml

as demonstrated above, is re-enabled.