`ModelConfig` is `ModelVersion`! #2044

avishniakov · 2023-11-13T13:32:48Z

Describe changes

This is a big refactoring PR containing many renames and breaking changes. The ultimate goal was to bring a simplified user-interface experience, so once ModelVersion is configured in @pipeline or @step it will travel into all other user-facing places: step context, client, etc.

[BREKING CHANGES]

ModelConfig -> ModelVersion
@pipeline(model_config=...) -> @pipeline(model_version=...)
Config YAML: model_config -> model_version
Client methods create_model_version, get_model_version and update_model_version now return ModelVersion
get_step_context().model_version now return ModelVersion
Included here renaming of linked artifacts asmodel_objects>model_artifacts, artifacts>data_artifacts and deployments>endpoint_artifacts in CLI and other places in code.
Previously public methods of response like get_or_create_model_version and get_or_create_model are now private and called under the hood of other functions, so meant to be non-user-facing going forward
ArtifactConfig -> DataArtifactConfig
DeploymentArtifactConfig -> EndpointArtifactConfig
Client methods for Model and ModelVersion were significantly reworked from input args PoV
Sunset of running model versions and surrounding complexity for users. Now if the version is None in model_version a new model version will get created and shared across all other model_version requests inside the same run.
Using DataArtifactConfig and others are no longer working with only model_name - model_version needs to be fed along. In case of misuse, a clear RuntimeError will be raised. These objects can still be used without model_name and model_version to get context from the step.

BUGFIXES

Prevent the creation of versions using reserved names, which happened before, if the "training" pipeline was calling ModelVersion(...,version="staging") in some steps
Properly refresh the template, now it just overwrites/appends files, but not removing old ones, so E2E is a bit of a mess. 9ad0778

SIDE CHANGES

ModelVersion is now equipped with extra public methods, which I find useful. Have a look, if something is missing in your opinion.

Incoming templates PR: zenml-io/template-e2e-batch#19

Pre-requisites

Please ensure you have done the following:

I have read the CONTRIBUTING.md document.
If my change requires a change to docs, I have updated the documentation accordingly.
If I have added an integration, I have updated the integrations table and the corresponding website section.
I have added tests to cover my changes.
I have based my new branch on develop and the open PR is targeting develop. If your branch wasn't based on develop read Contribution guide on rebasing branch to develop.

Types of changes

Bug fix (non-breaking change which fixes an issue)
New feature (non-breaking change which adds functionality)
Breaking change (fix or feature that would cause existing functionality to change)
Other (add details above)

…rsion

…-model-version

…' of https://github.com/zenml-io/zenml into feature/OSS-2609-OSS-2575-model-config-is-model-version

src/zenml/new/pipelines/pipeline.py

fa9r

Overall changes look good, just needs a bit more polish before merging 👍

fa9r · 2023-11-15T12:33:56Z

src/zenml/cli/model.py

@@ -375,7 +368,7 @@ def list_model_versions(model_name_or_id: str, **kwargs: Any) -> None:
        **kwargs: Keyword arguments to filter models.
    """
    model_id = Client().get_model(model_name_or_id=model_name_or_id).id
-    model_versions = Client().list_model_versions(
+    model_versions = Client().zen_store.list_model_versions(


Why are we bypassing the client method here?

Because I need the response model back, not ModelVersion.

As suggested in the other comment - the Client should return the response model anyway if we want to be consistent with the current Client interface, so I guess this would also solve this issue then?

Prefer to keep it as is, no one even knows about zenml.models.model_models.ModelVersionResponseModel thing - it is too internal to be a concerning, in my opinion.

src/zenml/cli/model.py

src/zenml/client.py

src/zenml/steps/step_decorator.py

src/zenml/model/model_version.py

github-actions · 2023-11-15T17:43:38Z

E2E template updates in examples/e2e have been pushed.

fa9r

Only major issue I see is that the Client methods are still inconsistent with other entities, other than that it looks ready to merge from my side

src/zenml/client.py

…' of https://github.com/zenml-io/zenml into feature/OSS-2609-OSS-2575-model-config-is-model-version

* Moved model versions to their own root route * Fix the delete endpoint * Fixed smaller mistakes * Fixed for Client as well * Fixed lint errors * More linting * More linting * More tiny fixes * Update src/zenml/zen_server/routers/model_versions_endpoints.py Co-authored-by: Andrei Vishniakov <[email protected]> * More linting * Fixed tests and solved conflicts * Fixed linting * Fixed more tests * Further refactoring * Added raises section * Fix one failing test * Take "latest" stage into account * Reformatted * Standardize use of list response * Rewrote some tests * Add clien tests * Fixed spelling * Tested to work with e2e pipeline * Ugly fixes to get response models in the CLI * Auto-update of E2E template * Access ModelVersionResponseModels in Client again * Another small fix * Linted --------- Co-authored-by: Andrei Vishniakov <[email protected]> Co-authored-by: GitHub Actions <[email protected]>

…-model-version

github-actions · 2023-11-16T16:48:26Z

E2E template updates in examples/e2e have been pushed.

avishniakov · 2023-11-17T07:06:17Z

@fa9r , any vetos merging this? Otherwise, I'll merge.

This reverts commit b518dcc.

fa9r

I would still suggest to change create / get / update to return ModelVersionResponseModel again, otherwise LGTM

fa9r · 2023-11-17T08:02:40Z

src/zenml/client.py

+        stage: Optional[Union[str, ModelStages]] = None,
+        force: bool = False,
+        name: Optional[str] = None,
+    ) -> "ModelVersion":


Still not a fan of this returning ModelVersion instead of ModelVersionResponseModel

Let's proceed like this then:

I will merge this one to unblock others (including you with rebasing your mega-branch).

Spin a feedback loop on Discord to clarify what we should return and if we should stick to responses here.

If we decide to use responses - I'll raise a separate ticket for that purpose.

avishniakov added 22 commits November 10, 2023 13:51

rename file

4a77b19

rename file

70a9b97

rename file

5518513

Merge branch 'develop' into feature/OSS-2609-model-config-is-model-ve…

5252b58

…rsion

big rebranding

7cd672c

big rebranding

26e0b3a

deprecate method

ed5bb8f

redesign context to return ModelVersion

8aafc95

add linked_artifact_ids property

0ff4b21

add __eq__

0138546

fix typo

c924456

add number and name props

e62100b

rename artifact types

3b4f1d5

prevent reserved mv names creation

f53dc24

typo

77f1155

more logging

ef25311

prevent reserved mv names creation

3a13ccd

handle reserved names properly

1a23383

rework client calls

a4ae3c8

remove version_description

052c3b8

remove version_name

5490aa9

delete_new_version_on_failure > with_recovery

5dd7c81

github-actions bot added internal To filter out internal PRs and issues enhancement New feature or request labels Nov 13, 2023

avishniakov mentioned this pull request Nov 13, 2023

Update template to follow changes in MCP zenml-io/template-e2e-batch#19

Merged

7 tasks

avishniakov changed the base branch from main to develop November 13, 2023 13:43

Merge branch 'develop' into feature/OSS-2609-OSS-2575-model-config-is…

cd211aa

…-model-version

avishniakov marked this pull request as ready for review November 13, 2023 13:43

avishniakov added 2 commits November 13, 2023 15:09

fix alembic false alarms

18ac906

Merge branch 'feature/OSS-2609-OSS-2575-model-config-is-model-version…

ccd70ca

…' of https://github.com/zenml-io/zenml into feature/OSS-2609-OSS-2575-model-config-is-model-version

schustmi reviewed Nov 15, 2023

View reviewed changes

src/zenml/new/pipelines/pipeline.py Outdated Show resolved Hide resolved

rename misleading method

6f069b6

fa9r suggested changes Nov 15, 2023

View reviewed changes

fa9r mentioned this pull request Nov 15, 2023

Feature/revamp model endpoints #2035

Merged

9 tasks

avishniakov added 5 commits November 15, 2023 16:09

remove all_model_versions

4185808

update list_models

c526d44

improve docstring

5d5c9fc

rephrase docstring

1ddb941

nits

a95c3ad

avishniakov requested a review from fa9r November 15, 2023 15:51

Auto-update of E2E template

1e0249f

fa9r suggested changes Nov 15, 2023

View reviewed changes

src/zenml/client.py Outdated Show resolved Hide resolved

avishniakov and others added 9 commits November 16, 2023 11:06

stick to unified list interface

ce5c34d

Merge branch 'feature/OSS-2609-OSS-2575-model-config-is-model-version…

374ca2f

…' of https://github.com/zenml-io/zenml into feature/OSS-2609-OSS-2575-model-config-is-model-version

resolve mess after merge

d37b423

Merge branch 'develop' into feature/OSS-2609-OSS-2575-model-config-is…

19cfbe9

…-model-version

after merge clean-up

886fc10

update outdated tests

6819991

Merge branch 'develop' into feature/OSS-2609-OSS-2575-model-config-is…

72d7412

…-model-version

Auto-update of E2E template

2fd2a36

avishniakov requested a review from fa9r November 16, 2023 20:46

avishniakov added 2 commits November 17, 2023 08:50

deep update of template

b518dcc

Revert "deep update of template"

7586b02

This reverts commit b518dcc.

fa9r reviewed Nov 17, 2023

View reviewed changes

avishniakov merged commit 3f06da9 into develop Nov 17, 2023
4 of 21 checks passed

avishniakov deleted the feature/OSS-2609-OSS-2575-model-config-is-model-version branch November 17, 2023 08:55

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

`ModelConfig` is `ModelVersion`! #2044

`ModelConfig` is `ModelVersion`! #2044

avishniakov commented Nov 13, 2023 •

edited

Loading

fa9r left a comment

fa9r Nov 15, 2023

avishniakov Nov 15, 2023

fa9r Nov 15, 2023

avishniakov Nov 16, 2023

github-actions bot commented Nov 15, 2023

fa9r left a comment

github-actions bot commented Nov 16, 2023

avishniakov commented Nov 17, 2023

fa9r left a comment

fa9r Nov 17, 2023

avishniakov Nov 17, 2023

ModelConfig is ModelVersion! #2044

ModelConfig is ModelVersion! #2044

Conversation

avishniakov commented Nov 13, 2023 • edited Loading

Describe changes

Pre-requisites

Types of changes

fa9r left a comment

Choose a reason for hiding this comment

fa9r Nov 15, 2023

Choose a reason for hiding this comment

avishniakov Nov 15, 2023

Choose a reason for hiding this comment

fa9r Nov 15, 2023

Choose a reason for hiding this comment

avishniakov Nov 16, 2023

Choose a reason for hiding this comment

github-actions bot commented Nov 15, 2023

fa9r left a comment

Choose a reason for hiding this comment

github-actions bot commented Nov 16, 2023

avishniakov commented Nov 17, 2023

fa9r left a comment

Choose a reason for hiding this comment

fa9r Nov 17, 2023

Choose a reason for hiding this comment

avishniakov Nov 17, 2023

Choose a reason for hiding this comment

`ModelConfig` is `ModelVersion`! #2044

`ModelConfig` is `ModelVersion`! #2044

avishniakov commented Nov 13, 2023 •

edited

Loading