-
Notifications
You must be signed in to change notification settings - Fork 3
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
General Questions #1
Comments
Yes I think that all makes sense and is correct. The tools to create the schema would need to be added or it would need to be created by hand by the data scientist. The connection with V2 schema is whether we want to extend it to allow this to be returned from |
I agree that a metadata format file can inform the API calls and protobuf/JSON body construction of a user. Without doling out work to anyone, I propose we think about stabilizing the V2 spec and design the client experience. I suggest using a good template for open source collaboration that encourages and documents conversation amongst many different stakeholders. There are two parts (maybe more?). First, is design of the metadata schema. We should start reaching out to the PyTorch, FastAI and XGB contributors as we formalize this into a spec for feedback and implementation (I can take this one). Second is a client experience. The bugs I discovered came from understandable and minor deltas between described behavior and actual. The client SDK should also serve an integral part of the testing suite for model servers that desire compliance. The simplest user story is that summarizes this is: As an application developer, I can be given the metadata file, endpoint URI, and body type (GRPC, JSON), and I don't have to write boilerplate code to run an inference invocation. A number of enterprise customers have something similar in heavier forms. Their internal SDKs also handle CRUD interaction with the model registry, model training, and model deployment infrastructure. |
GCP has a concept around this called Schemata https://cloud.google.com/vertex-ai/docs/reference/rest/v1/projects.locations.models#predictschemata |
MLFLow refers this as a Model Signature https://www.mlflow.org/docs/latest/_modules/mlflow/models/signature.html |
I want to check my understanding of this proposed schema:
The spec spans model design, model deployment, and model monitoring.
The json file originates when the PyTorch, XGB, or TF completes a model training. Typically a model artifact is created by doing
xgb.save()
. A complementary functionxgb.save_data_schema()
could be implemented that saves this JSON output.When the model is deployed on an inference server, data could be validated against this.
/metadata
URI on a model server would also return metadata in using subset of this structure.Within a model monitoring utility, the schema could be read from the provider to type the data attributes to inference outputs.
The text was updated successfully, but these errors were encountered: