On-prem data and models

[ryandawsonuk]

Let’s say my model or data is in a volume. This might be the case if I’m running on-prem and perhaps my data isn’t allowed to leave my company network. If so does ‘uri’ allow me to capture where it is? I then want a name of a volume and path on the volume. Perhaps those could be introduced as fields or the existing field called ‘location’?

@cvenets perhaps you have a view on this?

[jinchihe]

For on-prem volume case, Is that possiable to set the uri to pvc://pvc_name/path/for/models? First area between / can be pvc_name and following model path.

WDYT @zhenghuiwang ?

[zhenghuiwang]

Yes, you can set the uri with prefix pvc://

There is no schema for the uri. It is up to the users & integrated systems to interpret it. Some examples
https://github.com/kubeflow/metadata/blob/master/schema/alpha/docs/artifacts/model.md#uri-examples

[cvenets]

As @zhenghuiwang mentioned the metadata service doesn't interpret the URIs, it just stores them. The URI can be completely arbitrary, so yes, you can prefix it with pvc://. The other systems that interact with the metadata service need to know how to interpret it and consume it.

We have started a discussion with @zhenghuiwang in the design doc, for extending the metadata service's context execution, too. This will allow us to track PVCs and even K8s objects in an immutable way, so if you are running an intelligent data management layer underneath (like Arrikto's Rok), you will be able to reproduce the whole state from a URI in the metadata service. I will be coming back on this for v0.7, once v0.6 is released and we can test the Metadata functionality.

[jtfogarty]

/kind question
/area engprod
/priority p2