You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
to run data quality checks, python checks are going to need to be able to access data files directly on disk. This should be written in such a way that check code would undergo no changes were the method to access the files to change.
my plan is to implement this as an interface with the following components:
ObjectStore interface that defines get_object which will return a stream
HashStore implementation of the ObjectStore interface which accesses a hashstore backend
StoreManager that manages the backend store
so the calling code in the checks would look something like this, where store_configuration is a hashmap passed over from Java with configuration information including the store type
this work is mostly done but still needs implementation in check code. Leaving it on branch feature-store-interface and issue open until the corresponding metadig-checks (re-writing data checks to use it) and metadig-engine (add methods to pass pids to checks) is completed
implementation in check code has been completed, see metadig-checks branch feature-data-quality to see the drafted data.suite and it's corresponding checks. this is all just awaiting full integration testing on k8s and review
to run data quality checks, python checks are going to need to be able to access data files directly on disk. This should be written in such a way that check code would undergo no changes were the method to access the files to change.
my plan is to implement this as an interface with the following components:
ObjectStore
interface that definesget_object
which will return a streamHashStore
implementation of theObjectStore
interface which accesses a hashstore backendStoreManager
that manages the backend storeso the calling code in the checks would look something like this, where store_configuration is a
hashmap
passed over from Java with configuration information including the store typeThe text was updated successfully, but these errors were encountered: