This repository contains the demo code presented during the talk End-to-end ML streaming with Kubeflow, Kafka, and Redis at scale at O'Reilly OSCON 2019.
The code basically shows how to:
- train and save a simple model from the California Housing prices dataset from scikit-learn
- serve that model with seldon core kubernetes
- stream raw features to kafka
- aggregate features (perform feature engineering) and stream engineered features to redis
- invoke model directly from redis engineered features
To run this demo, follow the jupyter notebook
The slides from my talk are available here
The following great projects were used / referenced:
- Python, scikit-learn, MLFlow
- Kafka, Redis, Docker, Kubernetes, Kubeflow, Seldon Core
- Prometheus, Grafana
Troubleshooting:
- Make sure not to use some VPN software while running this demo as it may interfere with docker desktop and k8s routing
- If your network changes, you may need to restart kafka and the redis docker container
- Make sure
kubectl proxyisn't running
The code samples here are licensed under the Apache 2.0 License