Prediction Archiving at Picnic Asing Kafka and Snowpipe
Logging and storing predictions is crucial for developing and maintaining machine learning models. It provides an invaluable feedback loop for your models to investigate where model performance can be improved. But more importantly, it helps you to go back and see what prediction was made by which model and features. In a time where more and more decisions are made by models, prediction archiving is a critical part of your machine learning platform. At Picnic we built an in-house solution to archive each and every prediction made, both by batch processes as wel by real-time services. In this talk we will present our solution that uses Kafka and Snowpipe to ingest all predictions in Snowflake, our analytical data warehouse.
Tom Steenbergen is the ML Platform Lead at Picnic. Together with his team members, they are responsible for supporting data scientists working across the machine learning lifecycle. They build and provide the infrastructure and tooling used by all the machine learning models that are running in production at Picnic. Previously at Picnic, Tom worked as a full-stack Data Scientist building machine learning models in a variety of domains, ranging from time-series forecasting to natural language processing.