Use Data to Understand Your Data Pipelines
At Spotify, we use data to make sure Discover Weekly has new music for you each Monday, and so that you can know what you’ve been listening to all year with Wrapped. With the abundance of data being generated across organizations, it is crucial that the data is of high quality, reliable and available when needed. We will explore how Spotify enables data quality and better troubleshooting through data observability tools, to make sure data pipelines are first class citizens in the software ecosystem. That includes monitoring, reliability metrics and alerting. The challenge and difference from traditional software observability being - an engineer typically maintains a handful of backend services but hundreds of data pipelines.
Tonima Afroze is an Experienced Engineer working with data infrastructure at Spotify. She has previously developed software for medical companies, real-time credit risk assessments and fraud detection at a fintech company and built methods to detect bias in ML models in the same. Interests include building high quality software and using tech for good. She holds MScs in Medical Engineering and Software Engineering from KTH Sweden and the University of Oxford.