MLOps Summit MLOps Summit schedule

Making Code and Humans GPU-Capable at Mailchimp

What happens when you have a bunch of data scientists, a bunch of new and old projects, a big grab-bag of runtime environments, and you need to get all those humans and all that code access to GPUs? Come see how the ML Eng team at Mailchimp wrestled first with connecting abstract containerized processes to very-not-abstract hardware, then scaled that process across tons of humans and projects. We’ll talk through the technical how-to with Docker, Nvidia, and Kubernetes, but all good ML Engineers know that wrangling the tech is only half the battle and the human factors can be the trickiest part.

3 Key Takeaways • An overview of the call stack from container, orchestration framework, OS, and all the way down to real GPU hardware • How ML Eng at Mailchimp provides GPU-compatible dev environments for many different projects and data scientists • An experienced take on how to balance data scientist’s human needs against heavy system optimization (spoiler alert: favor the humans)

Emily May Curtin is a Senior Machine Learning Platform Engineer at Mailchimp, which is definitely what she thought she’d be doing back when she went to film school. She combines her wealth of experience in DevOps, data engineering, distributed systems, and “cloud stuff” to enable Data Scientists at Mailchimp to do their best work. Truthfully, she’d rather be at her easel painting hurricanes and UFOs. Emily lives (and paints) in her hometown of Atlanta, GA, the best city in the world, with her husband Ryan who’s a pretty cool guy.

Beyond Accuracy: Monitoring Models for Data Drift to Ensure Performance Over Time

To ensure that machine learning models are continually achieving business goals, companies must monitor their model performance constantly. Issues such as the degradation of model performance over time can be detected by spotting changes to model accuracy or data drift. In this talk, we'll cover the basics of model accuracy and data drift, as well as the most common metrics used for each analysis. We'll also cover how Arthur enables the continuous monitoring of these critical features for organizations with ML models in production.

Key Takeaways:

*What data drift is, how we can calculate it, and what types of data drift we need to look out for (e.g. multivariate vs. univariate, different metrics for different model types)

*How to combine drift monitoring with other analysis, such as bias or accuracy metrics, to get a full view of model performance

*How to incorporate model monitoring tools into your AI stack to automatically detect performance degradation and achieve AI maturity

Adam co-founded Arthur and serves as CEO. Adam has over 20 years of experience in the AI, Machine Learning and software development spaces. Prior to founding Arthur, he founded and acted as CEO for Anax Security, a DC-based startup focused on Machine Learning for large-scale defensive cybersecurity. After Anax’s acquisition by Capital One, Adam served as Capital One’s VP of AI & Data Innovation, leading transformative projects across the business. There, Adam helped bring AI observability, fairness, and explainability to high value areas such as credit, user experience, cybersecurity, marketing, fraud & financial crimes monitoring and operations automation. Adam holds a BS in Computer Science from the University of Maryland.

Édouard d´Archimbaud is a Data Scientist and a CTO of Kili Technology. He co-founded the company in 2018 after holding various positions in research and operational projects at several banking institutions and investment funds. He led the Data Science and Artificial Intelligence Lab at BNP Paribas CIB. He graduated from the École Polytechnique with a specialization in Applied Mathematics and Computer Science, and obtained a Master's degree in Machine Learning from the École Normale Supérieure de Cachan.

Advanced ML Methods For Automating Image Labeling

Training computer vision models require a constant feed of large and accurately labeled datasets. However, this typically requires large time and capital commitments, especially since most of the labeling and quality assurance is done manually by humans. Can most, if not all, of this workflow be automated intelligently? Join Superb AI's CEO, Hyun Kim, as he talks about how Superb AI uses advanced ML techniques like transfer and few-shot learning to help teams automate the labeling and auditing of computer vision datasets.

Key takeaways :

• Data labeling is a big bottleneck for teams, in both time and cost.

• Labeling automation isn't of much value on it's own if auditing is not intelligently automated as well

• Even with automation, teams need to craft precise and repeatable workflows around data preparation and data-ops

Hyunsoo (Hyun) Kim, Co-Founder and CEO of Superb AI, is an entrepreneur on a mission to democratize data and artificial intelligence. With a background in Deep Learning and Robotics during his Ph.D. studies at Duke University and career as a Machine Learning Engineer, Hyun saw the need for a more efficient way for companies to handle machine learning training data.

Hyun has also been selected as the featured honoree for the Enterprise Technology category of Forbes 30 Under 30 Asia 2020, and Superb AI graduated from Y Combinator, a prominent Silicon Valley startup accelerator.

Deploying ML in Production

An ML application in production must address all of the issues of modern software development methodology, as well as issues unique to ML and data science. Different types of ML data and modeling problems have different requirements, often driven by the different data lifecycles and sources of ground truth that are unique to each domain. Initial implementations often suffer from inherent limitations in testability, scalability across clusters, training/serving skew, and the modularity and reusability of components. In addition, ML application measurement often emphasizes top level metrics, leading to issues in model fairness as well as predictive performance across user segments. We discuss the use of ML pipeline architectures for implementing production ML applications, key lessons learned with 10+ years of deploying ML in production. In particular we review Google’s experience with TensorFlow Extended (TFX), Google’s primary platform for productionizing ML applications. Google uses TFX for a wide variety of ML applications, and offers an open-source version to the community. TFX enables strong software methodology including modularity, extensibility, scalability, and deep performance analysis.

Key Takeaways:

*Typically less than 5% of the code is trainer or modeling code in production ml, 95% of the code is plumbing code putting the system together (e.g.Data transformation & processing prior to training, model evaluation and validation post training)

*ML Engineering is an upcoming discipline which is a superset of Software Engineering. Treating Data (in addition to code) as a first class citizen is key to ML Engineering.

*Portability and interoperability is an important consideration while building MLOps platform. It can be challenging to support a wide variety of devices and environments, so requires a delicate balance.

Anusha Ramesh is a Machine Learning Product Manager at Google. She works on TensorFlow Extended which is a production scale machine learning platform. Previously, Anusha was a Product Lead at a fashion tech startup that builds personalized recommendations for women's fashion. She has a Master's degree in information networking from Carnegie Mellon. She worked as a Software Engineer in computer networks before venturing into product.

Sid is Platform Manager at Capital One. He leads the Machine Learning Operations Platform used for deploying acquisition models for Capital One’s Card business. Sid started his career as a Software Engineer and later worked as a Management Consultant advising Fortune 100 clients on growth and product strategy. Sid holds an MBA from Emory University, Atlanta.

Mapping MLOps Maturity Levels into Reality

At Roche we have a goal to deliver twice as many medical advances to society at half the cost. Artificial Intelligence is a key driver in order to achieve this and teams of MLOps engineers are emerging to turn this into reality. Leveraging Google MLOPs maturity levels is helping our teams speak a common language and achieve the same goals, mapping them has proven challenging and closing the gap to achieve one of these levels is non-trivial. Clearly prioritizing the risks associated with not doing MLOPs helps organize which gaps to fill first and help deliver value

Key Takeaways:

*Machine Learning is key in supporting Roche’s aim of delivering twice as many medical advances to society at half the cost.

*MLOps maturity levels help bring awareness and architectural guidance when building production-ready Machine Learning models.

*Identifying risks helps prioritize which component needs to be implemented first when gaps are identified between Google MLOps maturity levels and status quo.

Oswaldo Studied Physics at UNAM with a focus on Computational Physics and MScin Big Data Science at UCA. He is an experienced professional currently working as a Senior IT Professional - MLOps Engineer at Roche in Poland. He is passionate about the intersection between computing infrastructure and Artificial Intelligence. With experience in the IT, Financial and Pharmaceutical industries he has a wide spectrum of the different challenges that need to be taken into account when exposing ML in production. He is inspired by his current role since he can contribute back to society in a more direct manner by helping accelerate drug discovery from his MLOPs engineering trench. He works with Kubernetes, Kubeflow, DKube, Python, AWS, ArgoCD, Gitlaband constantly looking for new cloud-native technologies that can help bridge the gap to production while minimizing maintenance. Part of a novel team of MLOPs engineers at Roche with a common passion and a clear goal.

The aim of this panel is to explore the following areas:

• Understanding the importance of making MLOps more accessible

• The deployment of MLOps, at what stages should this be considered and what are the most important aspects to consider

• What are the common bumps in the road to deploying MLOps, what impacts do these have and how can these be eradicated

• Any common misconceptions leading to lack of accessibility

Sarah is a Partner at Amplify Partners where she focuses on early-stage investments in machine intelligence, data science, and data management. Sarah has several years of experience in developing data acquisition strategies and leading machine and deep learning-enabled product development. As head of data at Mattermark, she led a team to collect and organize information on over one million private companies; as a consultant at Palantir and as an analyst at Cyveillance, she implemented analytics solutions for municipal and federal agencies; and as a program manager at the Center for Advanced Defense Studies, she directed projects on adversary behavioral modeling and Somali pirate network analysis. Sarah earned a B.A. in International Security Studies from Stanford University.

Korri Jones is a Sr Lead Machine Learning Engineer and Innovation Coach at Chick-fil-A, Inc. in Atlanta, Georgia where he is focused on MLOps. Prior to his work at Chick-fil-A, he worked as a Business Analyst and Product Trainer for NavMD, Inc., was an adjunct professor at Roane State Community College, and instructor for the Project GRAD summer program at Pellissippi State Community College and the University of Tennessee, Knoxville. His accolades are just as diverse, and he was in the inaugural 40 under 40 for the University of Tennessee in 2021, Volunteer of the year with the Urban League of Greater Atlanta with over 1000 hours in a single calendar year and has received the “Looking to the Future” award within his department at Chick-fil-A among many others, including best speaker awards in business case competitions.

As a veteran in the Information Technology field Shantanu has overseen and directed the implementation of pioneering technologies within various organisations across multiple industries, including retail, manufacturing, banking, utilities, transportation, insurance, logistics and telecommunications. He has personally led projects and been responsible for commissioning use of data effectively to define and structure business cases for over twenty years. He has been involved in CRM, SCM and ERP transformations across his career. He had achieved mastery in various products and technologies by the following companies: SAP, Oracle, JDA, Google, IBM, Microsoft, and AWS.

Tatiana uses open-source and machine learning technology to create scalable and high-performance systems for the BBC.

She has always been passionate about improving people’s lives by using technology. Her career started over 16 years ago when she was still studying to become a Computer Engineer at Unicamp, Brazil. Her first job was to build a 3D visualisation software that helped surgeons planning complex medical procedures. This application helped thousands of professionals and patients across dozens of countries. She learned how to develop high-quality and scalable software while working for Globo, the third-largest media conglomerate in the world. Since then, she applied technology in several industries, including research and education. She has worked both in the Brazilian and British public and private sectors. Since February 2018, she has been part of the BBC Datalab team, contributing to how the organisation informs, educates and entertains the world.