AI Assistant Summit AI Assistant Summit schedule

While doing a PhD in Cognitive Science and Linguistics at UCSD, Ann's interest in phonetics and NLP led to a dissertation using neural networks to model how speakers of a language use paradigms and analogy to create new words in that language. Since then, her professional life has spanned R&D, product development and services across organizations, including Nuance, Amazon Lab 126, and smaller startups. Always curious about how people talk to machines—and how machines respond—she's co-author of the upcoming book "Mastering Voice Interfaces".

Advancing the State of the Art in Conversational AI for all Developers

Most people think of chatbots and voice assistants as either dumb or super smart (if they’ve never tried one :) ). Based on working with hundreds of developers in creating assistants, Rasa believes there are multiple levels of assistants – in fact, 5 levels of AI assistants. This talk discusses challenges when trying to advance from a simple to an advanced assistant, recent advances in NLP and machine learning research, and provides hands-on advice on how to setup in-house teams.

Alex Weidauer is co-founder and CEO of Rasa, provider of the standard infrastructure layer for conversational AI, which supplies the tools necessary to build better, more resilient contextual assistants. With over 1.5M+ downloads since launch, Rasa Open Source is loved by developers worldwide, with a friendly, fast-growing community learning from each other and working together to make better text- and voice-based AI assistants. Alex studied Computer Science and Management, has worked for McKinsey and various tech startups before. Alex was recently on Forbes’ 30 under 30 list.

On-Device Language Understanding for the Google Assistant

Improvements in mobile power have enable the move of the computing behind commercial products from datacenters to the users' devices. For the Google Assistant, this allows a faster and more reliable product, as well as new user experiences which take advantage of the reduced latency and increased responsiveness. A central problem in this new scenario is the on-device deployment of semantic parsing solutions which can identify the intent that the user has expressed. This talk discusses the challenges inherent to building such an on-device solution. These range from technical (NN architectures, quantization, differential privacy), infrastructural (releasing, versioning) and organizational (data collection and management, developer experience).

Rushin Shah is a senior manager at Google. Prior to Google, Rushin was leading the natural language understanding group at Facebook Conversational AI. Previously, he was at Siri at Apple for 5 years, where he built and headed the natural language understanding group and he also worked at the query understanding group at Yahoo. He has worked on a broad range of problems in the NLP area including parsing, information extraction, dialog and question answering. He holds degrees in language technologies and computer science from Carnegie Mellon and IIT Kharagpur.

Delivering Magical Customer Experiences through Advances in Conversational AI

Voice services like Alexa have opened a new channel to deliver greater customer value every day. In this session, we'll review the recent advances in dialogue technologies that have made multi-turn conversations sound natural to customers, and how Alexa’s conversational AI will bring a wave of new voice experiences that weren’t possible before.

Nikko Ström is a technologist and scientist with a deep background in speech technologies. He joined Amazon in 2011 as a Senior Principal Scientist and was a founding member of the team that built Amazon Echo and Alexa. In this role, he leads deep learning efforts and machine learning projects across the Alexa organization.

Nikko has more than twenty years of experience in the field of Automatic Speech Recognition from a few of the most prominent research laboratories and companies in the world, and has published extensively at international conferences, journals, patents, and books. He has worked as a research scientist at the MIT Lab for Computer Science and a speech scientist at the start-up company Tellme Networks in 2000, and in 2007 he transitioned to the Core Speech Recognition Team at Microsoft, pushing the limits of the state of the art in commercial speech recognition technology in collaboration with the Microsoft Research Speech group.

Nikko earned his PhD at the Speech Communications LAB at KTH in Stockholm. As part of his thesis work, Nikko developed the world's first continuous speech recognizer for the Swedish language and made significant contributions in speaker adaptation and Artificial Neural Network technologies. He also published Open Source Artificial Neural Network software, which has been downloaded by thousands of researchers worldwide. He was also an invited guest researcher at the Advanced Telephony Research Lab in Kyoto, Japan, where he contributed to world class research in the speaker adaptation field.

Emotionally and Semantically Conditioned Dialogue Response Generation

A major challenge for advancing the state-of-the-art dialogue systems is to improve the controllability and robustness of the neural dialogue response generation module. In this talk, I will introduce MojiTalk, our recent work towards building the next generation empathetic dialogue agent that can synthesize human emotion into dialogue generation, without the need of any human annotated data. Furthermore, I will also discuss, in a complex multi-domain task-oriented dialogue system, how we can use graph representations to encode dialogue act dependency graph, and generate controllable and explainable response with disentangled hierarchical self-attention models.

William Wang is the Director of UC Santa Barbara's Natural Language Processing group and Responsible Machine Learning Center. He is an Assistant Professor in the Department of Computer Science at the University of California, Santa Barbara. He received his PhD from the School of Computer Science, Carnegie Mellon University. He has broad interests in machine learning approaches to data science, including statistical relational learning, information extraction, computational social science, speech, and vision. He has published more than 80 papers at leading NLP/AI/ML conferences and journals, and received best paper awards (or nominations) at ASRU 2013, CIKM 2013, EMNLP 2015, and CVPR 2019, a DARPA Young Faculty Award (Class of 2018), a Google Faculty Research Award (2018), three IBM Faculty Awards (2017-2019), two Facebook Research Awards (2018, 2019), an Adobe Research Award in 2018, and the Richard King Mellon Presidential Fellowship in 2011. He frequently serves as an Area Chair for NAACL, ACL, EMNLP, and AAAI. He is an alumnus of Columbia University, and a former research scientist intern of Yahoo! Labs, Microsoft Research Redmond, and University of Southern California. In addition to research, William enjoys writing scientific articles that impact the broader online community: his microblog @王威廉 has 100,000+ followers and more than 2,000,000 views each month. His work and opinions appear at major tech media outlets such as Wired, VICE, Scientific American, Fast Company, NASDAQ, The Next Web, Law.com, and Mental Floss.

Just Ask: An Interactive Learning Framework for Vision and Language Navigation

In the vision and language navigation task, the agent may encounter ambiguous situations that are hard to interpret by just relying on visual information and natural language instructions. We propose an interactive learning framework to endow the agent with the ability to ask for users' help in such situations. As part of this framework, we investigate multiple learning approaches for the agent with different levels of complexity. The simplest model-confusion-based method lets the agent ask questions based on its confusion, relying on the predefined confidence threshold of a next action prediction model. To build on this confusion-based method, the agent is expected to demonstrate more sophisticated reasoning such that it discovers the timing and locations to interact with a human. We achieve this goal using reinforcement learning (RL) with a proposed reward shaping term, which enables the agent to ask questions only when necessary. The success rate can be boosted by at least 15% with only one question asked on average during the navigation. Furthermore, we show that the RL agent is capable of adjusting dynamically to noisy human responses. Finally, we design a continual learning strategy, which can be viewed as a data augmentation method, for the agent to improve further utilizing its interaction history with a human. We demonstrate the proposed strategy is substantially more realistic and data-efficient compared to previously proposed pre-exploration techniques.

Seokhwan Kim is currently a senior machine learning scientist at Amazon Alexa AI. He received his Ph.D. from Pohang University of Science and Technology. Prior to joining Amazon, he conducted work in natural language understanding and spoken dialog systems where he was a Research Scientist at Adobe Research and the Institute for Infocomm Research. He has authored more than 50 peer-reviewed publications in international journals and conferences in speech and language technology areas. In 2015, he joined the organizing team of Dialog System Technology Challenge (DSTC) and has contributed to the last five challenges. In addition, he has been acting as a program committee member of the major conferences in NLP, speech, dialog, and AI fields including ACL, NAACL-HLT, EMNLP, ICASSP, Interspeech, IWSDS, AAAI, IJCAI, and ICLR.

Incorporating Common Sense and Semantic Understanding within the Assistants

With advancements in Deep Learning and data collection techniques, we have built artificial agents which more functional such that they can perform significantly better than humans on well-defined and unambiguous tasks such as Atari games. However, they do poorly on tasks that are dynamic and seem straightforward to humans such as embodied navigation and performing open-ended conversations, even after training with millions of training samples. One key element which differentiates humans from artificial agents in performing various tasks is that humans have access to common sense and semantic understanding, which is learned from past experiences. In this talk, I will be presenting how incorporating common sense and semantic understanding significantly help the agents in performing a complex task such as house navigation. I will also showcase that the semantic embeddings learned by the agent mimic the structural and positional patterns of the environment.

Chandra Khatri is a Senior AI Scientist at Uber AI driving Conversational AI efforts at Uber. Prior to Uber, he was the Lead AI Scientist at Alexa and was driving the Science for the Alexa Prize Competition, which is a $3.5 Million university competition for advancing the state of Conversational AI. Some of his recent work involves Open-domain Dialog Planning and Evaluation, Conversational Speech Recognition, Conversational Natural Language Understanding, and Sequential Modeling.

Prior to Alexa, Chandra was a Research Scientist at eBay, wherein he led various Deep Learning and NLP initiatives such as Automatic Text Summarization and Automatic Content Generation within the eCommerce domain, which has lead to significant gains for eBay. He holds degrees in Machine Learning and Computational Science & Engineering from Georgia Tech and BITS Pilani.

Conversational AI on the Edge

Dr. Sujith Ravi is a Director at Amazon. Prior to that, he was leading and managing multiple ML and NLP teams and efforts in Google AI. He founded and headed Google’s large-scale graph-based semi-supervised learning platform, deep learning platform for structured and unstructured data as well as on-device machine learning efforts for products used by billions of people in Search, Ads, Assistant, Gmail, Photos, Android, Cloud and YouTube. These technologies power conversational AI (e.g., Smart Reply), Web and Image Search; On-Device predictions in Android and Assistant; and ML platforms like Neural Structured Learning in TensorFlow, Learn2Compress as Google Cloud service, TensorFlow Lite for edge devices.

Dr. Ravi has authored over 90 scientific publications and patents in top-tier machine learning and natural language processing conferences. His work has been featured in press: Wired, Forbes, Forrester, New York Times, TechCrunch, VentureBeat, Engadget, New Scientist, among others, and also won the SIGDIAL Best Paper Award in 2019 and ACM SIGKDD Best Research Paper Award in 2014. For multiple years, he was a mentor for Google Launchpad startups. Dr. Ravi was the Co-Chair (AI and deep learning) for the 2019 National Academy of Engineering (NAE) Frontiers of Engineering symposium. He was also the Co-Chair for ICML 2019, NAACL 2019, and NeurIPS 2018 ML workshops and regularly serves as Senior/Area Chair and PC of top-tier machine learning and natural language processing conferences like NeurIPS, ICML, ACL, NAACL, EMNLP, COLING, KDD, and WSDM.

Language and Interaction in Minecraft

I will discuss a research program aimed at building a Minecraft assistant, in order to facilitate the study of agents that can complete tasks specified by dialogue, and eventually, to learn from dialogue interactions. I will describe the tools and platform we have built allowing players to interact with the agents and to record those interactions, and the data we have collected. In addition, I will describe an initial agent from which we (and hopefully others in the community) can iterate.

I am a Research Engineer at Facebook AI Research. Prior to FAIR, I was a Machine Learning Engineer at the Silicon Valley AI Lab at Baidu Research led by Adam Coates and Andrew Ng, where I worked on speech and NLP problems. I was at the Allen Institute for Artificial Intelligence for a summer before that working on learning to rank for Semantic Scholar. I did my grad school from Language Technology Institute at Carnegie Mellon University, where I worked on areas of machine translation, question answering and learning to rank for graphs and knowledge bases.

Interpreting Expression from Voice in Human-Computer Interactions

Millions of people reach out to digital assistants such as Siri every day, asking for information, making phone calls, seeking assistance, and more. The expectation is that these assistants should understand the intent of the user’s query. Detecting the intent of a query from a short, isolated utterance is a difficult task. Intent cannot always be obtained from speech-recognized transcriptions. A transcription-driven approach can interpret what has been said but fails to acknowledge how it has been said, and as a consequence, may ignore the expression present in the voice. In this talk, we will explore if a machine-learned system can reliably detect vocal expression in queries using acoustic and primitive affective embeddings. We will further explore if it is possible to improve affective state detection from speech using a time-convolutional long-short time memory (TCLSTM) architecture. We will demonstrate that using intonation and affective state information can help to attain a relative equal error rate (EER) decrease of 60% compared to a bag-of-word based system, corroborating that expression is significantly represented by vocal attributes, rather than being purely lexical.

Vikramjit Mitra is a Senior Research Scientist at Apple who is working on speech science and machine learning for human-machine interactions. Previously he worked as an Advanced Research Scientist at SRI International's Speech Technology and Research Laboratory from 2011 to 2017. He received his PhD in Electrical Engineering from University of Maryland, College Park in 2010. His research interests include speech for health applications, robust signal processing for noise/channel/reverberation, speech recognition, production/perception-motivated signal processing, information retrieval, machine learning, and speech analytics. One of his major research contributions is the estimation of speech articulatory information from the acoustic signal, and using such information for recognition of both natural and clinical speech, and the detection of depressive symptoms in adults. He led SRI’s STAR lab’s efforts on robust acoustic feature research and development, which led to state-of-the-art results in keyword spotting and speech activity detection in DARPA’s Robust Automatic Transcription of Speech Program. He has served as the PI/co-PI of several projects funded by NSF and has worked on research efforts funded by DARPA, IARPA, AFRL, NSF and Sandia National Laboratories. He is a senior member of the IEEE and an affiliate member of the Speech and Language processing technical committee (SLTC), and he has served on the scientific committees of several workshops and technical conferences.

Avenging Clippy

In this talk Ryan will show how a decade of doodling on Google’s homepage lead him to found the team that expresses the Google Assistant’s personality. Moreover, this talk will explore the opportunity of character–including anthropomorphic paper clips ahead of their time–as the mental model for conversational interfaces.

Ryan Germick is a Principal Designer at Google where he leads the Conversation Design, UX Writing, and Assistant Personality teams. Previously, Ryan was a founding member and long-time lead of the Google Doodle team.

Artificial Emotional Intelligence – a new age of AI

For a fulfilling conversation, two components are essential: the chance to make yourself properly understood and a conversation partner responding to you appropriately both on the intellectual as well as the emotional level.

Thus, to provide successful and satisfying communication, a conversational AI must have emotional and verbal intelligence, i.e. a combination of empathy, reasoning, and advanced conversation skills. The ideal conversational AI must have empathy and own emotions, a user-related emotional and contextual memory, psychological knowledge, and human-like communication soft skills. It must be able to listen, learn, and adapt, and - in order to make sense in a conversation - connect the dots and draw conclusions. This way, it can engage the user, make appropriate recommendations, or take immediate appropriate action.

In this talk, I will argue that the ideal conversational AI should be a social animal: it strives to learn all about the user’s social surroundings and – if used within a community – can create significant additional value by pro-actively intercommunicating between users of all shapes. This way, communication obstacles between individuals and corporations, as well as between consumers and providers become easily eliminated – to the point where the conversational AI may serve as the user’s primary interface to his or her world.

A conversational AI like that is not a mere vision: In this session, I will - for the first time ever - introduce soulbot, the world’s first empathetic conversational AI with emotional intelligence. The unique features of this AEI are empathy, own emotions, and advanced, human-like communication skills and tactics. soulbot listens, adapts, and empathizes. soulbot continuously learns from and about the user, it connects the dots and draws conclusions to engage, support, make recommendations, or take immediate appropriate action.

An award-winning veteran of the games industry for more than 20 years, Kai Rosenkranz started his first job at game developer Piranha Bytes while still at school. As a software developer, composer, and participator of the studio, he worked on several AAA titles such as the Gothic series. Next, he established his own company, Nevigo/Articy Software, creating and trading architectural tools for software development and game design.

In 2016, Kai left Articy to take on the challenge as Chief Technology Officer for mindfulmachines A.I.. He poured his skill and experience into working with Dr. Klaus Lassert on the soulbot® engine, aiming to revolutionize the way we interact with voice assistants and other machines. As chief architect of the product, he has been a key player in creating soulbot®, the first conscious, mindful, and appreciative, artificial emotional intelligence.

On a mission to support other creative minds in the industry, and himself being a successful composer, Kai also founded the professional network “European Game Composers”. Kai is also an experienced speaker on games and technology, e.g. at GDC, devcom, Nordic Game, Develop.

The Power of Personalized Proactivity in AI Assistants

Hear from Ken Dodelin, Vice President of Conversational AI, about the increasing role proactivity has in truly intelligent AI assistants. In this presentation, Ken will share insights and lessons learned through his team's multi-year product development journey with Eno, Capital One's AI assistant. Today, Eno does more than 15 things proactively for customers, allowing them to spend less time banking and more time living. Topics covered will include:

• the human-assistant proactive tasks that AI Assistants can be great at • finding the balance between helpfully proactive and intrusively annoying • the importance of character and defining the role of a proactive AI Assistant for consumers

Ken Dodelin is Vice President of Conversational AI Products at Capital One, where he leads teams building intelligent assistants. Previously, Ken served as VP of Product Development at Cricket Media and Director of Mobile Products at The Washington Post. Ken also founded and led Mobile Surroundings, which produced the "It Happened Here" mobile app that reached #1 in the iTunes Travel category. Ken is an Adjunct Faculty member of Georgetown University’s McDonough School of Business, where he teaches MBA courses in mobile and AI product development. Ken holds a JD and MBA from the University of North Carolina – Chapel Hill and a BS in Psychology from the College of William & Mary.

Dialogues with Plato: Concurrent training of conversational agents

In this talk I will introduce our recently released Plato Research Dialogue System - a platform for developing conversational agents - and present a method for concurrently training two conversational agents, each with different role, by letting them interact with each other via self-generated language. This is achieved by employing multi-agent reinforcement learning methods to train our agents using DSTC2 data (in the domain of restaurant information) as a testbed and show the kinds of dialogues our system can generate.

Alex is currently with Uber AI, on the Conversational AI team; his interests include statistical dialogue management, natural language processing, and human-machine social interactions. Prior to Uber, he was with Toshiba Research Europe, leading the Cambridge Research Lab team on Statistical Spoken Dialogue. Before joining Toshiba, he was a post-doctoral fellow at CMU's Articulab, working with Justine Cassell on designing and developing the next generation of socially-skilled virtual agents. He received his PhD from the University of Texas at Arlington, MSc from University College London, and BSc from the University of Athens.

Building Language Technologies for Social Good

We live in an era where many aspects of our daily activities are recorded as textual and activity data, from social media posts, to medical and financial records, to work activities captured by Wikipedia and other online tools. My research combines techniques in natural language processing, machine learning, and theories in social science to study human behavior in online communities, with the goal of developing theories and systems to build better socio-technical systems. In this talk, I will explain my research from two specific studies. The first one focuses on modeling how people seek and offer support via language in online cancer support communities, and the second studies what makes language persuasive by introducing a semi-supervised neural network to recognize persuasion strategies in loan requests on crowdfunding platforms. Through these two examples, I show how we can accurately and efficiently model human communication to build better social systems.

Diyi Yang is an assistant professor in the School of Interactive Computing at Georgia Tech, also affiliated with the Machine Learning Center ([email protected]) at Georgia Tech. Diyi received her Ph.D. from the Language Technologies Institute at Carnegie Mellon University, and her bachelor's degree from Shanghai Jiao Tong University, China. She is interested in Computational Social Science, and Natural Language Processing. She has published at leading NLP/HCI conferences and journals, and received one Notable Dataset Award from EMNLP 2015, one Best Paper Honorable Mention from ICWSM 2016, and two Best Paper Honorable Mentions from SIGCHI 2019. Diyi was awarded Carnegie Mellon Presidential Fellowship and Facebook Ph.D. Fellowship.

Towards Interactive Learning of Spoken Dialog Systems

Spoken dialog system is a prominent component in today’s virtual personal assistant, which enables people to perform everyday tasks by interacting with devices via voice interfaces. Recent advances in deep learning enabled new research directions for end-to-end neural network based dialog modeling. Such data-driven learning systems address many limitations of the conventional dialog systems but also introduce new challenges. In this talk, we will discuss recent research work on deep and reinforcement learning for neural dialog systems. We will further discuss how we can the challenges on learning efficiency and scalability by combining offline training and online interactive learning with human-in-the-loop.

Bing is a research scientist at Facebook working on conversational AI. His area of work focuses on machine learning for spoken language processing, natural language understanding, and dialog systems. He develops conversational AI systems that learn continuously from user interactions with weak supervision via deep and reinforcement learning. Before joining Facebook, he interned at Google Research working on end-to-end learning of neural dialog systems. Bing received his Ph.D. from Carnegie Mellon University where he worked on deep learning and reinforcement learning for task-oriented dialog systems.