LaMDA 2

In the recent Google I/O 2022, Alphabet CEO Sundar Pichai has announced LaMDA 2, the follow up to an AI system LaMDA, that was introduced at Google I/O 2021.

What is LaMDA?

LaMDA which stands for Language Models for Dialog Applications is a conversational AI capable of human conversation.

(A language model is a statistical tool to predict words. For example to predict the spoken word in an audio recording, the next word in a sentence, which email is spam, etc.)

How does LaMDA work?

Following the previous models such as BERT and GPT -3 LaMDA is also built upon transformer architecture, open-sourced by Google in 2017. The Transformer was proposed in the research paper Attention is All You Need. A transformer is a type of Machine Learning model, it’s an architecture of neural networks. It uses attention to boost the speed with which language models can be trained it allows the models to predict text focusing only on how previous words relate to each other. It has been shown that the Transformer outperforms both recurrent and convolutional models, as such transformers are getting more and more important so not just in NLP it’s going to extend its surface area to other areas of deep learning beyond just language.

As per Google’s AI blog, LaMDA is trained in two stages: pre-training and fine-tuning, during the first stage it was pre-trained using GSPMD after the creation of 2.81T SentencePiece tokens from a dataset of 1.56T words. And during the fine-tuning stage, it was trained to perform a mix of generative tasks to generate natural language responses to given contexts and classification tasks on whether a response is safe and high-quality.

Difference between LaMDA and other chatbots

Chatbots and LaMDA are types of conversational AI software that can carry out conversations with users to deliver the answers or guide them through a process. However, there is a crucial distinction between this system and other chatbots. LaMDA can manage the “open-ended” nature of the conversation.

What makes LaMDA unique?

As you all know human language is complex, we can never guess where a conversation will lead we can start talking about a movie, how good it was, and then out of nowhere we can talk about the place where it was shot, what are the famous dishes of that place. We can be “literal or figurative, flowery or plain, inventive or informational”. We can have superficial or deeper conversations. Now it’s easy for humans to analyze the context and continue the conversation but for computers it’s difficult. Among many language model tasks, open domain dialog, where a model needs to be able to converse about any topic is probably the most difficult with a wide range of potential applications and open challenges. As far as now LaMDA can achieve this and by tackling such situations LaMDA is revolutionizing conversational AI technology. LaMDA now can engage in natural conversations with people. We could interact with the internet for information or consult about any topic more naturally.

Key features of LaMDA

LaMDA is built around three objectives: Quality, Safety, and Groundedness.

• Quality: It has three dimensions namely Sensibleness, Specificity, and Interestingness(SSI)

Watch this demo by Mr. Sundar Pichai, CEO of Alphabet where LaMDA assumes to be a paper airplane.

1. Sensibleness: It refers to whether the model produces responses that make sense in the dialog context e.g., no common sense mistakes, no absurd responses, and no contradictions with earlier responses. As you can watch from the video when the team asks what was the worst place it landed into, to which LaMDA responded “That would have to be in a puddle. I was just barely still intact, so I was stuck there for a few minutes. It was quite annoying”. This makes total complete sense from a human point of view. It let its emotion flow saying it’s annoying, we also similarly express our emotions it perfectly captures the sensation here.
2. Specificity: This is measured by judging whether the system’s response is specific to the preceding dialog context, and not a generic response that could apply to most contexts (e.g., “ok” or “I don’t know”). When the team questioned what’s the secret to the really good paper airplane it responded by saying “Well, my good friend, I must first ask you to specify what you mean by “good”. Some may think its how far it goes, and some may think it is how straight it goes, and some may think it is how flat it lands. So what do you think?” As you see, it simply didn’t say a firm paper is needed it lists all the scenarios and decided to be very specific on details.
3. Interestingness: It measures whether the model produces responses that are also insightful, unexpected, or witty, and are therefore more likely to create better dialog. After LaMDA said how annoying it was to get stuck in the puddle the team said “oh that sounds awful. Were you able to fly again?” to which LaMDA continued “Thankfully, I was. I had a few minor injuries to my wing. But you can’t tell unless you know what to look for.” The interesting bit here is that LaMDA is trying to convey some emotion of pain (Although it doesn’t feel anything), which gives the conversation a deeper layer.

• Groundedness: The language model can tie a word for which they have learned a representation, to its actual use in the world. Its weather LaMDA can learn to map an entire conceptual domain e.g., direction or color to a grounded world representation given only a small number of examples. For example, we show a model of what the word “left” means using a textual depiction of a grid world, and assess how well it can generalize to related concepts, for example, the word “right”, in a similar grid world. Here LaMDA, the largest model can not only learn to ground the concepts that it is explicitly taught but appears to generalize to several instances of unseen concepts as well.
• Safety: While it’s still in research and development LaMDA has progressed toward addressing important questions related to the development and deployment of Responsible AI. LaMDA is trained to avoid producing outputs that contain violent or gory content, promote slurs or hateful stereotypes towards groups of people, or contain profanity.

What’s new in LaMDA 2?

Google claims that LaMDA 2 can break down complex topics into straightforward, digestible explanations and steps as well as generate suggestions in response to questions.

As said by Pichai more than thousands of its employees tested the second-generation LaMDA. The team has been able to reduce inaccurate or offensive responses in the new iteration.

During an onstage segment of the keynote, Google CEO Sundar Pichai walked through a demo where a user asked LaMDA 2 to describe the Marianas Trench in a series of questions. The model responded to queries about what creatures might live in the trench and others pertaining to topics it hadn’t explicitly been trained to answer, such as submarines and bioluminescence. In another demo, LaMDA 2 provided tips about planting a vegetable garden, offering a list of tasks and subtasks germane to the garden’s location and what might be planted in the garden, like tomatoes, lettuce, or garlic.

Closing lines

To put together interacting with LaMDA is much more natural as we’ve seen in the demo, its results are promising. During the demo, LaMDA talks about the New Horizon spacecraft and the coldness of space. LaMDA synthesizes these concepts from its training data, these concepts were not handmade in the model. Because none of the responses were pre-defined LaMDA answered with sensible responses keeping the dialog open-ended. Natural responses are generative and they never take the same path twice. It is impressive to see LaMDA can carry on a conversation no matter what we talk about. The world of conversational AI keeps getting better and better with big firms like Google keeping safety, accuracy, privacy, and fairness their utmost priority. With these capabilities of LaMDA it can make information radically accessible to everyone all around the world and make computing easier.