If you've visited the Rasa documentation site lately, you may have noticed a familiar (feathery) face. Sara, the Rasa demo assistant, is back.
Sara's been a fixture on the Rasa website and documentation since 2018, but until recently, she's been offline undergoing a major renovation. Now that she's back, we wanted to sit down with two Rasa engineers, Ella Rohm-Ensing and Greg Stephens, to find out more about where she's been and what's been updated.
Some background: you could say that Sara's development runs in parallel to the development of Rasa. As new updates are made to the product, those updates are brought into Sara. In doing so, the Rasa team can test out features in a real-world laboratory and channel our own engineers' feedback into the product development process. That process is usually known as dogfooding-but in Sara's case, birdfooding might be a more accurate term.
In many ways, the challenge of keeping Sara up-to-date is similar to what Rasa users face in maintaining a complex assistant over time. Technical requirements change, as does the logic behind the dialogues. One of Sara's main tasks is to walk users through a getting started tutorial, and that tutorial has changed as the product developed. It was time to get Sara running on the latest version of Rasa Open Source and make sure she can provide the most up-to-date information about Rasa. In the process, we added a couple of brand new features to make her even more helpful.
A Brief History...
Sara was first developed in 2018 when the Rasa team set a goal to build a "badass bot" that would showcase the latest and greatest Rasa capabilities. Ella says, "We wanted to show something that included several different skills, that was complex enough that the machine learning policies really came into play."
Rasa wants every developer to have the chance to become an expert at building AI assistants - it's part of our mission. So it follows that Sara's purpose should be helping developers find information on how to build AI assistants. Sara's skills include helping users install and get started with Rasa, answering technical questions, subscribing users to the Rasa newsletter, and setting up a call with the Sales team.
But Sara has another purpose too: to provide an open source example that developers can use to understand the inner workings of a complicated contextual assistant. Developers can use Sara as a reference implementation for a number of useful techniques:
- Using Rasa Core and NLU to handle contextual conversations
- Extracting custom and pre-trained entities
- Slot filling and form actions
- Customizing the tensorflow_embedding pipeline to improve classification
- Generalizing previously unseen conversations with Rasa Core
- Using a React chat component on a website
- Using Rasa X to improve from real conversations
Whether developers chat with Sara about Rasa, look at her code as an example, or clone her as a starting point for building their own assistants, Sara helps developers fast track their way to building "badass bots" of their own.
Rasa NLU and Rasa Core become Rasa Open Source
So why did Sara go offline? In May 2019 we reached an important developmental milestone: the release of version 1.0, which merged Rasa NLU and Rasa Core into a single library, Rasa Open Source. While the two modules could still be decoupled and used independently, packaging both into a single library was necessary for building end-to-end models that can take the context of conversations into account, increasing code maintainability, and optimizing for the most common use case: an assistant where both NLU and dialogue management are needed. Another benefit to upgrading? Compatibility with Rasa X, which was released at the same time.
While updating a large bot with lots of historical conversation data is no small task, for Sara, it also meant the answers and tutorials she provided for users needed to be updated to include new content on Rasa X. "One of the things Sara does is help you get started," Ella says. "In order to update we had to completely revamp the whole Getting Started flow. It was a huge PR, like 1000 lines, changing all the content, the stories, and how the whole flow happens."
New and Improved
Customer Success Engineering, the Rasa team responsible for building and maintaining Sara, took Sara offline to bring her back up to date. The highest priority maintenance tasks included:
- Putting Sara on the latest version of Rasa Open Source
- Migrating her database from MongoDB to SQL
- Making sure the website connector could handle heavy traffic to the Rasa docs website
In the past, Sara's web chat connector opened a websocket connection every time a user navigated to a new page. Over time, with more and more people using Rasa and visiting the documentation, the traffic began to overload the connector. Luckily, new features in the webchat connector offered a solution for this issue: after the update, the socket doesn't open until you click the webchat widget to start a conversation with Sara-much more scalable.
In addition to data migration and maintenance, the team had several other items on the to-do list: using Rasa X to improve Sara, syncing her training data with GitHub (opening the door to CI/CD), and allowing Sara to search for answers to user questions in the forum and the documentation.
Rasa X
With the introduction of Rasa X, adding new educational content to Sara was only half of the operation. As Greg explains it, setting up Sara with a Rasa X instance was a big priority so the team could start using the tool to improve her based on conversation data collected from user interactions. And in doing so, the team was also able to collect valuable first-hand feedback about the experience of using Rasa X.
"We use Rasa X to continue to monitor the bot, review conversations, and make improvements. It really has been helpful for building out Sara. We brought Sara back online with all this new content, and then within a few weeks, we're starting to use Rasa X to enhance the assistant," Greg says.
Integrating with Git and CI/CD
More recently, with the release of Integrated Version Control in Rasa X, the team set out to optimize some of the engineering workflows around Sara. Using Integrated Version Control allows Rasa engineers to manage training data versions by keeping the NLU data in Rasa X in sync with the connected GitHub repository. Once changes are pushed to GitHub, the team can automate testing and deployment.
"When we open a pull request, it does a few different tests," Ella says. "We run tests for linting and code formatting. And we run rasa data validate which goes through and checks for training data errors, like intents in the NLU data that aren't defined in the domain. It will also throw warnings for things like having an utterance defined that isn't used anywhere, which helps keep the code clean."
Once the PR has been reviewed, merging triggers a sequence of automated processes. Ella explains, "When we merge to master, it deploys a new action server by taking that version's action code, building a new image, and pushing it to the container registry."
Recently, the team ported Sara's CI/CD configuration from Travis to GitHub Actions. Take a look at the code here.
Searching the forum and documentation
Sara has a list of technical questions she can answer directly, but when faced with a question she's never seen before, her previous behavior was to fall back to "Sorry, I haven't learned how to answer that yet."
The Customer Success Engineering team wanted to help users get closer to the answer they were looking for, even if Sara couldn't provide every answer herself. To do this, Greg implemented a new fallback behavior. Now, when Sara classifies a user input as a technical question but doesn't know how to answer it, she searches the documentation for a match. If the user says the results weren't helpful, she does another search, this time through the forum.
"We use the Algolia API to search our docs, retrieve the top search results, and then format that into a markdown response. If the user still wants help, we do a search of the forum using the Discourse API," Greg says.
Besides providing a much better user experience, this also gives Sara access to all of the knowledge contained in the docs and forum. Want to see how it works? Take a look at the code.
The Takeaway
Both Ella and Greg cited the opportunity to test techniques in a real world use case as one of the most valuable aspects of developing Sara. As Rasa engineers, it's one thing to understand the code base in depth, and another to build a bot the same way users do.
"Recently, I came across a situation where I needed to use test comparison mode," Ella says. "We noticed the NLU output was different whether the user typed Detroit with an upper or lowercase D. And we had to decide-do we want our config to be case sensitive or not?" As Ella explains, test comparison mode allows you to run test scripts with two different configs to compare results with both options. "So you can really easily see which configuration provides the best results." For Ella, it was a chance to view the feature not as a Rasa engineer, but as a user working on a real-life use case.
Greg notes that working on Sara has given him a greater appreciation for the assistant as a teaching tool. "After working on Sara, now when I address questions on the forum, a lot of times I'll point to action code in Sara or some other feature in our bot. All in all, there are a lot of chat bot techniques and Rasa development techniques that make good examples for anybody building an assistant."
What's Next?
As any machine learning developer will tell you, improving an AI assistant is an ongoing task, but the team has set their sights on one big roadmap item: updating Sara to use the ResponseSelector NLU component, introduced with Rasa 1.3. "The response selector is a completely different model that uses the actual text of an incoming user message to directly predict a response for it," Ella says. It introduces a new best practice for handling chitchat and FAQs, and "one of the things we're focused on is making sure that Sara uses the latest best practices."
Conclusion
Sara underwent a major version upgrade, a database migration, picked up a few enhancements -and, even got a fresh new look from the Rasa Design team.
You can talk to Sara on the Getting Started page and the Rasa documentation and visit the open source repository to see how it all works.
Want to get involved? Share your feedback on Sara or ask us a question on the forum, or check out Sara's project board for ideas on how to contribute!