Skip to content
/Tutorials

January 23rd, 2020

The Rasa Masterclass Handbook: Episode 10

  • portrait of Rasa

    Rasa

The Rasa Masterclass is a weekly video series that takes viewers through the process of building an AI assistant, all the way from idea to production. Hosted by Head of Developer Relations Justina Petraityte, each episode focuses on a key concept of building sophisticated AI assistants with Rasa and applies those learnings to a hands-on project. At the end of the series, viewers will have built a fully-functioning AI assistant that can locate medical facilities in US cities.

To supplement the video content, we'll be releasing handbooks to summarize each episode. You can follow along as you watch to reinforce your understanding, or you can use them as a quick reference. We'll also include links to additional resources you can use to help you along your journey.

Introduction

Welcome back to the Rasa Masterclass! In this episode, we will continue exploring the topic of how to use Rasa X to improve an assistant built with Rasa Open Source. This edition of the Masterclass Handbook will cover:

  • How to invite guest testers to talk to your assistant
  • How Rasa X enables you to review the collected conversations
  • How to improve your assistant based on those conversations

Opening your assistant to testers

Once your assistant is capable of handling the most important happy path stories, the next step is to invite testers-real users-to have conversations with your assistant. Using the conversations you collect from testing, you will be able to review conversations and make minor, major, and architectural changes to improve your assistant.

This is an important intermediary step that sits between the initial development-where you rely on manually generated training data-and the deployment stage-where you open up your assistant to a wider audience of users. Inviting guest testers to try out your assistant will help you to spot problems early, and make the improvements necessary to ensure that real users have a good experience when using your assistant in production.

Rasa X comes with a number of useful features that enable you to easily share your assistant with real testers, collect their conversations with your assistant, then review those conversations and decide which improvements you should work on next.

Reviewing the Rasa X UI will help to better understand these features.

Conversations tab

The Conversations tab is accessed on the left navigation within Rasa X. Clicking on this tab will show you the conversations that users have had with your assistant. This tab will also show how many conversations have taken place, when those conversations happened, and the content of the actual conversations themselves.

If you haven't shared your assistant with real users just yet, this section will be empty.

Sharing with guest testers

Rasa X allows you to invite real testers to talk to your assistant via a shareable link. At the top right, you will find an icon to share your assistant. Clicking on the icon will bring up a "Share with guest testers" dialog box.

Here you can provide a name for your assistant and a description about what your assistant does. Once you are happy with the details, hit Generate link. This link can be shared with the people that you want to test your assistant, such as your friends, family, colleagues or a specific group of people that you've selected for a user test.

As you can see, the link in the screenshot above includes our server IP address. Users will be able to test using this URL, but best practice would be to configure a custom domain and SSL for your server, so that users would see a more familiar URL format.

What the guest tester sees

Once an invited test user clicks on the link you shared, they will see a simple chat UI where they can immediately talk to your assistant. The guest tester doesn't need to install software or do any additional setup, making it very simple for them to start testing.

It's important to note that multiple guest testers can talk to your assistant, all at the same time. All of the conversations they have with your assistant are stored in a database on your server and are displayed in the Conversations tab of Rasa X for you to review.

Even more importantly, remember that all of the conversations you collect and view using Rasa X are stored on your server, which means that you completely own your data. Conversations are not shared with Rasa. You completely control access to these conversations.

Lastly, remember that you can find all of the files of your assistant on your server - including models, actions, configurations, and the database itself.

Reviewing test conversations

After your guest testers have had a chance to chat with your assistant, you can review their conversations in Rasa X.

Each conversation has a timestamp for when it took place. To make reviewing the conversations more efficient, you can apply filters to the collected conversations - for example, you can filter by intents or actions included in the stories, or the number of user messages included in a story.

Reviewing these collected conversations should be a great source of insight about your assistant, and will help in deciding which changes are needed to improve your assistant.

Improving your assistant

Changes within Rasa X

Minor changes are typically something that can be adjusted within Rasa X, like improving your NLU model to better classify intents. In the example below, the assistant failed to classify the message Hellooo as a greeting. This could be an indication that we should include more-and more unusual-examples of greetings in our training data.

From the NLU Training screen, located under the Training tab, we can add Hellooo as a new training example. Rasa X will automatically suggest an intent match, and we can verify that the intent for the message should be greet.

Major changes to your assistant, like creating a new intent or adjusting the logic of your stories, can be done either within Rasa X or on the command line. This blog post highlights how that works within Rasa X.

Architectural changes

Your test conversations may uncover more substantial issues that require architectural changes to your assistant, such as changing your custom actions. These changes usually require updates to your assistant's files on your local computer, made via a text editor, and then pushed to made via the command line in Rasa Open Source.

For example, in testing our medicare locator, we collected a conversation with a test user based in London. However, the custom action we created can handle only requests for US-based facilities. We can flag this conversation for later review.

In this case, we would either have to include location validation, or update our custom actions with a new API which would find health facilities in locations outside the US.

Improving the NLU model

The conversations you collect can also be used to improve your NLU model.

You can access the NLU Training section of Rasa X in the left navigation, and there you will find the inputs from test users who talked to your assistant. From this screen, you can review each of the inputs and annotate them. You can improve your NLU model by annotating a user input and its predicted intent as correct.

If some examples aren't useful - like input below Hey, which was extracted as an entity -- such inputs can simply be deleted.

After you make improvements to your model and annotate new training data, you should train a new model. In prior episodes of the Masterclass, we've been training our model using the command line, but training can also be done directly in Rasa X.

Once you make the new model active, you can start sharing your assistant with real testers again, and start collecting new conversations.

Version control

Best practice is to version control your assistant, using Rasa X's integrated version control feature.

When you make a change to your model in Rasa X, like annotating new training data or changing model configurations, you will see that the integrated version control icon is highlighted to the left with an orange bar. This indicates that there are some changes you made in Rasa X that are not reflected in your git server.

To include those changes, you can commit them to the master branch or create a new one.

Once the changes are added, you will find a pull request on your assistant's git repository with suggested changes. You can invite team members to review and merge it, and continue improving your assistant.

Conclusion

Rasa X is an essential tool for improving assistants built with Rasa Open Source. The process outlined in this Masterclass should be repeated until you're happy with how your assistant performs:

  • Share the assistant with real testers,
  • Make improvements based on the collected conversations, and
  • Repeat.

Once you feel your assistant can handle conversations with guest testers, it's time to take the next step-deployment. In deploying your assistant, you will connect your assistant to an outside messaging platform, your own website, or any other platform where you want real users to talk to your assistant. You will collect the conversations users have with your assistant, and continue making improvements based on what you can learn from these real-world conversations. We'll cover this in the next episode of the Rasa Masterclass, #11.

Additional Resources