Update: A lot of things have changed since this post was written. Rasa X, the freemium companion tool to Rasa Open Source, is no longer supported or maintained, and we are currently focused on the development of the Rasa Enterprise platform. To learn more about this, you can check out this blog post
As soon as you've started to share your AI assistant with test users, you're collecting valuable data that you can use to improve your assistant. Reviewing conversations is a great way to gather valuable insights about your assistant's performance but it can be time consuming, especially if you're working with a large dataset.
With our recent release of Rasa X 0.26 we've added new features and functionality including conversation tagging and additional filters that make it even easier to review conversations and improve your assistant with Rasa X. In this post we'll go over why reviewing conversations should be part of your workflow while developing contextual assistants; and we'll walk through how you can implement all of the new and existing tools Rasa X provides to make reviewing conversations easier and more effective.
Why review conversations in Rasa X?
Every conversation users have with your assistant is automatically collected by Rasa X and can be reviewed in the conversations screen. Reviewing conversations is important because it helps evaluate how your assistant is performing and allows you to catch and correct issues early on.
Reviewing conversations is also a good way to identify patterns that point to areas where your assistant can be improved. Detecting patterns in large data sets is a challenging problem-developers often rely on spreadsheets to export conversations and manually analyze hundreds or thousands of them. This approach is not only tedious, but it does not scale. Rasa X features like tags and filters provide a scalable way to review conversations and help narrow your search so you can surface patterns more quickly and identify issues early on such as:
- Did the assistant help the user achieve their goal?
- Did the conversation flow feel natural?
- How did the assistant handle unexpected user input?
- Where are users getting stuck and dropping off?
There are various ways to review conversations in Rasa X, including filtering, flagging, sharing conversations and our new feature-conversation tagging. We'll walk through each method below.
Tagging conversations
With the release of Rasa X 0.26, we've brought conversation tagging into the Rasa X UI to further improve developer workflows and enhance collaboration for teams working with large conversational datasets.
Conversation tagging in the UI allows users to define and apply labels to conversations and use those labels as custom filters. For example, an individual working with a team to review hundreds of conversations can add a `reviewed` tag as they work so their teammates can easily filter out conversations that have already been reviewed and avoid repeating work.
In Rasa X 0.25, we added the ability to programmatically assign tags to conversations via the Rasa X REST API. Tagging through the API is useful for tracking actions that aren't visible by simply viewing conversations themselves (e.g. tracking clicks) or tracking user behavior external to the conversation (e.g. whether a sale occurred). In contrast, tracking conversations where a specific feature is requested requires an element of human judgement and would be a good use case for applying tags through the UI.
Filtering Conversations
In Rasa X you could already filter on intents, entities, actions, and minimum conversation length, while in Rasa X 0.26 we've added the ability to filter on customizable tags, slots and slot value, and Core confidence level. You can also filter by stories that are not currently in your training data and by channels such as Share your bot or Facebook Messenger. Within channels, you can also filter on your own conversations that you had with your assistant in interactive learning-these conversations now show up in the Conversations screen in Rasa X.
Use filters to identify patterns in your conversations:
- Filter by a goodbye intent to see when and why users ended the conversation-did they accomplish their goal or get frustrated and leave?
- Filter on thank you messages to see where your assistant likely performed well
- Filter by intents that were recognized and verify if your assistant responded appropriately
Rasa X makes the challenging process of recognizing patterns in training data easier and more efficient; identifying these patterns surfaces valuable insight into how your assistant is performing and ways that it can be improved.
Flagging conversations
If you come across a conversation that requires further review or notes, you can flag it and filter for flagged conversations whenever you want to go back and review.
Hover to the side of an user input or response and a greyed out flag will appear. Click on the flag to highlight the message.
Sharing conversations
Sharing conversations is easy in Rasa X. If you come across a conversation or a specific message within a conversation that you want to share with another member of your team, simply click on the message. The browser URL will update with a reference that message, and you can copy the URL and share it with whomever you'd like.
Working with Rasa X to review conversations means more time can be spent gaining valuable insight into how to improve your assistant. Learn more about reviewing conversations in our docs.
We hope the new functionality we've built into Rasa X 0.26 improves the developer workflow and saves time on manually grouping, sorting, and filtering through conversations-please let us know what you think in our community forum. Thanks for being part of the Rasa community and stay tuned for more releases that improve the developer workflow!