Warning: This document is for an old version of Rasa X. The latest version is 0.32.5.

Review Conversations

Once you’ve deployed your assistant and users have started having conversations with it, you’ll need to review the conversations to see how they’re going. To evaluate the quality of the conversations, you may want to ask yourself:

  • Are user utterances being classified into the correct intent?

  • Are the assistant’s responses correct?

  • Are users able to complete their goals?

The conversations screen in Rasa X helps you identify insights in conversations, take notes on them, and share them with the team so that you can take steps to improve your assistant.

Filtering Conversations

When you have a lot of conversations, it gets challenging to read every single one. A good first step to finding patterns is to filter your conversations. You can filter based on intents, entities, slot values, and actions appearing in conversations, as well as by the date the conversation took place on, the channel it happened on and the minimum conversation length. All filters are found in the filters tab, except for the date filter, which is found at the top of the page:

Rasa X conversation filters

The goal in applying filters is to focus on the conversations most likely to need your attention. Sometimes a single filter is all you need; for example, if you want to find a conversation that has a story not yet in your training data in order to add to your conversation tests, you can select Stories not included in training data from the Other tab.

Another good use of filtering is to look for conversations where the fallback policy action was invoked. In this example, it’s action_default_fallback. This can help identify areas where your assistant didn’t understand the user, or didn’t know which action to take.

Rasa X filter conversations on fallback policy

It can also be useful to apply multiple filters. For example, to find out how well Sara, the Rasa demo bot, is identifying technical questions, we could first filter by the intent technical_question. We might then want to filter for the docs_search_unhelpful tag that was added via the API to identify where the documentation returned in response to the technical question did not satisfy the user’s goal:

Rasa X filter by multiple criteria

Or, we could filter for long conversations, which can be useful to detect users who got stuck while talking to the assistant. In this example, only conversations that have more than 20 user messages will appear.

Rasa X filter by length of a conversation

Tagging Conversations

Simply reading through filtered conversations can be useful to give you a general idea of how your assistant is performing. Usually, however, you’ll want to take notes about the problems you discover as you go. You also might want to take notes to make it easier for your team to process conversations together. You can achieve both with conversation tags.

Manually tagging in the Rasa X UI

Tagging in the UI is a helpful way to take notes as you review conversations and identify patterns in the ways your users struggle to complete their task. It also enables you to establish a schema and collaborate with your team as you figure out how to improve together.

You can add / remove tags from conversations via the Rasa X UI by clicking the gear icon on the right side panel, which brings up this Manage tags modal:

Create + add / remove tags from conversations

In the example above, the reviewer adds the “reviewed” tag, so they don’t or other people don’t look at it again. At the bottom of the modal is an option to create a new tag, if the one you need doesn’t exist yet.

In the following example, the assistant is collecting name, location, and if the user is an existing Rasa user. We have a tag for each of these stages, the different failures that occur, and one for if they complete the task.

Here you can see they completed the task, so the reviewer tagged it to reflect that:

Completed tag

While in this conversation, the assistant could not pick up the name of the user, so it got a tag for NLU failure at the name stage:

Name tag

When a team member does not know how to review a conversation, they add the tag expert, so a person who understands the assistant and the task being automated the best can know to review it.

Using expert tag to signal that a conversation needs expert review

That expert can come by later and filter for all of the conversations tagged with expert.

Automated tagging via the Rasa X API

Manually tagging conversations is useful for noting what you observe as you review. Automated tagging, on the other hand, is useful for identifying specific user behavior which helps you understand when your assistant is successful…

  • before a conversation (e.g. user comes from a certain marketing campaign)

  • during a conversation (e.g. user completes a task within the conversation)

  • after a conversation (e.g. user purchases following their conversation with your assistant)

For example, Carbon Bot uses tags to mark conversations where a user clicked on a link that indicates the user decided to look into purchasing carbon offsets:

Automated conversation tagging with Carbon bot

Another cool use case for assigning tags via the API is to write a script that divides conversations among team members by tagging each conversation with a specific name, so everyone knows what conversations are their responsibility. You will see many examples of this in the screenshots in Manually tagging in the Rasa X UI.

You can tag a conversation using a POST call to the Rasa X REST API:

curl --data '[{"value":"review today","color":"eb4034"}]' \
    http://{HOST}/api/conversations/{conversation_id}/tags

Note

Please make sure to replace {HOST} and {conversation_id} with your own values.

After that you can see that the conversation now has the new tag:

Rasa X Conversation With Tag Screenshot

Once you have assigned some tags to your conversations, you can filter them by their tags:

Rasa X Conversation Tag Filter Screenshot

You can also delete a tag, removing it from all conversations, using a DELETE call to the Rasa X API.

Flagging Messages

While reviewing conversations for issues, you may want to flag one or more bot or user messages in the conversation for review by a colleague or for further discussion. Hover to the side of an user input or bot response and a greyed out flag will appear. Click on the flag to highlight the message.

You can also filter for conversations containing flagged messages when you want to go back and review them.

Rasa X flag a conversation

Sharing Conversations

While reviewing conversations, you might want the other members of your team to look at a conversation. To do so, you can share a link to the conversation, or even a specific message within the conversation. For example, to share a specific user message, click on the message and the browser URL will update to reference that message in that conversation. You can then copy the URL and send it to your teammates.

Rasa X share a conversation