Warning: This document is for an old version of Rasa X. The latest version is 0.32.5.

Review Conversations

Once you’ve deployed your assistant and users have started having conversations with it, you’ll need to review the conversations to see how they’re going. To evaluate the quality of the conversations, here are some questions you may want to ask yourself:

  • How does the assistant handle unexpected user input?

  • Are user utterances being classified into the correct intent?

  • Are the assistant’s responses correct?

  • Does the conversation flow feel natural?

  • Are users able to complete their goals?

  • Do any forms and associated slot filling work?

  • Where do users get stuck and drop off?

  • What patterns are there in the conversations that go wrong?

Rasa X Conversations screen

The Conversations screen is where you review conversations. As you are reviewing conversations, you will notice patterns in flows that your assistant doesn’t handle well. Rasa X has tools to help you find more conversations that fit a certain pattern and share them with your team, so that you can make steps to improve your assistant.

Filtering Conversations

When you have a lot of conversations, a good first step to finding patterns is to filter your conversations. You can filter based on intents, entities, or actions, as well as the minimum conversation length.

Rasa X conversation filters

Start by filtering on an intent – for example, the ask_howbuilt intent in the gif below. You can review the conversations where this intent was recognized, verify that the user input matches the intent, and that your assistant responds correctly.

Rasa X filter conversation by intent

Intents can also tell you a bit about how your assistant is doing. For example, you might want to filter by a goodbye intent to see when users felt a conversation was complete, or a ‘thank you’ intent to see where the assistant performed well.

Rasa X filter conversations with goodbye intent

Searching for long conversations can be useful to detect users who got stuck while talking to the assistant. In this example, only conversations that have more than 20 user messages will appear.

Rasa X filter by length of conversation

You can also filter the 24,000 conversations shown above to show only those that reference the entity product, if you want to see how the assistant handles that entity specifically.

Rasa X filter conversation with entity

Select the All time dropdown in the upper right to filter based upon the date. In this example, conversations from yesterday are selected.

Rasa X filter conversation by time

Another good use of filtering is to look for conversations where the fallback policy action was invoked. In this example, it’s action_default_fallback. This can help identify areas where your assistant didn’t understand the user, or didn’t know which action to take.

Rasa X filter conversation on fallback policy

Tagging Conversations

While reviewing conversations, you might want to take notes about the problems you discover. You also might want to take notes to make it easier for your team to process conversations together. You can achieve both with conversation tags.

Currently, you can tag a conversation using a REST API call :

curl --data '[{"value":"review today","color":"eb4034"}]' \
    http://{HOST}/api/conversations/{conversation_id}/tags

Note

Please make sure to replace {HOST} and {conversation_id} with your own values.

After that you can see that the conversation now has the newly created tag:

Rasa X Conversation With Tag Screenshot

Once you have assigned some tags to your conversations, you can filter them by their tags:

Rasa X Conversation Tag Filter Screenshot

Some examples of how you could use tags include:

  • Mark conversations when desirable user behavior happens before, during, or after a conversation (e.g. Carbon Bot uses tags to mark conversations where a user clicked on a link that indicates the user decided to look into purchasing carbon offsets)

  • Write a script to divide conversations among team members by tagging each conversation with a specific name, so everyone knows what conversations are their responsibility

  • Make it clear when a conversation has been processed by adding a reviewed tag

  • Identify patterns of where users get stuck by using a system for the different stages of the conversation

  • Let domain experts know that their review of a conversation is needed by adding a tag that tells them their expert review is needed

Flagging Conversations

While reviewing conversations for issues, you may want to flag a conversation for review by a colleague or for further discussion. Hover to the side of an user input or response and a greyed out flag will appear. Click on the flag to highlight the message.

You can also filter for flagged conversations when you want to go back and review them.

Rasa X flag a conversation

Sharing Conversations

While reviewing conversations, you might want the other members of your team to look at a conversation. To do so, you can share a link to the conversation, or even a specific message within the conversation. For example, to share a specific user message, click on the message and the browser URL will update to reference that message in that conversation. You can then copy the URL and send it to your teammates.

Rasa X share a conversation