Review Conversations

Once you’ve deployed your assistant and users have started having conversations with it, you’ll need to review the conversations to see how they’re going. To evaluate the quality of the conversations, here are some questions you may want to ask yourself:

  • How does the assistant handle unexpected user input?
  • Are user utterances being classified into the correct intent?
  • Are the assistant’s responses correct?
  • Does the conversation flow feel natural?
  • Are users able to complete their goals?
  • Do any forms and associated slot filling work?
  • Where do users get stuck and drop off?
  • What patterns are there in the conversations that go wrong?
Rasa X Conversations screen

The Conversations screen is where you review conversations. As you are reviewing conversations, you will notice patterns in flows that your assistant doesn’t handle well. Rasa X has tools to help you find more conversations that fit a certain pattern and share them with your team, so that you can make steps to improve your assistant.

Filtering Conversations

When you have a lot of conversations, a good first step to finding patterns is to filter your conversations. You can filter based on intents, entities, or actions, as well as the minimum conversation length.

Rasa X conversation filters

Start by filtering on an intent – for example, the ask_howbuilt intent in the gif below. You can review the conversations where this intent was recognized, verify that the user input matches the intent, and that your assistant responds correctly.

Rasa X filter conversations by intent

Intents can also tell you a bit about how your assistant is doing. For example, you might want to filter by a goodbye intent to see when users felt a conversation was complete, or a ‘thank you’ intent to see where the assistant performed well.

Rasa X filter conversations with goodbye intent

Searching for long conversations can be useful to detect users who got stuck while talking to the assistant. In this example, only conversations that have more than 20 user messages will appear.

Rasa X filter by length of a conversation

You can also filter the 24,000 conversations shown above to show only those that reference the entity product, if you want to see how the assistant handles that entity specifically.

Rasa X filter conversations with entity

Select the All time dropdown in the upper right to filter based upon the date. In this example, conversations from yesterday are selected.

Rasa X filter conversations by time

Another good use of filtering is to look for conversations where the fallback policy action was invoked. In this example, it’s action_default_fallback. This can help identify areas where your assistant didn’t understand the user, or didn’t know which action to take.

Rasa X filter conversations on fallback policy

Similarly, if you want to find a conversation that has a story not yet in your training data, you can select Stories not included in training data from the Other tab.

Rasa X filter conversations by not in training data

From the same tab, you can filter by the confidence of the predicted action:

Rasa X filter conversations with confidence filter

To focus on conversations from a particular channel (e.g. facebook or socketio), you can use the channels filter.

Rasa X filter conversations with channels filter

If you want to only look at conversations that contain a specific slot or slot value, you can use the slots filter.

Rasa X filter conversations with slots filter

Tagging Conversations

While reviewing conversations, you might want to take notes about the problems you discover. You also might want to take notes to make it easier for your team to process conversations together. You can achieve both with conversation tags.

Automated tagging via the Rasa X API

A great use case for automated tagging is to mark conversations when certain user behavior happens that helps you understand when your assistant is successful…

  • before a conversation (e.g. user comes from a certain marketing campaign)
  • during a conversation (e.g. user completes a task within the conversation)
  • after a conversation (e.g. user purchases following their conversation with your assistant)

For example, Carbon Bot uses tags to mark conversations where a user clicked on a link that indicates the user decided to look into purchasing carbon offsets:

Automated conversation tagging with Carbon bot

Another cool use case for assigning tags via the API is to write a script that divides conversations among team members by tagging each conversation with a specific name, so everyone knows what conversations are their responsibility. You will see many examples of this in the screenshots in Manually tagging in the Rasa X UI.

You can tag a conversation using a POST call to the Rasa X REST API:

curl --data '[{"value":"review today","color":"eb4034"}]' \
    http://{HOST}/api/conversations/{conversation_id}/tags

Note

Please make sure to replace {HOST} and {conversation_id} with your own values.

After that you can see that the conversation now has the new tag:

Rasa X Conversation With Tag Screenshot

Once you have assigned some tags to your conversations, you can filter them by their tags:

Rasa X Conversation Tag Filter Screenshot

You can also delete a tag, removing it from all conversations, using a DELETE call to the Rasa X API.

Manually tagging in the Rasa X UI

Tagging in the UI is a helpful way to take notes as you review conversations and identify patterns in the ways your users struggle to complete their task. It also enables you to establish a schema and collaborate with your team as you figure out how to improve together.

You can create a tag and add / remove tags from conversations via the Rasa X UI by clicking the gear icon on the right side panel, which brings up this Manage tags modal:

Create + add / remove tags from conversations

For example, when a conversation has been reviewed, the reviewer adds that tag, so they don’t or other people don’t look at it again:

Reviewed tag

In the following example, the assistant is collecting name, location, and if the user is an existing Rasa user. We have a tag for each of these stages, the different failures that occur, and one for if they complete the task.

Here you can see they completed the task, so the reviewer tagged it to reflect that:

Completed tag

While in this conversation, the assistant could not pick up the name of the user, so it got a tag for NLU failure at the name stage:

Name tag

When a team member does not know how to process a conversation, they add the tag expert, so a person who understands the assistant and the task being automated the best can know to review it.

Using expert tag to signal that a conversation needs expert review

That expert can come by later and filter for all of the conversations tagged with expert:

Using expert tag to signal that a conversation needs expert review

Flagging Conversations

While reviewing conversations for issues, you may want to flag a conversation for review by a colleague or for further discussion. Hover to the side of an user input or response and a greyed out flag will appear. Click on the flag to highlight the message.

You can also filter for flagged conversations when you want to go back and review them.

Rasa X flag a conversation

Sharing Conversations

While reviewing conversations, you might want the other members of your team to look at a conversation. To do so, you can share a link to the conversation, or even a specific message within the conversation. For example, to share a specific user message, click on the message and the browser URL will update to reference that message in that conversation. You can then copy the URL and send it to your teammates.

Rasa X share a conversation