Review Conversations

Once you’ve deployed your assistant and users have started having conversations with it, you’ll need to review the conversations to see how they’re going. To evaluate the quality of the conversations, here are some questions you may want to ask yourself:

  • How does the assistant handle unexpected user input?
  • Are user utterances being classified into the correct intent?
  • Are the assistant’s responses correct?
  • Does the conversation flow feel natural?
  • Are users able to complete their goals?
  • Do any forms and associated slot filling work?
  • Where do users get stuck and drop off?
  • What patterns are there in the conversations that go wrong?
Rasa X Conversations screen

The Conversations screen is where you review conversations. As you are reviewing conversations, you will notice patterns in flows that your assistant doesn’t handle well. Rasa X has tools to help you find more conversations that fit a certain pattern and share them with your team, so that you can make steps to improve your assistant.

Filtering Conversations

When you have a lot of conversations, a good first step to finding patterns is to filter your conversations. You can filter based on intents, entities, or actions, as well as the minimum conversation length.

Rasa X conversation filters

Start by filtering on an intent – for example, the ask_howbuilt intent in the gif below. You can review the conversations where this intent was recognized, verify that the user input matches the intent, and that your assistant responds correctly.

Rasa X filter conversation by intent

Intents can also tell you a bit about how your assistant is doing. For example, you might want to filter by a goodbye intent to see when users felt a conversation was complete, or a ‘thank you’ intent to see where the assistant performed well.

Rasa X filter conversations with goodbye intent

Searching for long conversations can be useful to detect users who got stuck while talking to the assistant. In this example, only conversations that have more than 20 user messages will appear.

Rasa X filter by length of conversation

You can also filter the 24,000 conversations shown above to show only those that reference the entity product, if you want to see how the assistant handles that entity specifically.

Rasa X filter conversation with entity

Select the All time dropdown in the upper right to filter based upon the date. In this example, conversations from yesterday are selected.

Rasa X filter conversation by time

Another good use of filtering is to look for conversations where the fallback policy action was invoked. In this example, it’s action_default_fallback. This can help identify areas where your assistant didn’t understand the user, or didn’t know which action to take.

Rasa X filter conversation on fallback policy

Flagging Conversations

While reviewing conversations for issues, you may want to flag a conversation for review by a colleague or for further discussion. Hover to the side of an user input or response and a greyed out flag will appear. Click on the flag to highlight the message.

You can also filter for flagged conversations when you want to go back and review them.

Rasa X flag a conversation

Sharing Conversations

While reviewing conversations, you might want the other members of your team to look at a conversation. To do so, you can share a link to the conversation, or even a specific message within the conversation. For example, to share a specific user message, click on the message and the browser URL will update to reference that message in that conversation. You can then copy the URL and send it to your teammates.

Rasa X share a conversation