Taking time to read through the conversations that users have had with your assistant is the best way to understand how users are interacting with your assistant. Too often, teams developing assistants get caught up in watching certain metrics (e.g. “what percent of users express intent X?”) to see the big picture.
Reading through real user conversations has benefits at every stage of a project. These include:
Adding NLU data that reflects how real users talk (you’ll be surprised!)
Discovering and prioritizing new goals that users expect to accomplish with the help of the assistant
Finding new ways users deviate from the happy path and adding stories to handle them
Rasa X was built first and foremost to be a tool to help you learn from real conversations. The Conversations page has many different features to help you understand your users and figure out what to do next.
In the conversations inbox, you can review all of the conversations your assistant has had with users. This includes conversations from all external channels and from the Share your Bot link and the Talk to your Bot screens.
To best understand how your users interact with the assistant, take some time to read different conversations.
In any conversation, you can view the messages the assistant and user has sent, as well as the predicted intents and actions and their confidences. For more information, click on a message or action. You’ll see what the generated story up until that point looks like, as well as the values of any slots filled at that point in the conversation.
The conversations inbox provides a number of different filters to help you find interesting conversations. Filters are available in the top left corner of the screen. You can filter by conversation length, channel, confidence, actions, intents, slots, and review status. For more information about using filters, see Finding Relevant Conversations.
You can triage conversations as you go through the inbox by clicking the Mark as reviewed, Save for later, or Delete icons. By default, the conversations inbox will show all of your conversations. You can use the ‘Review status’ filter to restrict the view to reviewed, saved, or unread conversations.
When reviewing conversations, you’ll undoubtedly come across conversation turns that have gone wrong.
When you come across these, flagging messages is a good way to keep track of issues without interrupting your reviewing flow. If you’re tagging conversations, flags can also be a way to provide more specificity to what behavior led to a certain tag.
Each message in a conversation has a flag next to it when you hover near it. Clicking this flag will turn a message red. In addition, the conversation in the list on the left will also receive a flag.
You can use flags to indicate whatever you like, but it’s best to agree as a team on what they mean. You could use one or more of the following ideas, or come up with your own:
Flag messages where the bot selected the wrong action, despite a correct NLU prediction
Flag user inputs that the bot can’t currently answer, but should in the future
Once you get to fixing problems, flags make it easy to find which conversations have problematic behavior, and which messages within a conversation are causing issues.