August 2nd, 2021
Prioritise Labelling with the UnexpecTEDIntentPolicy
Let's say you've got an assistant running in production. It turned out that it's a popular interface so lots of users are interacting with it. You'd like to learn from all these interactions so you set yourself up to start labelling.
This can be intimidating though. Especially when your assistant is popular, you're bound to have many conversation logs to go through. Even with a large team, going through every conversation can be a daunting task.
Instead of going through every conversation, it would be better to have a system that can help you prioritize which conversations to label first. There's nothing wrong with labelling conversations that give a good user experience but they will likely only confirm the situations that your assistant can handle. It'd be much better if we could find examples where the assistant falls short because these examples might shed light on areas where the assistant could improve.
With that in mind, we're interested in finding conversations where the user did something unexpected. The idea here is that these conversations are more likely to teach you how to improve the assistant than a conversation that went as expected.
To help Rasa users who face this problem we've recently released a new policy that does exactly this. It's called the
UnexpecTEDIntentPolicy and it's available as of Rasa 2.8.
How Does it Work?
UnexpecTEDIntentPolicy uses a model architecture which is a slight variation of
TED (hence the name!) to detect conversations where an unexpected intent occurred. The model looks at the story path in the conversation so far and as well as the currently observed intent to figure out how likely it is to see the intent in the current conversation according to the training data which the assistant was trained on.
If this similarity is too low and exceeds a threshold, the policy will emit an
action_unlikely_intent-action. This action is logged and can therefore be used in hindsight in Rasa X to help you label conversations.
How would you use it?
If you want to use the
UnexpecTEDIntentPolicy you'll need to add it to your config.yml file first.
policies: - name: UnexpecTEDIntentPolicy epochs: 200
When you now train a new model it will contain the
UnexpecTEDIntentPolicy that will emit the
action_unlikely_intent-action. This action will be logged just like any other action in the conversation which means that you can use it for filtering in Rasa X. You can find the action in the "Actions" submenu when you filter.
Once the filter is active you'll be able to focus on the conversations that have an
action_unlikely_intent in them. Since something unexpected happens in these conversations these conversations would be good candidates to prioritize.
You can learn more about the
UnexpecTEDIntentPolicy on our documentation or on the Algorithm Whiteboard youtube channel. We've made three videos that explain all the details of the new system.
- Video 1: Introducing the UnexpecTEDIntentPolicy
- Video 2: Algorithmic Details in the UnexpecTEDIntentPolicy
- Video 3: Finding Unexpected Intents in Rasa X
We hope you'll be able to use this system to find conversations that improve your assistant. If you have any feedback on the algorithm, feel free to let us know on our forum.