Rasa Labs access - New in 3.7.0b1
Rasa Labs features are experimental. We introduce experimental features to co-create with our customers. If you are interested in using this feature, please contact us. These features might be changed or removed in the future.
The new intentless policy leverages large language models (LLMs) to complement existing rasa components and make it easier:
- to build assistants without needing to define a lot of intent examples
- to handle conversations where messages don't fit into intents and conversation context is necessary to choose a course of action.
question-answering bot can already understanding many different ways
that users could phrase their questions - even across a series of user messages:
This only requires appropriate responses to be defined in the domain file.
To eliminate hallucinations, the policy only chooses which response from your domain file to send. It does not generate new text.
In addition, you can control the LLM by:
- providing example conversations (end-to-end stories) which will be used in the prompt.
- setting the confidence threshold to determine when the intentless policy should kick in.
This repository contains a starter pack with a bot that uses the
IntentlessPolicy. It's a good starting point for trying out the policy and for
Webinar demo showing that this policy can already handle some advanced linguistic phenomena out of the box.
The examples in the webinar recording are also part of the end-to-end tests
defined in the example repository in (
Adding the Intentless Policy to your bot
IntentlessPolicy is part of the
rasa_plus package. To add it to your
bot, add it to your
Combining with NLU predictions
The intentless policy can be combined with NLU components which predict intents. This is useful if you want to use the intentless policy for some parts of your bot, but still want to use the traditional NLU components for other intents.
nlu_abstention_threshold can be set to a value between 0 and 1. If
the NLU prediction confidence is below this threshold, the intentless policy
will be used if it's confidence is higher than the NLU prediction. Above the
threshold, the NLU prediction will always be used.
The following example shows the default configuration in the
nlu_abstention_threshold defaults to
LLM / Embeddings configuration
You can customize the openai models used for generation and embedding.
By default, OpenAI will be used for embeddings. You can configure the
embeddings.model_name property in the
config.yml file to change the used
text-embedding-ada-002. The model name needs to be set to an
available embedidng model..
By default, OpenAI is used for LLM generation. You can configure the
llm.model_name property in the
config.yml file to specify which
OpenAI model to use:
text-davinci-003. The model name needs to be set to an
available GPT-3 LLM model.
If you want to use Azure OpenAI Service, you can configure the necessary parameters as described in the Azure OpenAI Service section.
Other LLMs / Embeddings
By default, OpenAI is used as the underlying LLM and embedding provider.
The used LLM provider and embeddings provider can be configured in the
config.yml file to use another provider, e.g.
For more information, see the LLM setup page on llms and embeddings.
For any rule-based policies in your pipeline, set
use_nlu_confidence_as_score: True. Otherwise, the rule-based policies will
always make predictions with confidence value 1.0, ignoring any uncertainty from
the NLU prediction:
This is important because the intentless policy kicks in only if the other policies are uncertain:
If there is a high-confidence NLU prediction and a matching story/rule, the
MemoizationPolicywill be used.
If there is a high-confidence NLU prediction but no matching story/ rule, the
IntentlessPolicywill kick in.
If the NLU prediction has low confidence, the
IntentlessPolicywill kick in.
IntentlessPolicyprediction has low confidence, the
RulePolicywill trigger fallback based on the
What about TED?
There is no reason why you can't also have TED in your configuration. However,
- TED frequently makes predictions with very high confidence values (~0.99) so
will often override what the
- TED and the
IntentlessPolicyare trying to solve similar problems, so your system is easier to reason about if you just use one or the other.
Steering the Intentless Policy
The first step to steering the intentless policy is adding and editing responses in the domain file. Any response in the domain file can be chosen as an response by the intentless policy. This whitelisting ensures that your assistant can never utter any inappropriate responses.
Beyond having the
utter_ prefix, the naming of the utterances is not relevant.
The second step is to add
data/e2e_stories.yml. These stories teach the LLM about your domain, so it
can figure out when to say what.
The stories and utterances in combination are used to steer the LLM. The difference here to the existing policies is, that you don't need to add a lot of intent examples to get this system going.
The policy is a usual Rasa Policy and can be tested in the same way as any other policy.
Once trained, you can test your assistant interactively by running the following command:
If a flow you'd like to implement doesn't already work out of the box, you can
add try to change the examples for the intentless policy. Don't forget that you
can also add and edit the traditional Rasa primitives like intents, entities,
slots, rules, etc. as you normally would. The
IntentlessPolicy will kick in
only when the traditional primitives have low confidence.
As part of the beta, we're also releasing a beta version of a new End-To-End
testing framework. The
rasa test e2e command allows you to test your bot
end-to-end, i.e. from the user's perspective. You can use it to test your bot in
a variety of ways, including testing the
To use the new testing framework, you need to define a set of test cases in a
test folder, e.g.
tests/e2e_test_stories.yml. The test cases are defined in a
similar format as stories are, but contain the user's messages and the bot's
responses. Here's an example:
Please ensure all your test stories have unique names! After setting the
beta feature flag for E2E testing in your current shell with
export RASA_PRO_BETA_E2E=true, you can run the tests with
rasa test e2e -f tests/e2e_test_stories.yml
The intentless policy uses the OpenAI API to create responses. This means that your users conversations are sent to OpenAI's servers.
The response generated by OpenAI is not send back to the bot's user. However, the user can craft messages that will misslead the intentless policy. These cases are handled gracefully and fallbacks are triggered.
The prompt used for classification won't be exposed to the user using prompt injection. This is because the generated response from the LLM is mapped to one of the existing responses from the domain, preventing any leakage of the prompt to the user.
More detailed information can be found in Rasa's webinar on LLM Security in the Enterprise.
What about entities?
Entities are currently not handled by the intentless policy. They have to still be dealt with using the traditional NLU approaches and slots.
What about custom actions?
At this point, the intentless policy can only predict utterances but not custom actions. Triggering custom actions needs to be done by traditional policies, such as the rule- or memoization policy.