Skip to main content

Test cases reference

Why testing your assistant?

For more information E2E testing, see the E2E testing product documentation.

To write test cases, you need to create a YAML file inside the tests directory of your project. The name of the file should be e2e_test_cases.yml. You can also create a subdirectory inside the tests directory and place your test case YAML files there. These files will be automatically discovered and run by Rasa Pro, however you need to provide the path to the subdirectory as positional argument to the rasa test e2e command.

Each input file must contain the test_cases key. The value of this key is a list of test cases. Each test case must include a name given to the test_case key and a list of test steps given to the steps key. A step can be either one of the following:

  • user: refers to a user's message.
  • bot: denotes a textual response generated by the bot.
  • utter: refers to a bot response name as defined in the domain file.
  • slot_was_set: indicates the successful setting of a slot, and depending on how it's defined:
    • if a slot name is provided, e.g. my_slot: checks that a slot with the given name is set.
    • if a key-value pair is provided, e.g. my_slot: value: checks that a slot with the given name is set with the expected value.
  • slot_was_not_set: specifies the failure to set a slot, and depending on the defenition:
    • if a slot name is provided, e.g. my_slot: checks that a slot with the given name is not set.
    • if a key-value pair is provided, e.g. my_slot: value: checks that a slot with the given name is not set or that its value differs from the expected value.

The following example illustrates how user, bot, utter and slot_was_set steps can be used in a test case:

tests/e2e_test_cases.yml
test_cases:
  - test_case: user books a restaurant # name of the test case must be provided and be unique
    steps:
      - user: I want to book a table for 4 at Xaigon for 8pm tonight
      - slot_was_set: # slot_was_set/slot_was_not_set can contain multiple slot names or key-value pairs
          - book_restaurant_name_of_restaurant: Xaigon
          - book_restaurant_number_of_people: "4"
          - book_restaurant_date: tonight
          - book_restaurant_time: 8pm
      - utter: utter_restaurant_available # utter is used for predefined domain utterances
      - utter: utter_ask_book_restaurant_reservation_name
      - user: Emil
      - slot_was_set:
          - book_restaurant_reservation_name: Emil
      - utter: utter_ask_book_restaurant_confirmation
      - user: yes
      - bot: Thank you for booking the table at Xaigon for 4 people at 8pm tonight. # bot is used to match textual responses

In each of the steps after the user step, you can specify multiple expected events. Any additional events that are found are ignored and do not cause the test to fail.

Slots​

Slots can be specified as a list of either string values (representing the slot name) or of slot name and slot value pairs. If the slot is specified as a key-value pair, the the values are also compared. If the slot step contains only the slot name, it is asserted that the slot was either set (when using slot_was_set step) or not (when using slot_was_not_set step).

tests/e2e_test_cases.yml
- slot_was_set:
    - book_restaurant_name_of_restaurant: Xaigon # check that the slot is set and the value is Xaigon
    - book_restaurant_number_of_people: "4"
    - book_restaurant_time # check that the slot is set regardless the value with which it is set
- slot_was_not_set:
    - book_restaurant_reservation_name # check that the slot is not set (no SlotSet event found)
    - book_restaurant_reservation_date: "2023-11-11" # check that the slot either is not set or the value is not 2023-11-11

It is not required to specify all the slots events in the slot_was_set or slot_was_not_set steps, you can indicate only a subset of slots that you want to check. You can think of slot_was_not_set as an inverse of slot_was_set and when specifing a slot_was_not_set step it looks for the absence of the SlotSet event in the tracker store but only for that particular user step not globally (more at order of test steps).

Fixtures for Pre-Filled Slots​

Using fixtures is an optional feature that enables the pre-filling of slots, ensuring specific context before individual test cases are run. The fixtures key at the top level of your test case configuration consists of a list of fixture names, each of which must be unique. These fixture names correspond to sets of slot key-value pairs. When a particular test case needs predefined slot values, you can reference the fixture name within the test case definition by adding it to the fixtures key.

Consider the following example, which includes a test case file with fixtures and two test cases that leverage these fixtures:

tests/fixture-tests.yml
fixtures:
  - premium: # name of the fixture must be provided and be unique
      - membership_type: premium # every fixture can contain multiple slot key-value pairs
      - logged_in: True
  - standard:
      - logged_in: False
      - membership_type: standard

test_cases:
  - test_case: "test_premium_booking"
    fixtures:
      - premium # re-use the name of the fixture provided in fixtures section
    steps:
      - user: "Hi!"
      - bot: "Welcome back! How can I help you?"
      - user: "I want to book a trip."
      - utter: utter_ask_location
      - user: "I would like to travel to Lisbon."
      - slot_was_set:
          - location: "Lisbon"
      - utter: utter_ask_date
      - user: "I would like to travel on 22nd of June."
      - slot_was_set:
          - travel_date: "2023-06-22"
      - bot: "Great! I will book your trip to Lisbon on 22nd of June."
      - bot: "You saved 20% by being a premium member."

  - test_case: "test_anonymous_booking"
    fixtures:
      - standard
    steps:
      - user: "Hi!"
      - bot: "Hey! How can I help you?"
      - user: "I want to book a trip."
      - utter: utter_ask_location
      - user: "I would like to travel to Paris."
      - slot_was_set:
          - location: "Paris"
      - utter: utter_ask_date
      - user: "I would like to travel on 2nd of April."
      - slot_was_set:
          - travel_date: "2023-04-02"
      - bot: "Great! I will book your trip to Paris on 2nd of April."
      - bot: "You can also choose to save 20% by becoming a premium member."

These slots in fixtures are set after the action_session_start action and before the first step is executed by the test runner.

Metadata on User Messages​

This feature is only relevant when you have used custom connectors with your assistant and have passed extra information from your front end in your custom actions using the metadata key of your user message and want to properly test the conversation flow based on the metadata provided at runtime. Please see the Metadata on messages section of the custom connectors documentation for more information.

Using Metadata is an optional feature that enables the testing of interactions that are dynamically influenced by external metadata such as API response headers, middleware communications, or other contextual information.

The metadata key at the top level of your test case configuration consists of a list of metadata names, each of which must be unique. These metadata names correspond to key-value pairs of metadata. When all user steps in a test case need predefined metadata, you can reference the metadata name within the test case definition by adding it to the metadata key. In addition to this, you can also use the metadata key in each user step to provide additional metadata. This will merge with the test case level metadata before being passed to the UserMessage object.

note

In case of a conflict between the metadata provided in the user step and that of the test case during a merge operation, the user step metadata takes precedence and will override the one provided by the test case.

Consider the following example, which includes a test case file with metadata and two test cases that leverage these metadata:

tests/metadata-tests.yml
metadata:
  - user_info:
      language: English
      location: Europe
  - device_info:
      os: linux

test_cases:
  - test_case: "test_standard_booking"
    metadata: user_info
    steps:
      - user: "Hi!"
      - utter: "utter_greet"
      - user: "I would like to book a trip."
      - bot: "Where would you like to travel?"
      - user: "I want to travel to Lisbon."
        metadata: device_info
      - bot: "Your trip to Lisbon has been booked."
      - bot: "You saved 15% by booking with a standard membership."
      - bot: "Upgrade for more savings."

  - test_case: "test_mood_great"
    steps:
      - user: "Hi!"
        metadata: user_info
      - bot: "Hey! How are you?"
      - user: "I am feeling wonderful."
      - bot: "Great, carry on!"

In the above example, all user steps in the test_standard_booking test case will have only the metadata user_info with the exception of the third user step, which will have the metadata device_info in addition to the user_info metadata. Also, only the first user step in the test_mood_great test case will have the user_info metadata, while other user steps have no metadata.

Stubbing Custom Actions​

New Beta Feature in 3.10

You can now stub custom actions in your test cases to simulate the execution of a custom action without actually running the action server.

This feature is only a beta (experimental) and may change in future Rasa Pro versions. To enable the feature, you must set the environment variable RASA_PRO_BETA_STUB_CUSTOM_ACTION to true in your testing environment.

We welcome your feedback on this feature through the Customer Office team.

You can stub regular custom actions in your test cases by defining the stub_custom_actions key at the top level of your test case file. This allows you to simulate the execution of a custom action without actually running the action server. The stub_custom_actions key consists of a dictionary of custom action names that you want to stub. The value of each custom action name is a dictionary of expected events and responses. This represents what the custom action would return and must follow the same format that the action server would return.

For example:

stub-tests.yml
stub_custom_actions:
  check_balance:
    events:
      - event: slot
        name: account_balance
        value: 1000
    responses:
        - text: "Your account balance is 1000."

When a custom action is stubbed, the test runner will not make a call to the action server but will instead look for the stub implementation. Note that a test run must stub all custom actions that are called in the test cases if this feature is used. If you'd like to stub only select custom actions, we recommend you create a separate test case file and run these custom actions separately with a development action server instance.

You can define multiple stubs for the same custom action. Please follow the naming convention of test_case_id::action_name for the custom action name to differentiate between these stubs.

For example:

stub-multiple-tests.yml
stub_custom_actions:
  test_account_balance_is_positive::check_balance:
    events:
      - event: slot
        name: account_balance
        value: 1000
    responses:
      - text: "Your account balance is 1000."

  test_account_balance_is_empty::check_balance:
    events:
      - event: slot
        name: account_balance
        value: 0
    responses:
      - text: "Your account balance is empty."
info

The current version of the stubbing feature does not support the stubbing of slot validation custom actions.

Limitations​

End-to-end testing is a powerful tool for conducting thorough assessments of your conversational AI system. However, it does come with certain limitations that are important to consider. These limitations are as follows:

Dependency on events and the tracker store​

End-to-end testing heavily relies on the availability of specific event types in the tracker store. In particular, it requires the presence of events such as BotUttered, UserUttered, and SlotSet to execute tests effectively. If your test scenario involves actions or events that do not generate these specific events, the testing algorithm is not able to evaluate them.

Order of test steps and events​

It is essential to structure your test cases to closely mimic real conversations to avoid potential issues. The test runner works by running the user steps and capturing the events generated by the bot from the tracker store, after which it will compare the events generated by the bot with the expected events be it bot or slot test steps. It's best to avoid creating test cases with mupltiple user steps followed by bot events, as it will only evaluate events created from the last user step.

faulty-tests.yml
test_cases:
  - test_case: user checks their balance and doesn't ask for anything else
    steps:
      - user: Show my balance
      - user: no
      - utter: utter_current_balance
      - utter: utter_can_do_something_else
      - utter: utter_noworries
  - test_case: user checks their balance and then about transactions
    steps:
      - user: Show my balance
      - user: Show my transactions please
      - utter: utter_current_balance
      - utter: utter_can_do_something_else
      - utter: utter_transactions

Note that the order of the bot, utter and slot steps which follow a user step is not important. The test case will pass as long as the bot, utter and slot events are executed after the user step.

Testing the start of a conversation​

The evaluation of actual events against the defined expected test steps begins after the action_session_start action and it's advisable to start the test with a user step. However it is possible to test before the first user utterance when the action_session_start has been customized.

tests/e2e_test_cases.yml
test_cases:
  - test_case: user books a restaurant
    steps:
      - utter: utter_welcome # action_session_start is modified to also utter_welcome
      - user: I want to book a table for 4 at Xaigon for 8pm tonight
    ...
Last updated on