Test Cases Reference - Rasa Documentation

Why testing your assistant?For more information E2E testing, see the E2E testing product documentation.

To write test cases, you need to create a YAML file inside the tests directory of your project. The name of the file should be e2e_test_cases.yml. You can also create a subdirectory inside the tests directory and place your test case YAML files there. These files will be automatically discovered and run by Rasa, however you need to provide the path to the subdirectory as positional argument to the rasa test e2e command. Each input file must contain the test_cases key. The value of this key is a list of test cases. Each test case must include a name given to the test_case key and a list of test steps given to the steps key. A step can be either one of the following:

user: refers to a user’s message.
bot: denotes a textual response generated by the bot.
utter: refers to a bot response name as defined in the domain file.
slot_was_set: indicates the successful setting of a slot, and depending on how it’s defined:
- if a slot name is provided, e.g. my_slot: checks that a slot with the given name is set.
- if a key-value pair is provided, e.g. my_slot: value: checks that a slot with the given name is set with the expected value.
slot_was_not_set: specifies the failure to set a slot, and depending on the defenition:
- if a slot name is provided, e.g. my_slot: checks that a slot with the given name is not set.
- if a key-value pair is provided, e.g. my_slot: value: checks that a slot with the given name is not set or that its value differs from the expected value.

The following example illustrates how user, bot, utter and slot_was_set steps can be used in a test case:

tests/e2e_test_cases.yml

test_cases:
  - test_case: user books a restaurant # name of the test case must be provided and be unique
    steps:
      - user: I want to book a table for 4 at Xaigon for 8pm tonight
      - slot_was_set: # slot_was_set/slot_was_not_set can contain multiple slot names or key-value pairs
          - book_restaurant_name_of_restaurant: Xaigon
          - book_restaurant_number_of_people: "4"
          - book_restaurant_date: tonight
          - book_restaurant_time: 8pm
      - utter: utter_restaurant_available # utter is used for predefined domain utterances
      - utter: utter_ask_book_restaurant_reservation_name
      - user: Emil
      - slot_was_set:
          - book_restaurant_reservation_name: Emil
      - utter: utter_ask_book_restaurant_confirmation
      - user: yes
      - bot: Thank you for booking the table at Xaigon for 4 people at 8pm tonight. # bot is used to match textual responses

In each of the steps after the user step, you can specify multiple expected events. Any additional events that are found are ignored and do not cause the test to fail.

Slots

Slots can be specified as a list of either string values (representing the slot name) or of slot name and slot value pairs. If the slot is specified as a key-value pair, the the values are also compared. If the slot step contains only the slot name, it is asserted that the slot was either set (when using slot_was_set step) or not (when using slot_was_not_set step).

tests/e2e_test_cases.yml

- slot_was_set:
    - book_restaurant_name_of_restaurant: Xaigon # check that the slot is set and the value is Xaigon
    - book_restaurant_number_of_people: "4"
    - book_restaurant_time # check that the slot is set regardless the value with which it is set
- slot_was_not_set:
    - book_restaurant_reservation_name # check that the slot is not set (no SlotSet event found)
    - book_restaurant_reservation_date: "2023-11-11" # check that the slot either is not set or the value is not 2023-11-11

It is not required to specify all the slots events in the slot_was_set or slot_was_not_set steps, you can indicate only a subset of slots that you want to check. You can think of slot_was_not_set as an inverse of slot_was_set and when specifing a slot_was_not_set step it looks for the absence of the SlotSet event in the tracker store but only for that particular user step not globally (more at order of test steps).

Fixtures for Pre-Filled Slots

Using fixtures is an optional feature that enables the pre-filling of slots, ensuring specific context before individual test cases are run. You can define fixtures in either:

Test case files — using the fixtures key at the top level of a test case YAML file (e.g. e2e_test_cases.yml).
Conftest files — using the same fixtures key in a conftest.yml or conftest.yaml file at the root of your test tree or in any subfolder (see Conftest-style fixture hierarchy).

Within a single file, each fixture name must be unique. Across files, the same fixture name can be used to override a fixture from a parent scope (see Override semantics). Fixture names correspond to sets of slot key-value pairs. When a particular test case needs predefined slot values, you can reference the fixture name within the test case definition by adding it to the fixtures key. Consider the following example, which includes a test case file with fixtures and two test cases that leverage these fixtures:

tests/fixture-tests.yml

fixtures:
  - premium: # name of the fixture must be provided and be unique
      - membership_type: premium # every fixture can contain multiple slot key-value pairs
      - logged_in: True
  - standard:
      - logged_in: False
      - membership_type: standard

test_cases:
  - test_case: "test_premium_booking"
    fixtures:
      - premium # re-use the name of the fixture provided in fixtures section
    steps:
      - user: "Hi!"
      - bot: "Welcome back! How can I help you?"
      - user: "I want to book a trip."
      - utter: utter_ask_location
      - user: "I would like to travel to Lisbon."
      - slot_was_set:
          - location: "Lisbon"
      - utter: utter_ask_date
      - user: "I would like to travel on 22nd of June."
      - slot_was_set:
          - travel_date: "2023-06-22"
      - bot: "Great! I will book your trip to Lisbon on 22nd of June."
      - bot: "You saved 20% by being a premium member."

  - test_case: "test_anonymous_booking"
    fixtures:
      - standard
    steps:
      - user: "Hi!"
      - bot: "Hey! How can I help you?"
      - user: "I want to book a trip."
      - utter: utter_ask_location
      - user: "I would like to travel to Paris."
      - slot_was_set:
          - location: "Paris"
      - utter: utter_ask_date
      - user: "I would like to travel on 2nd of April."
      - slot_was_set:
          - travel_date: "2023-04-02"
      - bot: "Great! I will book your trip to Paris on 2nd of April."
      - bot: "You can also choose to save 20% by becoming a premium member."

These slots in fixtures are set after the action_session_start action and before the first step is executed by the test runner.

Conftest-style fixture hierarchy

You can define fixtures in conftest files (conftest.yml or conftest.yaml) so that all e2e tests under a given path see them, without repeating definitions in every test file. Fixtures defined in tests/conftest.yml are visible to all test files under tests/.

Folder conftest — Place a conftest file in a subfolder (e.g. tests/admin/conftest.yml). Fixtures in a subfolder are visible to test files in that folder and its subfolders (e.g. tests/admin/foo.yml, tests/admin/bar/baz.yml).

Single-file run and fixture resolution

When you pass a single test file to rasa test e2e (e.g. rasa test e2e tests/admin/foo.yml), the runner still resolves fixtures from the full hierarchy: it loads conftest files from the root of the test tree and from every folder along the path to that file. For example, for tests/admin/foo.yml it will load tests/conftest.yml and tests/admin/conftest.yml if they exist, and merge their fixtures (with override semantics) before running the tests in foo.yml. So a run with a single file sees the same set of fixtures as when that file is run as part of the full suite (e.g. rasa test e2e tests/). You do not need to copy or re-define fixtures into the single file just to run it in isolation; fixture resolution is the same in both cases. Example layout:

tests/
  conftest.yml           # fixtures here apply to all tests
  e2e_test_cases.yml
  admin/
    conftest.yml         # fixtures here apply only to tests under admin/
    foo.yml
    bar.yml

Example root conftest:

tests/conftest.yml

fixtures:
  - default_user:
      - logged_in: True
      - membership_type: standard

Test files under tests/ can then reference default_user in their test cases without defining it locally.

Fixture override semantics

Fixtures are resolved in a fixed order; a definition that appears later in the hierarchy overrides one with the same name from a parent scope:

Root — Fixtures from the root conftest (e.g. tests/conftest.yml).
Folder — Fixtures from folder-level conftest files along the path to the test file (closer folders override parent folders).
File — Fixtures defined in the test case file itself.

So: file-level fixtures override folder-level; folder-level overrides root. This lets you reuse a fixture name in a folder or file to override the root definition for that scope.

Within a single file — Duplicate fixture names are not allowed. The runner reports an error if the same fixture name appears more than once in one file (test case file or conftest).
Across files — The same fixture name in different files is allowed: the definition that wins is the one closest to the test file (file > folder conftest > root conftest).

Mocking Datetime in LLM Prompts

New in 3.15The mocked_datetime slot for testing datetime-aware LLM components is available starting from Rasa 3.15.

When testing components that include datetime information in their LLM prompts (such as CompactLLMCommandGenerator, SearchReadyLLMCommandGenerator, EnterpriseSearchPolicy, and ReAct sub-agents), you can use the mocked_datetime slot to provide a fixed datetime value for deterministic testing. The mocked_datetime slot is a built-in slot that, when set in a fixture, overrides the actual current time for all datetime-aware components during that test case execution. This ensures that tests are deterministic and don’t depend on when they are run, and that the datetime context included in LLM prompts is consistent across test runs.

Format Requirements

The mocked_datetime slot value must be provided as a string in ISO 8601 format. The following formats are supported:

ISO 8601 with timezone (recommended)
- Format: YYYY-MM-DDTHH:MM:SS±HH:MM
- Example: "2025-01-15T14:30:45+00:00" # UTC
- Example: "2025-01-15T14:30:45-05:00" # EST
- Example: "2025-12-25T14:45:30+02:00" # CEST
ISO 8601 naive (converted to UTC)
- Format: YYYY-MM-DDTHH:MM:SS
- Example: "2025-01-15T14:30:45"
Date only (time set to 00:00:00 UTC)
- Format: YYYY-MM-DD
- Example: "2025-01-15"
Space-separated (converted to UTC)
- Format: YYYY-MM-DD HH:MM:SS
- Example: "2025-01-15 14:30:45"

Example Usage

tests/datetime-tests.yml

fixtures:
  - mocked_datetime_jan_15:
      - mocked_datetime: "2025-01-15T10:30:00"  # Monday, January 15, 2025, 10:30:00 UTC
  - mocked_datetime_dec_25:
      - mocked_datetime: "2025-12-25T14:45:30-05:00"  # Wednesday, December 25, 2025, 14:45:30 EST

test_cases:
  - test_case: test_temporal_reference_with_mocked_datetime
    fixtures:
      - mocked_datetime_jan_15
    steps:
      - user: What day is it today?
      # The bot will understand "today" as Monday, January 15, 2025
      - bot: Today is Monday, January 15, 2025.
  
  - test_case: test_appointment_booking_tomorrow
    fixtures:
      - mocked_datetime_jan_15
    steps:
      - user: Can I book an appointment tomorrow?
      # The bot will correctly interpret "tomorrow" as Tuesday, January 16, 2025
      - bot: I can help you book an appointment for Tuesday, January 16, 2025.

Validation

The e2e test runner validates the mocked_datetime value before test execution. If the value is invalid (not a string, wrong format, or invalid date/time), a ValidationError is raised with a clear error message indicating the expected format. Invalid examples:

# ❌ Invalid - not a string
mocked_datetime: 20240115

# ❌ Invalid - wrong format
mocked_datetime: "01-15-2025"

# ❌ Invalid - invalid date
mocked_datetime: "2025-13-01"  # Month 13 doesn't exist

# ❌ Invalid - invalid time
mocked_datetime: "2025-01-15 25:00:00"  # Hour 25 doesn't exist

When to Use

Use the mocked_datetime slot when:

Testing datetime-dependent behavior (e.g., “today”, “tomorrow”, “next week”)
Testing timezone-specific functionality
Ensuring deterministic test results regardless of when tests are run
Testing edge cases (midnight, year boundaries, etc.)

The mocked_datetime slot only affects components that have include_date_time enabled. If a component has include_date_time: false, it will not use the mocked datetime value.

Metadata on User Messages

This feature is only relevant when you have used custom connectors with your assistant and have passed extra information from your front end in your custom actions using the metadata key of your user message and want to properly test the conversation flow based on the metadata provided at runtime. Please see the Metadata on messages section of the custom connectors documentation for more information. Using Metadata is an optional feature that enables the testing of interactions that are dynamically influenced by external metadata such as API response headers, middleware communications, or other contextual information. The metadata key at the top level of your test case configuration consists of a list of metadata names, each of which must be unique. These metadata names correspond to key-value pairs of metadata. When all user steps in a test case need predefined metadata, you can reference the metadata name within the test case definition by adding it to the metadata key. In addition to this, you can also use the metadata key in each user step to provide additional metadata. This will merge with the test case level metadata before being passed to the UserMessage object.

In case of a conflict between the metadata provided in the user step and that of the test case during a merge operation, the user step metadata takes precedence and will override the one provided by the test case.

Consider the following example, which includes a test case file with metadata and two test cases that leverage these metadata:

tests/metadata-tests.yml

metadata:
  - user_info:
      language: English
      location: Europe
  - device_info:
      os: linux

test_cases:
  - test_case: "test_standard_booking"
    metadata: user_info
    steps:
      - user: "Hi!"
      - utter: "utter_greet"
      - user: "I would like to book a trip."
      - bot: "Where would you like to travel?"
      - user: "I want to travel to Lisbon."
        metadata: device_info
      - bot: "Your trip to Lisbon has been booked."
      - bot: "You saved 15% by booking with a standard membership."
      - bot: "Upgrade for more savings."

  - test_case: "test_mood_great"
    steps:
      - user: "Hi!"
        metadata: user_info
      - bot: "Hey! How are you?"
      - user: "I am feeling wonderful."
      - bot: "Great, carry on!"

In the above example, all user steps in the test_standard_booking test case will have only the metadata user_info with the exception of the third user step, which will have the metadata device_info in addition to the user_info metadata. Also, only the first user step in the test_mood_great test case will have the user_info metadata, while other user steps have no metadata.

Stubbing Custom Actions

New Beta Feature in 3.10You can now stub custom actions in your test cases to simulate the execution of a custom action without actually running the action server.This feature is only a beta (experimental) and may change in future Rasa versions. To enable the feature, you must set the environment variable RASA_PRO_BETA_STUB_CUSTOM_ACTION to true in your testing environment.We welcome your feedback on this feature through the Customer Office team.

You can stub regular custom actions in your test cases by defining the stub_custom_actions key at the top level of your test case file. This allows you to simulate the execution of a custom action without actually running the action server. The stub_custom_actions key consists of a dictionary of custom action names that you want to stub. The value of each custom action name is a dictionary of expected events and responses. This represents what the custom action would return and must follow the same format that the action server would return. For example:

stub-tests.yml

stub_custom_actions:
  check_balance:
    events:
      - event: slot
        name: account_balance
        value: 1000
    responses:
        - text: "Your account balance is 1000."

When a custom action is stubbed, the test runner will not make a call to the action server but will instead look for the stub implementation. Note that a test run must stub all custom actions that are called in the test cases if this feature is used. If you’d like to stub only select custom actions, we recommend you create a separate test case file and run these custom actions separately with a development action server instance. You can define multiple stubs for the same custom action. Please follow the naming convention of test_case_id::action_name for the custom action name to differentiate between these stubs. For example:

stub-multiple-tests.yml

stub_custom_actions:
  test_account_balance_is_positive::check_balance:
    events:
      - event: slot
        name: account_balance
        value: 1000
    responses:
      - text: "Your account balance is 1000."

  test_account_balance_is_empty::check_balance:
    events:
      - event: slot
        name: account_balance
        value: 0
    responses:
      - text: "Your account balance is empty."

The current version of the stubbing feature does not support the stubbing of slot validation custom actions.

Limitations

End-to-end testing is a powerful tool for conducting thorough assessments of your conversational AI system. However, it does come with certain limitations that are important to consider. These limitations are as follows:

Dependency on events and the tracker store

End-to-end testing heavily relies on the availability of specific event types in the tracker store. In particular, it requires the presence of events such as BotUttered, UserUttered, and SlotSet to execute tests effectively. If your test scenario involves actions or events that do not generate these specific events, the testing algorithm is not able to evaluate them.

Order of test steps and events

It is essential to structure your test cases to closely mimic real conversations to avoid potential issues. The test runner works by running the user steps and capturing the events generated by the bot from the tracker store, after which it will compare the events generated by the bot with the expected events be it bot or slot test steps. It’s best to avoid creating test cases with mupltiple user steps followed by bot events, as it will only evaluate events created from the last user step.

faulty-tests.yml

test_cases:
  - test_case: user checks their balance and doesn't ask for anything else
    steps:
      - user: Show my balance
      - user: no
      - utter: utter_current_balance
      - utter: utter_can_do_something_else
      - utter: utter_noworries
  - test_case: user checks their balance and then about transactions
    steps:
      - user: Show my balance
      - user: Show my transactions please
      - utter: utter_current_balance
      - utter: utter_can_do_something_else
      - utter: utter_transactions

Note that the order of the bot, utter and slot steps which follow a user step is not important. The test case will pass as long as the bot, utter and slot events are executed after the user step.

Testing the start of a conversation

The evaluation of actual events against the defined expected test steps begins after the action_session_start action and it’s advisable to start the test with a user step. However it is possible to test before the first user utterance when the action_session_start has been customized.

tests/e2e_test_cases.yml

test_cases:
  - test_case: user books a restaurant
    steps:
      - utter: utter_welcome # action_session_start is modified to also utter_welcome
      - user: I want to book a table for 4 at Xaigon for 8pm tonight
    ...

​Slots

​Fixtures for Pre-Filled Slots

​Conftest-style fixture hierarchy

​Single-file run and fixture resolution

​Fixture override semantics

​Mocking Datetime in LLM Prompts

​Format Requirements

​Example Usage

​Validation

​When to Use

​Metadata on User Messages

​Stubbing Custom Actions

​Limitations

​Dependency on events and the tracker store

​Order of test steps and events

​Testing the start of a conversation

Slots

Fixtures for Pre-Filled Slots

Conftest-style fixture hierarchy

Single-file run and fixture resolution

Fixture override semantics

Mocking Datetime in LLM Prompts

Format Requirements

Example Usage

Validation

When to Use

Metadata on User Messages

Stubbing Custom Actions

Limitations

Dependency on events and the tracker store

Order of test steps and events

Testing the start of a conversation