Skip to main content

​
Configuring Evaluations

eval/conftest.yml controls which LLM models power the simulation and evaluation steps:
eval/conftest.yml
Configure simulation and evaluation independently. We recommend using gpt-5.1-2025-11-13 for both.

​
Overriding Evaluation Prompts

There are three built-in prompt templates, each overridable with a custom Jinja2 template: the simulated user prompt (drives the LLM-as-user), the criteria judge prompt (scores your quality criteria), and the metrics judge prompt (computes quality metrics):
eval/conftest.yml
Paths are relative to your project root. If omitted, the built-in templates are used.

​
Limiting concurrent LLM calls

evaluate_agent runs scenarios concurrently. Use max_concurrent_llm_calls to cap how many requests each LLM role sends its provider at once. Set it separately for simulation and evaluation, because the two roles can use different providers with different rate limits:
eval/conftest.yml
Both keys default to 3 and must be at least 1. Existing conftest.yml files without these keys keep working with the defaults. This limit is independent of the parallelism argument of evaluate_agent. parallelism controls how many runs overlap. max_concurrent_llm_calls controls what each provider receives. Tune it from your provider’s rate limits. A run uses two evaluation slots when the scenario defines goals.criteria (the criteria and metrics judges run concurrently) and one slot otherwise. Generative assertions also count against the evaluation limit. If you set these values higher than your provider can sustain, runs fail after exhausting their retries. evaluate_agent reports them as harness failures in runs_errored.

​
Default built-in prompts

Rasa ships three default Jinja2 templates for simulation and evaluation. They are selected automatically when the corresponding *_prompt key is omitted from eval/conftest.yml. Copy one into your project (for example under prompt_templates/) and point eval/conftest.yml at it when you want to customize behavior.
Drives the LLM-as-user during simulation. Injects simulation_context from the scenario YAML and returns a JSON object with the next user message and a done flag.
simulated_user_system_prompt.jinja2

​
Adding custom metrics

Built-in scale metrics are helpfulness, repair_quality, coherence, and tone. The built-in binary metric is task_completion. These are informational and do not gate the run’s pass/fail verdict. To add your own metrics (for example politeness or language_consistency):
  1. Copy the default metrics judge prompt into your project, for example in prompt_templates/metrics_evaluation.jinja2.
  2. Add a descriptive paragraph for each custom metric in the prompt body, following the same rubric style as the built-in metrics.
  3. Extend the JSON response schema at the bottom of the template so the judge returns your new keys:
  1. Point metrics_judge_prompt in eval/conftest.yml at your customized file.
Custom metrics appear in the Quality Metrics section of each run_N.txt result file alongside the built-in metrics.

​
MCP Tools

Both tools are registered on the rasa tools run FastMCP server.

​
validate_scenario

Validates a scenario YAML file before running any simulation. The skill calls this automatically after generating a scenario. You can also invoke it directly to check a file you have written or edited by hand. Checks:
  • YAML syntax and structure
  • All assertion types are valid
  • Slot names referenced in initial_slots, slot_was_set, and slot_was_not_set assertions exist in the domain with compatible types
  • All validation errors are reported together, not one at a time

​
evaluate_agent

Runs the simulation and evaluation loop for one or more scenarios. For each run it:
  1. Loads eval/conftest.yml and the scenario YAML
  2. (If initial_slots are set) sends /session_start to bootstrap the session, then injects slots into the fresh sim-<uuid> conversation
  3. Runs an LLM-simulated multi-turn conversation against your running Rasa server
  4. Fetches the tracker and runs deterministic assertions against the full event history
  5. Passes the transcript to the LLM judge, which scores criteria and computes quality metrics
  6. Writes a per-run result file
After all runs finish, evaluate_agent updates the experiment summary once. Pass all scenarios for a session in scenario_paths (a list of up to 50 paths). Rasa runs them concurrently and they share one summary. Scenario file names must be unique within a call. Use run_count (default: 1, max: 10) to run each scenario multiple times and account for LLM non-determinism. A scenario that passes 3/3 is a more reliable signal than one that passes 1/1. You can control this in natural language - the agent will set run_count accordingly:
Use parallelism (default: 3, max: 8) to set how many runs execute at once across the whole batch. Raise it only when both your Rasa server and your LLM provider have headroom. The response contains one scenario_results entry per scenario with runs_passed, runs_total, and runs_errored. runs_errored counts runs that failed for harness reasons, such as timeouts or provider errors, rather than on their merits. success is false only when the call fails before runs start or when every run failed for harness reasons. See the evaluate_agent reference for the full parameter and response schema. Each run has a 10-minute timeout, and the batch has an overall timeout capped at 2 hours. Runs that time out get a placeholder result file and count toward runs_errored. evaluate_agent still writes the summary with the results that completed.

​
Reading Results

Results are written to eval/results/<experiment_id>/ and preserved across runs:
The experiment ID is a timestamp in YYYY-MM-DD_HH-MM-SS format, generated once per session and shared across all evaluate_agent calls so that multiple scenarios land under the same folder.

​
Per-run file

Each run_N.txt contains the full evaluation output for one simulated conversation:
Quality Metrics — bot_quality is the average of the four built-in scale metrics (helpfulness, repair_quality, coherence, tone). task_completion is a binary pass/fail verdict. Quality metrics are informational: they do not gate the run’s pass/fail result. If a metric outcome must be a hard requirement, express it as a criterion instead.

​
Summary file

summary.txt contains per-scenario pass/fail counts, run ratios, per-scenario run time, a direct path to each scenario’s result folder, and a “Scenarios requiring attention” section listing any scenarios where not all runs passed:
The Time line reports two figures:
  • elapsed — wall-clock time the experiment took. It accumulates across every evaluate_agent call that shares the experiment timestamp.
  • total run time — the sum of every run’s duration. The per-scenario times above add up to this figure. Because runs overlap, it is usually larger than the elapsed time.
When some runs failed for harness reasons, the summary adds a line under Total with the count, for example 1 failed for harness reasons (timed out, cancelled, or errored); counted as failures above. It also marks the affected scenario in “Scenarios requiring attention” with [N harness failures]. A summary.json file next to summary.txt stores the accumulated per-scenario results, including runs_errored and the elapsed wall-clock time. Later calls with the same experiment timestamp merge their results into it. Rasa still reads summary files written by earlier versions.

​
Inspecting a Simulated Conversation

Every result file includes an Inspector URL. Open it in your browser to step through the conversation turn by turn and inspect slot values and tracker events. Simulated conversations use a sim- prefix on the sender ID (e.g. sim-f3a1b2c4), so they are clearly distinguished from real user conversations.