rasa.core.test
WrongPredictionException Objects
Raised if a wrong prediction is encountered.
EvaluationStore Objects
Class storing action, intent and entity predictions and targets.
add_to_store
Add items or lists of items to the store
merge_store
Add the contents of other to self
serialise
Turn targets and predictions to lists of equal size for sklearn.
WronglyPredictedAction Objects
The model predicted the wrong action.
Mostly used to mark wrong predictions and be able to dump them as stories.
__init__
Creates event for a successful event execution.
See the docstring of the parent class ActionExecuted
for more information.
inline_comment
A comment attached to this event. Used during dumping.
EndToEndUserUtterance Objects
End-to-end user utterance.
Mostly used to print the full end-to-end user message in the
failed_test_stories.yml
output file.
WronglyClassifiedUserUtterance Objects
The NLU model predicted the wrong user utterance.
Mostly used to mark wrong predictions and be able to dump them as stories.
inline_comment
A comment attached to this event. Used during dumping.
as_story_string
Returns text representation of event.
emulate_loop_rejection
Add ActionExecutionRejected
event to the tracker.
During evaluation, we don't run action server, therefore in order to correctly test unhappy paths of the loops, we need to emulate loop rejection.
Arguments:
partial_tracker
- a :class:rasa.core.trackers.DialogueStateTracker
test
Run the evaluation of the stories, optionally plot the results.
Arguments:
stories
- the stories to evaluate onagent
- the agentmax_stories
- maximum number of stories to considerout_directory
- path to directory to results tofail_on_prediction_errors
- boolean indicating whether to fail on prediction errors or note2e
- boolean indicating whether to use end to end evaluation or notdisable_plotting
- boolean indicating whether to disable plotting or notsuccesses
- boolean indicating whether to write down successful predictions or noterrors
- boolean indicating whether to write down incorrect predictions or not
Returns:
Evaluation summary.
compare_models_in_dir
Evaluate multiple trained models in a directory on a test set.
Arguments:
model_dir
- path to directory that contains the models to evaluatestories_file
- path to the story fileoutput
- output directory to store results to
compare_models
Evaluate provided trained models on a test set.
Arguments:
models
- list of trained model pathsstories_file
- path to the story fileoutput
- output directory to store results to