Telemetry Event Reference

Telemetry events are only reported if telemetry is enabled. A detailed explanation on the reasoning behind collecting optional telemetry events can be found in our telemetry documentation.

CALM#

Response Rephrased#

backend Triggered when a response is rephrased. Event properties:

• rephrase_all (boolean): True if the rephraser is setup to rephrase all responses.
• custom_prompt_template (string): A custom prompt template, if specified.
• llm_type (string): The type of LLM.
• llm_model (string): The model name of the LLM.

Intentless Policy Training Started#

backend Triggered when a user trains the IntentlessPolicy. Event properties:

Intentless Policy Training Completed#

backend Triggered when the training of IntentlessPolicy completed. Event properties:

• embeddings_type (string): The type of embeddings.
• embeddings_model (string): The model name for embeddings.
• llm_type (string): The type of LLM.
• llm_model (string): The model name of the LLM.

Intentless Policy Predicted#

backend Triggered when the IntentlessPolicy makes a prediction. Event properties:

• embeddings_type (string): The type of embeddings.
• embeddings_model (string): The model name for embeddings.
• llm_type (string): The type of LLM.
• llm_model (string): The model name of the LLM.
• score (number): The prediction score.

Enterprise Search Policy Training Started#

backend Triggered when a user trains the EnterpriseSearchPolicy. Event properties:

Enterprise Search Policy Training Completed#

backend Triggered when the training of EnterpriseSearchPolicy completed. Event properties:

• vector_store (string): The vector store.
• embeddings_type (string): The type of embeddings.
• embeddings_model (string): The model name for embeddings.
• llm_type (string): The type of LLM.
• llm_model (string): The model name of the LLM.
• citation_enabled (boolean): Whether source_citation is enabled or not.

Enterprise Search Policy Predicted#

backend Triggered when the EnterpriseSearchPolicy makes a prediction. Event properties:

• vector_store (string): The vector store.
• embeddings_type (string): The type of embeddings.
• embeddings_model (string): The model name for embeddings.
• llm_type (string): The type of LLM.
• llm_model (string): The model name of the LLM.
• citation_enabled (boolean): Whether source_citation is enabled or not.

End-to-End Testing#

E2E Test Run Started#

backend Triggered when end-to-end testing has been started. Event properties:

• number_of_test_cases (integer): Number of test cases to be run.
• number_of_fixtures (integer): Number of fixtures defined globally.
• uses_fixtures (boolean): Indicates if any fixtures have been defined globally.
• uses_metadata (boolean): Indicates if any metadata has been defined globally.
• number_of_metadata (integer): Number of metadata defined globally.
• uses_assertions (boolean): Indicates if any assertions have been defined in test cases.
• flow_started_count (integer): Number of flow_started assertion type used in the test run.
• flow_completed_count (integer): Number of flow_completed assertion type used in the test run.
• flow_cancelled_count (integer): Number of flow_cancelled assertion type used in the test run.
• pattern_clarification_count (integer): Number of pattern_clarification assertion type used in the test run.
• action_executed_count (integer): Number of action_executed assertion type used in the test run.
• slot_was_set_count (integer): Number of slot_was_set assertion type used in the test run.
• slot_was_not_set_count (integer): Number of slot_was_not_set assertion type used in the test run.
• bot_uttered_count (integer): Number of bot_uttered assertion type used in the test run.
• generative_response_is_relevant_count (integer): Number of generative_response_is_relevant assertion type used in the test run.
• generative_response_is_grounded_count (integer): Number of generative_response_is_grounded assertion type used in the test run.

Model Training#

Training Started#

backend A training of a Rasa machine learning model got started. The event provides information on aggregated training data statistics. Event properties:

• language (string): Language model is trained with, e.g. 'en'.
• training_id (string): Generated unique identifier for this training.
• type (string): Type of model trained, either 'nlu', 'core' or 'rasa'.
• pipeline (undefined): List of the pipeline configurations used for training.
• policies (undefined): List of the policy configurations used for training.
• train_schema (undefined): Training graph schema for graph recipe
• predict_schema (undefined): Predict graph schema for graph recipe
• num_intent_examples (integer): Number of NLU examples.
• num_entity_examples (integer): Number of entity examples.
• num_actions (integer): Number of actions defined in the domain.
• num_templates (integer): Number of templates or responses defined in the domain.
• num_conditional_response_variations (integer): Number of conditional response variations defined in the domain.
• num_slot_mappings (integer): Number of total slot mappings defined in the domain.
• num_custom_slot_mappings (integer): Number of custom slot mappings defined in the domain.
• num_conditional_slot_mappings (integer): Number of slot mappings with conditions attached defined in the domain.
• num_slots (integer): Number of slots defined in the domain.
• num_forms (integer): Number of forms defined in the domain.
• num_intents (integer): Number of intents defined in the domain.
• num_entities (integer): Number of entities defined in the domain.
• num_story_steps (integer): Number of story steps available.
• num_lookup_tables (integer): Number of different lookup tables.
• num_synonyms (integer): Total number of entity synonyms defined.
• num_regexes (integer): Total number of regexes defined.
• is_finetuning (boolean): True if a model is trained by finetuning an existing model.
• recipe (string): Recipe used in training the model, either 'default.v1' or 'graph.v1'.
• num_flows (integer): Number of flows.
• num_flows_with_nlu_trigger (integer): Number of flows that have an NLU trigger defined.
• num_flows_with_flow_guards (integer): Number of flows that have a flow guard condition.
• num_flows_with_not_startable_flow_guards (integer): Number of flows with the flow guard condition 'if: False'.
• num_collect_steps (integer): Number of collect steps in flows.
• num_collect_steps_with_separate_utter (integer): Number of collect steps which have a different utterance defined.
• num_collect_steps_with_rejections (integer): Number of collect steps with rejections included.
• num_collect_steps_with_not_reset_after_flow_ends (integer): Number of collect steps with 'reset_after_flow_ends' set to 'False'.
• num_set_slot_steps (integer): Number of set slot steps in flows.
• num_link_steps (integer): Number of link steps in flows.
• num_call_steps (integer): Number of call steps in flows.
• max_depth_of_if_construct (integer): Maximum depth of an if construct.
• num_shared_slots_between_flows (integer): Number of slots being shared across flows.
• llm_command_generator_model_name (string): The name of the model used in the 'LLMCommandGenerator'.
• llm_command_generator_custom_prompt_used (boolean): True, if a custom prompt was configured for the 'LLMCommandGenerator', False otherwise.
• multi_step_llm_command_generator_custom_handle_flows_prompt_used (boolean): True, if a custom prompt was configured for handling flows in the 'MultiStepLLMCommandGenerator', False otherwise.
• multi_step_llm_command_generator_custom_fill_slots_prompt_used (boolean): True, if a custom prompt was configured for filling slots in the 'MultiStepLLMCommandGenerator', False otherwise.
• flow_retrieval_enabled (boolean): True, if flow retrieval is configured for the 'LLMCommandGenerator', False otherwise.
• flow_retrieval (string): The name of the embedding model used by flow retrieval within 'LLMCommandGenerator'.

Training Completed#

backend The training of a Rasa machine learning model finished. The event provides information about the resulting model. Event properties:

• training_id (string): Generated unique identifier for this training. Can be used to join with 'Training Started'.
• type (string): Type of model trained, either 'nlu', 'core' or 'rasa'.
• runtime (integer): The time in seconds it took to train the model.

Model Testing#

Model Core Tested#

backend Triggered when a Core model is getting tested. Event properties:

• project (string,null): Fingerprint of the project the tested model got trained in.
• num_story_steps (integer): Number of story steps used for testing
• end_to_end (boolean): Indicates if tests are running in end-to-end mode, testing message handling and dialogue handling at the same time

Model NLU Tested#

backend Triggered when an NLU model is getting tested. Event properties:

• num_intent_examples (integer): Number of NLU examples.
• num_entity_examples (integer): Number of entity examples.
• num_lookup_tables (integer): Number of different lookup tables.
• num_synonyms (integer): Total number of entity synonyms defined.
• num_regexes (integer): Total number of regexes defined.

Model Serving#

Interactive Learning Started#

backend Triggered when an interactive learning session got started. Event properties:

• skip_visualization (boolean): Whether the visualization of stories should be shown during the interactive learning session
• save_in_e2e (boolean): Whether the data should be stored in end-to-end format

Server Started#

backend Triggered when a Rasa server gets started. Event properties:

• input_channels (array): Names of the used input channels
• api_enabled (boolean): Indicator if the API is enabled or if only the input channel is running
• number_of_workers (integer): Amount of Sanic workers started as part of the server
• endpoints_nlg (string,null): Type of the used NLG endpoint
• endpoints_nlu (string,null): Type of the used NLU endpoint
• endpoints_action_server (string,null): Type of the used action server
• endpoints_model_server (string,null): Type of the used model server
• endpoints_tracker_store (string,null): Type of the used tracker store
• endpoints_lock_store (string,null): Type of the used lock store
• endpoints_event_broker (string,null): Type of the used event broker
• project (string,null): Hash of the deployed model the server is started with

Shell Started#

backend Triggered when a shell session is started to talk to a trained bot. Event properties:

• type (string): Type of the model, either 'nlu', 'core' or 'rasa'.

Markers Extraction#

Markers Extraction Initiated#

backend Triggered when marker extraction has been initiated. Event properties:

• strategy (string): Strategy to use when selecting trackers to extract from.
• only_extract (boolean): Indicates if path to write out statistics hasn't been specified.
• seed (boolean): The seed to initialise the random number generator for use with the 'sample' strategy.
• count (integer,null): Number of trackers to extract from (for any strategy except 'all').

Markers Extracted#

backend Triggered when markers have been extracted. Event properties:

• trackers_count (integer): Number of processed trackers.

Markers Parsed#

backend Triggered when markers have been successfully parsed. Event properties:

• marker_count (integer): Number of parsed markers.
• max_depth (integer): Maximum depth of the parsed markers.
• branching_factor (integer): Maximum number of children of any of the parsed markers.

Markers Statistics Computed#

backend Triggered when marker statistics have been computed. Event properties:

• trackers_count (integer): Number of processed trackers.

Data Handling#

Training Data Split#

backend Triggered when training data gets split. Event properties:

• fraction (number): Percentage of the data which goes into training data (the rest goes into the test set).
• type (string): Type of data, either 'nlu', 'core' or 'rasa'.

Training Data Validated#

backend Triggered when training data gets validated. Event properties:

• validation_success (boolean): whether the validation was successful

Training Data Converted#

backend Triggered when training data gets converted. Event properties:

• output_format (string): target format of the converter
• type (string): Type of data, either 'nlu', 'core', 'config' or 'nlg'.

Tracker Exported#

backend Triggered when conversations get exported from a tracker store through an event broker. Event properties:

• event_broker (string): Name of the used event broker
• tracker_store (string): Name of the used tracker store
• number_of_exported_events (integer): Number of events exported through the event broker

Story Visualization Started#

backend Triggered when stories are getting visualized.

Rasa Pro Services#

Analytics Started#

backend Triggered when the Analytics pipeline is started. Event properties:

• consumers (number): The number of Kafka consumers.

Assistant Session Started#

backend Triggered when the Analytics pipeline detects a new session start. Event properties:

Miscellaneous#

Telemetry Disabled#

backend Triggered when telemetry reporting gets disabled. Last event sent before disabling telemetry. This event is not sent, if the user never enabled telemetry reporting before deactivating it.

Project Created#

backend Triggered when a project is created using rasa init. Event properties:

• init_directory (string): Hash of the directory path the project is created in