> ## Documentation Index
> Fetch the complete documentation index at: https://rasa.com/docs/llms.txt
> Use this file to discover all available pages before exploring further.

# Tracing

> Trace model training, messages, action-server requests, and streaming voice calls with OpenTelemetry.

Distributed tracing tracks requests as they flow through a distributed
system (in this case: a Rasa assistant), sending data about the requests
to a **tracing backend** which collects all trace data and enables
inspecting it. Trace data helps you understand the flow of requests
through both the components of a single service (Rasa itself), and across
different distributed services, for example, your action server.

### Supported Tracing Backends/Collectors

To trace requests in Rasa, you can either use
[Jaeger](https://www.jaegertracing.io/) as a backend, or use
the [OTEL Collector (OpenTelemetry Collector)](https://opentelemetry.io/docs/collector/).
to collect traces and then send them to the backend of your choice.
See [Configuring a Tracing Backend or Collector](/docs/reference/integrations/tracing.md#configuring-a-tracing-backend-or-collector)
for instructions.

### Rasa Channels

Trace context sent along with requests using the [W3C Trace Context Specification](https://www.w3.org/TR/trace-context/)
via the REST channel is used to continue tracing in Rasa.

#### Rasa Inspector

If you have enabled tracing in Rasa and are using the [Rasa Inspector](/docs/pro/testing/trying-assistant.md)
debugging tool to try your assistant, note that in addition to the expected tracing span for the `Agent.handle_message`
method call, the tracing backend will collect independent tracing spans for the `MessageProcessor.get_tracker`
method calls. This is expected behaviour because the Rasa Inspector tool uses the Rasa [HTTP API endpoints](/docs/reference/api/pro/rasa-pro-rest-api.md) to
retrieve the conversation tracker which is required by the Inspector interface.

### Action Server

The trace context from Rasa is sent along with requests to the custom action server
using the [W3C Trace Context Specification](https://www.w3.org/TR/trace-context/) and
then used to continue tracing the request through the custom action server.

Tracing is continued in the action server by instrumenting the webhook that receives custom actions.
See [Action server attributes](/docs/reference/integrations/tracing.md#action-executor-attributes) for the attributes captured as part of the trace context.

See [traced events](/docs/reference/integrations/tracing.md#traced-events) for
details on what attributes are made available as part of the trace context in Rasa.

## Questions Tracing Can Help Answer

Tracing can help troubleshoot issues in development and production, by answering questions such as:

* How does a user message request get processed across different components i.e. dialogue understanding components (NLU, `CommandGenerator`, `CommandProcessorComponent`), policies, and action server?
* Why has my Rasa assistant decided to execute a certain action?
* Why has my Rasa assistant been slow to respond?
* Why have my custom actions been slow to execute?
* What is my OpenAI prompt token usage?
* What is the performance of my Rasa assistant across different flows?
* What is the performance of my Rasa assistant across different LLM models?
* What is the performance of my Rasa assistant across different vector stores?

## Configuring a Tracing Backend or Collector

To configure a tracing backend or collector, add a `tracing` entry to your endpoints
i.e. in your `endpoints.yml` file, or in the relevant section of your Helm values in a deployment.

> [!NOTE]
>
> Enabling tracing activates Rasa's instrumentation layer. That layer also records [Observability Metrics](/docs/pro/improve/observability-metrics.md). If you configure a `metrics` block in your endpoints file, you must also enable tracing or no metric measurements will be collected.

### Jaeger

To configure a Jaeger tracing backend, specify the `type` as `jaeger`.

```yaml title="endpoints.yml" theme={null}
tracing:
  type: jaeger
  host: localhost
  port: 4317
  service_name: rasa
  sync_export: ~
```

> [!TIP]
>
> If you come across the error "OSError: \[Errno 40] Message too long",
> read the instructions [here to resolve it](/docs/pro/installation/troubleshooting.md#oserror-errno-40-message-too-long)

### OTEL Collector

Collectors are components that collect traces in a vendor-agnostic way and then forward them to various backends.
For example, the OpenTelemetry Collector (OTEL) can collect traces from multiple different components and instrumentation libraries, and then export them to multiple different backends e.g. jaeger.

To configure an OTEL Collector, specify the `type` as `otlp`.

```yaml title="endpoints.yml" theme={null}
tracing:
  type: otlp
  endpoint: my-otlp-host:4317
  insecure: false
  service_name: rasa
  root_certificates: ./tests/unit/tracing/fixtures/ca.pem
```

## Traced Events

The Rasa service areas that are traceable cover the actions required to:

* **train a model** (i.e., the training of each graph component)
* **handle a message**
* **handle a streaming voice call**

### Model Training

Tracing is enabled for model training by instrumenting Rasa [`GraphTrainer`](https://legacy-docs-oss.rasa.com/docs/rasa/reference/rasa/engine/training/graph_trainer/#graphtrainer-objects) and [`GraphNode`](https://legacy-docs-oss.rasa.com/docs/rasa/reference/rasa/engine/graph/#graphnode-objects) classes.

#### `GraphTrainer` Attributes

The following attributes can be inspected during training of `GraphTrainer`:

* `training_type` of model configuration:
  * `"NLU"`
  * `"CORE"`
  * `"BOTH"`
  * `"END-TO-END"`
* `language` of model configuration
* `recipe_name` used in the `config.yml` file
* `output_filename`: the location where the packaged model is saved
* `is_finetuning`: boolean argument, if `True` enables incremental training

#### `GraphNode` Attributes

The following attributes are captured during the training (as well as prediction during message handling) of every graph node:

* `node_name`
* `component_class`
* `fn_name`: method of component class that gets called

### Message Handling

The following Rasa classes are instrumented to enable tracing during message handling:

* `Agent`
* `MessageProcessor`
* [`TrackerStore`](/docs/reference/integrations/tracker-stores)
* [`LockStore`](/docs/reference/integrations/lock-stores)
* [`CompactLLMCommandGenerator`](/docs/reference/integrations/tracing#compactllmcommandgenerator-attributes)
* [`SearchReadyLLMCommandGenerator`](/docs/reference/integrations/tracing#searchreadycommandgenerator-attributes)
* [`NLUCommandAdapter`](/docs/reference/integrations/tracing#nlucommandadapter-attributes)
* [`FlowPolicy`](/docs/reference/config/policies/flow-policy)
* [`EnterpriseSearchPolicy`](/docs/reference/integrations/tracing#enterprisesearchpolicy-attributes)
* [`InformationRetrieval`](/docs/reference/integrations/tracing#informationretrieval-attributes)
* [`EndpointConfig`](/docs/reference/integrations/tracing#endpointconfig-attributes)

In addition, the following Python modules were instrumented to enable tracing during message handling:

* command processor module, i.e. utility functions leveraged by the `CommandProcessorComponent` to pre-process [predicted commands](/docs/reference/config/components/llm-command-generators.md#command-reference)
* flow executor module, i.e. utility functions leveraged by `FlowPolicy` to advance flows

Namely, these operations are now traceable:

* receiving a message
* parsing the message
* predicting [commands](/docs/reference/config/components/llm-command-generators.md#command-reference)
* [pre-processing commands](/docs/reference/integrations/tracing.md#command-processor-module-attributes)
* predicting the next action
* running the action
* advancing [flows](/docs/reference/integrations/tracing.md#flow-executor-module-attributes)
* searching documents in [vector stores](/docs/reference/config/policies/enterprise-search-policy.md#vector-store) for enterprise search
* generating LLM answers by policies e.g. [`EnterpriseSearchPolicy`](/docs/reference/config/policies/enterprise-search-policy)
* tracing [prompt token usage](/docs/reference/integrations/tracing.md#tracing-prompt-token-usage)
* retrieving and saving the tracker
* locking the conversation
* publishing to the event broker
* making requests to the action server or nlg server
* passing the trace context to the action server

#### Tracing prompt token usage

> [!NOTE]
>
> **New in 3.8**
>
> Tracing prompt token usage for OpenAI models is available starting with version `3.8.0`.

Tracing prompt token usage is available for the following classes if you're using OpenAI models:

* `CompactLLMCommandGenerator` class
* `SearchReadyLLMCommandGenerator` class
* `EnterpriseSearchPolicy` class
* `ContextualResponseRephraser` class

The prompt token usage is captured as part of the trace context and can be used to monitor the usage of prompt tokens in the LLM answer generation process.
This is only captured if one of instrumented classes mentioned above is configured to enable capturing the length of the prompt tokens.
For example, the `CompactLLMCommandGenerator` can be configured to trace the length of the prompt tokens by setting the `trace_prompt_tokens` attribute to `true` in the `config.yml` file:

```yaml title="config.yml" theme={null}
pipeline:
  - name: CompactLLMCommandGenerator
    trace_prompt_tokens: true
```

It is highly recommended to enable tracing of prompt tokens only in development and not in production, because it could increase assistant response latency.

#### `Agent` Attributes

Tracing the `Agent` instance handling a message captures the following attributes:

* `input_channel`: the name of the channel connector
* `sender_id`: the conversation id
* `model_id`: a unique identifier for the model
* `model_name`: the model name

#### `MessageProcessor` Attributes

The following `MessageProcessor` attributes are extracted during the tracing:

* `number_of_events`: number of events in tracker
* `action_name`: the name of the predicted and executed action
* `sender_id`: the conversation id of the `DialogueStateTracker` object
* `message_id`: the unique message id

The latter three attributes are also injected in the trace context that gets passed to the requests made to the custom action server.

#### `TrackerStore` & `LockStore` Attributes

Observable `TrackerStore` and `LockStore` attributes include:

* `number_of_streamed_events`: number of new events to stream
* `broker_class`: the `EventBroker` on which the new events are published
* `lock_store_class`: Name of lock store used to lock conversations while messages are actively processed

#### `CompactLLMCommandGenerator` Attributes

The following attributes are captured as part of the trace context of the `CompactLLMCommandGenerator`:

* `class_name`: the name of the instrumented component class
* `llm_model`: the name of the LLM used
* `llm_type`: the type of LLM used
* `embeddings`: the embeddings used
* `llm_temperature`: the temperature used for LLM answer generation
* `request_timeout`: the timeout for the LLM request
* `llm_engine`: the engine used for LLM answer generation
* `len_prompt_tokens`: the token length of the prompt (optional, only supported for OpenAI models). To enable this attribute, see instructions in the [Tracing prompt token usage](/docs/reference/integrations/tracing.md#tracing-prompt-token-usage) section.

#### `SearchReadyCommandGenerator` Attributes

The following attributes are captured as part of the trace context of the `SearchReadyCommandGenerator`:

* `class_name`: the name of the instrumented component class
* `llm_model`: the name of the LLM used
* `llm_type`: the type of LLM used
* `embeddings`: the embeddings used
* `llm_temperature`: the temperature used for LLM answer generation
* `request_timeout`: the timeout for the LLM request
* `llm_engine`: the engine used for LLM answer generation
* `len_prompt_tokens`: the token length of the prompt (optional, only supported for OpenAI models). To enable this attribute, see instructions in the [Tracing prompt token usage](/docs/reference/integrations/tracing.md#tracing-prompt-token-usage) section.

#### `NLUCommandAdapter` Attributes

> [!NOTE]
>
> **New in 3.8**
>
> Tracing the described `NLUCommandAdapter` attributes is available starting with version `3.8.0`.

The following attributes are captured as part of the trace context of the `NLUCommandAdapter`:

* `commands`: the predicted commands
* `intent`: the predicted intent of the user message that the `NLUCommandAdapter` receives as input

#### Command Processor Module Attributes

> [!NOTE]
>
> **New in 3.8**
>
> Tracing the described command processor module attributes is available starting with version `3.8.0`.

The following attributes are captured as part of the trace context of the command processor module functions:

1. `execute_commands` function:
   * `number_of_events`: the number of events in the tracker
   * `sender_id`: the conversation id of the `DialogueStateTracker` object
2. `validate_state_of_commands` function:
   * `cleaned_up_commands`: list of cleaned up commands
3. `clean_up_commands` function:
   * `commands`: list of originally parsed commands from the LLM answer
   * `current_context`: the current context of the dialogue stack
4. `remove_duplicated_set_slots` function:
   * `resulting_events`: list of events prior to removing duplicated set slot events; note that slot values are removed to prevent PII leakage

#### Flow Executor Module Attributes

> [!NOTE]
>
> **New in 3.8**
>
> Tracing the described flow executor module attributes is available starting with version `3.8.0`.

The following attributes are captured as part of the trace context of the flow executor module functions:

1. `advance_flow` function:
   * `available_actions`: list of available actions
   * `current_context`: the current context of the dialogue stack
2. `advance_flows_until_next_action` function:
   * `action_name`: the name of the action to be executed
   * `score`: the score of the executed action
   * `metadata`: the prediction metadata
   * `events`: list of event names if available
3. `run_step` function:
   * `step_custom_id`: the custom id of the step if available
   * `step_description`: the description of the step if available
   * `current_flow_id`: the id of the current flow
   * `current_context`: the current context of the dialogue stack

#### `Policy` subclasses attributes

> [!NOTE]
>
> **New in 3.8**
>
> Tracing the described `Policy` subclasses' attributes is available starting with version `3.8.0`.

The following attributes are captured as part of the trace context of subclasses of the `Policy` interface, e.g. `FlowPolicy`, `EnterpriseSearchPolicy`:

* `priority`: the priority of the policy which made the prediction
* `events`: a list of event names which are applied independent of whether the policy wins against other policies or not
* `optional_events`: a list of optional event names if available else `None` - these events are applied if the policy wins against other policies
* `is_end_to_end_prediction`: a boolean indicating if the prediction used the text of the user message instead of the intent
* `is_no_user_prediction`: a boolean indicating if the prediction uses neither the text of the user message nor the intent
* `diagnostic_data`: intermediate results or other information that is not necessary for Rasa to function, but intended for debugging and fine-tuning purposes
* `action_metadata`: additional metadata that can be passed by policies

#### `EnterpriseSearchPolicy` Attributes

> [!NOTE]
>
> **New in 3.8**
>
> Tracing the described `EnterpriseSearchPolicy` attributes is available starting with version `3.8.0`.

The `EnterpriseSearchPolicy._generate_llm_answer` method captures the same attributes as the [`CompactLLMCommandGenerator` class](/docs/reference/integrations/tracing#compactllmcommandgenerator-attributes).

#### `InformationRetrieval` Attributes

> [!NOTE]
>
> **New in 3.8**
>
> Tracing the described `InformationRetrieval` subclasses' attributes is available starting with version `3.8.0`.

The following attributes are captured as part of the trace context of the `InformationRetrieval` subclasses, e.g. `Milvus_Store`, `Qdrant_Store`:

* `query`: the query used to search the vector store
* `document_metadata`: the metadata of the documents retrieved from the vector store

#### `EndpointConfig` Attributes

> [!NOTE]
>
> **New in 3.8**
>
> Tracing the described `EndpointConfig` attributes is available starting with version `3.8.0`.

The following attributes are captured as part of the trace context of the `EndpointConfig`:

* `url`: the url of the endpoint
* `request_body_size_in_bytes`: the size of the request body in bytes

### Voice calls

When voice tracing is enabled, each call produces a trace that follows provider
connections, input handling, speech synthesis, audio playback, and call recording
when recording is enabled.
This section catalogs those spans, their attributes, and the events recorded on
each span.

#### Typical trace

The tree shows common parent-child relationships, not execution order.

📞 [voice call](#voice-call-span)

├── 🔌 [connect asr](#connect-asr-span)

├── 🔌 [connect tts](#connect-tts-span)

├── 📝 [transcribe audio](#transcribe-audio-span)

├── ⚡ [handle barge-in](#handle-barge-in-span)

├── 🔢 [handle dtmf](#handle-dtmf-span)

├── ⏱️ [handle silence](#handle-silence-span)

├── ⚙️ MessageProcessor spans

│ ├── 🗣️ [synthesize speech](#synthesize-speech-span)

│ └── 🔊 [play audio](#play-audio-span)

├── ⚠️ [process channel message](#process-channel-message-span) 
(failures only)

├── ⚠️ [compose event](#compose-event-span) (failures only)

├── 💾 [store call recording](#store-call-recording-span) (when
recording is enabled)

├── 🔌 [disconnect asr](#disconnect-asr-span)

└── 🔌 [disconnect tts](#disconnect-tts-span)

#### Content capture

Content capture is off by default. Set the `RASA_VOICE_CAPTURE_CONTENT`
environment variable to `true`, `1`, or `yes` (case-insensitive, surrounding
whitespace ignored) to include supported transcript and generated-text
attributes in traces.

> [!NOTE]
>
> Content capture never includes raw audio, partial transcripts, or DTMF digits.
> Audio is reported only as byte and chunk totals. Call-recording storage
> locators are identifiers, not content, and are emitted without this gate.

#### Attributes shared by voice spans

The following correlation fields are added to every span created inside an active
voice call. They are not included when the span does not belong to a voice call.

| Attribute | Type | Availability | Data classification | Description |
| - | - | - | - | - |
| `rasa.conversation.id` | string | Inside a voice call | [Identifier](#privacy-identifier) | Conversation ID used to correlate the voice call with its conversation. |
| `voice.channel.call_id` | string | Inside a voice call | [Identifier](#privacy-identifier) | Channel-assigned call ID. It can differ from the conversation ID. |

Every span that Rasa marks `ERROR` also has:

| Attribute | Type | Availability | Data classification | Description |
| - | - | - | - | - |
| `error.type` | string | On error | [Operational](#privacy-operational) | Exception class name, such as `RuntimeError`. |

> [!NOTE]
>
> `exception` events can include messages and stack traces.
> `RASA_VOICE_CAPTURE_CONTENT` does not redact them.

#### `voice call` span

| Detail | Description |
| - | - |
| Span kind | `INTERNAL` |
| Lifetime | Opens before provider and session setup. Ends after active processing, provider disconnects, and transport teardown. |
| Typical parent | The active OpenTelemetry span, when present. Otherwise `voice call` is the trace root. |
| Error behavior | Failures during setup, active processing, or teardown mark it `ERROR`. Cancellation alone does not. |

**Attributes**

| Attribute | Type | Availability | Data classification | Description |
| - | - | - | - | - |
| `rasa.conversation.id` | string | Always | [Identifier](#privacy-identifier) | Conversation correlation ID. |
| `voice.channel.call_id` | string | Always | [Identifier](#privacy-identifier) | Channel call ID. |
| `voice.channel.name` | string | Always | [Operational](#privacy-operational) | Voice channel name. |
| `voice.asr.provider` | string | When configured | [Operational](#privacy-operational) | Configured ASR provider. |
| `voice.tts.provider` | string | When configured | [Operational](#privacy-operational) | Configured TTS provider. |
| `voice.locale` | string | When known | [Operational](#privacy-operational) | Call locale at span creation. |
| `audio.input.bytes.total` | integer | At span end | [Operational](#privacy-operational) | Total incoming audio bytes observed during the call. |
| `audio.input.chunk.count` | integer | At span end | [Operational](#privacy-operational) | Number of incoming audio chunks observed during the call. |
| `audio.output.bytes.total` | integer | At span end | [Operational](#privacy-operational) | Total outgoing audio bytes delivered during the call. |
| `audio.output.chunk.count` | integer | At span end | [Operational](#privacy-operational) | Number of outgoing audio chunks delivered during the call. |
| `error.type` | string | On error | [Operational](#privacy-operational) | Present only when the call is marked `ERROR`. |

**Events**

| Event | Description | Attribute | Type | Availability | Data classification |
| - | - | - | - | - | - |
| `voice.call_started` | Call setup completed and the session started. | None | — | — | — |
| `voice.hangup_requested` | The assistant requested a hangup. Goodbye audio can still finish first. | None | — | — | — |
| `voice.disconnect` | The channel reported a transport-level disconnect. | None | — | — | — |
| `voice.call_ended` | Active call processing handed over to teardown. | None | — | — | — |
| `voice.turn_event_enqueued` | A voice input was processed and sent to the assistant for handling. | `voice.turn_event.type` | string | Always | [Operational](#privacy-operational) |
| | | `voice.queue.size` | integer | Always | [Operational](#privacy-operational) |

> [!NOTE]
>
> `voice.call_ended` ends active processing, not the `voice call` span.
> Disconnect of ASR and TTS and teardown can follow.

#### Provider connection spans

The four connection spans are `CLIENT` spans. Each span covers one provider
connection or disconnection attempt. A failed attempt adds `error.type`,
records a standard `exception` event, and marks the span `ERROR`.

##### `connect asr` span

Opens for every ASR connection attempt, including reconnects, and ends when the
attempt succeeds or fails.

| Detail | Description |
| - | - |
| Span kind | `CLIENT` |
| Lifetime | One ASR connection attempt, including a reconnect. |
| Typical parent | `voice call`. |
| Error behavior | A failed attempt records `error.type` and a standard `exception` event and marks the span `ERROR`. |

**Attributes**

| Attribute | Type | Availability | Data classification | Description |
| - | - | - | - | - |
| `voice.asr.provider` | string | Always | [Operational](#privacy-operational) | ASR provider name. |
| `voice.locale` | string | When configured | [Operational](#privacy-operational) | Rasa language key. |
| `voice.asr.language` | string | When configured | [Operational](#privacy-operational) | Provider language key. |
| `voice.asr.model` | string | When configured | [Operational](#privacy-operational) | Provider model. |
| `voice.connect.reconnect` | boolean | On reconnect | [Operational](#privacy-operational) | `true` after an earlier successful connection. |
| `error.type` | string | On error | [Operational](#privacy-operational) | Present only on failure. |

**Events**

No custom span events. A standard `exception` event is recorded on failure.

##### `disconnect asr` span

Opens for every ASR disconnection attempt and ends when the attempt succeeds or
fails. It never carries `voice.connect.reconnect`.

| Detail | Description |
| - | - |
| Span kind | `CLIENT` |
| Lifetime | One ASR disconnection attempt. |
| Typical parent | `voice call`. |
| Error behavior | A failed attempt records `error.type` and a standard `exception` event and marks the span `ERROR`. |

**Attributes**

| Attribute | Type | Availability | Data classification | Description |
| - | - | - | - | - |
| `voice.asr.provider` | string | Always | [Operational](#privacy-operational) | ASR provider name. |
| `voice.locale` | string | When configured | [Operational](#privacy-operational) | Rasa language key. |
| `voice.asr.language` | string | When configured | [Operational](#privacy-operational) | Provider language key. |
| `voice.asr.model` | string | When configured | [Operational](#privacy-operational) | Provider model. |
| `error.type` | string | On error | [Operational](#privacy-operational) | Present only on failure. |

**Events**

No custom span events. A standard `exception` event is recorded on failure.

##### `connect tts` span

Opens for every TTS connection attempt, including reconnects, and ends when the
attempt succeeds or fails.

| Detail | Description |
| - | - |
| Span kind | `CLIENT` |
| Lifetime | One TTS connection attempt, including a reconnect. |
| Typical parent | `voice call`. |
| Error behavior | A failed attempt records `error.type` and a standard `exception` event and marks the span `ERROR`. |

**Attributes**

| Attribute | Type | Availability | Data classification | Description |
| - | - | - | - | - |
| `voice.tts.provider` | string | Always | [Operational](#privacy-operational) | TTS provider name. |
| `voice.locale` | string | When configured | [Operational](#privacy-operational) | Rasa language key. |
| `voice.tts.language` | string | When configured | [Operational](#privacy-operational) | Provider language key. |
| `voice.tts.model` | string | When configured | [Operational](#privacy-operational) | Provider model. |
| `voice.connect.reconnect` | boolean | On reconnect | [Operational](#privacy-operational) | `true` after an earlier successful connection. |
| `error.type` | string | On error | [Operational](#privacy-operational) | Present only on failure. |

**Events**

No custom span events. A standard `exception` event is recorded on failure.

##### `disconnect tts` span

Opens for every TTS disconnection attempt and ends when the attempt succeeds or
fails. It never carries `voice.connect.reconnect`.

| Detail | Description |
| - | - |
| Span kind | `CLIENT` |
| Lifetime | One TTS disconnection attempt. |
| Typical parent | `voice call`. |
| Error behavior | A failed attempt records `error.type` and a standard `exception` event and marks the span `ERROR`. |

**Attributes**

| Attribute | Type | Availability | Data classification | Description |
| - | - | - | - | - |
| `voice.tts.provider` | string | Always | [Operational](#privacy-operational) | TTS provider name. |
| `voice.locale` | string | When configured | [Operational](#privacy-operational) | Rasa language key. |
| `voice.tts.language` | string | When configured | [Operational](#privacy-operational) | Provider language key. |
| `voice.tts.model` | string | When configured | [Operational](#privacy-operational) | Provider model. |
| `error.type` | string | On error | [Operational](#privacy-operational) | Present only on failure. |

**Events**

No custom span events. A standard `exception` event is recorded on failure.

#### `transcribe audio` span

| Detail | Description |
| - | - |
| Span kind | `INTERNAL` |
| Lifetime | One user utterance. Opens on the first partial transcript, or on a final transcript when no partial arrived, and normally closes on the final transcript. An empty final with no open utterance creates no span. |
| Typical parent | `voice call`. |
| Error behavior | An ASR stream error marks it `ERROR`. Stream teardown closes an open span without treating the missing final as an error. |

**Attributes**

| Attribute | Type | Availability | Data classification | Description |
| - | - | - | - | - |
| `voice.asr.provider` | string | Always | [Operational](#privacy-operational) | ASR provider name. |
| `voice.locale` | string | When configured | [Operational](#privacy-operational) | Rasa language key. |
| `voice.asr.language` | string | When configured | [Operational](#privacy-operational) | Provider language key. |
| `voice.asr.model` | string | When configured | [Operational](#privacy-operational) | Provider model. |
| `partial_transcript.count` | integer | At span end | [Operational](#privacy-operational) | Total number of partial transcript updates, including updates without a sampled event. |
| `voice.transcript.final` | string | On a final transcript when content capture is enabled | [Content](#privacy-content) | Final transcript. |
| `error.type` | string | On error | [Operational](#privacy-operational) | Present only on ASR failure. |

**Events**

| Event | Description | Attribute | Type | Availability | Data classification |
| - | - | - | - | - | - |
| `voice.partial_transcript` | Sampled partial-transcript update. The first partial and then every tenth partial are recorded: 1, 11, 21, and so on. | None | — | — | — |
| `voice.final_transcript` | Final transcript arrived. Its text is never placed on the event. | None | — | — | — |
| `voice.asr_error` | The ASR stream failed. A standard `exception` event is also recorded. | None | — | — | — |

#### Input-decision spans

Input-decision spans show how Rasa handles input received during a call before
passing it to the assistant. They cover spoken interruptions, keypad input, and
silence timeouts. They record whether the input was accepted, ignored, or
buffered, as well as any failures.

##### `handle barge-in` span

| Detail | Description |
| - | - |
| Span kind | `INTERNAL` |
| Lifetime | One barge-in classification. Opens when interruptions are enabled and caller speech is evaluated as a possible interruption while assistant audio is playing or waiting to be played. |
| Typical parent | `voice call`. |
| Error behavior | A classification or interruption-processing failure records `voice.barge_in_failed` and a standard `exception` event, then marks the span `ERROR`. |

**Attributes**

None beyond the shared correlation and optional failure attributes.

**Events**

| Event | Description | Attribute | Type | Availability | Data classification |
| - | - | - | - | - | - |
| `voice.barge_in_attempted` | A gated transcript entered classification. | `voice.barge_in.transcript` | string | When content capture is enabled | [Content](#privacy-content) |
| `voice.turn_event_enqueued` | The accepted interruption was sent to the assistant for processing. | `voice.turn_event.type` | string | Always | [Operational](#privacy-operational) |
| | | `voice.queue.size` | integer | Always | [Operational](#privacy-operational) |
| `voice.barge_in_accepted` | The speech was accepted as a deliberate interruption. | `voice.barge_in.reason` | string | Optional | [Operational](#privacy-operational) |
| | | `voice.barge_in.audio_was_playing` | boolean | When known | [Operational](#privacy-operational) |
| | | `voice.barge_in.audio_was_buffered` | boolean | When known | [Operational](#privacy-operational) |
| | | `voice.barge_in.transcript` | string | When content capture is enabled | [Content](#privacy-content) |
| `voice.barge_in_ignored` | The speech was classified as not being a barge-in. | `voice.barge_in.reason` | string | Optional | [Operational](#privacy-operational) |
| | | `voice.barge_in.audio_was_playing` | boolean | When known | [Operational](#privacy-operational) |
| | | `voice.barge_in.audio_was_buffered` | boolean | When known | [Operational](#privacy-operational) |
| | | `voice.barge_in.transcript` | string | When content capture is enabled | [Content](#privacy-content) |
| `voice.barge_in_failed` | Classification or interruption processing failed. | None | — | — | — |

`voice.barge_in.audio_was_playing` is `true` when assistant audio was already audible
when the transcript arrived. `voice.barge_in.audio_was_buffered` is `true` when
audio was queued but not yet audible. Both can be `true` at once.

##### `handle dtmf` span

| Detail | Description |
| - | - |
| Span kind | `INTERNAL` |
| Lifetime | One DTMF keypress decision and its resulting action. |
| Typical parent | `voice call`. |
| Error behavior | A failure adds `error.type` and a standard `exception` event and marks the span `ERROR`. |

**Attributes**

| Attribute | Type | Availability | Data classification | Description |
| - | - | - | - | - |
| `voice.dtmf.buffer_length` | integer | After a digit is buffered | [Operational](#privacy-operational) | Number of buffered digits after this keypress. Digit values are not captured. |
| `error.type` | string | On error | [Operational](#privacy-operational) | Present only on failure. |

**Events**

| Event | Description | Attribute | Type | Availability | Data classification |
| - | - | - | - | - | - |
| `voice.dtmf_started` | This keypress opened a digit-collection window. | None | — | — | — |
| `voice.dtmf_digit` | One digit was buffered. The value is never attached. | None | — | — | — |
| `voice.turn_event_enqueued` | The completed DTMF input was sent to the assistant for processing. | `voice.turn_event.type` | string | Always | [Operational](#privacy-operational) |
| | | `voice.queue.size` | integer | Always | [Operational](#privacy-operational) |
| `voice.dtmf_accepted` | The buffered input completed the active DTMF collect step. | None | — | — | — |
| `voice.dtmf_ignored` | No active DTMF collect step accepted the keypress. | None | — | — | — |

##### `handle silence` span

| Detail | Description |
| - | - |
| Span kind | `INTERNAL` |
| Lifetime | Starts when the caller has been silent for the configured timeout and ends when Rasa finishes handling the timeout. |
| Typical parent | `voice call`. |
| Error behavior | A failure adds `error.type` and a standard `exception` event and marks the span `ERROR`. |

**Attributes**

None beyond the shared correlation and optional failure attributes.

**Events**

| Event | Description | Attribute | Type | Availability | Data classification |
| - | - | - | - | - | - |
| `voice.silence_detected` | The armed timer fired. | None | — | — | — |
| `voice.turn_event_enqueued` | The silence timeout was sent to the assistant for processing. | `voice.turn_event.type` | string | Always | [Operational](#privacy-operational) |
| | | `voice.queue.size` | integer | Always | [Operational](#privacy-operational) |
| `voice.silence_accepted` | Rasa accepted the detected silence as a timeout. | None | — | — | — |
| `voice.dtmf_timeout` | The caller did not complete keypad input before the timeout, so the buffered digits were discarded. | None | — | — | — |

#### `synthesize speech` span

| Detail | Description |
| - | - |
| Span kind | `INTERNAL` |
| Lifetime | One assistant message, from the start of speech synthesis through generated, cached, or fallback audio completion. Each message gets its own span. |
| Typical parent | A `MessageProcessor` span. |
| Error behavior | A TTS failure adds `error.type`, records `voice.tts_error` and a standard `exception` event, and marks the span `ERROR`. The span stays open while fallback silence is produced and closes afterward. |

**Attributes**

| Attribute | Type | Availability | Data classification | Description |
| - | - | - | - | - |
| `voice.tts.provider` | string | Always | [Operational](#privacy-operational) | TTS provider name. |
| `voice.locale` | string | When configured | [Operational](#privacy-operational) | Rasa language key. |
| `voice.tts.language` | string | When configured | [Operational](#privacy-operational) | Provider language key. |
| `voice.tts.model` | string | When configured | [Operational](#privacy-operational) | Provider model. |
| `voice.tts.cache_hit` | boolean | After cache lookup | [Operational](#privacy-operational) | Whether audio came from the TTS cache. |
| `voice.latency.tts_first_byte_ms` | double | When first audio is available | [Operational](#privacy-operational) | Speech synthesis start to the first generated, cached, or fallback audio chunk. |
| `audio.bytes.total` | integer | At span end | [Operational](#privacy-operational) | Total generated, cached, or fallback audio bytes for this message. |
| `audio.chunk.count` | integer | At span end | [Operational](#privacy-operational) | Number of generated, cached, or fallback audio chunks for this message. |
| `voice.tts.text` | string | When content capture is enabled | [Content](#privacy-content) | Synthesized text. For streamed messages, this is the text accumulated from chunks sent to TTS. |
| `error.type` | string | On error | [Operational](#privacy-operational) | Present only on failure. |

> [!NOTE]
>
> Audio totals are added when the span ends and can be zero.

**Events**

| Event | Description | Attribute | Type | Availability | Data classification |
| - | - | - | - | - | - |
| `voice.tts_request` | A synthesis request was sent to the provider. Absent on a cache hit. | None | — | — | — |
| `voice.tts_cache_hit` | The audio was served from the TTS cache. Recorded instead of `voice.tts_request`. | None | — | — | — |
| `voice.tts_first_audio_generated` | The first generated, cached, or fallback audio chunk became available. | `voice.tts.audio_source` | string | Always | [Operational](#privacy-operational) |
| `voice.tts_complete` | Generated, cached, or fallback audio processing completed. | None | — | — | — |
| `voice.tts_error` | Synthesis failed. | None | — | — | — |

`voice.tts.audio_source` is `cached`, `generated`, or `fallback`.

> [!NOTE]
>
> After a TTS failure, fallback audio can still produce completion and
> audio-total signals. The span remains `ERROR`.

#### `play audio` span

| Detail | Description |
| - | - |
| Span kind | `INTERNAL` |
| Lifetime | One assistant message's audio delivery. |
| Typical parent | A `MessageProcessor` span. |
| Error behavior | A delivery failure adds `error.type` and a standard `exception` event and marks the span `ERROR`. |

**Attributes**

| Attribute | Type | Availability | Data classification | Description |
| - | - | - | - | - |
| `voice.interrupted` | boolean | When interrupted | [Operational](#privacy-operational) | Present and `true` when barge-in cut playback short. |
| `voice.playback.text` | string | At span end, when content capture is enabled and TTS text is known | [Content](#privacy-content) | Text that was synthesized for this delivery, including streamed chunks. |
| `audio.bytes.total` | integer | At span end | [Operational](#privacy-operational) | Bytes delivered before the span ended. |
| `audio.chunk.count` | integer | At span end | [Operational](#privacy-operational) | Chunks delivered before the span ended. |
| `error.type` | string | On error | [Operational](#privacy-operational) | Present only on delivery failure. |

**Events**

| Event | Description | Attribute | Type | Availability | Data classification |
| - | - | - | - | - | - |
| `voice.playback_start` | First audio chunk was delivered. Emitted at most once. | None | — | — | — |
| `voice.playback_interrupted` | A user barge-in interrupted playback. Emitted at most once. | None | — | — | — |
| `voice.playback_cancelled` | Audio delivery was cancelled during teardown or shutdown. | None | — | — | — |
| `voice.playback_finished` | Audio delivery finished. It can follow an interruption, but is suppressed after cancellation. | None | — | — | — |

#### `store call recording` span

Opens when Rasa finalizes a call recording: encode, upload, and report the
outcome. It is created only when recording is enabled for the call.

| Detail | Description |
| - | - |
| Span kind | `INTERNAL` |
| Lifetime | One finalize of the call recording, from encode and upload through the terminal outcome. |
| Typical parent | `voice call`. |
| Error behavior | Encode or upload failure records `voice.recording_failed`, adds `error.type`, and marks the span `ERROR`. It does not record a standard `exception` event or put the exception message on the span status. An empty recording is skipped and is not an error. |

**Attributes**

| Attribute | Type | Availability | Data classification | Description |
| - | - | - | - | - |
| `voice.recording.storage.provider` | string | Always | [Operational](#privacy-operational) | Configured storage provider, such as `s3`, `gcs`, `azure`, or `local`. |
| `voice.recording.chunk_count` | integer | Always | [Operational](#privacy-operational) | Number of audio chunks captured before finalization. |
| `voice.recording.status` | string | At span end | [Operational](#privacy-operational) | Terminal outcome: `available`, `failed`, or `skipped`. |
| `voice.recording.content_type` | string | When the recording is available | [Operational](#privacy-operational) | MIME type of the stored file, such as `audio/wav`. |
| `voice.recording.bucket` | string | When the recording is available | [Identifier](#privacy-identifier) | Storage bucket or container. Empty for local files. |
| `voice.recording.key` | string | When the recording is available | [Identifier](#privacy-identifier) | Object key or filename of the stored recording. |
| `voice.recording.uri` | string | When the recording is available | [Identifier](#privacy-identifier) | Locator for the stored recording (`s3://`, `gs://`, `https://`, or `file://`). Does not include credentials. |
| `error.type` | string | On error | [Operational](#privacy-operational) | Exception class name. Present only when status is `failed`. |

> [!NOTE]
>
> Storage locators (`bucket`, `key`, `uri`) identify where the recording file
> lives. They are not the audio itself and are not gated by content capture. A
> `file://` URI can include a local filesystem path.

**Events**

| Event | Description | Attribute | Type | Availability | Data classification |
| - | - | - | - | - | - |
| `voice.recording_available` | The recording was encoded and stored. | None | — | — | — |
| `voice.recording_skipped` | No audio was captured, so nothing was stored. | None | — | — | — |
| `voice.recording_failed` | Encoding or storage failed. | None | — | — | — |

Each span records exactly one of these events.

#### Failure-only spans

These spans appear only when an operation fails. They are backdated to the
start of the failed operation and end when the failure is recorded.

##### `process channel message` span

Covers a failed channel-message processing operation. It has only the shared
correlation and failure attributes.

| Detail | Description |
| - | - |
| Span kind | `INTERNAL` |
| Lifetime | The failed channel operation, backdated to its start and ended when the failure is recorded. |
| Typical parent | `voice call`. |
| Error behavior | Always records `error.type` and a standard `exception` event and has `ERROR` status. |

**Attributes**

| Attribute | Type | Availability | Data classification | Description |
| - | - | - | - | - |
| `error.type` | string | Always | [Operational](#privacy-operational) | Exception class name. |

**Events**

No custom span events. A standard `exception` event is recorded.

##### `compose event` span

Appears when Rasa fails while interpreting a voice-channel input, such as caller
speech, keypad input, or a silence timeout, and preparing it for the assistant.

| Detail | Description |
| - | - |
| Span kind | `INTERNAL` |
| Lifetime | The failed input-processing attempt, from when Rasa began interpreting the input until the failure was recorded. |
| Typical parent | `voice call`. |
| Error behavior | Always records `error.type` and a standard `exception` event and has `ERROR` status. |

**Attributes**

| Attribute | Type | Availability | Data classification | Description |
| - | - | - | - | - |
| `voice.turn_event.type` | string | When event type is available | [Operational](#privacy-operational) | Type of voice input that could not be processed. |
| `error.type` | string | Always | [Operational](#privacy-operational) | Exception class name. |

**Events**

No custom span events. A standard `exception` event is recorded.

#### How to read voice attribute tables

The `Availability` column describes when an attribute is eligible or written
relative to an emitted span.

The `Data classification` column uses these privacy labels:

* **Operational** — contains no spoken content
  or direct call identifier.
* **Identifier** — correlates a call or conversation,
  locates a stored recording, or otherwise may be customer data.
* **Content** — contains transcript or generated
  text and is emitted only when content capture is enabled.

## Tracing in the Action Server

API Requests are traced as they flow through the action server by instrumenting the webhook that receives custom actions
and other classes involved in the execution of custom actions.

> [!NOTE]
>
> **New in 3.8**
>
> Additional classes are now instrumented to improve tracing in the action server.

The following classes are instrumented;

* [ValidationAction](/docs/reference/integrations/action-server/validation-action.md#validationaction-class):
  the base class for custom actions extracting and validating slots.
* [ActionExecutor](/docs/reference/integrations/tracing.md#action-executor-attributes) - the class that executes the custom actions.

### Webhook Attributes

The following attributes are captured as part of the trace context of the webhook that receives custom actions;

* `http.method`: the http method used to make the request
* `http.route`: the endpoint of the request
* `next_action`: the name of the next action to be executed
* `version`: the rasa version used
* `sender_id`: the id of the conversation
* `message_id`: the unique message id

### Action Executor Attributes

The following attributes are captured as part of the trace context of the action executor;

* `action_name`: the name of the action to be executed
* `sender_id`: the id of the conversation
* `events`: a list of returned events
* `slots`: a list of filled slots by the executed custom action
* `utters`: a list of executed utterances

### Slot Validation Action Attributes

The following attributes are captured as part of the trace context of Slot Validation Actions;

* `class_name`: the name of the instrumented component class
* `action_name`: the name of the action to be executed
* `sender_id`: the id of the conversation
* `events`: a list of returned events
* `slots`: a list of filled slots by the executed custom action
* `utters`: a list of executed utterances
* `message_count`: the number of messages
* `slots_to_validate`: a list of recently filled slots to validate

### Debugging custom actions performance

> [!NOTE]
>
> **New in 3.8**
>
> You can now continue tracing the request further along your custom actions code.

It is now possible to debug the performance of your custom actions by tracing specific parts of your custom actions code.
This can be achieved by creating spans to trace the execution of these parts.

In order to create more spans, you can retrieve the [tracer object](https://opentelemetry.io/docs/languages/python/instrumentation/#acquire-tracer)
from the `ActionExecutorTracerRegister` component.

```python title="actions.py" theme={null}
# import the ActionExecutorTracerRegister component
from rasa_sdk.tracing.tracer_register import ActionExecutorTracerRegister
```

To create a span as documented in the [OTEL documentation](https://opentelemetry.io/docs/languages/python/instrumentation/#creating-spans),
corresponding to traces from a specific part of your custom actions code, you can embed the following code snippet in the `run` method of the custom action:

```python title="actions.py" theme={null}
# retrieve the tracer object
tracer = ActionExecutorTracerRegister().get_tracer()

# create a span
with tracer.start_as_current_span("span_name") as span:
  # your code here
  span.set_attribute("attribute_name", "attribute_value")
```

For example, a complete custom action that implements a custom span is shown below:

```python title="actions.py" theme={null}
import requests
import json
from rasa_sdk import Action
from rasa_sdk.tracing.tracer_register import ActionExecutorTracerRegister

class ActionCheckSufficientFunds(Action):
  def name(self):
    return "action_check_sufficient_funds"

  def run(
    self,
    dispatcher: CollectingDispatcher,
    tracker: Tracker,
    domain: Dict[Text, Any]
  ) -> List[Dict[Text, Any]]:
    tracer = ActionExecutorTracerRegister().get_tracer()

    with tracer.start_as_current_span("span_name"):
      balance = 1000 # hardcoded balance from tutorial purposes
      transfer_amount = tracker.get_slot("amount")
      has_sufficient_funds = transfer_amount <= balance

      # set trace attributes
      span.set_attribute("has_sufficient_funds", has_sufficient_funds)

      return [SlotSet("has_sufficient_funds", has_sufficient_funds)]
```

Enabling and disabling tracing in the action server is also done in the same way as described [below](/docs/reference/integrations/tracing.md#enabling--disabling).
The same Tracing Backends/Collectors listed [above](/docs/reference/integrations/tracing.md#supported-tracing-backendscollectors) are also supported for the action server.
See [Configuring a Tracing Backend or Collector](/docs/reference/integrations/tracing.md#configuring-a-tracing-backend-or-collector)
for further instructions.

### Enabling / Disabling

Tracing is automatically enabled in Rasa by
[configuring a supported tracing backend](/docs/reference/integrations/tracing.md#configuring-a-tracing-backend-or-collector).
No further action is required to enable tracing.

You can disable tracing by leaving the `tracing:` configuration key empty
in your endpoints file.


## Related topics

- [Tracing](/docs/pro/improve/tracing.md)
- [Langfuse Integration](/docs/reference/integrations/langfuse.md)
- [Rasa Pro Version Migration Guide](/docs/reference/changelogs/rasa-pro-migration-guide.md)
- [Observability Metrics](/docs/pro/improve/observability-metrics.md)
- [Rasa Pro Change Log](/docs/reference/changelogs/rasa-pro-changelog.md)


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.