Observability Metrics - Rasa Documentation

Metrics are runtime measurements that capture indicators of a service’s availability and performance. Unlike tracing—which helps you understand the sequence of operations for a single request—metrics provide an aggregated statistical view across multiple requests or conversations. Typical examples include average response time, throughput, and CPU/memory consumption. Monitoring these helps you:

Track the health of your service.
Quickly detect and alert on outages or anomalies.
Quantify the impact of code or infrastructure changes on performance.

Tracing must also be enabled for metrics to be recorded. Metric measurements are emitted from Rasa’s instrumentation layer, which is only activated when tracing is configured. With both enabled, metrics and traces together give you a complete view of your deployment behavior, making it easier to debug issues and optimize resource usage.

How To Use Metrics

Enabling Metrics in Rasa

Rasa uses an OpenTelemetry (OTEL) Collector to collect metrics and send them to your desired backend (e.g., Prometheus, Datadog, etc.).

Tracing is requiredConfiguring the metrics block alone does not activate metric collection. Rasa records measurements through the same instrumentation layer that powers tracing, and that layer starts only when a tracing block is present in your endpoints configuration. You must enable tracing in addition to metrics.

Enable tracing in your endpoints file (or Helm values): Add a tracing block to activate Rasa’s instrumentation layer. For example, using an OTLP collector:

endpoints.yml

tracing:
  - type: otlp
    endpoint: my-otlp-host:4318
    insecure: false
    service_name: rasa
    root_certificates: ./path/to/ca.pem

See Tracing for other supported backends (Jaeger, Langfuse, and more). 2. Configure metrics in the same endpoints file (or Helm values):

endpoints.yml

metrics:
  type: otlp
  endpoint: my-otlp-host:4318
  insecure: false
  service_name: rasa
  root_certificates: ./path/to/ca.pem

type: otlp indicates you are using OpenTelemetry’s OTLP format.
endpoint is the URL of the OTEL Collector or metrics backend.
service_name is an identifier for your Rasa Pro service.
insecure/root_certificates specify how TLS is handled.

Tracing surfaces the sequence of internal method calls for individual requests; metrics aggregate their performance across many requests. Enabling both gives you the most complete observability picture.

Custom Metrics Collected by Rasa

Once tracing and metrics are both configured, Rasa automatically collects several custom metrics relevant to large language model (LLM) usage and overall assistant performance:

CPU and Memory Usage of any LLM-based command generator (e.g., CompactLLMCommandGenerator, SearchReadyLLMCommandGenerator) at the time of making an LLM call.
Prompt Token Usage for LLM-based command generators, provided the trace_prompt_tokens config property is enabled.
Method Call Durations for LLM-specific components, such as:
- EnterpriseSearchPolicy
- ContextualResponseRephraser
- CompactLLMCommandGenerator
- SearchReadyLLMCommandGenerator
HTTP Request Metrics for the Rasa client:
- Duration of requests to external services (action server, NLG server, etc.).
- Request size in bytes.

Sub-agents (ReAct and A2A)

When a flow uses autonomous steps to hand off to a ReAct or A2A sub-agent, Rasa emits extra OpenTelemetry histograms (in addition to the spans described in Tracing):

agent_execution_duration — Wall time for each sub-agent run (_call_agent_with_retry in the agent executor). Useful for end-to-end latency and error rates per sub-agent. Attribute labels include:
- agent_name — Sub-agent id from configuration.
- protocol_type — How the sub-agent is connected: mcp_open or mcp_task for ReAct-style MCP sub-agents, or a2a for A2A sub-agents.
- status — Final status of the sub-agent result.
mcp_tool_execution_duration — Time spent inside an MCP tool call. The same metric name is used in two execution paths; use the execution_context label to distinguish them:
- execution_context = flow — Tool invoked from a flow MCP tool step. Labels include tool_id, mcp_server, and success.
- execution_context = agent — Tool invoked while a ReAct MCP sub-agent runs (_execute_tool_call). Labels include tool_name, agent_name, protocol_type, and success.
ReAct MCP sub-agent LLM usage — For MCP-based sub-agents, each LLM send_message round-trip also records resource-style histograms (CPU and memory sampled like other LLM components, plus estimated prompt size and response duration):
- mcp_agent_llm_cpu_usage
- mcp_agent_llm_memory_usage
- mcp_agent_llm_prompt_token_usage
- mcp_agent_llm_response_duration

By collecting these telemetry metrics, you gain robust insights into how your assistant performs under real-world usage. You can proactively detect issues, understand resource consumption, and tailor your assistant’s architecture to provide the best possible experience for your users.

Voice metrics

Voice metrics cover calls handled through streaming voice channels.

`rasa.voice.call.duration`

Records the time from the start of a streaming voice call until the call ends. Rasa records it once at the end of the call, so ongoing calls are not included. Use it to compare call durations and outcomes.

Detail	Value
Kind	Histogram
Unit	Seconds (`s`)
Histogram boundaries	`5`, `15`, `30`, `60`, `120`, `300`, `600`, `1200`, `1800`, `2700`, `3600`

Dimensions

Dimension	Type	Required	Values
`rasa.voice.outcome`	string	Yes	`completed` — the call ended normally, at the assistant’s request, or while Rasa was shutting down; cancellation is not a separate outcome `disconnected` — the connection dropped `failed` — an unhandled error ended the call

`rasa.voice.tts.time_to_first_audio`

Records the time to the first audio chunk for one assistant message. For a cache hit, timing starts when Rasa finds the cached audio. For a cache miss, it starts when Rasa sends the request to the TTS provider. The metric is not recorded if no audio is produced or TTS fails, even if fallback audio plays after the failure. In that case, the voice.latency.tts_first_byte_ms span attribute can still measure the fallback audio when tracing is enabled. Use it to compare time to first audio by provider and cache status.

Detail	Value
Kind	Histogram
Unit	Seconds (`s`)
Histogram boundaries	`0.005`, `0.01`, `0.025`, `0.05`, `0.1`, `0.25`, `0.5`, `1`, `2.5`, `5`, `10`, `30`

Dimensions

Dimension	Type	Required	Values
`rasa.voice.provider`	string	No	TTS provider name; free-form
`rasa.voice.tts.cache_hit`	boolean	No	`true` / `false`

`rasa.voice.tts.synthesis.duration`

Records the time to complete TTS for one assistant message. For a cache hit, timing starts when Rasa finds the cached audio and ends when cached-audio delivery completes. For a cache miss, it starts when Rasa sends the provider request and ends when audio generation completes. On failure, timing stops at the first failure and excludes fallback audio. If TTS neither completes nor fails, no measurement is recorded. Use it to compare TTS completion times by outcome, provider, and cache status.

Detail	Value
Kind	Histogram
Unit	Seconds (`s`)
Histogram boundaries	`0.005`, `0.01`, `0.025`, `0.05`, `0.1`, `0.25`, `0.5`, `1`, `2.5`, `5`, `10`, `30`

Dimensions

Dimension	Type	Required	Values
`rasa.voice.outcome`	string	Yes	`completed` `error`
`rasa.voice.provider`	string	No	TTS provider name; free-form
`rasa.voice.tts.cache_hit`	boolean	No	`true` / `false`

`rasa.voice.playback.duration`

Records the time from the first delivered audio chunk until playback completes, is interrupted, is cancelled, or fails. Rasa records it only after at least one audio chunk is delivered. If playback ends before delivering audio, this metric is not recorded, but rasa.voice.operation.failures can still count an unexpected failure. Use it to compare playback times by outcome.

Detail	Value
Kind	Histogram
Unit	Seconds (`s`)
Histogram boundaries	`0.25`, `0.5`, `1`, `2.5`, `5`, `10`, `20`, `30`, `60`, `120`, `300`

Dimensions

Dimension	Type	Required	Values
`rasa.voice.outcome`	string	Yes	`completed` `interrupted` — the caller interrupted the assistant `cancelled` — the channel cancelled playback `error`

`rasa.voice.operation.failures`

Counts unexpected failures in streaming voice operations. Expected outcomes, such as an ignored barge-in, a DTMF timeout, or interrupted or cancelled playback, are not counted. Use it to identify failures by component, operation, provider, and error type.

Detail	Value
Kind	Counter
Unit	Failures (`{failure}`)

Dimensions

Dimension	Type	Required	Values
`rasa.voice.component`	string	Yes	`asr` `tts` `playback` `composer` `channel`
`rasa.voice.operation`	string	Yes	`connect` `disconnect` `transcribe` `synthesize` `play` `process_channel_message` `compose_event`
`rasa.voice.provider`	string	No	ASR or TTS provider name, when applicable; free-form
`rasa.voice.error_type`	string	Yes	Exception class name or explicit error value; free-form. Exception messages are excluded.

`rasa.voice.barge_in.decisions`

Counts one accepted or ignored barge-in decision for each transcript update Rasa evaluates, including partial transcripts. Decisions are made when interruptions are enabled and the caller speaks while assistant audio is playing or waiting to play. One caller interruption can produce several ignored decisions before an accepted one. Other caller speech that does not result in a decision is not counted. Use it to compare accepted and ignored decisions and their reasons.

Detail	Value
Kind	Counter
Unit	Decisions (`{decision}`)

Dimensions

Dimension	Type	Required	Values
`rasa.voice.outcome`	string	Yes	`accepted` / `ignored`
`rasa.voice.reason`	string	No	`below_min_words` — the transcript did not reach the configured minimum word count `word_threshold_reached` — the transcript reached the configured minimum word count

`rasa.voice.dtmf.decisions`

Counts accepted DTMF input, keypresses received when Rasa is not waiting for DTMF input, and DTMF input attempts that time out. Keypresses are not counted individually while Rasa waits for the full input. Use it to compare keypad interaction outcomes without recording the digits entered.

Detail	Value
Kind	Counter
Unit	Decisions (`{decision}`)

Dimensions

Dimension	Type	Required	Values
`rasa.voice.outcome`	string	Yes	`accepted` `ignored` `timeout`
`rasa.voice.reason`	string	No	Reason for the decision, when provided; free-form. Rasa currently omits this dimension for DTMF decisions.

`rasa.voice.silence_timeouts`

Counts silence timers that expire and trigger a turn. Timers cancelled before they expire are not counted. Use it to compare counts across configured timeout values.

Detail	Value
Kind	Counter
Unit	Timeouts (`{timeout}`)

Dimensions

Dimension	Type	Required	Values
`rasa.voice.outcome`	string	Yes	`accepted`
`rasa.voice.silence.timeout_s`	double	No	Configured timeout in seconds, rounded to one decimal place; this is a configuration value, not elapsed time

`rasa.voice.audio.bytes`

Counts audio bytes processed during streaming voice calls. At the end of a call, Rasa records the total for each direction with more than zero bytes. Directions with no audio and ongoing calls are not included. Use it to compare incoming and outgoing audio volume.

Detail	Value
Kind	Counter
Unit	Bytes (`By`)

Dimensions

Dimension	Type	Required	Values
`rasa.voice.audio.direction`	string	Yes	`input` — caller audio received by Rasa `output` — audio sent to the caller

​How To Use Metrics

​Enabling Metrics in Rasa

​Custom Metrics Collected by Rasa

​Sub-agents (ReAct and A2A)

​Voice metrics

​rasa.voice.call.duration

​rasa.voice.tts.time_to_first_audio

​rasa.voice.tts.synthesis.duration

​rasa.voice.playback.duration

​rasa.voice.operation.failures

​rasa.voice.barge_in.decisions

​rasa.voice.dtmf.decisions

​rasa.voice.silence_timeouts

​rasa.voice.audio.bytes

How To Use Metrics

Enabling Metrics in Rasa

Custom Metrics Collected by Rasa

Sub-agents (ReAct and A2A)

Voice metrics

`rasa.voice.call.duration`

`rasa.voice.tts.time_to_first_audio`

`rasa.voice.tts.synthesis.duration`

`rasa.voice.playback.duration`

`rasa.voice.operation.failures`

`rasa.voice.barge_in.decisions`

`rasa.voice.dtmf.decisions`

`rasa.voice.silence_timeouts`

`rasa.voice.audio.bytes`