> ## Documentation Index > Fetch the complete documentation index at: https://rasa.com/docs/llms.txt > Use this file to discover all available pages before exploring further. # ASR Integrations > Configure the built-in automatic speech recognition providers for Rasa Voice Stream channels. ## Automatic Speech Recognition (ASR) This section describes the supported integrations with Automatic Speech Recognition (ASR) or speech-to-text (STT) services. Configure an ASR engine under `asr` in a Voice Stream channel in `credentials.yml`. Each engine accepts `keep_alive_interval` (seconds, default `5`) and a `language_map` that maps the assistant language keys from `config.yml` to provider settings. The top-level `language` and `model` options are deprecated; set provider-specific language and model values inside `language_map` instead. ### Deepgram Use the environment variable `DEEPGRAM_API_KEY` for Deepgram API Key. You can request a key from [Deepgram](https://deepgram.com/). It can be configured in a Voice Stream channel as follows: ```yaml title="credentials.yml" theme={null} browser_audio: # ... other configuration asr: name: deepgram ``` #### Deepgram v1 models (Nova) Deepgram v1 models, including Nova models, use Deepgram's `/v1/listen` endpoint. > [!NOTE] > > **Turn Detection** > > Deepgram v1 models use two mechanisms to detect when a speaker has finished talking: > > 1. **Endpointing**: Uses Voice Activity Detection (VAD) to detect silence after speech > 2. **UtteranceEnd**: Looks at word timings to detect gaps between words > > The configuration parameters `endpointing` and `utterance_end_ms` control these features respectively. For noisy environments, `utterance_end_ms` may be more reliable as it ignores non-speech audio. [Read more on Deepgram Documentation](https://developers.deepgram.com/docs/understanding-end-of-speech-detection) ##### Configuration parameters * `endpoint`: Optional, defaults to `api.deepgram.com` - The endpoint URL for the Deepgram API. * `endpointing`: Optional, defaults to `400` - Number of milliseconds of silence to determine the end of speech. * `smart_format`: Optional, defaults to `true` - Boolean value to enable or disable [Deepgram's smart formatting](https://developers.deepgram.com/docs/smart-format). * `utterance_end_ms`: Optional, defaults to `1000` - Time in milliseconds to wait before considering an utterance complete. * `language_map`: Optional. By default, Rasa maps the assistant language to Deepgram language `en` and model `nova-3`. Use each entry's `language` and `model` fields to configure another provider language or model. > [!WARNING] > > **Language and Model Configuration** > > The deprecated top-level `language` and `model` options cannot be combined with `language_map`. Put those values inside each `language_map` entry instead. ##### Deepgram v1 multilingual support To configure Deepgram v1 for multilingual agent use `language_map` property. Keys listed in the map must match languages (`language` and `additional_languages`) mentioned in `config.yml`. Deepgram v1 supports `language` and `model` as configuration parameters inside `language_map`. ```yaml title="config.yml" theme={null} recipe: default.v1 language: en additional_languages: - de-DE pipeline: - name: SingleStepLLMCommandGenerator policies: - name: FlowPolicy assistant_id: 20240418-073244-narrow-archive ``` ```yaml title="credentials.yml" theme={null} browser_audio: asr: name: deepgram language_map: en: language: eng model: mistv2 de-DE: language: de model: mistv2 ``` To see the full list of supported languages and models for Deepgram visit [Models & Languages Overview](https://developers.deepgram.com/docs/models-languages-overview). #### Deepgram Flux (v2) Use the separate `deepgram_flux` engine for Deepgram Flux models. Flux uses Deepgram's `/v2/listen` endpoint and model-integrated end-of-turn detection. > [!NOTE] > > **Turn Detection** > > Deepgram v2 models use model-integrated end-of-turn detection. ##### Configuration parameters * `endpoint`: Optional, defaults to `api.deepgram.com` - The endpoint URL for the Deepgram API. * `eot_threshold`: Optional - Confidence threshold for end-of-turn detection. When omitted, Deepgram uses its default. * `eot_timeout_ms`: Optional - Maximum silence before forcing end-of-turn detection. When omitted, Deepgram uses its default. * `language_map`: Optional. By default, Rasa maps the assistant language to model `flux-general-en`. Each entry supports `model` and an optional `language` hint. ##### Deepgram Flux multilingual support To configure Deepgram Flux for a multilingual agent, use the `language_map` property. Keys in the map must match the languages (`language` and `additional_languages`) defined in `config.yml`. Rasa's Deepgram Flux integration supports `model` and an optional `language` inside `language_map`. When `language` is set, Rasa passes it to Deepgram as a `language_hint`. > [!NOTE] > > **Omit language for single-language models** > > Only set `language` for the multilingual `flux-general-multi` model. For single-language > models such as `flux-general-en`, omit `language`, as the model is already English-only. To use Deepgram Flux, set a `flux-*` model inside `language_map`: ```yaml title="credentials.yml" theme={null} browser_audio: asr: name: deepgram_flux language_map: en: model: flux-general-en eot_threshold: 0.7 eot_timeout_ms: 5000 ``` For multilingual agents, use the `flux-general-multi` model and set `language` on each entry to pass a Deepgram `language_hint`: ```yaml title="credentials.yml" theme={null} browser_audio: asr: name: deepgram_flux language_map: en: language: en model: flux-general-multi de-DE: language: de model: flux-general-multi ``` For more on configuring Flux, see [Flux Configuration](https://developers.deepgram.com/docs/flux/configuration). ### Azure Requires the python library `azure-cognitiveservices-speech`. The API Key can be set with the environment variable `AZURE_SPEECH_API_KEY`. Sample configuration looks as follows: ```yaml title="credentials.yml" theme={null} browser_audio: # ... other configuration asr: name: azure ``` #### Configuration parameters * `speech_region`: Optional. The region identifier for the Azure Speech service, such as `westus`. If you omit all connection fields, Rasa uses `eastus`. * `speech_endpoint`: Optional. A custom Azure Speech service endpoint, such as an endpoint behind a reverse proxy. * `speech_host`: Optional. A custom service host in `protocol://host:port` format. The port is optional. * `language_map`: Optional. Each entry must set Azure's `language` code. By default, Rasa maps the assistant language to `en-US`. If you set `speech_region`, `speech_endpoint`, or `speech_host`, make sure the value matches your Azure Speech resource. If you omit all three, Rasa uses the `eastus` region. When connecting to Azure Cloud, `speech_region` is enough. For example: ```yaml title="credentials.yml" theme={null} browser_audio: asr: name: azure speech_region: germanywestcentral language_map: de-DE: language: de-DE ``` > [!WARNING] > > **Language Configuration** > > Use `language_map` for language selection. The top-level `language` option is deprecated and cannot be combined with `language_map`. #### Azure ASR Multilingual Support > [!NOTE] > > New in Rasa 3.16.0 To configure Azure ASR for multilingual agent use `language_map` property. Keys listed in the map must match languages (`language` and `additional_languages`) mentioned in `config.yml`. Azure ASR supports `language` as configuration parameter inside `language_map`. ```yaml title="config.yml" theme={null} recipe: default.v1 language: en additional_languages: - de-DE pipeline: - name: SingleStepLLMCommandGenerator policies: - name: FlowPolicy assistant_id: 20240418-073244-narrow-archive ``` ```yaml title="credentials.yml" theme={null} browser_audio: asr: name: azure speech_region: germanywestcentral language_map: en: language: en-US de-DE: language: de-DE ``` To see the full list of supported languages and models for Azure ASR visit [Azure ASR Languages](https://learn.microsoft.com/en-us/azure/ai-services/speech-service/language-support?tabs=stt). ### ElevenLabs ElevenLabs Scribe v2 Realtime transcribes streamed voice audio. Set the `ELEVENLABS_API_KEY` environment variable. The default endpoint is `wss://api.elevenlabs.io/v1/speech-to-text/realtime`. ```yaml title="credentials.yml" theme={null} browser_audio: asr: name: elevenlabs ``` #### Configuration parameters * `endpoint`: Optional. The ElevenLabs WebSocket endpoint. Defaults to `wss://api.elevenlabs.io/v1/speech-to-text/realtime`. * `commit_strategy`: Optional. Use `vad` (default) to commit transcripts based on voice activity detection, or `manual` to commit when the audio turn ends. * `include_timestamps`: Optional, defaults to `false`. Ask ElevenLabs to include word timestamps. * `include_language_detection`: Optional, defaults to `false`. Ask ElevenLabs to detect the language from audio. When enabled, Rasa does not send the configured language code. * `language_map`: Optional. Each entry supports `language` and `model`. The default model is `scribe_v2_realtime`. ### Others Looking for integration with a different ASR service? You can [create your own custom ASR component](#custom-asr). ## Custom ASR > [!TIP] > > **Create Custom ASR with Skills** > > [Rasa MCP Tools](https://rasa.com/docs/pro/installation/rasa-mcp-tools.md) includes AI Skills to scaffold and audit your custom ASR component with your IDE copilot. > Run `rasa tools init` in your project root to install the agent skills. You can implement your own custom ASR component as a Python class to integrate with any third-party speech recognition service. A custom ASR component must subclass the `ASREngine` class from `rasa.core.channels.voice_stream.asr.asr_engine`. Your custom ASR component receives audio in the [RasaAudioBytes format](https://rasa.com/docs/reference/integrations/speech-integrations.md#audio-format) and may need to convert it to your service's expected format. ### Required Methods Your custom ASR component must implement the following methods: * `open_websocket_connection()`: Establish a websocket connection to your ASR service * `from_config_dict(config: Dict)`: Class method to create an instance from configuration dictionary * `signal_audio_done()`: Signal to the ASR service that audio input has ended * `rasa_audio_bytes_to_engine_bytes(chunk: RasaAudioBytes)`: Convert Rasa audio format to your engine's expected format * `engine_event_to_asr_event(event: Any)`: Convert your engine's events to Rasa's `ASREvent` format * `get_default_config()`: Static method that returns the default configuration for your component ### Optional Methods You may also override these methods as needed: * `send_keep_alive()`: Send keep-alive messages to maintain the connection. The default implementation is only a `pass` statement. * `close_connection()`: Custom cleanup when closing the connection. Default implementation is as follows, ``` async def close_connection(self) -> None: if self.asr_socket: await self.asr_socket.close() ``` ### ASR Events Your `engine_event_to_asr_event` method should return appropriate `ASREvent` objects: * `UserIsSpeaking(transcript)`: For interim/partial transcripts while the user is speaking * `NewTranscript(transcript)`: For final transcripts when the user has finished speaking See [Configure a custom ASR component](#configure-a-custom-asr-component) for the credentials configuration. ### Example Implementation Here's an example based on the Deepgram implementation structure: ```python title="custom_asr.py" theme={null} import json import os from dataclasses import dataclass from typing import Any, Dict, Optional from urllib.parse import urlencode import websockets from websockets.legacy.client import WebSocketClientProtocol from rasa.core.channels.voice_stream.asr.asr_engine import ASREngine, ASREngineConfig from rasa.core.channels.voice_stream.asr.asr_event import ( ASREvent, NewTranscript, UserIsSpeaking, ) from rasa.core.channels.voice_stream.audio_bytes import RasaAudioBytes @dataclass class MyASRConfig(ASREngineConfig): api_key: str = "" endpoint: str = "wss://api.example.com/v1/speech" language: str = "en-US" class MyASR(ASREngine[MyASRConfig]): required_env_vars = ("MY_ASR_API_KEY",) # Optional: required environment variables required_packages = ("my_asr_package",) # Optional: required Python packages def __init__(self, config: Optional[MyASRConfig] = None): super().__init__(config) self.accumulated_transcript = "" async def open_websocket_connection(self) -> WebSocketClientProtocol: """Connect to the ASR system.""" api_key = os.environ["MY_ASR_API_KEY"] headers = {"Authorization": f"Bearer {api_key}"} return await websockets.connect( self._get_api_url_with_params(), extra_headers=headers ) def _get_api_url_with_params(self) -> str: """Build API URL with query parameters.""" query_params = { "language": self.config.language, "encoding": "mulaw", "sample_rate": "8000", "interim_results": "true" } return f"{self.config.endpoint}?{urlencode(query_params)}" @classmethod def from_config_dict(cls, config: Dict) -> "MyASR": """Create instance from configuration dictionary.""" asr_config = MyASRConfig.from_dict(config) return cls(asr_config) async def signal_audio_done(self) -> None: """Signal to the ASR service that audio input has ended.""" await self.asr_socket.send(json.dumps({"type": "stop_audio"})) def rasa_audio_bytes_to_engine_bytes(self, chunk: RasaAudioBytes) -> bytes: """Convert Rasa audio format to engine format.""" # For most services, you can return the chunk directly # since it's already in mulaw format return chunk def engine_event_to_asr_event(self, event: Any) -> Optional[ASREvent]: """Convert engine response to ASREvent.""" data = json.loads(event) if data.get("type") == "transcript": transcript = data.get("text", "") if data.get("is_final"): # Final transcript - user finished speaking full_transcript = self.accumulated_transcript + " " + transcript self.accumulated_transcript = "" return NewTranscript(full_transcript.strip()) elif transcript: # Interim transcript - user is still speaking return UserIsSpeaking(transcript) return None @staticmethod def get_default_config() -> MyASRConfig: """Get default configuration.""" return MyASRConfig( endpoint="wss://api.example.com/v1/speech", language="en-US" ) async def send_keep_alive(self) -> None: """Send keep-alive message if supported by your service.""" if self.asr_socket is not None: await self.asr_socket.send(json.dumps({"type": "keep_alive"})) ``` This structure allows you to integrate any speech recognition service with Rasa's voice capabilities while maintaining compatibility with the existing voice stream infrastructure. ## Configure a custom ASR component To use a custom ASR component, configure its module path and settings in your `credentials.yml` file. The configuration should contain the **module path** of your custom class and any required configuration parameters. The module path follows the format `path.to.module.ClassName`. For example, a class `MyASR` in `addons/custom_asr.py` has module path `addons.custom_asr.MyASR`. ### Custom ASR Configuration Example ```yaml title="credentials.yml" theme={null} browser_audio: # ... other configuration asr: name: addons.custom_asr.MyASR api_key: "your_api_key" endpoint: "wss://api.example.com/v1/speech" language: "en-US" # any other custom parameters your ASR needs ``` ## Related topics - [Speech Integrations](https://rasa.com/docs/reference/integrations/speech-integrations.md) - [Rasa Tutorial](https://rasa.com/docs/pro/tutorial.md) - [Genesys Cloud](https://rasa.com/docs/reference/channels/genesys-cloud-voice.md) - [Voice Assistants](https://rasa.com/docs/pro/build/voice-assistants.md) - [WebSocket Voice Channel](https://rasa.com/docs/reference/channels/websockets-voice.md) This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.