WebSocket Voice Channel

New in 3.19

From Rasa Pro 3.19, you can stream conversation audio to your Rasa assistant from your own mobile or web app over a plain WebSocket.

This channel is a voice-stream connector that accepts audio from any client that can open a WebSocket and exchange base64-encoded audio, such as an iOS or Android app, a browser, or a desktop app. Unlike the telephony stream channels (Jambonz Stream, AudioCodes Voice Stream), it is not tied to a carrier or phone number. Your app captures microphone audio, sends it to Rasa, and plays back the synthesized response.

Configure Rasa Assistant

Use the built-in channel websockets_voice to configure your Rasa assistant. Create or edit the credentials.yml file at the root of your assistant directory to add the websockets_voice channel. Here's an example:

credentials.yml

websockets_voice:
  server_url: "<your-domain>"
  asr:
    name: deepgram
  tts:
    name: cartesia

  # Optional configurations
  interruptions:
    enabled: true
    min_words: 3
  silence_timeout: 3.0

The channel configuration accepts the following properties:

• server_url (required): The domain at which the Rasa server is available. Do not include the protocol (ws:// or wss://). For example, if your server is deployed on https://example.ngrok.app, server_url should be example.ngrok.app.

• asr (required): Configuration for Automatic Speech Recognition. See Speech Integrations for the list of ASR engines Rasa integrates with.

• tts (required): Configuration for Text-To-Speech. See Speech Integrations for the list of TTS engines Rasa integrates with.

• interruptions (optional): Configuration for interruption handling. This lets the assistant detect when a user speaks over it and respond more naturally. See Interruption Handling for more information.

• silence_timeout (optional): Number of seconds of silence after which the current user turn is closed. Set this when you want the assistant to react to a pause in speech.

Run the assistant with rasa run. Your app needs a URL that can reach the Rasa server. For development, you can expose your local server with tools like ngrok or Cloudflare Tunnel.

Bot URLs for development

Visit this section to learn how to generate the required bot URL when testing the channel on your local machine.

Connecting your app

Your app connects to the WebSocket endpoint exposed by the channel:

wss://<your-domain>/webhooks/websockets_voice/websocket

A GET request to /webhooks/websockets_voice/ returns {"status": "ok"}, which you can use as a health check.

Audio format

Audio is exchanged in both directions as Linear PCM (L16), 24 kHz, mono, 16-bit. Rasa supports 24 kHz natively, so no server-side transcoding is required. Your app must capture and play back audio at this sample rate.

WebSocket protocol

The client and server exchange JSON text frames. Audio payloads are base64-encoded PCM data.

Client to server

Send captured audio:

{ "audio": "<base64-encoded-pcm-data>" }

Send a text message instead of audio (the assistant treats it as a user turn):

{ "text": "I want to book a flight" }

Acknowledge a playback marker (see Playback markers):

{ "marker": "<marker-id>" }

Server to client

Synthesized assistant audio to play back:

{ "audio": "<base64-encoded-pcm-data>" }

A playback marker to acknowledge once the corresponding audio has finished playing:

{ "marker": "<marker-id>" }

A signal to stop playback immediately, sent when the assistant detects an interruption:

{ "interruptPlayback": true }

A custom payload from a custom action's custom response. The channel sends it as data rather than speaking it, so your app can act on it silently (for example, to refresh an expired token):

{ "custom": { "your": "payload" } }

Playback markers

The server tags bot audio with markers so it can track what the client has actually played. When your app finishes playing a chunk of audio, send the matching marker frame back. Rasa uses these acknowledgements to know when the assistant has stopped speaking and, for example, when it is safe to end the call after a goodbye message. An app that ignores markers still plays audio, but the assistant cannot reliably tell when playback has finished.

Sending and receiving messages

When a user speaks, your app streams the audio frames to Rasa. The configured ASR engine converts speech to text, and Rasa interprets the message like any other channel. The assistant responds with text, the configured TTS engine converts it to speech, and the audio streams back to your app.

utter_greet:
  - text: "Hello! How can I help you today?"

note

Only text and audio are supported. Images, attachments, and buttons cannot be used with voice stream channels. Custom (custom) responses are delivered to the client as data frames instead of being spoken.

Passing client data to actions

Your app can attach per-user data when it opens the WebSocket, and that data is carried into every user turn. There are two ways to send it:

• The Authorization header on the WebSocket upgrade request, available under the authorization key.
• Any query argument on the connection URL, except language.

wss://<your-domain>/webhooks/websockets_voice/websocket?token=abc123&user_id=456

The channel collects these values into the extra field of the call metadata. A custom action can read them from the latest message or from the session_started_metadata slot:

def run(self, dispatcher, tracker, domain):
    metadata = tracker.latest_message["metadata"]["extra"]
    token = metadata.get("token")      # "abc123"
    user_id = metadata.get("user_id")  # "456"
    # Use for authentication, personalization, or session context.

The extra field only forwards these values to your assistant. It does not authenticate the WebSocket connection, and returning a token here does not restrict who can connect. Enforce access control in your custom actions or in front of the Rasa server. Client-supplied values are confined to extra, so they cannot overwrite call parameters such as call_id.

The language query argument is reserved: it sets the language of the call rather than being forwarded in extra.

wss://<your-domain>/webhooks/websockets_voice/websocket?language=en

Call metadata

Metadata about the call can be accessed through the session_started_metadata slot at the beginning of the conversation. The following fields are available:

Field Name	Description
call_id	A unique identifier generated for the connection, prefixed with app-.
stream_id	The unique stream identifier (same as call_id).
language	The call language, taken from the language query argument if provided.
extra	Client-supplied data from the Authorization header and query arguments.

A custom action_session_start can be used to store this information to a slot.