Build a voice agent - Rasa Documentation

In this tutorial you build a card replacement skill, talk to it in the Inspector, and then make it sound right on a call. Each step adds one change and explains what the caller hears differently. You start from the project that rasa init --engine mantle creates. If you have not made one yet, follow Getting Started up to step 3, and configure the Inspector’s speech providers as in step 3. Keep the generated model_groups configuration: the speech keys do not replace your LLM provider key or Rasa license. Restart the Inspector after changing integrations.yml or your environment.

The card data and order operation below are mocks. They do not authenticate a caller, lock a real card, or place a real order. Before connecting a card API, add authorization and retry protection in your tools.

What you’ll build

A skill that:

greets the caller with a recording notice,
finds the card from its last four digits,
locks a stolen card straight away,
confirms the order out loud before placing it,
reads the order number back digit by digit.

The skill also keeps working in chat, unchanged.

1. Write the skill

Create skills/card_replace/ with three files. The instructions are plain prose with no voice-specific wording:

skills/card_replace/skill.md

---
name: Card Replace
description: Credit card - replacement when lost, stolen, damaged, or not received
---

Help the customer replace a credit card. Never invent card details or order
numbers; use the tools.

1. Ask why they need a replacement: lost, stolen, damaged, or not received.
   Record the reason as soon as they say it.
2. Ask for the last four digits of the card, then call @tool.find_card.
3. If the card was stolen, call @tool.lock_card straight away and tell them the
   card is locked.
4. Ask whether they want standard or rush shipping, and record it.
5. Call @tool.process_card_replacement and tell them the order is placed.

The memory the skill tracks. The model records the reason and shipping choice; the tools write the rest:

skills/card_replace/memory.yml

schema:
  public:
    replacement_reason:
      type: categorical
      enum_values: [lost, stolen, damaged, not_received]
      llm_settable: true
      description: Why the card is being replaced.
    shipping_type:
      type: categorical
      enum_values: [standard, rush]
      llm_settable: true
      description: Standard or rush shipping for the replacement card.
    selected_card_id:
      type: text
    selected_card_label:
      type: text
    card_locked:
      type: bool
    order_reference:
      type: text
    order_reference_spoken:
      type: text

And the tools, with mock data in place of a card API:

skills/card_replace/tools.py

import asyncio
import random

from rasa.mantle.tools.decorator import ToolContext, tool
from rasa.mantle.tools.result import ToolResult

_CARDS = {
    "4821": {"id": "card_visa", "label": "River Visa ending 4821"},
    "7390": {"id": "card_mc", "label": "Summit Mastercard ending 7390"},
}


@tool(description="Find the customer's card by its last four digits.")
async def find_card(last_four: str, context: ToolContext = None) -> ToolResult:
    card = _CARDS.get(last_four.strip())
    if card is None:
        return ToolResult(llm_response={"found": False})
    context.memory.set("selected_card_id", card["id"])
    context.memory.set("selected_card_label", card["label"])
    return ToolResult(llm_response={"found": True, "card": card["label"]})


@tool(description="Lock the selected card so no new charges go through.")
async def lock_card(context: ToolContext = None) -> ToolResult:
    context.memory.set("card_locked", True)
    return ToolResult(llm_response={"locked": True})


@tool(description="Order a replacement for the selected card.")
async def process_card_replacement(context: ToolContext = None) -> ToolResult:
    await asyncio.sleep(1)  # stands in for a call to your card system
    reference = f"{random.randint(0, 999999):06d}"
    context.memory.set("order_reference", reference)
    context.memory.set("order_reference_spoken", " ".join(reference))
    return ToolResult(llm_response={"ordered": True, "order_reference": reference})

Train and open the Inspector:

rasa train
rasa inspect

Select Start voice conversation and say My card was stolen. Give the last four digits as four eight two one. Listen for what Mantle already does on a call. The replies are short and spoken, with no lists or formatting. You hear a brief acknowledgement while find_card runs. If you talk over the agent, it stops and listens. What is still missing: the agent says nothing until you speak, and the order goes through without the caller agreeing to it.

2. Greet the caller

A caller expects to hear the agent first. The bundled session-start skill stays silent, so replace it with one that plays a recording notice and a greeting. Voice channels start every call with /session_start, which runs this skill:

skills/default_session_start/skill.md

---
name: Session Start
description: Greet the user at the start of a session.
routing:
  engine_managed: true
---

:::ordered_block id=main
steps:
  - id: notice
    action: utter_recording_notice
  - id: greet
    action: utter_greet
:::

skills/default_session_start/responses.yml

responses:
  utter_recording_notice:
    - text: This call may be recorded for quality and training purposes.
  utter_greet:
    - text: Hi, thanks for calling Acme Bank. How can I help you today?

The notice is a response, so it is spoken exactly as written every time. See Start a conversation for how session start works.

3. Give the pauses your agent’s voice

The acknowledgement you heard in step 1 is generated by the model. Give it a few examples in your agent’s tone so it sounds like your brand rather than a generic agent. Add a fillers block to agent.yml:

agent.yml

fillers:
  examples:
    - user: I lost my card
      ack: Oh no, let me take a look.
    - user: It was stolen
      ack: I'm sorry to hear that. Let me see what I can do.

Acknowledgements are on for voice and off for chat by default, so this only changes what callers hear. See fillers.

4. Confirm before placing the order

On chat, the customer can scroll back and check what they agreed to. On a call, they hear it once. Make the engine ask for a spoken yes before the order goes through, and have the question restate what is being ordered. Add tool_constraints to the skill frontmatter, and change step 5 of the prose so the model calls the tool and leaves the asking to the engine:

skills/card_replace/skill.md

---
name: Card Replace
description: Credit card - replacement when lost, stolen, damaged, or not received
complete_when:
  condition: "session.card_replace.order_reference"
tool_constraints:
  - lock_card:
      requires: >
        session.card_replace.selected_card_id
        and session.card_replace.replacement_reason == 'stolen'
      on_success: utter_card_locked
  - process_card_replacement:
      requires: >
        session.card_replace.selected_card_id
        and session.card_replace.shipping_type
      requires_confirmation:
        enabled: true
        utter_for_confirmation: utter_confirm_order
        utter_on_user_denial: utter_order_cancelled
      on_success: utter_order_placed
---

Help the customer replace a credit card. Never invent card details or order
numbers; use the tools.

1. Ask why they need a replacement: lost, stolen, damaged, or not received.
   Record the reason as soon as they say it.
2. Ask for the last four digits of the card, then call @tool.find_card.
3. If the card was stolen, call @tool.lock_card straight away.
4. Ask whether they want standard or rush shipping, and record it.
5. Call @tool.process_card_replacement as soon as the card and shipping are
   known.

Then add the wording:

skills/card_replace/responses.yml

responses:
  utter_card_locked:
    - text: I've locked your card, so no new charges can go through.
  utter_confirm_order:
    - text: >-
        I'll order a replacement for your {session.card_replace.selected_card_label}
        with {session.card_replace.shipping_type} shipping. Shall I go ahead?
  utter_order_cancelled:
    - text: No problem, I haven't placed the order.
  utter_order_placed:
    - text: >-
        Your replacement is on its way. Your order number is
        {session.card_replace.order_reference_spoken}.

What changed:

requires: keeps each tool unavailable until its facts exist, so the model cannot lock a card that was not stolen or order before shipping is chosen.
requires_confirmation: pauses the order and asks utter_confirm_order. The tool runs on a yes and is cancelled on a no. See Explicit confirmations.
on_success: speaks the lock and order lines exactly as written.
complete_when: finishes the skill once an order number exists.

5. Read the order number back clearly

Read aloud, 482193 can come out as “four hundred eighty-two thousand…”. The process_card_replacement tool already stores a second, spaced-out copy in order_reference_spoken (“4 8 2 1 9 3”), and utter_order_placed uses that copy, so the caller hears one digit at a time. The plain order_reference stays available for your systems and for chat. Speech providers differ in how they read numbers, so listen to your own identifiers in the Inspector. For more on exact wording, see Use a response when the words have to be exact.

6. Make the order safe to interrupt

A caller can talk over the agent while process_card_replacement is waiting on your card system. When Mantle accepts the interruption, it ends the turn and cancels the tool at its current await. Check context.is_cancelled right before starting an irreversible step:

skills/card_replace/tools.py

@tool(description="Order a replacement for the selected card.")
async def process_card_replacement(context: ToolContext = None) -> ToolResult:
    await asyncio.sleep(1)  # stands in for a call to your card system
    if context.is_cancelled:
        return ToolResult(llm_response={"ordered": False, "reason": "cancelled"})
    reference = f"{random.randint(0, 999999):06d}"
    context.memory.set("order_reference", reference)
    context.memory.set("order_reference_spoken", " ".join(reference))
    return ToolResult(llm_response={"ordered": True, "order_reference": reference})

This check does not undo an order that your card system has already accepted. A tool cancelled during an await can run again on the next turn. For a real order API, keep a stable idempotency key for that order attempt and use it to reconcile an uncertain result before retrying. Do not treat a saved key as proof that the order succeeded. See Cancellation.

7. Say goodbye once

When the caller says they are done, the agent should close the call, not ask another question. Add a rule to agent.yml:

agent.yml

rules:
  - When the customer says they are done, give exactly one brief goodbye and stop. Do not ask another question after that.

If the caller goes quiet instead, the bundled silence-timeout skill checks in three times, then says goodbye and hangs up. See Silence timeout to change that wording.

8. Try it

Train, restart the Inspector, and start a new voice conversation:

rasa train
rasa inspect

You hear the recording notice and the greeting.
Say My card was stolen, then four eight two one. The card is locked and you hear the exact lock message.
Choose rush shipping. The agent restates the card and shipping and asks whether to go ahead.
Say yes. You hear the order number one digit at a time.
Say That’s all, thanks. The agent says goodbye once.

Also try no at the confirmation, an unknown card number, and a correction after you choose shipping. Check the tool events and memory, not only the spoken reply: a denied order must not call process_card_replacement. Repeat yes and no as short answers, then interrupt while the mock order tool is waiting. See Test the speech path separately for a reusable test checklist. Now type the same conversation in the chat box. The skill is unchanged, and the replies take the shape of text instead.

Continue with a community project

To apply these patterns across several skills, follow the Rasa Community travel-agent tutorial. The runnable companion project includes demo data and setup instructions; its chapter snippets show how each step changes the project. These links leave the Mantle docs: the walkthrough is on rasa.community, and the code is in the public rasa-community-resources repository. Check their tested versions before adapting them to your project.

Next steps

Connect a phone line

Put this agent behind a real phone number.

Designing for voice

More patterns: lists, buttons, keypad input, and ending calls.

Speech providers

Change the recognition model, the voice, or the provider.

Test and monitor voice

Check latency and interruptions before and after launch.

​What you’ll build

​1. Write the skill

​2. Greet the caller

​3. Give the pauses your agent’s voice

​4. Confirm before placing the order

​5. Read the order number back clearly

​6. Make the order safe to interrupt

​7. Say goodbye once

​8. Try it

​Continue with a community project

​Next steps

Connect a phone line

Designing for voice

Speech providers

Test and monitor voice

What you’ll build

1. Write the skill

2. Greet the caller

3. Give the pauses your agent’s voice

4. Confirm before placing the order

5. Read the order number back clearly

6. Make the order safe to interrupt

7. Say goodbye once

8. Try it

Continue with a community project

Next steps