Introduction
My name is Muhammed Emin Tetik. I am a data scientist at Softtech. I was awarded third place in the Rasa Agent Building Challenge 2025. In this blog post, I will share the details of my project and my first experiences using the Rasa platform. If you're considering starting development on the Rasa platform, this blog post may be helpful for you.
Project links:
Problem Definition
I’d like to share the story behind the idea for this project. The goal was to support dental clinic staff who work long hours. After the competition announcement, I started thinking about the problem and came up with this idea. In fact, this idea was inspired by real-life challenges I’ve observed. My older brother is a dentist who runs his own clinic, which allowed me to closely observe the daily operations of dental clinics.
From my observations, I noticed a significant amount of effort is spent on non-treatment-related patient tasks, many requiring the dentist’s direct involvement. Since dentists are responsible for both treatment and non-treatment tasks, this process is very taxing for them. It also negatively impacts the performance of the treatment process, which is their primary responsibility. Having witnessed and listened to these challenges firsthand, I could clearly define the problem and the project’s requirements.
Project Overview
The dental assistant AI agent consists of four main scenarios:
- Creating a patient record to register a new patient.
- Scheduling an appointment for an existing patient.
- Sending SMS notifications to patients with scheduled appointments.
- Analyzing panoramic X-ray images of patients.
For the AI agent to be product-ready, additional cases must be integrated. At the start of the competition, I planned to develop other scenarios as well. You can explore these case ideas by reviewing the GitHub page. If you have additional ideas, feel free to contact me or open a topic in the Rasa Forum. I’m genuinely curious to hear your suggestions.
I selected these four scenarios from a broader set of concepts because they demonstrate the Rasa Platform’s ability to execute actions of different types.
- Using Rasa, a new record can be added to the database via REST API.
- Duckling is used to convert time expressions into structured time formats.
- Twilio API is utilized to send SMS notifications.
- The Grok-2-Vision model is employed to analyze X-ray images through technical image analysis.
You can examine the overall structure of the project and the services used in Image 1.
The blog post will continue to explain some of the structures not yet mentioned in the visual.
Image 1
Project Development
After settling on the project topic, I wrote a requirements document that listed the scenarios I wanted to build and used that list as my roadmap. Choosing which scenarios to tackle first was the hardest step, mainly because I’d never built anything with Rasa before. I skimmed the official docs and jumped in with the Rasa Quickstart Codespace. I opened a fresh GitHub repo, launched a codespace, and bootstrapped a new Rasa project by following the quick-start example.
When I started developing the project, I used the “contacts” example from the rasa-calm-demo repository as a template. That template was a lifesaver in the early stages. Once the project was set up, I added my OPENAI_API_KEY to the environment and ran rasa inspect to spin everything up. Boom, it worked!
I’ll be honest: the Rasa platform looked a bit scary before the first run. All those YAML files felt overwhelming. But once the app was up and running, the rest of the journey was far smoother than I expected.
Gemma Integration
After running the project once with the GPT API, I wanted to run it with the Gemma3 model operating on my local computer. I saw in the live broadcast introducing the competition that one of the evaluation criteria for projects was the use of open-source models. I already used the Gemma3 model with Ollama on my local computer for my daily tasks, so I thought integrating this feature wouldn't be too difficult. After a short investigation, I found that the Rasa Platform had an Ollama integration. However, I needed to connect my local Ollama server to Codespace.
For this, I used the ZeroTier service. With ZeroTier, it's possible to run remote machines on the same network. Since it's off-topic, I won't include details about the ZeroTier installation in this blog post. After completing the installation and adding them to the same network, I pinged my local computer from Codespace, and it was successful. All that was left was introducing Ollama to the Rasa application as an LLM command generator. After trying a few different configurations, this configuration worked:
model_groups:
- id: ollama_llm
models:
- provider: ollama
model: "gemma3:12b"
api_base: http://<ip>:11434
After defining this configuration in the endpoints.yml file, I observed that the tests I ran in Rasa Inspector were making requests to my local Ollama.
Fastapi Backend
I used the contact management template in the tests I had conducted up to this point. Now, I needed to develop other scenarios. To create these, I had to establish a database connection. Instead of connecting directly from the Rasa application, I chose a more manageable approach by developing a REST API with FastAPI and connecting the Rasa application to this REST API. Since backend development is also off-topic, I will not explain this part in detail. As a result, an endpoint where I could manually register patients, as shown in Image 2, became ready.
Image 2
Create a Patient Record
After developing the backend endpoint, I needed to develop the "create patient" flow and trigger the "create patient" action as a result of this flow. To build my first flow, I utilized the template files.
I used this add_contact.yml flow file as a template:
flows:
add_contact:
description: add a contact to your contact list
name: add a contact
steps:
- collect: "add_contact_handle"
description: "a user handle starting with @"
- collect: "add_contact_name"
description: "a name of a person"
- collect: "add_contact_confirmation"
ask_before_filling: true
next:
- if: "slots.add_contact_confirmation is not true"
then:
- action: utter_add_contact_cancelled
next: END
- else: add_contact
- id: add_contact
action: add_contact
next:
- if: "slots.return_value = 'success'"
then:
- action: utter_contact_added
next: END
- if: "slots.return_value = 'already_exists'"
then:
- action: utter_contact_already_exists
next: END
- else:
- action: utter_add_contact_error
next: END
Looking at this file, I developed my own flow:
flows:
add_contact:
description: add a contact to your contact list
name: add a contact
steps:
- collect: "add_contact_handle"
description: "a user handle starting with @"
- collect: "add_contact_name"
description: "a name of a person"
- collect: "add_contact_confirmation"
ask_before_filling: true
next:
- if: "slots.add_contact_confirmation is not true"
then:
- action: utter_add_contact_cancelled
next: END
- else: add_contact
- id: add_contact
action: add_contact
next:
- if: "slots.return_value = 'success'"
then:
- action: utter_contact_added
next: END
- if: "slots.return_value = 'already_exists'"
then:
- action: utter_contact_already_exists
next: END
- else:
- action: utter_add_contact_error
next: END
After that, I also performed the same copy-paste operation for the action and domain sections.
Template action file:
from typing import Any, Dict, List, Text
from rasa_sdk import Action, Tracker
from rasa_sdk.events import SlotSet
from rasa_sdk.executor import CollectingDispatcher
from actions.db import add_contact, get_contacts, Contact
class AddContact(Action):
def name(self) -> str:
return "add_contact"
def run(
self, dispatcher: CollectingDispatcher, tracker: Tracker, domain: Dict[str, Any]
) -> List[Dict[Text, Any]]:
contacts = get_contacts(tracker.sender_id)
name = tracker.get_slot("add_contact_name")
handle = tracker.get_slot("add_contact_handle")
if name is None or handle is None:
return [SlotSet("return_value", "data_not_present")]
existing_handles = {c.handle for c in contacts}
if handle in existing_handles:
return [SlotSet("return_value", "already_exists")]
new_contact = Contact(name=name, handle=handle)
add_contact(tracker.sender_id, new_contact )
return [SlotSet("return_value", "success")]
My action file:
from typing import Any, Dict, List, Text
from rasa_sdk import Action, Tracker
from rasa_sdk.events import SlotSet
from rasa_sdk.executor import CollectingDispatcher
from actions.services.patient_services import add_patient, get_patients, Patient
class CreatePatientRecord(Action):
def name(self) -> str:
return "create_patient_record"
def run(
self, dispatcher: CollectingDispatcher, tracker: Tracker, domain: Dict[str, Any]
) -> List[Dict[Text, Any]]:
patients = get_patients()
patient_identity_number = tracker.get_slot("add_patient_identity_number")
patient_first_name = tracker.get_slot("add_patient_first_name")
patient_last_name = tracker.get_slot("add_patient_last_name")
patient_age = tracker.get_slot("add_patient_age")
if patient_identity_number is None:
return [SlotSet("return_value", "data_not_present")]
existing_idn = {c["identity_number"] for c in patients}
if patient_identity_number in existing_idn:
return [SlotSet("return_value", "already_exists")]
new_patient = Patient(identity_number=patient_identity_number,
first_name=patient_first_name,
last_name=patient_last_name,
age=patient_age)
result = add_patient(new_patient)
if result:
return [SlotSet("return_value", "success")]
else:
return [SlotSet("return_value", "failed")]
Template domain.yml:
version: "3.1"
actions:
- add_contact
slots:
add_contact_confirmation:
type: bool
mappings:
- type: from_llm
add_contact_name:
type: text
mappings:
- type: from_llm
add_contact_handle:
type: text
mappings:
- type: from_llm
responses:
utter_ask_add_contact_confirmation:
- text: Do you want to add {add_contact_name}({add_contact_handle}) to your contacts?
buttons:
- payload: "/SetSlots(add_contact_confirmation=true)"
title: Yes
- payload: "/SetSlots(add_contact_confirmation=false)"
title: No, cancel
utter_ask_add_contact_handle:
- text: What's the handle of the user you want to add?
utter_ask_add_contact_name:
- text: What's the name of the user you want to add?
utter_add_contact_error:
- text: "Something went wrong, please try again."
utter_add_contact_cancelled:
- text: "Okay, I am cancelling this adding of a contact."
utter_contact_already_exists:
- text: "There's already a contact with that handle in your list."
utter_contact_added:
- text: "Contact added successfully."
My domain.yml file:
version: "0.1"
actions:
- create_patient_record
slots:
add_patient_confirmation:
type: bool
mappings:
- type: from_llm
add_patient_identity_number:
type: text
mappings:
- type: from_llm
add_patient_first_name:
type: text
mappings:
- type: from_llm
add_patient_last_name:
type: text
mappings:
- type: from_llm
add_patient_age:
type: text
mappings:
- type: from_llm
responses:
utter_ask_add_patient_confirmation:
- text: Do you want to add {add_patient_first_name} {add_patient_last_name} to the system?
buttons:
- payload: "/SetSlots(add_patient_confirmation=true)"
title: Yes
- payload: "/SetSlots(add_patient_confirmation=false)"
title: No, cancel
utter_ask_add_patient_first_name:
- text: What's the first name of the patient you want to add?
utter_ask_add_patient_last_name:
- text: What's the last name of the patient you want to add?
utter_ask_add_patient_identity_number:
- text: What's the government identity number of the patient you want to add?
utter_ask_add_patient_age:
- text: What's the age of the patient you want to add?
utter_patient_data_error:
- text: "There was an error processing the patient data. Please check the information and try again."
utter_patient_creation_error:
- text: "Something went wrong, please try again."
utter_add_patient_cancelled:
- text: "Okay, I am cancelling this adding of a patient."
utter_patient_already_exists:
- text: "There's already a patient with that identity number in the system."
utter_patient_added:
- text: "Patient added successfully."
Scheduling Appointments
In the previous scenario, I implemented an enhancement that prioritized triggering a REST API action with Rasa. In the current scenario, I wanted to add the ability to process unstructured data as structured data. For this, the use case of booking an appointment for a patient was very suitable.
Imagine booking an appointment over the phone with a human assistant. In that scenario, we don’t always use the same format for time expressions. Most of the time, we use phrases like “Tomorrow at 3” or “Thursday afternoon.” While researching how to convert these unstructured date expressions into a uniform format with Rasa, I contacted the Rasa team through the competition’s Slack support channel. They sent me the date-related parts from Rasa’s examples as a reply. Thanks to them, I was guided in the right direction. Additionally, the Duckling-related threads on Rasa’s forum were extremely helpful. As a result, after starting up the Duckling server, I could convert the incoming string into a datetime object using the following function.
import os
from datetime import datetime
from typing import List, Optional
from rasa.shared.nlu.training_data.message import Message
from rasa.nlu.extractors.duckling_entity_extractor import DucklingEntityExtractor
duckling_url = os.environ.get("RASA_DUCKLING_HTTP_URL")
duckling_config = {
**DucklingEntityExtractor.get_default_config(),
"url": duckling_url,
"dimensions": ["time"]
}
duckling_entity_extractor = DucklingEntityExtractor(duckling_config)
def parse_datetime(text: str) -> Optional[datetime]:
# If the text is already a date slot value extracted from Duckling,
# we can just use it
try:
result = datetime.fromisoformat(text)
return result.replace(tzinfo=None)
except ValueError:
pass
# Otherwise, we need to parse the value set by the LLM
# using Duckling
msg = Message.build(text)
duckling_entity_extractor.process([msg])
if len(msg.data.get("entities", [])) == 0:
return None
parsed_value = msg.data["entities"][0]["value"]
if isinstance(parsed_value, dict):
parsed_value = parsed_value["from"]
result = datetime.fromisoformat(parsed_value)
return result.replace(tzinfo=None)
Image 3
Image 4
You can see the date-conversion process with Duckling by referring to Image 3 and Image 4. When I reply “tomorrow 1:30 pm” in the chatbot interface, the parse_date function in the code is called. This way, even though the user hasn’t provided tomorrow’s exact date, the full date is still recorded correctly in the database.
Sending a Notification SMS
In the first scenario, we saved our patient to the database, and in the second scenario, we created an appointment for that saved patient. In this scenario, we will send an SMS message to the patient whose appointment we created so they don’t forget their appointment time.
A quick bit of research was enough to determine how to implement this scenario. Twilio is generally used for SMS-sending scenarios. Twilio offers a trial version with a phone number and a $15 balance for sending SMS messages.
In the code snippet, after constructing the SMS content, a function is used to send the SMS.
import tomllib
from twilio.rest import Client
CONFIG_PATH = “config.toml”
def get_config():
with open(CONFIG_PATH, "rb") as f:
config = tomllib.load(f)
return config
config = get_config()
def send_sms(sms_to, sms_from, sms_content):
account_sid = config["TWILIO_SID"]
auth_token = config["TWILIO_TOKEN"]
client = Client(account_sid, auth_token)
message = client.messages.create(
body=sms_content,
from_=sms_from,
to=sms_to,
)
print(f"SMS sent to {sms_to} from {sms_from}: \n{sms_content}")
return message.error_code is None
X-Ray Image Analysis
Now we come to the most exciting part of the project. Panoramic dental radiography is an X-ray technique that allows all teeth and the entire jaw to be displayed in a single image. During a panoramic X-ray exam, the imaging device traces a half-circle around the patient’s head to obtain an image that includes the entire jaw. Using this panoramic image, the dentist can quickly view the patient’s oral structure and gain a general understanding.
I thought that by using an AI model for panoramic image analysis, I could help the clinician form a preliminary idea before treatment. I first researched AI models designed for panoramic X-ray analysis to explore this. I came across many specialized models and tools for this purpose, but they were all commercial and paid. Additionally, since they were offered as packaged software, integration was difficult.
Next, I investigated large language models with image-processing capabilities for this task. At the time of my research, two successful models stood out: OpenAI’s GPT-4o and Grok2-Vision. In my own tests, Grok2-Vision performed somewhat better for this specific use case. I decided to use the Grok2-Vision model.
For the X-ray example, I used my own previously captured X-ray image. One challenge I faced at this stage was sending the X-ray image from the chatbot interface to Grok. I arrived at the following solution:
- Paste the link to the image into the chatbot interface to submit it via the chatbot.
- In the Actions section, download the image using its link and convert it to Base64 format.
- Send the Base64 string to the Grok2-Vision model along with a system prompt.
- Return the resulting analysis text to the user as the chatbot’s response.
You can inspect the action class I use for this in the code snippet:
from typing import Any, Dict, List, Text
from rasa_sdk import Action, Tracker
from rasa_sdk.events import SlotSet
from rasa_sdk.executor import CollectingDispatcher
from actions.services.patient_services import find_patient
from actions.services.image_analysis_services import analyze_image_with_llm
class AnalyzeImage(Action):
def name(self) -> str:
return "analyze_image"
def run(
self, dispatcher: CollectingDispatcher, tracker: Tracker, domain: Dict[str, Any]
) -> List[Dict[Text, Any]]:
patient_first_name = tracker.get_slot("analyze_image_patient_first_name")
patient_last_name = tracker.get_slot("analyze_image_patient_last_name")
image_url = tracker.get_slot("analyze_image_image_url")
image_analysis_instruction = tracker.get_slot("analyze_image_instruction")
patient = find_patient(patient_first_name, patient_last_name, None)
if patient is None:
return [SlotSet("return_value", "patient_not_found")]
image_analysis = analyze_image_with_llm(image_url, image_analysis_instruction)
print("Image analysis result:", image_analysis)
# dispatcher.utter_message(
# response="utter_image_analysis_report",
# image_analysis=image_analysis
# )
if image_analysis is not None:
return [
SlotSet("image_analysis", image_analysis),
SlotSet("return_value", "success")
]
else:
return [SlotSet("return_value", "failed")]
Image 5: My panoramic X-ray image
Image 6: The analysis of my panoramic X-ray image
Conclusion
In this blog post, I discussed the project I developed for the 2025 Rasa Agent Development Challenge. I hope it serves as a helpful guide for anyone getting started with the Rasa CALM framework.
My deepest thanks go to the Rasa team for organizing the competition and their unwavering support, as well as to the other competitors who opened new horizons for me.