Skip to content
ChatGPT and Rasa

March 16th, 2023

Answering Questions About Structured Data with Rasa and ChatGPT

  • portrait of Alan Nichol

    Alan Nichol

A few years ago we introduced the ability to integrate Rasa with knowledge bases, allowing assistants to answer detailed follow-up questions, like in the conversation below.

While powerful, the knowledge base feature is quite a lot of work to set up. I’ve been impressed with ChatGPT’s ability to answer questions about structured data, so wanted to explore if we could do this more easily and get better results if we leverage instruction-tuned LLMs like ChatGPT.

Compared to knowledge base actions, using an LLM to answer these types of questions:

  • Requires much less work to set up
  • Can easily be extended to new fields (often without re-training)
  • Produces more natural responses

However, it also comes with limitations:

  • You can't control exactly what the bot says
  • Your bot has the potential to hallucinate facts
  • You are potentially vulnerable to prompt injection attacks.

I’ll first explain how this works and then we’ll look at each of these points in turn. Here’s the repository in case you want to get started right away.

Using ChatGPT in a Rasa Custom Action

Most chatbots query APIs and/or databases to interact with the world outside of the conversation. Custom actions are a convenient way to build that functionality in Rasa.

For this example, we have two custom actions. The first action, action_show_restaurants, fetches restaurant recommendations from an API, presents them to the user, and also stores the results in a slot. The second action answers follow-up questions, taking the results of the previous API call from the “results” slot and using the ChatGPT API to answer the question.

In the example conversation above, the user asks “which of those have Wifi?”. The prompt that gets sent to the ChatGPT API combines instructions, structured data, and the user’s question.

Answer the following question, based on the data shown. 
Answer in a complete sentence and don't say anything else.


Restaurants,Rating,Has WiFi,cuisine
Mama's Pizza,4.5,False,italian
Maria Bonita,4.6,True,mexican
China Palace,3.2,False,chinese
The Cheesecake Factory,4.2,True,american
Pizza Hut,3.8,True,italian

Which of those have Wifi?

This prompt is very simple and I’m sure you can get better results by tweaking it for your use case. Providing more detailed instructions would be a great place to start. And while ChatGPT does quite a good job interpreting the CSV format, there’s no reason it has to be presented that way.

Nonexistent Fields

What if you ask about something that’s not in the data? I was pleasantly surprised that ChatGPT’s response acknowledged that the information is missing, rather than just making something up. But from a conversation design perspective, the phrasing is very technical and not especially friendly. Again, you should be able to get better results by tweaking the prompt.

Adding Support for New Fields

Let’s say you’re doing some conversation analysis and discover that, actually, quite a few people are asking about reservations and takeout options. And luckily, the API you’re using actually provides you with this data.

The nice thing about this LLM-based approach is that we can start answering these questions right away, without re-training the bot! We just have to make sure that the additional fields end up in the CSV representation we put in the prompt. To try this out, modify the actions.py file in the repo to read the restaurants_extra.csv file instead. Restart the action server, and your bot will now be able to answer these questions.

Even with this simple prompt, we can get more natural responses than we can with knowledge base actions:

Hallucination

ChatGPT, for all its strengths, is still prone to hallucination. Just look at the conversation below. The data show Maria Bonita as a Mexican restaurant, not a pizza place. It also doesn’t have the highest rating outright. So this is a clear example of the model making an error. Depending on the stakes of the AI assistant you’re building, this may or may not be a dealbreaker.

Prompt Injection

I have very much enjoyed watching the internet try to trick ChatGPT, Bing Chat, and others into doing their bidding using a prompt injection attack. Here I tried something very unsophisticated and got the result I wanted.

A simple remediation to this would be to add a prompt_injection intent to your NLU model and create a rule that triggers an appropriate response. It seems like it would be straightforward enough for a classifier to detect the difference between a good-faith user message and a prompt injection attack, but I haven’t tried this in production so can’t vouch for its effectiveness.

Next Steps

We’ve explored just one of the ways to use LLMs like ChatGPT to make bots smarter. Here, we're directly sending LLM-generated text back to bot users. For some applications (e.g. customer support for large brands), the reputational risk that comes from hallucination and prompt injection is a dealbreaker. But for many other chat and voice-bot use cases, this is a great way to leverage the power of LLMs inside a bot that can interact with databases and APIs, while providing more natural conversation. Do take the code and try it out and extend it for your own use case, and let us know in the forum how you’re getting on!

P.S. Read about our work on using LLMs to break free from intents.