Improve Assistant

Improving an assistant involves making incremental updates both on the Rasa X server and on your local machine. By incorporating automated testing (e.g. End-to-End Testing) into every update, you will immediately know how any changes to your training data affect the performance of your assistant.

Rasa X is not intended to be an all-in-one bot builder; many things are easier to do in a text editor on your local machine, and that’s where you should do them. Rasa X is meant to help you improve some parts of your assistant using a UI, while still keeping it easy to make changes locally.

Making Incremental Updates

In order to improve your assistant, you will need to make updates inside and outside of Rasa X based on what you learn from your conversations. There are 3 types of updates you can make to improve your assistant: minor, major and architectural updates.

Minor Updates

The most efficient way to make iterative improvements to your model, such as fixing incorrect NLU / Core predictions or updating response templates, is to do it on the Rasa X server. From there, you can push your changes to the remote Git repository via the Integrated Version Control feature and run automated tests before redeploying.

One example of a minor update is annotating new messages that come from users talking to your assistant. You can improve your assistant by annotating new data from these users. You can classify the data to new or existing intents.

Data that is displayed in the NLU Inbox comes from a few different sources. These sources include:

  • User conversations from sharing your assistant or via a custom channel like Slack, Twilio, etc.
  • Conversations you have with the assistant on the Talk to your bot screen.

In the NLU Inbox, you can review & fix intent and entity predictions, mark them as correct, or delete them.

Rasa X NLU Train Annotate Screenshot

You can also make use of keyboard shortcuts to navigate the NLU Inbox table rows and perform actions. The j and k keys allow you to navigate down and up respectively. Pressing s will mark the currently selected item as correct. Pressing d will delete the currently selected item from the inbox.

Rasa X NLU Train Annotate Screenshot

Once you select Mark Correct, the data will be moved from the NLU Inbox to the Training Data screen.

Rasa X NLU Train Annotate Ok Button

To annotate entities, you select one or multiple words in a sentence. A menu will pop up where you select the Entity type and an optional Entity Synonym to be used instead of the actual text in the sentence:

Rasa X NLU Train Annotate Ok Button

Note

In order to use this training data you will need to retrain your assistant via the Train button.

Major Updates

For more disruptive changes to your assistant, like merging two intents into one or changing the logic of stories, whether you will want to use Rasa X or not depends on the situation. Usually smaller changes are easier in Rasa X, while larger changes are often faster in a text editor on your local machine. After you make these changes, you will want to push your updates to the Git repo connected to your Rasa X instance, and let a continuous integration (CI) server ensure that everything works as expected.

A good example of a major update that you can do with Rasa X is the creation of a new story using Interactive Learning mode on the Talk to your bot screen.

With Interactive Learning, you create a story by teaching your assistant as you talk to it. At each step in the conversation, you validate and correct the predictions that are made by your assistant. At the end, the new story can be added directly to the training data. In addition, the same story will be available to use as an end-to-end test.

To start, on the Talk to your bot screen you select the Interactive Learning button:

Interactive Learning Button Tooltip

In this example, when you enter hello, the assistant correctly predicts the intent greet. However, because you are in Interactive Learning mode, the conversation with the assistant pauses until you mark this as a correct prediction by clicking the check box.

Interactive Learning Hello Greet

As soon as you click the check box, the sentence with it’s prediction will be added to the Training Data screen. If a sentence happens to be already in the Training Data, it will not be added again to avoid duplication.

You can check if a sentence like ‘Yoh!’ also triggers the greet intent, by refreshing the page, switching again to Interactive Learning mode, and typing it in to check the prediction:

Interactive Learning Hello Update

In this case, the assistant predicts the intent affirm, which is wrong. You can change the predicted intent in the dropdown to greet, and then click on the checkbox.

Interactive Learning Sup Update

After clicking the checkbox, you will see the next predicted Core action. The conversation will again pause until you validate or correct this. In this example, the next action of the assistant is correct, and you click the checkbox.

Interactive Learning Greeting Action

The conversation is moving along nicely, and after the assistant has completed the utter_great action, you will see the next predicted step in the conversation. It is predicted that the assistant should take as Next Action: action_listen. This is correct, so you click the checkbox.

Interactive Learning Listen Action

You continue this flow until you have created a new story that you want to add to the Training Data.

On the right hand side of the screen you can find the Story till now and you can save it in the Training Data by clicking the Save Story button.

Interactive Learning Story

This completes the major update of creating a new story via Interactive Learning, and you can now retrain your assistant via the Train button.

To enable End-to-End Testing, Rasa X also provides the story in an end-to-end story format, which includes both the NLU output and the original text. If Rasa X is connected to a remote Git repository, you can save the end-to-end story by clicking on the Save end-to-end test button.

End 2 End Story

Architectural Updates

If the improvement requires altering the structure of your assistant that surrounds and defines your model, such as implementing a custom NLU component or custom action, you will want to use a text editor on your local machine. After you have improved your assistant, you will once again want to go through the process of pushing your changes to the Git repository and then testing and deploying with CI/CD.

Using Integrated Version Control

Integrating your Rasa X server with Git enables you and your team to apply best practices from software engineering to the development of AI assistants.

Once you have connected your assistant’s repository to your Rasa X server, Rasa X will automatically pull data from your remote repository and reflect those updates in Rasa X. At this initial stage, Rasa X will have the latest data from your remote repository on the Git server. Rasa X will indicate that you are up to date with the latest changes with a green state in the side bar.

Rasa X server in a clean Git state

When you make a change to your assistant in Rasa X, the side bar state will transition to an orange state. This means that you have changes that can be pushed to the remote repository when you are ready. During this time, Rasa X won’t fetch new changes from the remote repository to avoid overwriting your changes.

Rasa X server with changes which can be committed and pushed.

When you are finished making changes and are ready to add them to your Git server, you can commit and push them by clicking on the Git icon in the sidebar. If your remote branch has not changed while you were annotating, you can choose between

  • pushing changes directly to the remote branch
  • pushing them to a new branch

If new changes were added to the remote branch while you were annotating, you will only be able to push your changes to a new branch to avoid potential merge conflicts. Rasa X will notify you that there have been changes to the remote branch with a red state on the side bar.

Rasa X server with changes while remote is ahead.

When you are ready, click on the side bar icon and follow its instructions. Once your changes are pushed, Rasa X will return to being in sync with your remote branch and will pull in the latest changes from your remote as they become available.

Note

If you pushed your changes to a new branch, these changes won’t be reflected in Rasa X until you have merged them into the target branch.

You can of course also discard your changes. Rasa X will then be reset to the latest state of your remote branch.

Leveraging Git

Integrating Rasa X with Git enables your team to use the full power of the Git ecosystem:

  • Apply a review process for any changes to your model
  • Automatically evaluate dialogues end-to-end, running through test conversations and making sure that both NLU and Core make correct predictions, as part of a CI pipeline (check out End-to-End Testing in the Rasa docs for more information)
  • Automatically generate reports, a confusion matrix, and a confidence histogram on your new model as part of a CI pipeline (check out Testing Your Assistant in the Rasa docs for more information)
  • Minimize the time from change to production with continuous deployment