notice

This is documentation for Rasa Open Source Documentation v2.3.x, which is no longer actively maintained.
For up-to-date documentation, see the latest version (2.4.x).

Version: 2.3.x

rasa.nlu.tokenizers.tokenizer

Tokenizer Objects

class Tokenizer(Component)

__init__

| __init__(component_config: Dict[Text, Any] = None) -> None

Construct a new tokenizer using the WhitespaceTokenizer framework.

tokenize

| tokenize(message: Message, attribute: Text) -> List[Token]

Tokenizes the text of the provided attribute of the incoming message.

train

| train(training_data: TrainingData, config: Optional[RasaNLUModelConfig] = None, **kwargs: Any, ,) -> None

Tokenize all training data.

process

| process(message: Message, **kwargs: Any) -> None

Tokenize the incoming message.