notice

This is unreleased documentation for Rasa Open Source Documentation Master/Unreleased version.
For the latest released documentation, see the latest version (2.2.x).

Version: Master/Unreleased

rasa.nlu.tokenizers.lm_tokenizer

LanguageModelTokenizer Objects

class LanguageModelTokenizer(WhitespaceTokenizer)

This tokenizer is deprecated and will be removed in the future.

Use the LanguageModelFeaturizer with any other Tokenizer instead.

__init__

| __init__(component_config: Dict[Text, Any] = None) -> None

Initializes LanguageModelTokenizer for tokenization.

Arguments:

  • component_config - Configuration for the component.