notice
This is unreleased documentation for Rasa Open Source Documentation Master/Unreleased version.
For the latest released documentation, see the latest version (2.5.x).
Version: Master/Unreleased
rasa.nlu.tokenizers.lm_tokenizer
LanguageModelTokenizer Objects
class LanguageModelTokenizer(WhitespaceTokenizer)
This tokenizer is deprecated and will be removed in the future.
Use the LanguageModelFeaturizer with any other Tokenizer instead.
__init__
| __init__(component_config: Dict[Text, Any] = None) -> None
Initializes LanguageModelTokenizer for tokenization.
Arguments:
component_config- Configuration for the component.