notice
This is unreleased documentation for Rasa Open Source Documentation Master/Unreleased version.
For the latest released documentation, see the latest version (2.8.x).
Version: Master/Unreleased
rasa.nlu.tokenizers.whitespace_tokenizer
WhitespaceTokenizer Objects
class WhitespaceTokenizer(Tokenizer)
__init__
| __init__(component_config: Dict[Text, Any] = None) -> None
Construct a new tokenizer using the WhitespaceTokenizer framework.
remove_emoji
| remove_emoji(text: Text) -> Text
Remove emoji if the full text, aka token, matches the emoji regex.