This is unreleased documentation for Rasa Open Source Documentation Master/Unreleased version.
For the latest released documentation, see the latest version (2.0.x).

Version: Master/Unreleased


LanguageModelTokenizer Objects

class LanguageModelTokenizer(Tokenizer)

Tokenizer using transformer based language models.

Uses the output of HFTransformersNLP component to set the tokens for dense featurizable attributes of each message object.