This is documentation for Rasa Open Source Documentation v2.0.x, which is no longer actively maintained.
For up-to-date documentation, see the latest version (2.6.x).

Version: 2.0.x


LanguageModelTokenizer Objects

class LanguageModelTokenizer(Tokenizer)

Tokenizer using transformer based language models.

Uses the output of HFTransformersNLP component to set the tokens for dense featurizable attributes of each message object.