notice

This is documentation for Rasa Documentation v2.x, which is no longer actively maintained.
For up-to-date documentation, see the latest version (3.x).

Version: 2.x

rasa.nlu.tokenizers.convert_tokenizer

ConveRTTokenizer Objects

class ConveRTTokenizer(WhitespaceTokenizer)

This tokenizer is deprecated and will be removed in the future.

The ConveRTFeaturizer component now sets the sub-token information for dense featurizable attributes of each message object.

__init__

| __init__(component_config: Dict[Text, Any] = None) -> None

Initializes ConveRTTokenizer with the ConveRT model.

Arguments:

  • component_config - Configuration for the component.