Version: 3.x
rasa.nlu.tokenizers.convert_tokenizer
ConveRTTokenizer Objects
class ConveRTTokenizer(WhitespaceTokenizer)
This tokenizer is deprecated and will be removed in the future.
The ConveRTFeaturizer component now sets the sub-token information for dense featurizable attributes of each message object.
__init__
| __init__(component_config: Dict[Text, Any] = None) -> None
Initializes ConveRTTokenizer with the ConveRT model.
Arguments:
component_config- Configuration for the component.