This tokenizer is a wrapper for Jieba (https://github.com/fxsjy/jieba).
Construct a new intent classifier using the MITIE framework.
Load all the custom dictionaries stored in the path.
More information about the dictionaries file format can be found in the documentation of jieba. https://github.com/fxsjy/jieba#load-dictionary
Persist this model into the passed directory.