Releases: naivenlp/naive-tokenizers
Releases · naivenlp/naive-tokenizers
Release v0.0.1
Contents:
- Add abstract interface for
Tokenizer - Wrapper
jiebaasJiebaTokenizer - Add
TransformerTokenizer - Add
BertTokenizer
Contents:
Tokenizerjieba as JiebaTokenizerTransformerTokenizerBertTokenizer