[Tokenizers] Tokenizer fix decode #767
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…h and without string support in OV core. Moved StringTensorUnpack and reworked it to be aligned with the new approach. Reworked sentece piece op and translation code to be compatible with several variants of string tensor representation and the plugin wrapping hack.
…ranch to contrib in form compatible with both master and the branch with string tensors support. Added CaseFoldUTF8 from that branch.
…pty constants, register StringTensorPack and StringTensorUnpack as OV operations to be able to read IRs with those operations
…den Const translator for TF to intercept string constants
…r conditional compilation based on available features in OpenVINO
…combination of WordpieceTokenizeWithOffsets and LookupTableFindV2 from TensorFlow
…ute initialization optional (needed for core.make_node)
…ean mask extra output
…n and RegexSplit based on paddle fast_tokenizer lib. Limited implementation, not all of the features of ops and TF translated ops are implemented.
… necessary steps to complete HF bert preprocessing conversion (not validated)
…dling of model name
…kenizer and main model is fixed partially (still produces topologically incorrect model)
…uts, now Bert and its tokenizer are connected together correctly
…ding, fix bugs for batches processing
…bled debug output
ilya-lavrenov
force-pushed
the
tokenizer-fix-decode
branch
from
7a9448f to
f68123f
Compare
|
/azp run openvino_contrib-mac |
|
Azure Pipelines successfully started running 1 pipeline(s). |
ilya-lavrenov
force-pushed
the
tokenizer-fix-decode
branch
6 times, most recently
from
13fdedd to
f738fcb
Compare
ilya-lavrenov
force-pushed
the
tokenizer-fix-decode
branch
from
f738fcb to
5209f50
Compare
ilya-lavrenov
commented
Nov 29, 2023
removed build of fast tokenizers
mryzhov
approved these changes
Nov 29, 2023
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
No description provided.