[128] Add OpenVINO Tokenziers Notebook (openvinotoolkit#1697)

A noteook with examples of OpenVINO Tokenizers usage
CVS-99455

---------

Co-authored-by: Yaroslav Tarkan <yaroslav.tarkan@intel.com>

Author: apaniukov

.ci/ignore_treon_mac.txt

@@ -3,6 +3,7 @@
 117-model-server
 122-quantizing-model-with-accuracy-control
 127-big-transfer-quantization
+128-openvino-tokenizers
 204-segmenter-semantic-segmentation
 216-attention-center
 220-cross-lingual-books-alignment

.ci/spellcheck/.pyspelling.wordlist.txt

@@ -1,4 +1,5 @@
 3D
+abstractive
 accelerometers
 accelerometer
 acknowledgement
@@ -155,6 +156,8 @@ denormalized
 depainting
 DepthAnything
 detections
+detokenization
+detokenizer
 Dettmers
 dev
 detectron
@@ -230,6 +233,7 @@ gcc
 GEC
 GELU
 genai
+genAI
 Girshick
 Gitee
 GitHub
@@ -694,6 +698,7 @@ SwiGLU
 SwinV
 TaskManager
 TartanAir
+tbb
 TensorBoard
 tensorflow
 tf
@@ -710,6 +715,7 @@ tokenized
 tokenizer
 Tokenizer
 tokenizers
+Tokenizers
 tokenizes
 tokenizing
 ToneColorConverter

README.md

@@ -144,6 +144,7 @@ Tutorials that explain how to optimize and quantize models with OpenVINO tools.
 | [125-torchvision-zoo-to-openvino](notebooks/125-torchvision-zoo-to-openvino/)<br>Classification<br>[![Binder](https://mybinder.org/badge_logo.svg)](https://mybinder.org/v2/gh/openvinotoolkit/openvino_notebooks/HEAD?filepath=notebooks%2F125-torchvision-zoo-to-openvino%2F125-convnext-classification.ipynb)<br>[![Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/openvinotoolkit/openvino_notebooks/blob/main/notebooks/125-torchvision-zoo-to-openvino/125-convnext-classification.ipynb)<br>Semantic Segmentation<br>[![Binder](https://mybinder.org/badge_logo.svg)](https://mybinder.org/v2/gh/openvinotoolkit/openvino_notebooks/HEAD?filepath=notebooks%2F125-torchvision-zoo-to-openvino%2F125-lraspp-segmentation.ipynb)<br>[![Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/openvinotoolkit/openvino_notebooks/blob/main/notebooks/125-torchvision-zoo-to-openvino/125-lraspp-segmentation.ipynb)| Convert torchvision classification and semantic segmentation models to OpenVINO IR |
 | [126-tensorflow-hub](notebooks/126-tensorflow-hub/)<br>[![Binder](https://mybinder.org/badge_logo.svg)](https://mybinder.org/v2/gh/openvinotoolkit/openvino_notebooks/HEAD?filepath=notebooks%2F126-tensorflow-hub%2F126-tensorflow-hub.ipynb)<br>[![Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/openvinotoolkit/openvino_notebooks/blob/main/notebooks/126-tensorflow-hub/126-tensorflow-hub.ipynb) | Convert TensorFlow Hub models to OpenVINO IR |
 | [127-big-transfer-quantization](notebooks/127-big-transfer-quantization/)| BiT Image Classification OpenVINO IR model Quantization with NNCF |
+| [128-openvino-tokenizers](notebooks/128-openvino-tokenizers/)<br>[![Binder](https://mybinder.org/badge_logo.svg)](https://mybinder.org/v2/gh/openvinotoolkit/openvino_notebooks/HEAD?filepath=notebooks%2F128-opnevino-tokenizers%2F128-opnevino-tokenizers.ipynb)<br>[![Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/openvinotoolkit/openvino_notebooks/blob/main/notebooks/128-openvino-tokenizers/128-opnevino-tokenizers.ipynb)| Incorporate Text Processing Into OpenVINO Pipelines with OpenVINO Tokenizers |
 
 <div id='-model-demos'></div>
 

README_cn.md

@@ -125,7 +125,7 @@ Jupyter notebooks 分为四个大类，选择一个跟你需求相关的开始
 | [124-hugging-face-hub](notebooks/124-hugging-face-hub/)<br>[![Binder](https://mybinder.org/badge_logo.svg)](https://mybinder.org/v2/gh/openvinotoolkit/openvino_notebooks/HEAD?filepath=notebooks%2F124-hugging-face-hub%2F124-hugging-face-hub.ipynb)<br>[![Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/openvinotoolkit/openvino_notebooks/blob/main/notebooks/124-hugging-face-hub/124-hugging-face-hub.ipynb) | 使用OpenVINO™加载Hugging Face Model Hub模型 |
 | [125-torchvision-zoo-to-openvino](notebooks/125-torchvision-zoo-to-openvino/)<br>Classification<br>[![Binder](https://mybinder.org/badge_logo.svg)](https://mybinder.org/v2/gh/openvinotoolkit/openvino_notebooks/HEAD?filepath=notebooks%2F125-torchvision-zoo-to-openvino%2F125-convnext-classification.ipynb)<br>[![Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/openvinotoolkit/openvino_notebooks/blob/main/notebooks/125-torchvision-zoo-to-openvino/125-convnext-classification.ipynb)<br>Semantic Segmentation<br>[![Binder](https://mybinder.org/badge_logo.svg)](https://mybinder.org/v2/gh/openvinotoolkit/openvino_notebooks/HEAD?filepath=notebooks%2F125-torchvision-zoo-to-openvino%2F125-lraspp-segmentation.ipynb)<br>[![Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/openvinotoolkit/openvino_notebooks/blob/main/notebooks/125-torchvision-zoo-to-openvino/125-lraspp-segmentation.ipynb)| 将torchvision 分类和语义分割模型转换为OpenVINO IR |
 | [126-tensorflow-hub](notebooks/126-tensorflow-hub/)<br>[![Binder](https://mybinder.org/badge_logo.svg)](https://mybinder.org/v2/gh/openvinotoolkit/openvino_notebooks/HEAD?filepath=notebooks%2F126-tensorflow-hub%2F126-tensorflow-hub.ipynb)<br>[![Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/openvinotoolkit/openvino_notebooks/blob/main/notebooks/126-tensorflow-hub/126-tensorflow-hub.ipynb) | 将TensorFlow Hub 模型转换到OpenVINO IR |
-
+| [128-openvino-tokenizers](notebooks/128-openvino-tokenizers/)<br>[![Binder](https://mybinder.org/badge_logo.svg)](https://mybinder.org/v2/gh/openvinotoolkit/openvino_notebooks/HEAD?filepath=notebooks%2F128-opnevino-tokenizers%2F128-opnevino-tokenizers.ipynb)<br>[![Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/openvinotoolkit/openvino_notebooks/blob/main/notebooks/128-openvino-tokenizers/128-opnevino-tokenizers.ipynb)| 使用 OpenVINO 分词器将文本处理集成到 OpenVINO 管道中 |
 <div id='-model-demos'/>
 
 ### 🎯 模型演示