Code

Add OpenVINO Tokenizers#513

Code

Merged

echarlaix merged 26 commits intohuggingface:mainhuggingface/optimum-intel:mainfrom apaniukov:openvino_tokenizersapaniukov/optimum-intel:openvino_tokenizersCopy head branch name to clipboardFeb 8, 2024

Commits

Commits on Jan 18, 2024

• Convert tokenizers with openvino_tokenizers

  

  slyalin

  authored and

  apaniukov

  committed
  94bc226

  Copy full SHA for 94bc226

• Update optimum/exporters/openvino/__main__.py

  

  slyalin

  authored and

  apaniukov

  committed
  6bb395f

  Copy full SHA for 6bb395f

• Refactor and Add Tests

  apaniukov

  committed
  7d16ec7

  Copy full SHA for 7d16ec7

• Fix t5 Test

  apaniukov

  committed
  f0933ad

  Copy full SHA for f0933ad

• Add Warning

  apaniukov

  committed
  24cc616

  Copy full SHA for 24cc616

• Return Tests

  apaniukov

  committed
  49337b0

  Copy full SHA for 49337b0

• Move export_tokenizer to convert.py

  Show description for 7709043
  apaniukov

  committed
  7709043

  Copy full SHA for 7709043

• Avoid Double Tokenizer Save

  apaniukov

  committed
  dbd609b

  Copy full SHA for dbd609b

• Fix Style

  apaniukov

  committed
  7e24f10

  Copy full SHA for 7e24f10

• Refactor After Review

  apaniukov

  committed
  2cf460d

  Copy full SHA for 2cf460d

• Skip Tokenizers Tests If No Package Installed

  Show description for 57782d1
  apaniukov

  committed
  57782d1

  Copy full SHA for 57782d1

• Merge branch 'main' into openvino_tokenizers

  apaniukov

  authored
  db668da

  Copy full SHA for db668da

Commits on Jan 19, 2024

• Style Fix

  apaniukov

  committed
  e7cd70f

  Copy full SHA for e7cd70f

• Fix OV Tokenizers Check

  apaniukov

  committed
  40cf117

  Copy full SHA for 40cf117

• Fix Tests

  apaniukov

  committed
  901f48a

  Copy full SHA for 901f48a

Commits on Jan 23, 2024

• Add Missing return

  apaniukov

  committed
  1a76bd4

  Copy full SHA for 1a76bd4

• Turn off tokenizer message if not installed

  apaniukov

  committed
  f2b2237

  Copy full SHA for f2b2237

Commits on Jan 24, 2024

• Merge branch 'main' into openvino-tokenizers

  apaniukov

  committed
  3066ade

  Copy full SHA for 3066ade

Commits on Jan 29, 2024

• Merge branch 'main' into openvino-tokenizers

  apaniukov

  committed
  9a2e271

  Copy full SHA for 9a2e271

• Move tokenizers to OV dependencies

  apaniukov

  committed
  7ee347e

  Copy full SHA for 7ee347e

Commits on Jan 30, 2024

• Check OV Compatibility

  apaniukov

  committed
  8d2ec41

  Copy full SHA for 8d2ec41

• Bump OV Version

  apaniukov

  committed
  32a7274

  Copy full SHA for 32a7274

Commits on Feb 1, 2024

• Move OpenVINO Tokenizers To Optional Dependencies

  apaniukov

  committed
  8c029e0

  Copy full SHA for 8c029e0

Commits on Feb 8, 2024

• Add --convert-tokenizer Option to CLI

  apaniukov

  committed
  09b067f

  Copy full SHA for 09b067f

• Merge branch 'main' into openvino_tokenizers

  apaniukov

  authored
  f3b8ce8

  Copy full SHA for f3b8ce8

• Fix SD Tokenizer

  apaniukov

  committed
  3c27fbd

  Copy full SHA for 3c27fbd