|
| 1 | +## Convert and Optimize Model |
| 2 | + |
| 3 | +<Tabs groupId="model-source"> |
| 4 | + <TabItem label="From Hugging Face" value="huggingface"> |
| 5 | + Use `optimum-intel` package to convert and optimize models: |
| 6 | + ```bash |
| 7 | + pip install optimum-intel[openvino] |
| 8 | + ``` |
| 9 | + |
| 10 | + Download and convert a model to the OpenVINO IR format: |
| 11 | + <Tabs groupId="export-precision"> |
| 12 | + <TabItem label="Compress weights to the int4 precision" value="int4"> |
| 13 | + ```bash |
| 14 | + optimum-cli export openvino --model meta-llama/Llama-2-7b-chat-hf --weight-format int4 ov_llama_2_7b_int4 --trust-remote-code |
| 15 | + ``` |
| 16 | + </TabItem> |
| 17 | + <TabItem label="Keep full model precision" value="fp16"> |
| 18 | + ```bash |
| 19 | + optimum-cli export openvino --model meta-llama/Llama-2-7b-chat-hf --weight-format fp16 ov_llama_2_7b_fp16 --trust-remote-code |
| 20 | + ``` |
| 21 | + </TabItem> |
| 22 | + </Tabs> |
| 23 | + |
| 24 | + :::info |
| 25 | + |
| 26 | + Check a full list of conversion options [here](https://huggingface.co/docs/optimum/en/intel/openvino/export). |
| 27 | + |
| 28 | + ::: |
| 29 | + |
| 30 | + :::tip |
| 31 | + |
| 32 | + You can also use [pre-converted LLMs](https://huggingface.co/collections/OpenVINO/llm-6687aaa2abca3bbcec71a9bd). |
| 33 | + |
| 34 | + ::: |
| 35 | + </TabItem> |
| 36 | + <TabItem label="From Model Scope" value="modelscope"> |
| 37 | + Use `modelscope` and `optimum-intel` packages to convert and optimize models: |
| 38 | + ```bash |
| 39 | + pip install modelscope optimum-intel[openvino] |
| 40 | + ``` |
| 41 | + |
| 42 | + Download the required model to a local folder: |
| 43 | + ```bash |
| 44 | + modelscope download --model 'Qwen/Qwen2-7b' --local_dir model_path |
| 45 | + ``` |
| 46 | + |
| 47 | + :::tip |
| 48 | + |
| 49 | + Convert the model and compress weights: |
| 50 | + |
| 51 | + <Tabs groupId="export-precision"> |
| 52 | + <TabItem label="INT4" value="int4"> |
| 53 | + ```bash |
| 54 | + optimum-cli export openvino -m model_path --weight-format int4 ov_qwen2_7b_int4 --task text-generation-with-past |
| 55 | + ``` |
| 56 | + </TabItem> |
| 57 | + <TabItem label="INT8" value="int8"> |
| 58 | + ```bash |
| 59 | + optimum-cli export openvino -m model_path --weight-format int8 ov_qwen2_7b_int8 --task text-generation-with-past |
| 60 | + ``` |
| 61 | + </TabItem> |
| 62 | + <TabItem label="FP16" value="fp16"> |
| 63 | + ```bash |
| 64 | + optimum-cli export openvino -m model_path --weight-format fp16 ov_qwen2_7b_fp16 --task text-generation-with-past |
| 65 | + ``` |
| 66 | + </TabItem> |
| 67 | + </Tabs> |
| 68 | + |
| 69 | + ::: |
| 70 | + </TabItem> |
| 71 | +</Tabs> |
0 commit comments