[OV] Add --all-layers argument to CLI #713
Conversation
|
The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update. |
optimum/commands/export/openvino.py
Outdated
| action="store_true", | ||
| default=None, | ||
| help=( | ||
| "Whether embeddings and last MatMul layers should be compressed to a primary precision (usually, INT4)." |
There was a problem hiding this comment.
Is there a non-INT4 usecase? I see below that if it is provided, all_layers is set to None if is_int8 else self.args.all_layers so it seems like it's ignored for INT8? If it is only for INT4 it would be good to clarify that in the help message. And possibly make it mutually exclusive with INT8.
There was a problem hiding this comment.
This is for INT4 only.
There was a problem hiding this comment.
Thanks for your comment! By default we compress those layers to INT8, even if number of bits is set to 4. This flag allows to compress those layer to INT4 as well. I've updated the description, hopefully it is more clear now (since currently the only supported primary precision is INT4, I've rephrased it a bit).
What does this PR do?
Add
--all-layersquantization argument to openvino export CLI interface.Before submitting