load_in_4bit option for OVModelForCausalLM

[AlexKoff88]

• API extension:

import nncf
from transformers import AutoTokenizer
from optimum.intel.openvino import OVModelForCausalLM

MODEL_ID = "databricks/dolly-v2-3b" 
tokenizer = AutoTokenizer.from_pretrained(MODEL_ID)

model = OVModelForCausalLM.from_pretrained(
    MODEL_ID,
    export=True,
    load_in_4bit=True,
    quantization_config={"dataset": "ptb", "mode": nncf.CompressWeightsMode.INT4_ASYM, "ratio": 0.8},
) 

• A set of default configs for 4-bit weights-only quantization of popular models
• Aligned tests with the latest version of NNCF (+Conv 8-bit quantization)

[HuggingFaceDocBuilderDev]

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

[echarlaix]

looks really great @AlexKoff88 !

[echarlaix]

Not sure we need both an ov_config and quantization_config, would prefer to keep only one if that's feasible

[AlexKoff88]

@echarlaix, how about deprecating ov_config here and migrating to different types of QuantizationConfigMixin so that there will be only quantization_config parameter?

[echarlaix]

was actually thinking about the opposite as ov_config have a quantization section, need to think a bit about this

[AlexKoff88]

@echarlaix, I've changed the code according to your suggestion.

[echarlaix]

was actually thinking about the opposite as ov_config have a quantization section, need to think a bit about this

[AlexKoff88]

CI is ready. The two failed quantization tests will be fixed by #535.

[AlexKoff88]

@echarlaix, fyi, if I name it quantization_config I get an error during the serialization process due to the logic in transformers library that checks the availability of quantization_config field and fails if it is None.

[echarlaix]

Could having only one argument make sense instead of having both compression and weight_quantization_config (we could create an instance of WeightQuantizationConfig if needed in the init directly). Also could you add a test to verify serialization is working ?

[AlexKoff88]

Fixed

[echarlaix]

thanks for iterating, it looks great

[echarlaix]

for when is the next nncf release planned ?

[AlexKoff88]

The release should be in 2-3 weeks. I would like to migrate to develop version for now to have time for integration.

[echarlaix]

OK would prefer to wait for nncf next release but 2/3 weeks sounds reasonable

[echarlaix]

Could having only one argument make sense instead of having both compression and weight_quantization_config (we could create an instance of WeightQuantizationConfig if needed in the init directly). Also could you add a test to verify serialization is working ?

[AlexKoff88]

@echarlaix, PR is ready and all the comments are resolved.