int4 config for Mixtral-8x7B

ljaljushkin · ljaljushkin · commit 142e6cdc912a · 2024-02-28T11:26:43.000+01:00
diff --git a/optimum/intel/openvino/configuration.py b/optimum/intel/openvino/configuration.py
@@ -96,6 +96,7 @@
     "openlm-research/open_llama_3b": {"bits": 4, "sym": True, "group_size": 64, "all_layers": True},
     "tiiuae/falcon-7b": {"bits": 4, "sym": True, "group_size": 64, "all_layers": True},
     "psmathur/orca_mini_3b": {"bits": 4, "sym": True, "group_size": 64, "all_layers": True},
+    "mistralai/Mixtral-8x7B-v0.1": {"bits": 4, "sym": True, "group_size": 128, "ratio": 0.8},
 }
 
 

Original file line number	Diff line number	Diff line change
`@@ -96,6 +96,7 @@`
`96`	`96`	`"openlm-research/open_llama_3b": {"bits": 4, "sym": True, "group_size": 64, "all_layers": True},`
`97`	`97`	`"tiiuae/falcon-7b": {"bits": 4, "sym": True, "group_size": 64, "all_layers": True},`
`98`	`98`	`"psmathur/orca_mini_3b": {"bits": 4, "sym": True, "group_size": 64, "all_layers": True},`
	`99`	`+ "mistralai/Mixtral-8x7B-v0.1": {"bits": 4, "sym": True, "group_size": 128, "ratio": 0.8},`
`99`	`100`	`}`
`100`	`101`
`101`	`102`