JSONDecodeError in samples of phi-4-multimodal and multimodal-rag

[JamieVC]

JSONDecodeError#1,
In sample https://github.com/openvinotoolkit/openvino_notebooks/blob/latest/notebooks/phi-4-multimodal/phi-4-multimodal.ipynb, failed on convert_phi4mm()

---

JSONDecodeError                           Traceback (most recent call last)
Cell In[6], line 16
     12     quantization_config = None
     13     model_path = base_model_path.parent / (base_model_path.name + "-fp16-ov")
---> 16 convert_phi4mm(model_id, model_path, quantization_config=quantization_config)

JSONDecodeError#2,
In sample https://github.com/openvinotoolkit/openvino_notebooks/tree/latest/notebooks/multimodal-rag, failed on
compressed_ov_model = nncf.compress_weights(ov_model, **compression_config)

Installation instructions (Please mark the checkbox)
[yes ] I followed the installation guide at https://github.com/openvinotoolkit/openvino_notebooks#-installation-guide to install the notebooks.

** Environment information **
Please run python check_install.py in the openvino_notebooks directory. If the output is NOT OK for any of the checks, please follow the instructions to fix that. If that does not work, or if you still encounter the issue, please paste the output of check_install.py here.

Additional context
Add any other context about the problem here.

openvino_notebooks>pip list
Package Version

---

about-time 4.2.1
accelerate 1.4.0
aiofiles 23.2.1
alive-progress 3.2.0
annotated-types 0.7.0
anyio 4.8.0
argon2-cffi 23.1.0
argon2-cffi-bindings 21.2.0
arrow 1.3.0
asttokens 3.0.0
async-lru 2.0.4
attrs 25.1.0
audioread 3.0.1
autograd 1.7.0
babel 2.17.0
backoff 2.2.1
beautifulsoup4 4.13.3
bleach 6.2.0
certifi 2025.1.31
cffi 1.17.1
charset-normalizer 3.4.1
click 8.1.8
cma 3.2.2
colorama 0.4.6
comm 0.2.2
contourpy 1.3.1
cycler 0.12.1
debugpy 1.8.13
decorator 5.2.1
defusedxml 0.7.1
Deprecated 1.2.18
dill 0.3.9
exceptiongroup 1.2.2
executing 2.2.0
fastapi 0.115.11
fastjsonschema 2.21.1
ffmpy 0.5.0
filelock 3.17.0
fonttools 4.56.0
fqdn 1.5.1
fsspec 2025.3.0
gradio 5.20.1
gradio_client 1.7.2
grapheme 0.6.0
groovy 0.1.2
h11 0.14.0
httpcore 1.0.7
httpx 0.28.1
huggingface-hub 0.29.3
idna 3.10
ipykernel 6.29.5
ipython 8.34.0
ipywidgets 8.1.5
isoduration 20.11.0
jedi 0.19.2
Jinja2 3.1.6
joblib 1.4.2
json5 0.10.0
jsonpointer 3.0.0
jsonschema 4.23.0
jsonschema-specifications 2024.10.1
jstyleson 0.0.2
jupyter_client 8.6.3
jupyter_core 5.7.2
jupyter-events 0.12.0
jupyter-lsp 2.2.5
jupyter_server 2.15.0
jupyter_server_terminals 0.5.3
jupyterlab 4.3.5
jupyterlab_pygments 0.3.0
jupyterlab_server 2.27.3
jupyterlab_widgets 3.0.13
kiwisolver 1.4.8
lazy_loader 0.4
librosa 0.11.0
llvmlite 0.44.0
markdown-it-py 3.0.0
MarkupSafe 2.1.5
matplotlib 3.10.1
matplotlib-inline 0.1.7
mdurl 0.1.2
mistune 3.1.2
mpmath 1.3.0
msgpack 1.1.0
natsort 8.4.0
nbclient 0.10.2
nbconvert 7.16.6
nbformat 5.10.4
nest-asyncio 1.6.0
networkx 3.3
ninja 1.11.1.3
nncf 2.15.0
notebook_shim 0.2.4
numba 0.61.0
numpy 2.1.3
openvino 2025.1.0.dev20250311
openvino-telemetry 2025.1.0
orjson 3.10.15
overrides 7.7.0
packaging 24.2
pandas 2.2.3
pandocfilters 1.5.1
parso 0.8.4
peft 0.13.2
pillow 11.1.0
pip 25.0.1
platformdirs 4.3.6
pooch 1.8.2
prometheus_client 0.21.1
prompt_toolkit 3.0.50
psutil 7.0.0
pure_eval 0.2.3
py-cpuinfo 9.0.0
pycparser 2.22
pydantic 2.10.6
pydantic_core 2.27.2
pydot 2.0.0
pydub 0.25.1
Pygments 2.19.1
pymoo 0.6.1.3
pyparsing 3.2.1
python-dateutil 2.9.0.post0
python-json-logger 3.3.0
python-multipart 0.0.20
pytz 2025.1
pywin32 309
pywinpty 2.0.15
PyYAML 6.0.2
pyzmq 26.2.1
referencing 0.36.2
regex 2024.11.6
requests 2.32.3
rfc3339-validator 0.1.4
rfc3986-validator 0.1.1
rich 13.9.4
rpds-py 0.23.1
ruff 0.9.10
safehttpx 0.1.6
safetensors 0.5.3
scikit-learn 1.6.1
scipy 1.15.2
semantic-version 2.10.0
Send2Trash 1.8.3
setuptools 75.8.2
shellingham 1.5.4
six 1.17.0
sniffio 1.3.1
soundfile 0.13.1
soupsieve 2.6
soxr 0.5.0.post1
stack-data 0.6.3
starlette 0.46.1
sympy 1.13.1
tabulate 0.9.0
terminado 0.18.1
threadpoolctl 3.5.0
tinycss2 1.4.0
tokenizers 0.21.0
tomli 2.2.1
tomlkit 0.13.2
torch 2.6.0+cpu
torchaudio 2.6.0+cpu
torchvision 0.21.0+cpu
tornado 6.4.2
tqdm 4.67.1
traitlets 5.14.3
transformers 4.49.0
typer 0.15.2
types-python-dateutil 2.9.0.20241206
typing_extensions 4.12.2
tzdata 2025.1
uri-template 1.3.0
urllib3 2.3.0
uvicorn 0.34.0
wcwidth 0.2.13
webcolors 24.11.1
webencodings 0.5.1
websocket-client 1.8.0
websockets 15.0.1
wheel 0.45.1
widgetsnbextension 4.0.13
wrapt 1.17.2

[JamieVC]

hi @eaidova

Thanks for your suggestion. I rolled back the version of nncf to 2.14.1 that doesn't use cpuinfo.
And then the JSONDecodeError was gone.

However, in sample https://github.com/openvinotoolkit/openvino_notebooks/tree/latest/notebooks/multimodal-rag, failed on
text_data = audio_to_text(output_audio_path)

---

RuntimeError Traceback (most recent call last)
Cell In[12], line 14
12 except Exception as e:
13 print("e:", e)
---> 14 raise e

Cell In[12], line 4
2 video_to_images(filepath, output_folder)
3 video_to_audio(filepath, output_audio_path)
----> 4 text_data = audio_to_text(output_audio_path)
6 with open(output_folder + "output_text.txt", "w") as file:
7 file.write(text_data)

It looks like cpu inference part has an issue.
Then I tried to use gpu but it had another error.

[JamieVC]

After I remove it from pipeline(), the multimodal-rag runs well on iGPU.

The "device=torch.device("cpu")," from pipeline() was added is only due to issues with running pipeline on MacOS where torch select diffuerst device by default.

[eaidova]

fix for pipeline device #2816

[nikita-savelyevv]

@JamieVC Could you run the reproducer with NNCF installed from openvinotoolkit/nncf#3357 ?