Falcon-40B convert successfully but output_dir has not been created

[JunxiChhen]

Convert cmd:

python3 convert.py         --model_id tiiuae/falcon-40b         --output_dir /root/.cache/huggingface/hub/tiiuae/falcon-40b-ov         --stateful --precision FP16

Benchmarking cmd:

numactl -C 0-55 -m 0 python benchmark.py     -m /root/.cache/huggingface/hub/falcon-40b-ov/pytorch/dldt/FP16  -p "It is done" -n 3     -bs 1     -d CPU --torch_compile_backend openvino     -ic 128 --num_beams 1 -lc bfloat16_config.json 2>&1 | tee -a ./logs/0.log

1. Model converting successfully to /root/.cache/huggingface/hub/tiiuae/falcon-40b-ov:
   
2. Model cannot be found during benchmarking:
   
3. Dir /root/.cache/huggingface/hub/tiiuae/falcon-40b-ov does not exist.

[eaidova]

@JunxiChhen I do not see in your logs that model successfully converted (here only logs begins), is there possibility that this directory does not have write permissions?

[JunxiChhen]

We have write permission actually. We can successfully convert models in same directory including llama2, gpt-j, ... Could you have a try and check if there's same issue? Thanks. @eaidova

[eaidova]

@JunxiChhen Unfortunately, my working machine does not have enough RAM for conversion falcon-40b, so I can not check it, but if you can share more logs for conversion (better in text format instead of screenshot), it will be very helpful)

[JunxiChhen]

benchmark_latency__bfloat16___04-23-24-06-25-42.log
Here is the log file.

[eaidova]

@JunxiChhen possibly it is some logging issue, but on my side I can not see the same behaviour. On my end model conversion failed with
RuntimeError: Sizes of tensors must match except in dimension 2. Expected size 8 but got size 128 for tensor number 1 in the list.

I prepared fix for that huggingface/optimum-intel#685

[eaidova]

@JunxiChhen could you please check falcon with the latest openvino.genai? the fix in optimum-intel was merged, commit is moved in llm_bench requirements.txt

[JunxiChhen]

Thanks. It passed when there's internet connection.
However, in offline mode, we met issue (We pre-downloaded the falocn model from huggingface):

[eaidova]

@JunxiChhen maybe setting of these variables may help avoiding reaching hf hub https://huggingface.co/docs/transformers/main/en/installation#offline-mode?

[JunxiChhen]

We downgraded the transformers version to 4.39.1 and passed. Thanks @eaidova