Support hallucination score #218
Conversation
WenjingKangIntel
commented
Mar 7, 2025
•
WenjingKangIntel
commented
- Use deepeval to calculate hallucination score
- Use SelfCheckGPT to calculate hallucination score
- Use both deepeval and SelfCheckGPT to calculate hallucination score (time consuming)
WenjingKangIntel
force-pushed
the
issue-199-dev-hallucination
branch
from
d016944 to
9dd0884
Compare
|
Resolves #199 |
yzheng124
force-pushed
the
issue-199-dev-hallucination
branch
3 times, most recently
from
589959e to
f01ab94
Compare
* Support hallucination score, deepeval part * Support hallucination score, selfcheckgpt part * Add workflow --------- Signed-off-by: Kang Wenjing <wenjing.kang@intel.com> Signed-off-by: yzheng124 <yi.zheng@intel.com> Co-authored-by: yzheng124 <yi.zheng@intel.com>
WenjingKangIntel
force-pushed
the
issue-199-dev-hallucination
branch
3 times, most recently
from
d152a45 to
9b2bf55
Compare
| @@ -0,0 +1,20 @@ | |||
| Can you suggest some popular fruit-based drinks that are healthy and refreshing? | |||
There was a problem hiding this comment.
I wouldn't call this file a "personality", rather "questions". Similar other files
|
|
||
| 3. Run Ollama, taking `deepseek-r1` as an example: | ||
| ``` | ||
| ollama run deepseek-r1 |
There was a problem hiding this comment.
Using deepseek-r1 will take forever. Is it possible to use any of the distilled models? https://huggingface.co/collections/deepseek-ai/deepseek-r1-678e1e131c0169c0bc89728d
There was a problem hiding this comment.
Actually, the command ollama run deepseek-r1 will run deepseek-r1-distill-qwen-7B model, not the 681B version model. Within the deepseek-r1 series, the only smaller model available is the deepseek-r1-distill-qwen-1.5B, but its performance is not so good.
| return dataset[dataset_info["col"]], ov_chat_engine | ||
|
|
||
|
|
||
| def load_chat_model(model_name: str, token: str = None) -> OpenVINOLLM: |
There was a problem hiding this comment.
Wouldn't it be better to use the function from main.py instead of copying it here? What if the function changes?
There was a problem hiding this comment.
We have carefully considered and implemented your valuable advice.
