feat: add complete implementation of CLEAR dataset #3466
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
yifanmai
approved these changes
Mar 24, 2025
There was a problem hiding this comment.
Looks good, thanks!
One thing to note is that this will send 1000 requests per condition for a total of 12k requests. If you want to reduce the total number of requests, you can set a lower number of requests on the run entry using the max_eval_instances=100 run expander.
|
|
||
| self.condition = condition | ||
| self.name = f"clear_{condition}" | ||
| self.description = f"A dataset for evaluating {self.CONDITION_PROMPTS[condition]} detection from patient notes with yes/no/maybe classifications." |
There was a problem hiding this comment.
Add # noqa: E501 at the end of this line to make the linter happy.
There was a problem hiding this comment.
Thank you! Fixed!
|
You need two spaces before the |
yifanmai
reviewed
Mar 24, 2025
| ), | ||
| input_noun=None, | ||
| output_noun="Respond only with 'A', 'B', or 'C'. Do not add any other text, punctuation, or symbols", | ||
| max_train_instances=0, | ||
| max_eval_instances=100, |
There was a problem hiding this comment.
Do this in the run entry instead of the run expander.
| {description: "clear:model=qwen/qwen2.5-7b-instruct,model_deployment=huggingface/qwen2.5-7b-instruct-4bit", priority: 1}, | ||
| {description: "clear:model=microsoft/phi-3.5-mini-instruct,model_deployment=huggingface/phi-3.5-mini-instruct-4bit", priority: 1}, | ||
| #Alcohol Dependence | ||
| {description: "clear:condition=alcohol_dependence,model=google/gemini-1.5-pro-001,model_deployment=stanfordhealthcare/gemini-1.5-pro-001", priority: 1}, |
There was a problem hiding this comment.
add max_eval_instances=100, here and below. (if desired)
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Added the full implementation of the CLEAR Dataset (https://www.nature.com/articles/s41746-024-01377-1). Earlier, we had just implemented it for checking "alcohol dependence". Now, it integrates all 13 conditions from the source paper (including bipolar, chronic pain etc.)
@MiguelAFH, @aunell, @HennyJie (tagging just as FYI)
@yifanmai - let me know if any additional changes are needed!