GraphEval: A Lightweight Graph-Based LLM Framework for Idea Evaluation

🌐 Project Page | 📜 arXiv

News

[2025.01.22] 🌟 GraphEval is accepted for ICLR 2025.

📌Preliminary

Environment Setup

# create a new environment
conda create -n grapheval python=3.10
conda activate grapheval

# install pytorch. Modify the command to align with your own CUDA version.
pip3 install torch  --index-url https://download.pytorch.org/whl/cu118

# install related libraries
pip install -r requirements.txt


# install pyg
pip install torch_geometric
pip install pyg_lib torch_scatter torch_sparse torch_cluster torch_spline_conv -f https://data.pyg.org/whl/torch-2.1.0+cu118.html

Dataset Preparation

We have three datasets to choose from, namely ICLR_Dataset, AI_Researcher and Review_Advisor. You can find the dataset files in the Data/[DATASET_NAME] folder.

⭐Experiments

Viewpoint-Graph Extraction

Generate viewpoint nodes and edges.

# DATASET Choices: ICLR_Dataset, AI_Researcher and Review_Advisor
python models/Graph_Extraction.py

Training and Evaluation

Run experiments and print/save evaluation results on metrics accuracy, precision, recall, and F1 Score.

# Method Choices: GraphEval_GNN, GraphEval_LP, Baselines
python run_exp.py --method_name [Method]

Citation

@inproceedings{fenggrapheval,
  title={GraphEval: A Lightweight Graph-Based LLM Framework for Idea Evaluation},
  author={Feng, Tao and Sun, Yihang and You, Jiaxuan},
  booktitle={The Thirteenth International Conference on Learning Representations}
}