README.md

Data

This directory contains data I created earlier in this repository while preparing for my final project, as well as new data created while working on this project.

Structure

• documents.json: The datasets in json format
• documents-with-id1.json to documents-with-id5.json Old data that will not be used later.
• documents-with-id.json: dataset with id
• llm_answer_cosine.csv: Data created to evaluate LLM answers.
• question_vector_pickle, question_context_answer_vector_pickle, answer_vector_pickle, context_vector_pickle: Directories containing pickle files that store vector embeddings of questions, questions + context + answers, answers, and context from the documents.
• llm_answer: This directory contains answers generated by the LLM model.
• evaluations_aqa, evaluations_qa: llm generated evaluation data
• ground_truth_data : Ground truth data created earlier.