notebooks/notebook_sequential_binary.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Sequential binary classification"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Introduction\n",
    "\n",
    "In this notebook, we will perform sequential binary classification analysis using LLMs. Binary classification refers to the task of categorizing data entries into one of two predefined categories. Here, we explore a specific case where binary classifications are conducted sequentially, with the final classification outcome depending on the results of prior classifications.\n",
    "\n",
    "### Dataset Description\n",
    "\n",
    "The dataset used in this notebook is an anonymized subset from the study available at [X]. In the original experiment, children were tasked with reading a reference text and engaging in four sequential interactions with an interactive app. The goal of these steps was to help the children formulate a divergent question. A question is considered divergent if its answer is not explicitly stated in the reference text.\n",
    "\n",
    "The four steps, include:\n",
    "1. **Identify**: The child identifies a knowledge gap related to the reference text.\n",
    "2. **Guess**: The child makes a guess about what the answer to the knowledge gap could be.\n",
    "3. **Seek**: The child formulates a question to seek the answer.\n",
    "4. **Assess**: The child evaluates whether the app provides an answer to their question.\n",
    "\n",
    "This process is called a **cycle**. An annotator evaluates the validity of a cycle by answering a series of binary Yes/No questions (binary classifications). A cycle is deemed valid if all binary questions can be answered by \"Yes\"; otherwise, it is considered invalid. For more details, see the codebook provided in the prompt cell.\n",
    "\n",
    "### Dataset Structure\n",
    "\n",
    "The dataset includes the following key components:\n",
    "- **Reference Text**: A column containing the text that children were asked to read beforehand.\n",
    "- **Identify**: A column containing the entries posed by the children during the Identify step.\n",
    "- **Guess**: A column containing the entries posed by the children during the Guess step.\n",
    "- **Seek**: A column containing the entries posed by the children during the Seek step.\n",
    "- **Assess**: A column containing the entries posed by the children during the Assess step.\n",
    "\n",
    "To classify a cycle, both the reference text and the entries from all four steps are required.\n",
    "\n",
    "Additionally, the dataset includes ratings from two human annotators. These ratings enable us to compute inter-annotator agreement metrics, such as Cohen's kappa, to assess the reliability of the annotations once the analysis is complete."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Import the libraries"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import os\n",
    "import pandas as pd\n",
    "from qualitative_analysis import (\n",
    "    load_data,\n",
    "    clean_and_normalize,\n",
    "    sanitize_dataframe,\n",
    "    get_llm_client,\n",
    "    compute_all_kappas,\n",
    "    plot_confusion_matrices,\n",
    "    extract_global_validity,\n",
    "    process_verbatims_for_binary_criteria,\n",
    ")\n",
    "import qualitative_analysis.config as config"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Load the data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>ID</th>\n",
       "      <th>reference</th>\n",
       "      <th>Identify</th>\n",
       "      <th>Guess</th>\n",
       "      <th>Seek</th>\n",
       "      <th>Assess</th>\n",
       "      <th>corr_cycle1</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>BC23</td>\n",
       "      <td>À des dizaines de kilomètres sous nos pieds, l...</td>\n",
       "      <td>Le manteau,le centre de la Terre</td>\n",
       "      <td>Je pense que le manteau et le centre de la Ter...</td>\n",
       "      <td>Le manteau et le centre de la Terre ne font il...</td>\n",
       "      <td>Je n'ai pas pu trouver la réponse à  ma question.</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>BC14</td>\n",
       "      <td>À des dizaines de kilomètres sous nos pieds, l...</td>\n",
       "      <td>Température du magma</td>\n",
       "      <td>La température du magma s'élève à  plus de 100...</td>\n",
       "      <td>à combien de  °cle magma est?</td>\n",
       "      <td>La température du magma atteint les 1000 °C</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>BC5</td>\n",
       "      <td>À des dizaines de kilomètres sous nos pieds, l...</td>\n",
       "      <td>Pourqoi on l'appelle le manteau.</td>\n",
       "      <td>le magma et une pierre.</td>\n",
       "      <td>quel et la temperature magma</td>\n",
       "      <td>1 000c</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>BC22</td>\n",
       "      <td>À des dizaines de kilomètres sous nos pieds, l...</td>\n",
       "      <td>La température du magma</td>\n",
       "      <td>La température du magma dépasse les 500 °C</td>\n",
       "      <td>Quel est la température du magma ?</td>\n",
       "      <td>La température du magma atteint les 1000 °C</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>BC20</td>\n",
       "      <td>À des dizaines de kilomètres sous nos pieds, l...</td>\n",
       "      <td>Combien de couches y-t-il ?</td>\n",
       "      <td>Il existe d'autre couches dans la terre à  par...</td>\n",
       "      <td>Quelles sont les autres couches et combien son...</td>\n",
       "      <td>La terre contient 7 couches  dont le noyau.</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "     ID                                          reference  \\\n",
       "0  BC23  À des dizaines de kilomètres sous nos pieds, l...   \n",
       "1  BC14  À des dizaines de kilomètres sous nos pieds, l...   \n",
       "2   BC5  À des dizaines de kilomètres sous nos pieds, l...   \n",
       "3  BC22  À des dizaines de kilomètres sous nos pieds, l...   \n",
       "4  BC20  À des dizaines de kilomètres sous nos pieds, l...   \n",
       "\n",
       "                           Identify  \\\n",
       "0  Le manteau,le centre de la Terre   \n",
       "1              Température du magma   \n",
       "2  Pourqoi on l'appelle le manteau.   \n",
       "3           La température du magma   \n",
       "4       Combien de couches y-t-il ?   \n",
       "\n",
       "                                               Guess  \\\n",
       "0  Je pense que le manteau et le centre de la Ter...   \n",
       "1  La température du magma s'élève à  plus de 100...   \n",
       "2                            le magma et une pierre.   \n",
       "3         La température du magma dépasse les 500 °C   \n",
       "4  Il existe d'autre couches dans la terre à  par...   \n",
       "\n",
       "                                                Seek  \\\n",
       "0  Le manteau et le centre de la Terre ne font il...   \n",
       "1                      à combien de  °cle magma est?   \n",
       "2                       quel et la temperature magma   \n",
       "3                 Quel est la température du magma ?   \n",
       "4  Quelles sont les autres couches et combien son...   \n",
       "\n",
       "                                              Assess  corr_cycle1  \n",
       "0  Je n'ai pas pu trouver la réponse à  ma question.            1  \n",
       "1        La température du magma atteint les 1000 °C            1  \n",
       "2                                             1 000c            0  \n",
       "3        La température du magma atteint les 1000 °C            1  \n",
       "4        La terre contient 7 couches  dont le noyau.            1  "
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Define data directory\n",
    "data_dir = 'data'\n",
    "os.makedirs(data_dir, exist_ok=True)\n",
    "\n",
    "# Define the path to your dataset\n",
    "data_file_path = os.path.join(data_dir, 'sequential_binary_sample.csv')\n",
    "\n",
    "# Load the data\n",
    "data = load_data(data_file_path, file_type='csv', delimiter=';')\n",
    "\n",
    "# Preview the data\n",
    "data.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Data Preprocessing"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "# 1a) Define a mapping from old column names to new names\n",
    "# rename_map = {\n",
    "#     \"ref\": \"reference\",\n",
    "#     \"txt1.ctrl1\": \"Identify\",\n",
    "#     \"txt1.det\": \"Guess\",\n",
    "#     \"txt1.exp\": \"Seek\",\n",
    "#     \"txt1.ctrl2\": \"Assess\"\n",
    "# }\n",
    "\n",
    "# # 1b) Rename the columns in the DataFrame\n",
    "# data = data.rename(columns=rename_map)\n",
    "\n",
    "# 2) Now define the new column names for cleaning\n",
    "text_columns = [\"reference\", \"Identify\", \"Guess\", \"Seek\", \"Assess\"]\n",
    "\n",
    "# 3) Clean and normalize the new columns\n",
    "for col in text_columns:\n",
    "    data[col] = clean_and_normalize(data[col])\n",
    "\n",
    "# 4) Sanitize the DataFrame\n",
    "data = sanitize_dataframe(data)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Combine Texts and Questions"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Total number of verbatims: 45\n",
      "Verbatim example:\n",
      "Id: BC23\n",
      "\n",
      "Text: A des dizaines de kilometres sous nos pieds, la terre contient une couche quon appelle le manteau. Il fait tellement chaud sur cette couche que les roches deviennent liquides : cest ce qu'on appelle le magma. Le magma est responsable de l'explosion des volcans: c'est ce que les scientifiques appellent aussi une eruption d'un volcan. On dit quun volcan est endormi sil ny a eu aucune eruption dans les 10 000 dernieres annees. Au dela de 10 000 ans, on peut dire que le volcan est eteint.\n",
      "\n",
      "Identify: Le manteau,le centre de la Terre\n",
      "\n",
      "Guess: Je pense que le manteau et le centre de la Terre sont les maames choses\n",
      "\n",
      "Seek: Le manteau et le centre de la Terre ne font ils qu'un?\n",
      "\n",
      "Assess: Je n'ai pas pu trouver la reponse a  ma question.\n",
      "\n",
      "\n"
     ]
    }
   ],
   "source": [
    "# Combine texts and entries\n",
    "\n",
    "data['verbatim'] = data.apply(\n",
    "    lambda row: (\n",
    "        f\"Id: {row['ID']}\\n\\n\"\n",
    "        f\"Text: {row['reference']}\\n\\n\"\n",
    "        f\"Identify: {row['Identify']}\\n\\n\"\n",
    "        f\"Guess: {row['Guess']}\\n\\n\"\n",
    "        f\"Seek: {row['Seek']}\\n\\n\"\n",
    "        f\"Assess: {row['Assess']}\\n\\n\"\n",
    "    ),\n",
    "    axis=1\n",
    ")\n",
    "\n",
    "# Extract the list of verbatims\n",
    "verbatims = data['verbatim'].tolist()\n",
    "\n",
    "print(f\"Total number of verbatims: {len(verbatims)}\")\n",
    "print(f\"Verbatim example:\\n{verbatims[0]}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Choose the LLM"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Choose the provider and model\n",
    "provider = 'openai'\n",
    "model_name = 'gpt-4o'\n",
    "\n",
    "# Initialize the client\n",
    "llm_client = get_llm_client(provider=provider, config=config.MODEL_CONFIG[provider])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Prompt Construction and Classification"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "=== Processing Verbatim 1/4 ===\n",
      "\n",
      "--- Evaluating Theme: Identify Step ---\n",
      "Prompt:\n",
      "\n",
      "You are an evaluator provided with data entries in the following format:\n",
      "ID: Unique identifiant of the participant\n",
      "Text: The reference text that participants must read beforehand. Their responses for the different steps must be semantically related to this text (same topic), but the answer to the question they are asking should not be found in the text.\n",
      "Identify: Response for the IDENTIFY step\n",
      "Guess: Response for the GUESS step\n",
      "Seek: Response for the SEEK step\n",
      "Assess: Response for the ASSESS step\n",
      "\n",
      "You are tasked with evaluating the following entry.\n",
      "\n",
      "Entry:\n",
      "Id: BC23\n",
      "\n",
      "Text: A des dizaines de kilometres sous nos pieds, la terre contient une couche quon appelle le manteau. Il fait tellement chaud sur cette couche que les roches deviennent liquides : cest ce qu'on appelle le magma. Le magma est responsable de l'explosion des volcans: c'est ce que les scientifiques appellent aussi une eruption d'un volcan. On dit quun volcan est endormi sil ny a eu aucune eruption dans les 10 000 dernieres annees. Au dela de 10 000 ans, on peut dire que le volcan est eteint.\n",
      "\n",
      "Identify: Le manteau,le centre de la Terre\n",
      "\n",
      "Guess: Je pense que le manteau et le centre de la Terre sont les maames choses\n",
      "\n",
      "Seek: Le manteau et le centre de la Terre ne font ils qu'un?\n",
      "\n",
      "Assess: Je n'ai pas pu trouver la reponse a  ma question.\n",
      "\n",
      "\n",
      "\n",
      "You should evaluate it like this:\n",
      "Does the Identify step indicate a topic of interest?\n",
      "Minor errors in spelling, grammar or wording should not be penalised as long as the intention of the entry is clear and meets the inclusion criteria. Focus on the content and purpose of the entry rather than on linguistic perfection.\n",
      "\n",
      "\n",
      "Reply with '1' if the entry meets the criterion or '0' otherwise.\n",
      "\n",
      "\n",
      "=== LLM Response ===\n",
      "1\n",
      "\n",
      "\n",
      "--- Evaluating Theme: Guess Step ---\n",
      "Prompt:\n",
      "\n",
      "You are an evaluator provided with data entries in the following format:\n",
      "ID: Unique identifiant of the participant\n",
      "Text: The reference text that participants must read beforehand. Their responses for the different steps must be semantically related to this text (same topic), but the answer to the question they are asking should not be found in the text.\n",
      "Identify: Response for the IDENTIFY step\n",
      "Guess: Response for the GUESS step\n",
      "Seek: Response for the SEEK step\n",
      "Assess: Response for the ASSESS step\n",
      "\n",
      "You are tasked with evaluating the following entry.\n",
      "\n",
      "Entry:\n",
      "Id: BC23\n",
      "\n",
      "Text: A des dizaines de kilometres sous nos pieds, la terre contient une couche quon appelle le manteau. Il fait tellement chaud sur cette couche que les roches deviennent liquides : cest ce qu'on appelle le magma. Le magma est responsable de l'explosion des volcans: c'est ce que les scientifiques appellent aussi une eruption d'un volcan. On dit quun volcan est endormi sil ny a eu aucune eruption dans les 10 000 dernieres annees. Au dela de 10 000 ans, on peut dire que le volcan est eteint.\n",
      "\n",
      "Identify: Le manteau,le centre de la Terre\n",
      "\n",
      "Guess: Je pense que le manteau et le centre de la Terre sont les maames choses\n",
      "\n",
      "Seek: Le manteau et le centre de la Terre ne font ils qu'un?\n",
      "\n",
      "Assess: Je n'ai pas pu trouver la reponse a  ma question.\n",
      "\n",
      "\n",
      "\n",
      "You should evaluate it like this:\n",
      "Does the Guess step suggest a possible explanation?\n",
      "Minor errors in spelling, grammar or wording should not be penalised as long as the intention of the entry is clear and meets the inclusion criteria. Focus on the content and purpose of the entry rather than on linguistic perfection.\n",
      "\n",
      "\n",
      "Reply with '1' if the entry meets the criterion or '0' otherwise.\n",
      "\n",
      "\n",
      "=== LLM Response ===\n",
      "1\n",
      "\n",
      "\n",
      "--- Evaluating Theme: Seek Step ---\n",
      "Prompt:\n",
      "\n",
      "You are an evaluator provided with data entries in the following format:\n",
      "ID: Unique identifiant of the participant\n",
      "Text: The reference text that participants must read beforehand. Their responses for the different steps must be semantically related to this text (same topic), but the answer to the question they are asking should not be found in the text.\n",
      "Identify: Response for the IDENTIFY step\n",
      "Guess: Response for the GUESS step\n",
      "Seek: Response for the SEEK step\n",
      "Assess: Response for the ASSESS step\n",
      "\n",
      "You are tasked with evaluating the following entry.\n",
      "\n",
      "Entry:\n",
      "Id: BC23\n",
      "\n",
      "Text: A des dizaines de kilometres sous nos pieds, la terre contient une couche quon appelle le manteau. Il fait tellement chaud sur cette couche que les roches deviennent liquides : cest ce qu'on appelle le magma. Le magma est responsable de l'explosion des volcans: c'est ce que les scientifiques appellent aussi une eruption d'un volcan. On dit quun volcan est endormi sil ny a eu aucune eruption dans les 10 000 dernieres annees. Au dela de 10 000 ans, on peut dire que le volcan est eteint.\n",
      "\n",
      "Identify: Le manteau,le centre de la Terre\n",
      "\n",
      "Guess: Je pense que le manteau et le centre de la Terre sont les maames choses\n",
      "\n",
      "Seek: Le manteau et le centre de la Terre ne font ils qu'un?\n",
      "\n",
      "Assess: Je n'ai pas pu trouver la reponse a  ma question.\n",
      "\n",
      "\n",
      "\n",
      "You should evaluate it like this:\n",
      "Is the Seek step formulated as a question?\n",
      "Minor errors in spelling, grammar or wording should not be penalised as long as the intention of the entry is clear and meets the inclusion criteria. Focus on the content and purpose of the entry rather than on linguistic perfection.\n",
      "\n",
      "\n",
      "Reply with '1' if the entry meets the criterion or '0' otherwise.\n",
      "\n",
      "\n",
      "=== LLM Response ===\n",
      "1\n",
      "\n",
      "\n",
      "--- Evaluating Theme: Assess Step ---\n",
      "Prompt:\n",
      "\n",
      "You are an evaluator provided with data entries in the following format:\n",
      "ID: Unique identifiant of the participant\n",
      "Text: The reference text that participants must read beforehand. Their responses for the different steps must be semantically related to this text (same topic), but the answer to the question they are asking should not be found in the text.\n",
      "Identify: Response for the IDENTIFY step\n",
      "Guess: Response for the GUESS step\n",
      "Seek: Response for the SEEK step\n",
      "Assess: Response for the ASSESS step\n",
      "\n",
      "You are tasked with evaluating the following entry.\n",
      "\n",
      "Entry:\n",
      "Id: BC23\n",
      "\n",
      "Text: A des dizaines de kilometres sous nos pieds, la terre contient une couche quon appelle le manteau. Il fait tellement chaud sur cette couche que les roches deviennent liquides : cest ce qu'on appelle le magma. Le magma est responsable de l'explosion des volcans: c'est ce que les scientifiques appellent aussi une eruption d'un volcan. On dit quun volcan est endormi sil ny a eu aucune eruption dans les 10 000 dernieres annees. Au dela de 10 000 ans, on peut dire que le volcan est eteint.\n",
      "\n",
      "Identify: Le manteau,le centre de la Terre\n",
      "\n",
      "Guess: Je pense que le manteau et le centre de la Terre sont les maames choses\n",
      "\n",
      "Seek: Le manteau et le centre de la Terre ne font ils qu'un?\n",
      "\n",
      "Assess: Je n'ai pas pu trouver la reponse a  ma question.\n",
      "\n",
      "\n",
      "\n",
      "You should evaluate it like this:\n",
      "Does it identify a possible answer or state that no answer was found ('no' is ok)?\n",
      "Minor errors in spelling, grammar or wording should not be penalised as long as the intention of the entry is clear and meets the inclusion criteria. Focus on the content and purpose of the entry rather than on linguistic perfection.\n",
      "\n",
      "\n",
      "Reply with '1' if the entry meets the criterion or '0' otherwise.\n",
      "\n",
      "\n",
      "=== LLM Response ===\n",
      "1\n",
      "\n",
      "\n",
      "--- Evaluating Theme: Consistency ---\n",
      "Prompt:\n",
      "\n",
      "You are an evaluator provided with data entries in the following format:\n",
      "ID: Unique identifiant of the participant\n",
      "Text: The reference text that participants must read beforehand. Their responses for the different steps must be semantically related to this text (same topic), but the answer to the question they are asking should not be found in the text.\n",
      "Identify: Response for the IDENTIFY step\n",
      "Guess: Response for the GUESS step\n",
      "Seek: Response for the SEEK step\n",
      "Assess: Response for the ASSESS step\n",
      "\n",
      "You are tasked with evaluating the following entry.\n",
      "\n",
      "Entry:\n",
      "Id: BC23\n",
      "\n",
      "Text: A des dizaines de kilometres sous nos pieds, la terre contient une couche quon appelle le manteau. Il fait tellement chaud sur cette couche que les roches deviennent liquides : cest ce qu'on appelle le magma. Le magma est responsable de l'explosion des volcans: c'est ce que les scientifiques appellent aussi une eruption d'un volcan. On dit quun volcan est endormi sil ny a eu aucune eruption dans les 10 000 dernieres annees. Au dela de 10 000 ans, on peut dire que le volcan est eteint.\n",
      "\n",
      "Identify: Le manteau,le centre de la Terre\n",
      "\n",
      "Guess: Je pense que le manteau et le centre de la Terre sont les maames choses\n",
      "\n",
      "Seek: Le manteau et le centre de la Terre ne font ils qu'un?\n",
      "\n",
      "Assess: Je n'ai pas pu trouver la reponse a  ma question.\n",
      "\n",
      "\n",
      "\n",
      "You should evaluate it like this:\n",
      "Are the Identify, Guess, and Seek steps related to the same question?\n",
      "Minor errors in spelling, grammar or wording should not be penalised as long as the intention of the entry is clear and meets the inclusion criteria. Focus on the content and purpose of the entry rather than on linguistic perfection.\n",
      "\n",
      "\n",
      "Reply with '1' if the entry meets the criterion or '0' otherwise.\n",
      "\n",
      "\n",
      "=== LLM Response ===\n",
      "1\n",
      "\n",
      "\n",
      "--- Evaluating Theme: Reference Link ---\n",
      "Prompt:\n",
      "\n",
      "You are an evaluator provided with data entries in the following format:\n",
      "ID: Unique identifiant of the participant\n",
      "Text: The reference text that participants must read beforehand. Their responses for the different steps must be semantically related to this text (same topic), but the answer to the question they are asking should not be found in the text.\n",
      "Identify: Response for the IDENTIFY step\n",
      "Guess: Response for the GUESS step\n",
      "Seek: Response for the SEEK step\n",
      "Assess: Response for the ASSESS step\n",
      "\n",
      "You are tasked with evaluating the following entry.\n",
      "\n",
      "Entry:\n",
      "Id: BC23\n",
      "\n",
      "Text: A des dizaines de kilometres sous nos pieds, la terre contient une couche quon appelle le manteau. Il fait tellement chaud sur cette couche que les roches deviennent liquides : cest ce qu'on appelle le magma. Le magma est responsable de l'explosion des volcans: c'est ce que les scientifiques appellent aussi une eruption d'un volcan. On dit quun volcan est endormi sil ny a eu aucune eruption dans les 10 000 dernieres annees. Au dela de 10 000 ans, on peut dire que le volcan est eteint.\n",
      "\n",
      "Identify: Le manteau,le centre de la Terre\n",
      "\n",
      "Guess: Je pense que le manteau et le centre de la Terre sont les maames choses\n",
      "\n",
      "Seek: Le manteau et le centre de la Terre ne font ils qu'un?\n",
      "\n",
      "Assess: Je n'ai pas pu trouver la reponse a  ma question.\n",
      "\n",
      "\n",
      "\n",
      "You should evaluate it like this:\n",
      "Are the Identify, Guess, and Seek steps related to the topic of the reference text?\n",
      "Minor errors in spelling, grammar or wording should not be penalised as long as the intention of the entry is clear and meets the inclusion criteria. Focus on the content and purpose of the entry rather than on linguistic perfection.\n",
      "\n",
      "\n",
      "Reply with '1' if the entry meets the criterion or '0' otherwise.\n",
      "\n",
      "\n",
      "=== LLM Response ===\n",
      "1\n",
      "\n",
      "\n",
      "--- Evaluating Theme: Seek Question Originality ---\n",
      "Prompt:\n",
      "\n",
      "You are an evaluator provided with data entries in the following format:\n",
      "ID: Unique identifiant of the participant\n",
      "Text: The reference text that participants must read beforehand. Their responses for the different steps must be semantically related to this text (same topic), but the answer to the question they are asking should not be found in the text.\n",
      "Identify: Response for the IDENTIFY step\n",
      "Guess: Response for the GUESS step\n",
      "Seek: Response for the SEEK step\n",
      "Assess: Response for the ASSESS step\n",
      "\n",
      "You are tasked with evaluating the following entry.\n",
      "\n",
      "Entry:\n",
      "Id: BC23\n",
      "\n",
      "Text: A des dizaines de kilometres sous nos pieds, la terre contient une couche quon appelle le manteau. Il fait tellement chaud sur cette couche que les roches deviennent liquides : cest ce qu'on appelle le magma. Le magma est responsable de l'explosion des volcans: c'est ce que les scientifiques appellent aussi une eruption d'un volcan. On dit quun volcan est endormi sil ny a eu aucune eruption dans les 10 000 dernieres annees. Au dela de 10 000 ans, on peut dire que le volcan est eteint.\n",
      "\n",
      "Identify: Le manteau,le centre de la Terre\n",
      "\n",
      "Guess: Je pense que le manteau et le centre de la Terre sont les maames choses\n",
      "\n",
      "Seek: Le manteau et le centre de la Terre ne font ils qu'un?\n",
      "\n",
      "Assess: Je n'ai pas pu trouver la reponse a  ma question.\n",
      "\n",
      "\n",
      "\n",
      "You should evaluate it like this:\n",
      "Is the answer to the Seek question not found (even vaguely) in the reference text?\n",
      "Minor errors in spelling, grammar or wording should not be penalised as long as the intention of the entry is clear and meets the inclusion criteria. Focus on the content and purpose of the entry rather than on linguistic perfection.\n",
      "\n",
      "\n",
      "Reply with '1' if the entry meets the criterion or '0' otherwise.\n",
      "\n",
      "\n",
      "=== LLM Response ===\n",
      "1\n",
      "\n",
      "\n",
      "=== Processing Verbatim 2/4 ===\n",
      "\n",
      "--- Evaluating Theme: Identify Step ---\n",
      "Prompt:\n",
      "\n",
      "You are an evaluator provided with data entries in the following format:\n",
      "ID: Unique identifiant of the participant\n",
      "Text: The reference text that participants must read beforehand. Their responses for the different steps must be semantically related to this text (same topic), but the answer to the question they are asking should not be found in the text.\n",
      "Identify: Response for the IDENTIFY step\n",
      "Guess: Response for the GUESS step\n",
      "Seek: Response for the SEEK step\n",
      "Assess: Response for the ASSESS step\n",
      "\n",
      "You are tasked with evaluating the following entry.\n",
      "\n",
      "Entry:\n",
      "Id: BC14\n",
      "\n",
      "Text: A des dizaines de kilometres sous nos pieds, la terre contient une couche quon appelle le manteau. Il fait tellement chaud sur cette couche que les roches deviennent liquides : cest ce qu'on appelle le magma. Le magma est responsable de l'explosion des volcans: c'est ce que les scientifiques appellent aussi une eruption d'un volcan. On dit quun volcan est endormi sil ny a eu aucune eruption dans les 10 000 dernieres annees. Au dela de 10 000 ans, on peut dire que le volcan est eteint.\n",
      "\n",
      "Identify: Temperature du magma\n",
      "\n",
      "Guess: La temperature du magma s'eleve a  plus de 1000 c\n",
      "\n",
      "Seek: a combien de  cle magma est?\n",
      "\n",
      "Assess: La temperature du magma atteint les 1000 C\n",
      "\n",
      "\n",
      "\n",
      "You should evaluate it like this:\n",
      "Does the Identify step indicate a topic of interest?\n",
      "Minor errors in spelling, grammar or wording should not be penalised as long as the intention of the entry is clear and meets the inclusion criteria. Focus on the content and purpose of the entry rather than on linguistic perfection.\n",
      "\n",
      "\n",
      "Reply with '1' if the entry meets the criterion or '0' otherwise.\n",
      "\n",
      "\n",
      "=== LLM Response ===\n",
      "1\n",
      "\n",
      "\n",
      "--- Evaluating Theme: Guess Step ---\n",
      "Prompt:\n",
      "\n",
      "You are an evaluator provided with data entries in the following format:\n",
      "ID: Unique identifiant of the participant\n",
      "Text: The reference text that participants must read beforehand. Their responses for the different steps must be semantically related to this text (same topic), but the answer to the question they are asking should not be found in the text.\n",
      "Identify: Response for the IDENTIFY step\n",
      "Guess: Response for the GUESS step\n",
      "Seek: Response for the SEEK step\n",
      "Assess: Response for the ASSESS step\n",
      "\n",
      "You are tasked with evaluating the following entry.\n",
      "\n",
      "Entry:\n",
      "Id: BC14\n",
      "\n",
      "Text: A des dizaines de kilometres sous nos pieds, la terre contient une couche quon appelle le manteau. Il fait tellement chaud sur cette couche que les roches deviennent liquides : cest ce qu'on appelle le magma. Le magma est responsable de l'explosion des volcans: c'est ce que les scientifiques appellent aussi une eruption d'un volcan. On dit quun volcan est endormi sil ny a eu aucune eruption dans les 10 000 dernieres annees. Au dela de 10 000 ans, on peut dire que le volcan est eteint.\n",
      "\n",
      "Identify: Temperature du magma\n",
      "\n",
      "Guess: La temperature du magma s'eleve a  plus de 1000 c\n",
      "\n",
      "Seek: a combien de  cle magma est?\n",
      "\n",
      "Assess: La temperature du magma atteint les 1000 C\n",
      "\n",
      "\n",
      "\n",
      "You should evaluate it like this:\n",
      "Does the Guess step suggest a possible explanation?\n",
      "Minor errors in spelling, grammar or wording should not be penalised as long as the intention of the entry is clear and meets the inclusion criteria. Focus on the content and purpose of the entry rather than on linguistic perfection.\n",
      "\n",
      "\n",
      "Reply with '1' if the entry meets the criterion or '0' otherwise.\n",
      "\n",
      "\n",
      "=== LLM Response ===\n",
      "1\n",
      "\n",
      "\n",
      "--- Evaluating Theme: Seek Step ---\n",
      "Prompt:\n",
      "\n",
      "You are an evaluator provided with data entries in the following format:\n",
      "ID: Unique identifiant of the participant\n",
      "Text: The reference text that participants must read beforehand. Their responses for the different steps must be semantically related to this text (same topic), but the answer to the question they are asking should not be found in the text.\n",
      "Identify: Response for the IDENTIFY step\n",
      "Guess: Response for the GUESS step\n",
      "Seek: Response for the SEEK step\n",
      "Assess: Response for the ASSESS step\n",
      "\n",
      "You are tasked with evaluating the following entry.\n",
      "\n",
      "Entry:\n",
      "Id: BC14\n",
      "\n",
      "Text: A des dizaines de kilometres sous nos pieds, la terre contient une couche quon appelle le manteau. Il fait tellement chaud sur cette couche que les roches deviennent liquides : cest ce qu'on appelle le magma. Le magma est responsable de l'explosion des volcans: c'est ce que les scientifiques appellent aussi une eruption d'un volcan. On dit quun volcan est endormi sil ny a eu aucune eruption dans les 10 000 dernieres annees. Au dela de 10 000 ans, on peut dire que le volcan est eteint.\n",
      "\n",
      "Identify: Temperature du magma\n",
      "\n",
      "Guess: La temperature du magma s'eleve a  plus de 1000 c\n",
      "\n",
      "Seek: a combien de  cle magma est?\n",
      "\n",
      "Assess: La temperature du magma atteint les 1000 C\n",
      "\n",
      "\n",
      "\n",
      "You should evaluate it like this:\n",
      "Is the Seek step formulated as a question?\n",
      "Minor errors in spelling, grammar or wording should not be penalised as long as the intention of the entry is clear and meets the inclusion criteria. Focus on the content and purpose of the entry rather than on linguistic perfection.\n",
      "\n",
      "\n",
      "Reply with '1' if the entry meets the criterion or '0' otherwise.\n",
      "\n",
      "\n",
      "=== LLM Response ===\n",
      "0\n",
      "\n",
      "\n",
      "--- Evaluating Theme: Assess Step ---\n",
      "Prompt:\n",
      "\n",
      "You are an evaluator provided with data entries in the following format:\n",
      "ID: Unique identifiant of the participant\n",
      "Text: The reference text that participants must read beforehand. Their responses for the different steps must be semantically related to this text (same topic), but the answer to the question they are asking should not be found in the text.\n",
      "Identify: Response for the IDENTIFY step\n",
      "Guess: Response for the GUESS step\n",
      "Seek: Response for the SEEK step\n",
      "Assess: Response for the ASSESS step\n",
      "\n",
      "You are tasked with evaluating the following entry.\n",
      "\n",
      "Entry:\n",
      "Id: BC14\n",
      "\n",
      "Text: A des dizaines de kilometres sous nos pieds, la terre contient une couche quon appelle le manteau. Il fait tellement chaud sur cette couche que les roches deviennent liquides : cest ce qu'on appelle le magma. Le magma est responsable de l'explosion des volcans: c'est ce que les scientifiques appellent aussi une eruption d'un volcan. On dit quun volcan est endormi sil ny a eu aucune eruption dans les 10 000 dernieres annees. Au dela de 10 000 ans, on peut dire que le volcan est eteint.\n",
      "\n",
      "Identify: Temperature du magma\n",
      "\n",
      "Guess: La temperature du magma s'eleve a  plus de 1000 c\n",
      "\n",
      "Seek: a combien de  cle magma est?\n",
      "\n",
      "Assess: La temperature du magma atteint les 1000 C\n",
      "\n",
      "\n",
      "\n",
      "You should evaluate it like this:\n",
      "Does it identify a possible answer or state that no answer was found ('no' is ok)?\n",
      "Minor errors in spelling, grammar or wording should not be penalised as long as the intention of the entry is clear and meets the inclusion criteria. Focus on the content and purpose of the entry rather than on linguistic perfection.\n",
      "\n",
      "\n",
      "Reply with '1' if the entry meets the criterion or '0' otherwise.\n",
      "\n",
      "\n",
      "=== LLM Response ===\n",
      "1\n",
      "\n",
      "\n",
      "--- Evaluating Theme: Consistency ---\n",
      "Prompt:\n",
      "\n",
      "You are an evaluator provided with data entries in the following format:\n",
      "ID: Unique identifiant of the participant\n",
      "Text: The reference text that participants must read beforehand. Their responses for the different steps must be semantically related to this text (same topic), but the answer to the question they are asking should not be found in the text.\n",
      "Identify: Response for the IDENTIFY step\n",
      "Guess: Response for the GUESS step\n",
      "Seek: Response for the SEEK step\n",
      "Assess: Response for the ASSESS step\n",
      "\n",
      "You are tasked with evaluating the following entry.\n",
      "\n",
      "Entry:\n",
      "Id: BC14\n",
      "\n",
      "Text: A des dizaines de kilometres sous nos pieds, la terre contient une couche quon appelle le manteau. Il fait tellement chaud sur cette couche que les roches deviennent liquides : cest ce qu'on appelle le magma. Le magma est responsable de l'explosion des volcans: c'est ce que les scientifiques appellent aussi une eruption d'un volcan. On dit quun volcan est endormi sil ny a eu aucune eruption dans les 10 000 dernieres annees. Au dela de 10 000 ans, on peut dire que le volcan est eteint.\n",
      "\n",
      "Identify: Temperature du magma\n",
      "\n",
      "Guess: La temperature du magma s'eleve a  plus de 1000 c\n",
      "\n",
      "Seek: a combien de  cle magma est?\n",
      "\n",
      "Assess: La temperature du magma atteint les 1000 C\n",
      "\n",
      "\n",
      "\n",
      "You should evaluate it like this:\n",
      "Are the Identify, Guess, and Seek steps related to the same question?\n",
      "Minor errors in spelling, grammar or wording should not be penalised as long as the intention of the entry is clear and meets the inclusion criteria. Focus on the content and purpose of the entry rather than on linguistic perfection.\n",
      "\n",
      "\n",
      "Reply with '1' if the entry meets the criterion or '0' otherwise.\n",
      "\n",
      "\n",
      "=== LLM Response ===\n",
      "1\n",
      "\n",
      "\n",
      "--- Evaluating Theme: Reference Link ---\n",
      "Prompt:\n",
      "\n",
      "You are an evaluator provided with data entries in the following format:\n",
      "ID: Unique identifiant of the participant\n",
      "Text: The reference text that participants must read beforehand. Their responses for the different steps must be semantically related to this text (same topic), but the answer to the question they are asking should not be found in the text.\n",
      "Identify: Response for the IDENTIFY step\n",
      "Guess: Response for the GUESS step\n",
      "Seek: Response for the SEEK step\n",
      "Assess: Response for the ASSESS step\n",
      "\n",
      "You are tasked with evaluating the following entry.\n",
      "\n",
      "Entry:\n",
      "Id: BC14\n",
      "\n",
      "Text: A des dizaines de kilometres sous nos pieds, la terre contient une couche quon appelle le manteau. Il fait tellement chaud sur cette couche que les roches deviennent liquides : cest ce qu'on appelle le magma. Le magma est responsable de l'explosion des volcans: c'est ce que les scientifiques appellent aussi une eruption d'un volcan. On dit quun volcan est endormi sil ny a eu aucune eruption dans les 10 000 dernieres annees. Au dela de 10 000 ans, on peut dire que le volcan est eteint.\n",
      "\n",
      "Identify: Temperature du magma\n",
      "\n",
      "Guess: La temperature du magma s'eleve a  plus de 1000 c\n",
      "\n",
      "Seek: a combien de  cle magma est?\n",
      "\n",
      "Assess: La temperature du magma atteint les 1000 C\n",
      "\n",
      "\n",
      "\n",
      "You should evaluate it like this:\n",
      "Are the Identify, Guess, and Seek steps related to the topic of the reference text?\n",
      "Minor errors in spelling, grammar or wording should not be penalised as long as the intention of the entry is clear and meets the inclusion criteria. Focus on the content and purpose of the entry rather than on linguistic perfection.\n",
      "\n",
      "\n",
      "Reply with '1' if the entry meets the criterion or '0' otherwise.\n",
      "\n",
      "\n",
      "=== LLM Response ===\n",
      "1\n",
      "\n",
      "\n",
      "--- Evaluating Theme: Seek Question Originality ---\n",
      "Prompt:\n",
      "\n",
      "You are an evaluator provided with data entries in the following format:\n",
      "ID: Unique identifiant of the participant\n",
      "Text: The reference text that participants must read beforehand. Their responses for the different steps must be semantically related to this text (same topic), but the answer to the question they are asking should not be found in the text.\n",
      "Identify: Response for the IDENTIFY step\n",
      "Guess: Response for the GUESS step\n",
      "Seek: Response for the SEEK step\n",
      "Assess: Response for the ASSESS step\n",
      "\n",
      "You are tasked with evaluating the following entry.\n",
      "\n",
      "Entry:\n",
      "Id: BC14\n",
      "\n",
      "Text: A des dizaines de kilometres sous nos pieds, la terre contient une couche quon appelle le manteau. Il fait tellement chaud sur cette couche que les roches deviennent liquides : cest ce qu'on appelle le magma. Le magma est responsable de l'explosion des volcans: c'est ce que les scientifiques appellent aussi une eruption d'un volcan. On dit quun volcan est endormi sil ny a eu aucune eruption dans les 10 000 dernieres annees. Au dela de 10 000 ans, on peut dire que le volcan est eteint.\n",
      "\n",
      "Identify: Temperature du magma\n",
      "\n",
      "Guess: La temperature du magma s'eleve a  plus de 1000 c\n",
      "\n",
      "Seek: a combien de  cle magma est?\n",
      "\n",
      "Assess: La temperature du magma atteint les 1000 C\n",
      "\n",
      "\n",
      "\n",
      "You should evaluate it like this:\n",
      "Is the answer to the Seek question not found (even vaguely) in the reference text?\n",
      "Minor errors in spelling, grammar or wording should not be penalised as long as the intention of the entry is clear and meets the inclusion criteria. Focus on the content and purpose of the entry rather than on linguistic perfection.\n",
      "\n",
      "\n",
      "Reply with '1' if the entry meets the criterion or '0' otherwise.\n",
      "\n",
      "\n",
      "=== LLM Response ===\n",
      "1\n",
      "\n",
      "\n",
      "=== Processing Verbatim 3/4 ===\n",
      "\n",
      "--- Evaluating Theme: Identify Step ---\n",
      "Prompt:\n",
      "\n",
      "You are an evaluator provided with data entries in the following format:\n",
      "ID: Unique identifiant of the participant\n",
      "Text: The reference text that participants must read beforehand. Their responses for the different steps must be semantically related to this text (same topic), but the answer to the question they are asking should not be found in the text.\n",
      "Identify: Response for the IDENTIFY step\n",
      "Guess: Response for the GUESS step\n",
      "Seek: Response for the SEEK step\n",
      "Assess: Response for the ASSESS step\n",
      "\n",
      "You are tasked with evaluating the following entry.\n",
      "\n",
      "Entry:\n",
      "Id: BC5\n",
      "\n",
      "Text: A des dizaines de kilometres sous nos pieds, la terre contient une couche quon appelle le manteau. Il fait tellement chaud sur cette couche que les roches deviennent liquides : cest ce qu'on appelle le magma. Le magma est responsable de l'explosion des volcans: c'est ce que les scientifiques appellent aussi une eruption d'un volcan. On dit quun volcan est endormi sil ny a eu aucune eruption dans les 10 000 dernieres annees. Au dela de 10 000 ans, on peut dire que le volcan est eteint.\n",
      "\n",
      "Identify: Pourqoi on l'appelle le manteau.\n",
      "\n",
      "Guess: le magma et une pierre.\n",
      "\n",
      "Seek: quel et la temperature magma\n",
      "\n",
      "Assess: 1 000c\n",
      "\n",
      "\n",
      "\n",
      "You should evaluate it like this:\n",
      "Does the Identify step indicate a topic of interest?\n",
      "Minor errors in spelling, grammar or wording should not be penalised as long as the intention of the entry is clear and meets the inclusion criteria. Focus on the content and purpose of the entry rather than on linguistic perfection.\n",
      "\n",
      "\n",
      "Reply with '1' if the entry meets the criterion or '0' otherwise.\n",
      "\n",
      "\n",
      "=== LLM Response ===\n",
      "1\n",
      "\n",
      "\n",
      "--- Evaluating Theme: Guess Step ---\n",
      "Prompt:\n",
      "\n",
      "You are an evaluator provided with data entries in the following format:\n",
      "ID: Unique identifiant of the participant\n",
      "Text: The reference text that participants must read beforehand. Their responses for the different steps must be semantically related to this text (same topic), but the answer to the question they are asking should not be found in the text.\n",
      "Identify: Response for the IDENTIFY step\n",
      "Guess: Response for the GUESS step\n",
      "Seek: Response for the SEEK step\n",
      "Assess: Response for the ASSESS step\n",
      "\n",
      "You are tasked with evaluating the following entry.\n",
      "\n",
      "Entry:\n",
      "Id: BC5\n",
      "\n",
      "Text: A des dizaines de kilometres sous nos pieds, la terre contient une couche quon appelle le manteau. Il fait tellement chaud sur cette couche que les roches deviennent liquides : cest ce qu'on appelle le magma. Le magma est responsable de l'explosion des volcans: c'est ce que les scientifiques appellent aussi une eruption d'un volcan. On dit quun volcan est endormi sil ny a eu aucune eruption dans les 10 000 dernieres annees. Au dela de 10 000 ans, on peut dire que le volcan est eteint.\n",
      "\n",
      "Identify: Pourqoi on l'appelle le manteau.\n",
      "\n",
      "Guess: le magma et une pierre.\n",
      "\n",
      "Seek: quel et la temperature magma\n",
      "\n",
      "Assess: 1 000c\n",
      "\n",
      "\n",
      "\n",
      "You should evaluate it like this:\n",
      "Does the Guess step suggest a possible explanation?\n",
      "Minor errors in spelling, grammar or wording should not be penalised as long as the intention of the entry is clear and meets the inclusion criteria. Focus on the content and purpose of the entry rather than on linguistic perfection.\n",
      "\n",
      "\n",
      "Reply with '1' if the entry meets the criterion or '0' otherwise.\n",
      "\n",
      "\n",
      "=== LLM Response ===\n",
      "To evaluate the entry, we need to determine if the \"Guess\" step suggests a possible explanation related to the topic of the text, which is about the Earth's mantle, magma, and volcanic eruptions.\n",
      "\n",
      "- The \"Identify\" step asks why the mantle is called the mantle.\n",
      "- The \"Guess\" step states \"le magma et une pierre\" (the magma and a stone), which does not suggest a possible explanation for why the mantle is called the mantle. Instead, it seems to be a statement about magma and stone, which is not directly related to the question posed in the \"Identify\" step.\n",
      "- The \"Seek\" step asks about the temperature of magma, which is related to the topic of magma.\n",
      "- The \"Assess\" step provides a temperature, \"1 000c,\" which is a reasonable assessment of magma temperature.\n",
      "\n",
      "However, the key criterion here is whether the \"Guess\" step suggests a possible explanation. Since the \"Guess\" step does not provide an explanation related to the \"Identify\" step's question, the entry does not meet the criterion.\n",
      "\n",
      "Therefore, the entry should be evaluated as '0'.\n",
      "\n",
      "\n",
      "--- Evaluating Theme: Seek Step ---\n",
      "Prompt:\n",
      "\n",
      "You are an evaluator provided with data entries in the following format:\n",
      "ID: Unique identifiant of the participant\n",
      "Text: The reference text that participants must read beforehand. Their responses for the different steps must be semantically related to this text (same topic), but the answer to the question they are asking should not be found in the text.\n",
      "Identify: Response for the IDENTIFY step\n",
      "Guess: Response for the GUESS step\n",
      "Seek: Response for the SEEK step\n",
      "Assess: Response for the ASSESS step\n",
      "\n",
      "You are tasked with evaluating the following entry.\n",
      "\n",
      "Entry:\n",
      "Id: BC5\n",
      "\n",
      "Text: A des dizaines de kilometres sous nos pieds, la terre contient une couche quon appelle le manteau. Il fait tellement chaud sur cette couche que les roches deviennent liquides : cest ce qu'on appelle le magma. Le magma est responsable de l'explosion des volcans: c'est ce que les scientifiques appellent aussi une eruption d'un volcan. On dit quun volcan est endormi sil ny a eu aucune eruption dans les 10 000 dernieres annees. Au dela de 10 000 ans, on peut dire que le volcan est eteint.\n",
      "\n",
      "Identify: Pourqoi on l'appelle le manteau.\n",
      "\n",
      "Guess: le magma et une pierre.\n",
      "\n",
      "Seek: quel et la temperature magma\n",
      "\n",
      "Assess: 1 000c\n",
      "\n",
      "\n",
      "\n",
      "You should evaluate it like this:\n",
      "Is the Seek step formulated as a question?\n",
      "Minor errors in spelling, grammar or wording should not be penalised as long as the intention of the entry is clear and meets the inclusion criteria. Focus on the content and purpose of the entry rather than on linguistic perfection.\n",
      "\n",
      "\n",
      "Reply with '1' if the entry meets the criterion or '0' otherwise.\n",
      "\n",
      "\n",
      "=== LLM Response ===\n",
      "1\n",
      "\n",
      "\n",
      "--- Evaluating Theme: Assess Step ---\n",
      "Prompt:\n",
      "\n",
      "You are an evaluator provided with data entries in the following format:\n",
      "ID: Unique identifiant of the participant\n",
      "Text: The reference text that participants must read beforehand. Their responses for the different steps must be semantically related to this text (same topic), but the answer to the question they are asking should not be found in the text.\n",
      "Identify: Response for the IDENTIFY step\n",
      "Guess: Response for the GUESS step\n",
      "Seek: Response for the SEEK step\n",
      "Assess: Response for the ASSESS step\n",
      "\n",
      "You are tasked with evaluating the following entry.\n",
      "\n",
      "Entry:\n",
      "Id: BC5\n",
      "\n",
      "Text: A des dizaines de kilometres sous nos pieds, la terre contient une couche quon appelle le manteau. Il fait tellement chaud sur cette couche que les roches deviennent liquides : cest ce qu'on appelle le magma. Le magma est responsable de l'explosion des volcans: c'est ce que les scientifiques appellent aussi une eruption d'un volcan. On dit quun volcan est endormi sil ny a eu aucune eruption dans les 10 000 dernieres annees. Au dela de 10 000 ans, on peut dire que le volcan est eteint.\n",
      "\n",
      "Identify: Pourqoi on l'appelle le manteau.\n",
      "\n",
      "Guess: le magma et une pierre.\n",
      "\n",
      "Seek: quel et la temperature magma\n",
      "\n",
      "Assess: 1 000c\n",
      "\n",
      "\n",
      "\n",
      "You should evaluate it like this:\n",
      "Does it identify a possible answer or state that no answer was found ('no' is ok)?\n",
      "Minor errors in spelling, grammar or wording should not be penalised as long as the intention of the entry is clear and meets the inclusion criteria. Focus on the content and purpose of the entry rather than on linguistic perfection.\n",
      "\n",
      "\n",
      "Reply with '1' if the entry meets the criterion or '0' otherwise.\n",
      "\n",
      "\n",
      "=== LLM Response ===\n",
      "To evaluate the entry, we need to check if each step is semantically related to the reference text and if the answer to the question asked in the \"Identify\" step is not found in the text.\n",
      "\n",
      "1. **Identify**: \"Pourqoi on l'appelle le manteau.\" - This question is related to the text as it asks about the term \"manteau,\" which is mentioned in the text. The text does not provide an explanation for why it is called \"le manteau,\" so this is a valid question.\n",
      "\n",
      "2. **Guess**: \"le magma et une pierre.\" - This guess is related to the topic of the text, which discusses magma and rocks. Although the guess is not very clear, it is related to the topic.\n",
      "\n",
      "3. **Seek**: \"quel et la temperature magma\" - This question is related to the text, which discusses magma. The text does not specify the temperature of magma, so this is a valid question to seek further information.\n",
      "\n",
      "4. **Assess**: \"1 000c\" - This assessment provides a possible temperature for magma, which is a reasonable follow-up to the \"Seek\" step.\n",
      "\n",
      "Each step is semantically related to the text, and the questions asked in the \"Identify\" and \"Seek\" steps are not directly answered by the text. Therefore, the entry meets the criterion.\n",
      "\n",
      "The entry should be evaluated as '1'.\n",
      "\n",
      "\n",
      "--- Evaluating Theme: Consistency ---\n",
      "Prompt:\n",
      "\n",
      "You are an evaluator provided with data entries in the following format:\n",
      "ID: Unique identifiant of the participant\n",
      "Text: The reference text that participants must read beforehand. Their responses for the different steps must be semantically related to this text (same topic), but the answer to the question they are asking should not be found in the text.\n",
      "Identify: Response for the IDENTIFY step\n",
      "Guess: Response for the GUESS step\n",
      "Seek: Response for the SEEK step\n",
      "Assess: Response for the ASSESS step\n",
      "\n",
      "You are tasked with evaluating the following entry.\n",
      "\n",
      "Entry:\n",
      "Id: BC5\n",
      "\n",
      "Text: A des dizaines de kilometres sous nos pieds, la terre contient une couche quon appelle le manteau. Il fait tellement chaud sur cette couche que les roches deviennent liquides : cest ce qu'on appelle le magma. Le magma est responsable de l'explosion des volcans: c'est ce que les scientifiques appellent aussi une eruption d'un volcan. On dit quun volcan est endormi sil ny a eu aucune eruption dans les 10 000 dernieres annees. Au dela de 10 000 ans, on peut dire que le volcan est eteint.\n",
      "\n",
      "Identify: Pourqoi on l'appelle le manteau.\n",
      "\n",
      "Guess: le magma et une pierre.\n",
      "\n",
      "Seek: quel et la temperature magma\n",
      "\n",
      "Assess: 1 000c\n",
      "\n",
      "\n",
      "\n",
      "You should evaluate it like this:\n",
      "Are the Identify, Guess, and Seek steps related to the same question?\n",
      "Minor errors in spelling, grammar or wording should not be penalised as long as the intention of the entry is clear and meets the inclusion criteria. Focus on the content and purpose of the entry rather than on linguistic perfection.\n",
      "\n",
      "\n",
      "Reply with '1' if the entry meets the criterion or '0' otherwise.\n",
      "\n",
      "\n",
      "=== LLM Response ===\n",
      "0\n",
      "\n",
      "\n",
      "--- Evaluating Theme: Reference Link ---\n",
      "Prompt:\n",
      "\n",
      "You are an evaluator provided with data entries in the following format:\n",
      "ID: Unique identifiant of the participant\n",
      "Text: The reference text that participants must read beforehand. Their responses for the different steps must be semantically related to this text (same topic), but the answer to the question they are asking should not be found in the text.\n",
      "Identify: Response for the IDENTIFY step\n",
      "Guess: Response for the GUESS step\n",
      "Seek: Response for the SEEK step\n",
      "Assess: Response for the ASSESS step\n",
      "\n",
      "You are tasked with evaluating the following entry.\n",
      "\n",
      "Entry:\n",
      "Id: BC5\n",
      "\n",
      "Text: A des dizaines de kilometres sous nos pieds, la terre contient une couche quon appelle le manteau. Il fait tellement chaud sur cette couche que les roches deviennent liquides : cest ce qu'on appelle le magma. Le magma est responsable de l'explosion des volcans: c'est ce que les scientifiques appellent aussi une eruption d'un volcan. On dit quun volcan est endormi sil ny a eu aucune eruption dans les 10 000 dernieres annees. Au dela de 10 000 ans, on peut dire que le volcan est eteint.\n",
      "\n",
      "Identify: Pourqoi on l'appelle le manteau.\n",
      "\n",
      "Guess: le magma et une pierre.\n",
      "\n",
      "Seek: quel et la temperature magma\n",
      "\n",
      "Assess: 1 000c\n",
      "\n",
      "\n",
      "\n",
      "You should evaluate it like this:\n",
      "Are the Identify, Guess, and Seek steps related to the topic of the reference text?\n",
      "Minor errors in spelling, grammar or wording should not be penalised as long as the intention of the entry is clear and meets the inclusion criteria. Focus on the content and purpose of the entry rather than on linguistic perfection.\n",
      "\n",
      "\n",
      "Reply with '1' if the entry meets the criterion or '0' otherwise.\n",
      "\n",
      "\n",
      "=== LLM Response ===\n",
      "1\n",
      "\n",
      "\n",
      "--- Evaluating Theme: Seek Question Originality ---\n",
      "Prompt:\n",
      "\n",
      "You are an evaluator provided with data entries in the following format:\n",
      "ID: Unique identifiant of the participant\n",
      "Text: The reference text that participants must read beforehand. Their responses for the different steps must be semantically related to this text (same topic), but the answer to the question they are asking should not be found in the text.\n",
      "Identify: Response for the IDENTIFY step\n",
      "Guess: Response for the GUESS step\n",
      "Seek: Response for the SEEK step\n",
      "Assess: Response for the ASSESS step\n",
      "\n",
      "You are tasked with evaluating the following entry.\n",
      "\n",
      "Entry:\n",
      "Id: BC5\n",
      "\n",
      "Text: A des dizaines de kilometres sous nos pieds, la terre contient une couche quon appelle le manteau. Il fait tellement chaud sur cette couche que les roches deviennent liquides : cest ce qu'on appelle le magma. Le magma est responsable de l'explosion des volcans: c'est ce que les scientifiques appellent aussi une eruption d'un volcan. On dit quun volcan est endormi sil ny a eu aucune eruption dans les 10 000 dernieres annees. Au dela de 10 000 ans, on peut dire que le volcan est eteint.\n",
      "\n",
      "Identify: Pourqoi on l'appelle le manteau.\n",
      "\n",
      "Guess: le magma et une pierre.\n",
      "\n",
      "Seek: quel et la temperature magma\n",
      "\n",
      "Assess: 1 000c\n",
      "\n",
      "\n",
      "\n",
      "You should evaluate it like this:\n",
      "Is the answer to the Seek question not found (even vaguely) in the reference text?\n",
      "Minor errors in spelling, grammar or wording should not be penalised as long as the intention of the entry is clear and meets the inclusion criteria. Focus on the content and purpose of the entry rather than on linguistic perfection.\n",
      "\n",
      "\n",
      "Reply with '1' if the entry meets the criterion or '0' otherwise.\n",
      "\n",
      "\n",
      "=== LLM Response ===\n",
      "1\n",
      "\n",
      "\n",
      "=== Processing Verbatim 4/4 ===\n",
      "\n",
      "--- Evaluating Theme: Identify Step ---\n",
      "Prompt:\n",
      "\n",
      "You are an evaluator provided with data entries in the following format:\n",
      "ID: Unique identifiant of the participant\n",
      "Text: The reference text that participants must read beforehand. Their responses for the different steps must be semantically related to this text (same topic), but the answer to the question they are asking should not be found in the text.\n",
      "Identify: Response for the IDENTIFY step\n",
      "Guess: Response for the GUESS step\n",
      "Seek: Response for the SEEK step\n",
      "Assess: Response for the ASSESS step\n",
      "\n",
      "You are tasked with evaluating the following entry.\n",
      "\n",
      "Entry:\n",
      "Id: BC22\n",
      "\n",
      "Text: A des dizaines de kilometres sous nos pieds, la terre contient une couche quon appelle le manteau. Il fait tellement chaud sur cette couche que les roches deviennent liquides : cest ce qu'on appelle le magma. Le magma est responsable de l'explosion des volcans: c'est ce que les scientifiques appellent aussi une eruption d'un volcan. On dit quun volcan est endormi sil ny a eu aucune eruption dans les 10 000 dernieres annees. Au dela de 10 000 ans, on peut dire que le volcan est eteint.\n",
      "\n",
      "Identify: La temperature du magma\n",
      "\n",
      "Guess: La temperature du magma depasse les 500 C\n",
      "\n",
      "Seek: Quel est la temperature du magma ?\n",
      "\n",
      "Assess: La temperature du magma atteint les 1000 C\n",
      "\n",
      "\n",
      "\n",
      "You should evaluate it like this:\n",
      "Does the Identify step indicate a topic of interest?\n",
      "Minor errors in spelling, grammar or wording should not be penalised as long as the intention of the entry is clear and meets the inclusion criteria. Focus on the content and purpose of the entry rather than on linguistic perfection.\n",
      "\n",
      "\n",
      "Reply with '1' if the entry meets the criterion or '0' otherwise.\n",
      "\n",
      "\n",
      "=== LLM Response ===\n",
      "1\n",
      "\n",
      "\n",
      "--- Evaluating Theme: Guess Step ---\n",
      "Prompt:\n",
      "\n",
      "You are an evaluator provided with data entries in the following format:\n",
      "ID: Unique identifiant of the participant\n",
      "Text: The reference text that participants must read beforehand. Their responses for the different steps must be semantically related to this text (same topic), but the answer to the question they are asking should not be found in the text.\n",
      "Identify: Response for the IDENTIFY step\n",
      "Guess: Response for the GUESS step\n",
      "Seek: Response for the SEEK step\n",
      "Assess: Response for the ASSESS step\n",
      "\n",
      "You are tasked with evaluating the following entry.\n",
      "\n",
      "Entry:\n",
      "Id: BC22\n",
      "\n",
      "Text: A des dizaines de kilometres sous nos pieds, la terre contient une couche quon appelle le manteau. Il fait tellement chaud sur cette couche que les roches deviennent liquides : cest ce qu'on appelle le magma. Le magma est responsable de l'explosion des volcans: c'est ce que les scientifiques appellent aussi une eruption d'un volcan. On dit quun volcan est endormi sil ny a eu aucune eruption dans les 10 000 dernieres annees. Au dela de 10 000 ans, on peut dire que le volcan est eteint.\n",
      "\n",
      "Identify: La temperature du magma\n",
      "\n",
      "Guess: La temperature du magma depasse les 500 C\n",
      "\n",
      "Seek: Quel est la temperature du magma ?\n",
      "\n",
      "Assess: La temperature du magma atteint les 1000 C\n",
      "\n",
      "\n",
      "\n",
      "You should evaluate it like this:\n",
      "Does the Guess step suggest a possible explanation?\n",
      "Minor errors in spelling, grammar or wording should not be penalised as long as the intention of the entry is clear and meets the inclusion criteria. Focus on the content and purpose of the entry rather than on linguistic perfection.\n",
      "\n",
      "\n",
      "Reply with '1' if the entry meets the criterion or '0' otherwise.\n",
      "\n",
      "\n",
      "=== LLM Response ===\n",
      "1\n",
      "\n",
      "\n",
      "--- Evaluating Theme: Seek Step ---\n",
      "Prompt:\n",
      "\n",
      "You are an evaluator provided with data entries in the following format:\n",
      "ID: Unique identifiant of the participant\n",
      "Text: The reference text that participants must read beforehand. Their responses for the different steps must be semantically related to this text (same topic), but the answer to the question they are asking should not be found in the text.\n",
      "Identify: Response for the IDENTIFY step\n",
      "Guess: Response for the GUESS step\n",
      "Seek: Response for the SEEK step\n",
      "Assess: Response for the ASSESS step\n",
      "\n",
      "You are tasked with evaluating the following entry.\n",
      "\n",
      "Entry:\n",
      "Id: BC22\n",
      "\n",
      "Text: A des dizaines de kilometres sous nos pieds, la terre contient une couche quon appelle le manteau. Il fait tellement chaud sur cette couche que les roches deviennent liquides : cest ce qu'on appelle le magma. Le magma est responsable de l'explosion des volcans: c'est ce que les scientifiques appellent aussi une eruption d'un volcan. On dit quun volcan est endormi sil ny a eu aucune eruption dans les 10 000 dernieres annees. Au dela de 10 000 ans, on peut dire que le volcan est eteint.\n",
      "\n",
      "Identify: La temperature du magma\n",
      "\n",
      "Guess: La temperature du magma depasse les 500 C\n",
      "\n",
      "Seek: Quel est la temperature du magma ?\n",
      "\n",
      "Assess: La temperature du magma atteint les 1000 C\n",
      "\n",
      "\n",
      "\n",
      "You should evaluate it like this:\n",
      "Is the Seek step formulated as a question?\n",
      "Minor errors in spelling, grammar or wording should not be penalised as long as the intention of the entry is clear and meets the inclusion criteria. Focus on the content and purpose of the entry rather than on linguistic perfection.\n",
      "\n",
      "\n",
      "Reply with '1' if the entry meets the criterion or '0' otherwise.\n",
      "\n",
      "\n",
      "=== LLM Response ===\n",
      "1\n",
      "\n",
      "\n",
      "--- Evaluating Theme: Assess Step ---\n",
      "Prompt:\n",
      "\n",
      "You are an evaluator provided with data entries in the following format:\n",
      "ID: Unique identifiant of the participant\n",
      "Text: The reference text that participants must read beforehand. Their responses for the different steps must be semantically related to this text (same topic), but the answer to the question they are asking should not be found in the text.\n",
      "Identify: Response for the IDENTIFY step\n",
      "Guess: Response for the GUESS step\n",
      "Seek: Response for the SEEK step\n",
      "Assess: Response for the ASSESS step\n",
      "\n",
      "You are tasked with evaluating the following entry.\n",
      "\n",
      "Entry:\n",
      "Id: BC22\n",
      "\n",
      "Text: A des dizaines de kilometres sous nos pieds, la terre contient une couche quon appelle le manteau. Il fait tellement chaud sur cette couche que les roches deviennent liquides : cest ce qu'on appelle le magma. Le magma est responsable de l'explosion des volcans: c'est ce que les scientifiques appellent aussi une eruption d'un volcan. On dit quun volcan est endormi sil ny a eu aucune eruption dans les 10 000 dernieres annees. Au dela de 10 000 ans, on peut dire que le volcan est eteint.\n",
      "\n",
      "Identify: La temperature du magma\n",
      "\n",
      "Guess: La temperature du magma depasse les 500 C\n",
      "\n",
      "Seek: Quel est la temperature du magma ?\n",
      "\n",
      "Assess: La temperature du magma atteint les 1000 C\n",
      "\n",
      "\n",
      "\n",
      "You should evaluate it like this:\n",
      "Does it identify a possible answer or state that no answer was found ('no' is ok)?\n",
      "Minor errors in spelling, grammar or wording should not be penalised as long as the intention of the entry is clear and meets the inclusion criteria. Focus on the content and purpose of the entry rather than on linguistic perfection.\n",
      "\n",
      "\n",
      "Reply with '1' if the entry meets the criterion or '0' otherwise.\n",
      "\n",
      "\n",
      "=== LLM Response ===\n",
      "1\n",
      "\n",
      "\n",
      "--- Evaluating Theme: Consistency ---\n",
      "Prompt:\n",
      "\n",
      "You are an evaluator provided with data entries in the following format:\n",
      "ID: Unique identifiant of the participant\n",
      "Text: The reference text that participants must read beforehand. Their responses for the different steps must be semantically related to this text (same topic), but the answer to the question they are asking should not be found in the text.\n",
      "Identify: Response for the IDENTIFY step\n",
      "Guess: Response for the GUESS step\n",
      "Seek: Response for the SEEK step\n",
      "Assess: Response for the ASSESS step\n",
      "\n",
      "You are tasked with evaluating the following entry.\n",
      "\n",
      "Entry:\n",
      "Id: BC22\n",
      "\n",
      "Text: A des dizaines de kilometres sous nos pieds, la terre contient une couche quon appelle le manteau. Il fait tellement chaud sur cette couche que les roches deviennent liquides : cest ce qu'on appelle le magma. Le magma est responsable de l'explosion des volcans: c'est ce que les scientifiques appellent aussi une eruption d'un volcan. On dit quun volcan est endormi sil ny a eu aucune eruption dans les 10 000 dernieres annees. Au dela de 10 000 ans, on peut dire que le volcan est eteint.\n",
      "\n",
      "Identify: La temperature du magma\n",
      "\n",
      "Guess: La temperature du magma depasse les 500 C\n",
      "\n",
      "Seek: Quel est la temperature du magma ?\n",
      "\n",
      "Assess: La temperature du magma atteint les 1000 C\n",
      "\n",
      "\n",
      "\n",
      "You should evaluate it like this:\n",
      "Are the Identify, Guess, and Seek steps related to the same question?\n",
      "Minor errors in spelling, grammar or wording should not be penalised as long as the intention of the entry is clear and meets the inclusion criteria. Focus on the content and purpose of the entry rather than on linguistic perfection.\n",
      "\n",
      "\n",
      "Reply with '1' if the entry meets the criterion or '0' otherwise.\n",
      "\n",
      "\n",
      "=== LLM Response ===\n",
      "1\n",
      "\n",
      "\n",
      "--- Evaluating Theme: Reference Link ---\n",
      "Prompt:\n",
      "\n",
      "You are an evaluator provided with data entries in the following format:\n",
      "ID: Unique identifiant of the participant\n",
      "Text: The reference text that participants must read beforehand. Their responses for the different steps must be semantically related to this text (same topic), but the answer to the question they are asking should not be found in the text.\n",
      "Identify: Response for the IDENTIFY step\n",
      "Guess: Response for the GUESS step\n",
      "Seek: Response for the SEEK step\n",
      "Assess: Response for the ASSESS step\n",
      "\n",
      "You are tasked with evaluating the following entry.\n",
      "\n",
      "Entry:\n",
      "Id: BC22\n",
      "\n",
      "Text: A des dizaines de kilometres sous nos pieds, la terre contient une couche quon appelle le manteau. Il fait tellement chaud sur cette couche que les roches deviennent liquides : cest ce qu'on appelle le magma. Le magma est responsable de l'explosion des volcans: c'est ce que les scientifiques appellent aussi une eruption d'un volcan. On dit quun volcan est endormi sil ny a eu aucune eruption dans les 10 000 dernieres annees. Au dela de 10 000 ans, on peut dire que le volcan est eteint.\n",
      "\n",
      "Identify: La temperature du magma\n",
      "\n",
      "Guess: La temperature du magma depasse les 500 C\n",
      "\n",
      "Seek: Quel est la temperature du magma ?\n",
      "\n",
      "Assess: La temperature du magma atteint les 1000 C\n",
      "\n",
      "\n",
      "\n",
      "You should evaluate it like this:\n",
      "Are the Identify, Guess, and Seek steps related to the topic of the reference text?\n",
      "Minor errors in spelling, grammar or wording should not be penalised as long as the intention of the entry is clear and meets the inclusion criteria. Focus on the content and purpose of the entry rather than on linguistic perfection.\n",
      "\n",
      "\n",
      "Reply with '1' if the entry meets the criterion or '0' otherwise.\n",
      "\n",
      "\n",
      "=== LLM Response ===\n",
      "1\n",
      "\n",
      "\n",
      "--- Evaluating Theme: Seek Question Originality ---\n",
      "Prompt:\n",
      "\n",
      "You are an evaluator provided with data entries in the following format:\n",
      "ID: Unique identifiant of the participant\n",
      "Text: The reference text that participants must read beforehand. Their responses for the different steps must be semantically related to this text (same topic), but the answer to the question they are asking should not be found in the text.\n",
      "Identify: Response for the IDENTIFY step\n",
      "Guess: Response for the GUESS step\n",
      "Seek: Response for the SEEK step\n",
      "Assess: Response for the ASSESS step\n",
      "\n",
      "You are tasked with evaluating the following entry.\n",
      "\n",
      "Entry:\n",
      "Id: BC22\n",
      "\n",
      "Text: A des dizaines de kilometres sous nos pieds, la terre contient une couche quon appelle le manteau. Il fait tellement chaud sur cette couche que les roches deviennent liquides : cest ce qu'on appelle le magma. Le magma est responsable de l'explosion des volcans: c'est ce que les scientifiques appellent aussi une eruption d'un volcan. On dit quun volcan est endormi sil ny a eu aucune eruption dans les 10 000 dernieres annees. Au dela de 10 000 ans, on peut dire que le volcan est eteint.\n",
      "\n",
      "Identify: La temperature du magma\n",
      "\n",
      "Guess: La temperature du magma depasse les 500 C\n",
      "\n",
      "Seek: Quel est la temperature du magma ?\n",
      "\n",
      "Assess: La temperature du magma atteint les 1000 C\n",
      "\n",
      "\n",
      "\n",
      "You should evaluate it like this:\n",
      "Is the answer to the Seek question not found (even vaguely) in the reference text?\n",
      "Minor errors in spelling, grammar or wording should not be penalised as long as the intention of the entry is clear and meets the inclusion criteria. Focus on the content and purpose of the entry rather than on linguistic perfection.\n",
      "\n",
      "\n",
      "Reply with '1' if the entry meets the criterion or '0' otherwise.\n",
      "\n",
      "\n",
      "=== LLM Response ===\n",
      "1\n",
      "\n",
      "\n",
      "=== Processing Complete ===\n",
      "Total Tokens Used: 11337\n",
      "Total Cost for Processing: $0.0326\n",
      "{'Id': 'BC23', 'Text': \"A des dizaines de kilometres sous nos pieds, la terre contient une couche quon appelle le manteau. Il fait tellement chaud sur cette couche que les roches deviennent liquides : cest ce qu'on appelle le magma. Le magma est responsable de l'explosion des volcans: c'est ce que les scientifiques appellent aussi une eruption d'un volcan. On dit quun volcan est endormi sil ny a eu aucune eruption dans les 10 000 dernieres annees. Au dela de 10 000 ans, on peut dire que le volcan est eteint.\", 'Identify': 'Le manteau,le centre de la Terre', 'Guess': 'Je pense que le manteau et le centre de la Terre sont les maames choses', 'Seek': \"Le manteau et le centre de la Terre ne font ils qu'un?\", 'Assess': \"Je n'ai pas pu trouver la reponse a  ma question.\", 'Theme': 'Identify Step', 'Label': 1}\n",
      "{'Id': 'BC23', 'Text': \"A des dizaines de kilometres sous nos pieds, la terre contient une couche quon appelle le manteau. Il fait tellement chaud sur cette couche que les roches deviennent liquides : cest ce qu'on appelle le magma. Le magma est responsable de l'explosion des volcans: c'est ce que les scientifiques appellent aussi une eruption d'un volcan. On dit quun volcan est endormi sil ny a eu aucune eruption dans les 10 000 dernieres annees. Au dela de 10 000 ans, on peut dire que le volcan est eteint.\", 'Identify': 'Le manteau,le centre de la Terre', 'Guess': 'Je pense que le manteau et le centre de la Terre sont les maames choses', 'Seek': \"Le manteau et le centre de la Terre ne font ils qu'un?\", 'Assess': \"Je n'ai pas pu trouver la reponse a  ma question.\", 'Theme': 'Guess Step', 'Label': 1}\n",
      "{'Id': 'BC23', 'Text': \"A des dizaines de kilometres sous nos pieds, la terre contient une couche quon appelle le manteau. Il fait tellement chaud sur cette couche que les roches deviennent liquides : cest ce qu'on appelle le magma. Le magma est responsable de l'explosion des volcans: c'est ce que les scientifiques appellent aussi une eruption d'un volcan. On dit quun volcan est endormi sil ny a eu aucune eruption dans les 10 000 dernieres annees. Au dela de 10 000 ans, on peut dire que le volcan est eteint.\", 'Identify': 'Le manteau,le centre de la Terre', 'Guess': 'Je pense que le manteau et le centre de la Terre sont les maames choses', 'Seek': \"Le manteau et le centre de la Terre ne font ils qu'un?\", 'Assess': \"Je n'ai pas pu trouver la reponse a  ma question.\", 'Theme': 'Seek Step', 'Label': 1}\n",
      "{'Id': 'BC23', 'Text': \"A des dizaines de kilometres sous nos pieds, la terre contient une couche quon appelle le manteau. Il fait tellement chaud sur cette couche que les roches deviennent liquides : cest ce qu'on appelle le magma. Le magma est responsable de l'explosion des volcans: c'est ce que les scientifiques appellent aussi une eruption d'un volcan. On dit quun volcan est endormi sil ny a eu aucune eruption dans les 10 000 dernieres annees. Au dela de 10 000 ans, on peut dire que le volcan est eteint.\", 'Identify': 'Le manteau,le centre de la Terre', 'Guess': 'Je pense que le manteau et le centre de la Terre sont les maames choses', 'Seek': \"Le manteau et le centre de la Terre ne font ils qu'un?\", 'Assess': \"Je n'ai pas pu trouver la reponse a  ma question.\", 'Theme': 'Assess Step', 'Label': 1}\n",
      "{'Id': 'BC23', 'Text': \"A des dizaines de kilometres sous nos pieds, la terre contient une couche quon appelle le manteau. Il fait tellement chaud sur cette couche que les roches deviennent liquides : cest ce qu'on appelle le magma. Le magma est responsable de l'explosion des volcans: c'est ce que les scientifiques appellent aussi une eruption d'un volcan. On dit quun volcan est endormi sil ny a eu aucune eruption dans les 10 000 dernieres annees. Au dela de 10 000 ans, on peut dire que le volcan est eteint.\", 'Identify': 'Le manteau,le centre de la Terre', 'Guess': 'Je pense que le manteau et le centre de la Terre sont les maames choses', 'Seek': \"Le manteau et le centre de la Terre ne font ils qu'un?\", 'Assess': \"Je n'ai pas pu trouver la reponse a  ma question.\", 'Theme': 'Consistency', 'Label': 1}\n",
      "{'Id': 'BC23', 'Text': \"A des dizaines de kilometres sous nos pieds, la terre contient une couche quon appelle le manteau. Il fait tellement chaud sur cette couche que les roches deviennent liquides : cest ce qu'on appelle le magma. Le magma est responsable de l'explosion des volcans: c'est ce que les scientifiques appellent aussi une eruption d'un volcan. On dit quun volcan est endormi sil ny a eu aucune eruption dans les 10 000 dernieres annees. Au dela de 10 000 ans, on peut dire que le volcan est eteint.\", 'Identify': 'Le manteau,le centre de la Terre', 'Guess': 'Je pense que le manteau et le centre de la Terre sont les maames choses', 'Seek': \"Le manteau et le centre de la Terre ne font ils qu'un?\", 'Assess': \"Je n'ai pas pu trouver la reponse a  ma question.\", 'Theme': 'Reference Link', 'Label': 1}\n",
      "{'Id': 'BC23', 'Text': \"A des dizaines de kilometres sous nos pieds, la terre contient une couche quon appelle le manteau. Il fait tellement chaud sur cette couche que les roches deviennent liquides : cest ce qu'on appelle le magma. Le magma est responsable de l'explosion des volcans: c'est ce que les scientifiques appellent aussi une eruption d'un volcan. On dit quun volcan est endormi sil ny a eu aucune eruption dans les 10 000 dernieres annees. Au dela de 10 000 ans, on peut dire que le volcan est eteint.\", 'Identify': 'Le manteau,le centre de la Terre', 'Guess': 'Je pense que le manteau et le centre de la Terre sont les maames choses', 'Seek': \"Le manteau et le centre de la Terre ne font ils qu'un?\", 'Assess': \"Je n'ai pas pu trouver la reponse a  ma question.\", 'Theme': 'Seek Question Originality', 'Label': 1}\n",
      "{'Id': 'BC14', 'Text': \"A des dizaines de kilometres sous nos pieds, la terre contient une couche quon appelle le manteau. Il fait tellement chaud sur cette couche que les roches deviennent liquides : cest ce qu'on appelle le magma. Le magma est responsable de l'explosion des volcans: c'est ce que les scientifiques appellent aussi une eruption d'un volcan. On dit quun volcan est endormi sil ny a eu aucune eruption dans les 10 000 dernieres annees. Au dela de 10 000 ans, on peut dire que le volcan est eteint.\", 'Identify': 'Temperature du magma', 'Guess': \"La temperature du magma s'eleve a  plus de 1000 c\", 'Seek': 'a combien de  cle magma est?', 'Assess': 'La temperature du magma atteint les 1000 C', 'Theme': 'Identify Step', 'Label': 1}\n",
      "{'Id': 'BC14', 'Text': \"A des dizaines de kilometres sous nos pieds, la terre contient une couche quon appelle le manteau. Il fait tellement chaud sur cette couche que les roches deviennent liquides : cest ce qu'on appelle le magma. Le magma est responsable de l'explosion des volcans: c'est ce que les scientifiques appellent aussi une eruption d'un volcan. On dit quun volcan est endormi sil ny a eu aucune eruption dans les 10 000 dernieres annees. Au dela de 10 000 ans, on peut dire que le volcan est eteint.\", 'Identify': 'Temperature du magma', 'Guess': \"La temperature du magma s'eleve a  plus de 1000 c\", 'Seek': 'a combien de  cle magma est?', 'Assess': 'La temperature du magma atteint les 1000 C', 'Theme': 'Guess Step', 'Label': 1}\n",
      "{'Id': 'BC14', 'Text': \"A des dizaines de kilometres sous nos pieds, la terre contient une couche quon appelle le manteau. Il fait tellement chaud sur cette couche que les roches deviennent liquides : cest ce qu'on appelle le magma. Le magma est responsable de l'explosion des volcans: c'est ce que les scientifiques appellent aussi une eruption d'un volcan. On dit quun volcan est endormi sil ny a eu aucune eruption dans les 10 000 dernieres annees. Au dela de 10 000 ans, on peut dire que le volcan est eteint.\", 'Identify': 'Temperature du magma', 'Guess': \"La temperature du magma s'eleve a  plus de 1000 c\", 'Seek': 'a combien de  cle magma est?', 'Assess': 'La temperature du magma atteint les 1000 C', 'Theme': 'Seek Step', 'Label': 0}\n",
      "{'Id': 'BC14', 'Text': \"A des dizaines de kilometres sous nos pieds, la terre contient une couche quon appelle le manteau. Il fait tellement chaud sur cette couche que les roches deviennent liquides : cest ce qu'on appelle le magma. Le magma est responsable de l'explosion des volcans: c'est ce que les scientifiques appellent aussi une eruption d'un volcan. On dit quun volcan est endormi sil ny a eu aucune eruption dans les 10 000 dernieres annees. Au dela de 10 000 ans, on peut dire que le volcan est eteint.\", 'Identify': 'Temperature du magma', 'Guess': \"La temperature du magma s'eleve a  plus de 1000 c\", 'Seek': 'a combien de  cle magma est?', 'Assess': 'La temperature du magma atteint les 1000 C', 'Theme': 'Assess Step', 'Label': 1}\n",
      "{'Id': 'BC14', 'Text': \"A des dizaines de kilometres sous nos pieds, la terre contient une couche quon appelle le manteau. Il fait tellement chaud sur cette couche que les roches deviennent liquides : cest ce qu'on appelle le magma. Le magma est responsable de l'explosion des volcans: c'est ce que les scientifiques appellent aussi une eruption d'un volcan. On dit quun volcan est endormi sil ny a eu aucune eruption dans les 10 000 dernieres annees. Au dela de 10 000 ans, on peut dire que le volcan est eteint.\", 'Identify': 'Temperature du magma', 'Guess': \"La temperature du magma s'eleve a  plus de 1000 c\", 'Seek': 'a combien de  cle magma est?', 'Assess': 'La temperature du magma atteint les 1000 C', 'Theme': 'Consistency', 'Label': 1}\n",
      "{'Id': 'BC14', 'Text': \"A des dizaines de kilometres sous nos pieds, la terre contient une couche quon appelle le manteau. Il fait tellement chaud sur cette couche que les roches deviennent liquides : cest ce qu'on appelle le magma. Le magma est responsable de l'explosion des volcans: c'est ce que les scientifiques appellent aussi une eruption d'un volcan. On dit quun volcan est endormi sil ny a eu aucune eruption dans les 10 000 dernieres annees. Au dela de 10 000 ans, on peut dire que le volcan est eteint.\", 'Identify': 'Temperature du magma', 'Guess': \"La temperature du magma s'eleve a  plus de 1000 c\", 'Seek': 'a combien de  cle magma est?', 'Assess': 'La temperature du magma atteint les 1000 C', 'Theme': 'Reference Link', 'Label': 1}\n",
      "{'Id': 'BC14', 'Text': \"A des dizaines de kilometres sous nos pieds, la terre contient une couche quon appelle le manteau. Il fait tellement chaud sur cette couche que les roches deviennent liquides : cest ce qu'on appelle le magma. Le magma est responsable de l'explosion des volcans: c'est ce que les scientifiques appellent aussi une eruption d'un volcan. On dit quun volcan est endormi sil ny a eu aucune eruption dans les 10 000 dernieres annees. Au dela de 10 000 ans, on peut dire que le volcan est eteint.\", 'Identify': 'Temperature du magma', 'Guess': \"La temperature du magma s'eleve a  plus de 1000 c\", 'Seek': 'a combien de  cle magma est?', 'Assess': 'La temperature du magma atteint les 1000 C', 'Theme': 'Seek Question Originality', 'Label': 1}\n",
      "{'Id': 'BC5', 'Text': \"A des dizaines de kilometres sous nos pieds, la terre contient une couche quon appelle le manteau. Il fait tellement chaud sur cette couche que les roches deviennent liquides : cest ce qu'on appelle le magma. Le magma est responsable de l'explosion des volcans: c'est ce que les scientifiques appellent aussi une eruption d'un volcan. On dit quun volcan est endormi sil ny a eu aucune eruption dans les 10 000 dernieres annees. Au dela de 10 000 ans, on peut dire que le volcan est eteint.\", 'Identify': \"Pourqoi on l'appelle le manteau.\", 'Guess': 'le magma et une pierre.', 'Seek': 'quel et la temperature magma', 'Assess': '1 000c', 'Theme': 'Identify Step', 'Label': 1}\n",
      "{'Id': 'BC5', 'Text': \"A des dizaines de kilometres sous nos pieds, la terre contient une couche quon appelle le manteau. Il fait tellement chaud sur cette couche que les roches deviennent liquides : cest ce qu'on appelle le magma. Le magma est responsable de l'explosion des volcans: c'est ce que les scientifiques appellent aussi une eruption d'un volcan. On dit quun volcan est endormi sil ny a eu aucune eruption dans les 10 000 dernieres annees. Au dela de 10 000 ans, on peut dire que le volcan est eteint.\", 'Identify': \"Pourqoi on l'appelle le manteau.\", 'Guess': 'le magma et une pierre.', 'Seek': 'quel et la temperature magma', 'Assess': '1 000c', 'Theme': 'Guess Step', 'Label': 1}\n",
      "{'Id': 'BC5', 'Text': \"A des dizaines de kilometres sous nos pieds, la terre contient une couche quon appelle le manteau. Il fait tellement chaud sur cette couche que les roches deviennent liquides : cest ce qu'on appelle le magma. Le magma est responsable de l'explosion des volcans: c'est ce que les scientifiques appellent aussi une eruption d'un volcan. On dit quun volcan est endormi sil ny a eu aucune eruption dans les 10 000 dernieres annees. Au dela de 10 000 ans, on peut dire que le volcan est eteint.\", 'Identify': \"Pourqoi on l'appelle le manteau.\", 'Guess': 'le magma et une pierre.', 'Seek': 'quel et la temperature magma', 'Assess': '1 000c', 'Theme': 'Seek Step', 'Label': 1}\n",
      "{'Id': 'BC5', 'Text': \"A des dizaines de kilometres sous nos pieds, la terre contient une couche quon appelle le manteau. Il fait tellement chaud sur cette couche que les roches deviennent liquides : cest ce qu'on appelle le magma. Le magma est responsable de l'explosion des volcans: c'est ce que les scientifiques appellent aussi une eruption d'un volcan. On dit quun volcan est endormi sil ny a eu aucune eruption dans les 10 000 dernieres annees. Au dela de 10 000 ans, on peut dire que le volcan est eteint.\", 'Identify': \"Pourqoi on l'appelle le manteau.\", 'Guess': 'le magma et une pierre.', 'Seek': 'quel et la temperature magma', 'Assess': '1 000c', 'Theme': 'Assess Step', 'Label': 1}\n",
      "{'Id': 'BC5', 'Text': \"A des dizaines de kilometres sous nos pieds, la terre contient une couche quon appelle le manteau. Il fait tellement chaud sur cette couche que les roches deviennent liquides : cest ce qu'on appelle le magma. Le magma est responsable de l'explosion des volcans: c'est ce que les scientifiques appellent aussi une eruption d'un volcan. On dit quun volcan est endormi sil ny a eu aucune eruption dans les 10 000 dernieres annees. Au dela de 10 000 ans, on peut dire que le volcan est eteint.\", 'Identify': \"Pourqoi on l'appelle le manteau.\", 'Guess': 'le magma et une pierre.', 'Seek': 'quel et la temperature magma', 'Assess': '1 000c', 'Theme': 'Consistency', 'Label': 0}\n",
      "{'Id': 'BC5', 'Text': \"A des dizaines de kilometres sous nos pieds, la terre contient une couche quon appelle le manteau. Il fait tellement chaud sur cette couche que les roches deviennent liquides : cest ce qu'on appelle le magma. Le magma est responsable de l'explosion des volcans: c'est ce que les scientifiques appellent aussi une eruption d'un volcan. On dit quun volcan est endormi sil ny a eu aucune eruption dans les 10 000 dernieres annees. Au dela de 10 000 ans, on peut dire que le volcan est eteint.\", 'Identify': \"Pourqoi on l'appelle le manteau.\", 'Guess': 'le magma et une pierre.', 'Seek': 'quel et la temperature magma', 'Assess': '1 000c', 'Theme': 'Reference Link', 'Label': 1}\n",
      "{'Id': 'BC5', 'Text': \"A des dizaines de kilometres sous nos pieds, la terre contient une couche quon appelle le manteau. Il fait tellement chaud sur cette couche que les roches deviennent liquides : cest ce qu'on appelle le magma. Le magma est responsable de l'explosion des volcans: c'est ce que les scientifiques appellent aussi une eruption d'un volcan. On dit quun volcan est endormi sil ny a eu aucune eruption dans les 10 000 dernieres annees. Au dela de 10 000 ans, on peut dire que le volcan est eteint.\", 'Identify': \"Pourqoi on l'appelle le manteau.\", 'Guess': 'le magma et une pierre.', 'Seek': 'quel et la temperature magma', 'Assess': '1 000c', 'Theme': 'Seek Question Originality', 'Label': 1}\n",
      "{'Id': 'BC22', 'Text': \"A des dizaines de kilometres sous nos pieds, la terre contient une couche quon appelle le manteau. Il fait tellement chaud sur cette couche que les roches deviennent liquides : cest ce qu'on appelle le magma. Le magma est responsable de l'explosion des volcans: c'est ce que les scientifiques appellent aussi une eruption d'un volcan. On dit quun volcan est endormi sil ny a eu aucune eruption dans les 10 000 dernieres annees. Au dela de 10 000 ans, on peut dire que le volcan est eteint.\", 'Identify': 'La temperature du magma', 'Guess': 'La temperature du magma depasse les 500 C', 'Seek': 'Quel est la temperature du magma ?', 'Assess': 'La temperature du magma atteint les 1000 C', 'Theme': 'Identify Step', 'Label': 1}\n",
      "{'Id': 'BC22', 'Text': \"A des dizaines de kilometres sous nos pieds, la terre contient une couche quon appelle le manteau. Il fait tellement chaud sur cette couche que les roches deviennent liquides : cest ce qu'on appelle le magma. Le magma est responsable de l'explosion des volcans: c'est ce que les scientifiques appellent aussi une eruption d'un volcan. On dit quun volcan est endormi sil ny a eu aucune eruption dans les 10 000 dernieres annees. Au dela de 10 000 ans, on peut dire que le volcan est eteint.\", 'Identify': 'La temperature du magma', 'Guess': 'La temperature du magma depasse les 500 C', 'Seek': 'Quel est la temperature du magma ?', 'Assess': 'La temperature du magma atteint les 1000 C', 'Theme': 'Guess Step', 'Label': 1}\n",
      "{'Id': 'BC22', 'Text': \"A des dizaines de kilometres sous nos pieds, la terre contient une couche quon appelle le manteau. Il fait tellement chaud sur cette couche que les roches deviennent liquides : cest ce qu'on appelle le magma. Le magma est responsable de l'explosion des volcans: c'est ce que les scientifiques appellent aussi une eruption d'un volcan. On dit quun volcan est endormi sil ny a eu aucune eruption dans les 10 000 dernieres annees. Au dela de 10 000 ans, on peut dire que le volcan est eteint.\", 'Identify': 'La temperature du magma', 'Guess': 'La temperature du magma depasse les 500 C', 'Seek': 'Quel est la temperature du magma ?', 'Assess': 'La temperature du magma atteint les 1000 C', 'Theme': 'Seek Step', 'Label': 1}\n",
      "{'Id': 'BC22', 'Text': \"A des dizaines de kilometres sous nos pieds, la terre contient une couche quon appelle le manteau. Il fait tellement chaud sur cette couche que les roches deviennent liquides : cest ce qu'on appelle le magma. Le magma est responsable de l'explosion des volcans: c'est ce que les scientifiques appellent aussi une eruption d'un volcan. On dit quun volcan est endormi sil ny a eu aucune eruption dans les 10 000 dernieres annees. Au dela de 10 000 ans, on peut dire que le volcan est eteint.\", 'Identify': 'La temperature du magma', 'Guess': 'La temperature du magma depasse les 500 C', 'Seek': 'Quel est la temperature du magma ?', 'Assess': 'La temperature du magma atteint les 1000 C', 'Theme': 'Assess Step', 'Label': 1}\n",
      "{'Id': 'BC22', 'Text': \"A des dizaines de kilometres sous nos pieds, la terre contient une couche quon appelle le manteau. Il fait tellement chaud sur cette couche que les roches deviennent liquides : cest ce qu'on appelle le magma. Le magma est responsable de l'explosion des volcans: c'est ce que les scientifiques appellent aussi une eruption d'un volcan. On dit quun volcan est endormi sil ny a eu aucune eruption dans les 10 000 dernieres annees. Au dela de 10 000 ans, on peut dire que le volcan est eteint.\", 'Identify': 'La temperature du magma', 'Guess': 'La temperature du magma depasse les 500 C', 'Seek': 'Quel est la temperature du magma ?', 'Assess': 'La temperature du magma atteint les 1000 C', 'Theme': 'Consistency', 'Label': 1}\n",
      "{'Id': 'BC22', 'Text': \"A des dizaines de kilometres sous nos pieds, la terre contient une couche quon appelle le manteau. Il fait tellement chaud sur cette couche que les roches deviennent liquides : cest ce qu'on appelle le magma. Le magma est responsable de l'explosion des volcans: c'est ce que les scientifiques appellent aussi une eruption d'un volcan. On dit quun volcan est endormi sil ny a eu aucune eruption dans les 10 000 dernieres annees. Au dela de 10 000 ans, on peut dire que le volcan est eteint.\", 'Identify': 'La temperature du magma', 'Guess': 'La temperature du magma depasse les 500 C', 'Seek': 'Quel est la temperature du magma ?', 'Assess': 'La temperature du magma atteint les 1000 C', 'Theme': 'Reference Link', 'Label': 1}\n",
      "{'Id': 'BC22', 'Text': \"A des dizaines de kilometres sous nos pieds, la terre contient une couche quon appelle le manteau. Il fait tellement chaud sur cette couche que les roches deviennent liquides : cest ce qu'on appelle le magma. Le magma est responsable de l'explosion des volcans: c'est ce que les scientifiques appellent aussi une eruption d'un volcan. On dit quun volcan est endormi sil ny a eu aucune eruption dans les 10 000 dernieres annees. Au dela de 10 000 ans, on peut dire que le volcan est eteint.\", 'Identify': 'La temperature du magma', 'Guess': 'La temperature du magma depasse les 500 C', 'Seek': 'Quel est la temperature du magma ?', 'Assess': 'La temperature du magma atteint les 1000 C', 'Theme': 'Seek Question Originality', 'Label': 1}\n"
     ]
    }
   ],
   "source": [
    "# Define instructions for multiclass classification\n",
    "# Classification settings\n",
    "reasoning = False  # If True, perform two-step reasoning. If False, single-step classification.\n",
    "verbose = True # If True, show the prompt\n",
    "\n",
    "# Select subset of verbatims for testing\n",
    "subset = 4\n",
    "verbatims_subset = verbatims[:subset]\n",
    "\n",
    "codebook = {\n",
    "    \"Identify Step\": \"Does the Identify step indicate a topic of interest?\",\n",
    "    \"Guess Step\": \"Does the Guess step suggest a possible explanation?\",\n",
    "    \"Seek Step\": \"Is the Seek step formulated as a question?\",\n",
    "    \"Assess Step\": \"Does it identify a possible answer or state that no answer was found ('no' is ok)?\",\n",
    "    \"Consistency\": \"Are the Identify, Guess, and Seek steps related to the same question?\",\n",
    "    \"Reference Link\": \"Are the Identify, Guess, and Seek steps related to the topic of the reference text?\",\n",
    "    \"Seek Question Originality\": \"Is the answer to the Seek question not found (even vaguely) in the reference text?\",\n",
    "}\n",
    "\n",
    "# \"\"\"\n",
    "# Exemple:\n",
    "\n",
    "# Identify: \"I don’t understand how rain forms.\"\n",
    "# Guess: \"Maybe rain condenses in the sky, forming droplets.\"\n",
    "# Seek: \"How does rain form?\"\n",
    "# Assess: \"No\"\n",
    "# Reference: \"Rain forms when water evaporates into the atmosphere, condenses into droplets, and falls due to gravity.\"\n",
    "\n",
    "# Identify Step: Does the Identify step indicate a topic of interest?\n",
    "# Yes: The topic is the formation of rain. 1\n",
    "# - Guess Step: Does the Guess step suggest a possible explanation?\n",
    "# Yes: It proposes condensation as the mechanism for rain formation. 1\n",
    "# - Seek Step: Is the Seek step formulated as a question?\n",
    "# Yes: It is explicitly phrased as a question with an interrogative structure. 1\n",
    "# - Assess Step: Does it identify a possible answer or state that no answer where found (\"no\" is ok) ?\n",
    "# Yes: It is stated that the answer to the question was not found, which is a valid answer in the Assess step. 1\n",
    "# - Consistency: Are the Identify, Guess, and Seek steps related to the same question?\n",
    "# Yes: They all pertain to the process of rain formation. 1\n",
    "# - Reference Link: Are the Identify, Guess, and Seek steps related to the topic of the reference text?\n",
    "# Yes: The text discusses rain and explains its formation. 1\n",
    "# - Seek Question Originality: Is the answer to the Seek question not found (even vaguely) in the reference text?\n",
    "# No: The answer is explicitly provided in the reference text. 0\n",
    "# - If the Assess step state an answer, does it answer to the question in the Seek step ?\n",
    "# Yes: \"No\" is a possible response, indicating that a response to the answer was not found. 1\n",
    "# - Cycle Resolution: Does the answer to the Seek question resolve the topic identified in the Identify step?\n",
    "# Yes: The Seek question directly addresses the knowledge gap in the Identify step. 1\n",
    "# \"\"\"\n",
    "\n",
    "# Define column descriptions\n",
    "column_descriptions = (\n",
    "    \"ID: Unique identifiant of the participant\\n\"\n",
    "    \"Text: The reference text that participants must read beforehand. Their responses for the different steps must be semantically related to this text (same topic), but the answer to the question they are asking should not be found in the text.\\n\"\n",
    "    \"Identify: Response for the IDENTIFY step\\n\"\n",
    "    \"Guess: Response for the GUESS step\\n\"\n",
    "    \"Seek: Response for the SEEK step\\n\"\n",
    "    \"Assess: Response for the ASSESS step\"\n",
    ")\n",
    "\n",
    "# Define queries for multiclass classification and reasoning\n",
    "binary_query = \"Reply with '1' if the entry meets the criterion or '0' otherwise.\"\n",
    "reasoning_query = \"First, generate a one-sentence reasoning about the classification.\"\n",
    "\n",
    "# To be sure to have an integer only when doing 1 call request\n",
    "command = (\n",
    "    \"Minor errors in spelling, grammar or wording should not be penalised as long as the intention of the entry is clear and meets the inclusion criteria. Focus on the content and purpose of the entry rather than on linguistic perfection.\"\n",
    "    if not reasoning\n",
    "    else \"Minor errors in spelling, grammar or wording should not be penalised as long as the intention of the entry is clear and meets the inclusion criteria. Focus on the content and purpose of the entry rather than on linguistic perfection.\"\n",
    ")\n",
    "\n",
    "prompt_template = f\"\"\"\n",
    "You are an evaluator provided with data entries in the following format:\n",
    "{column_descriptions}\n",
    "\n",
    "You are tasked with evaluating the following entry.\n",
    "\n",
    "Entry:\n",
    "{{verbatim_text}}\n",
    "\n",
    "You should evaluate it like this:\n",
    "{{codebook}}\n",
    "{command}\n",
    "\"\"\"\n",
    "\n",
    "results_binary, verbatim_costs_binary = process_verbatims_for_binary_criteria(\n",
    "    verbatims_subset=verbatims_subset,\n",
    "    codebooks=codebook,\n",
    "    llm_client=llm_client,\n",
    "    model_name=model_name,\n",
    "    prompt_template=prompt_template,\n",
    "    reasoning_query=reasoning_query,\n",
    "    binary_query=binary_query,\n",
    "    reasoning=reasoning,\n",
    "    verbose=verbose\n",
    ")\n",
    "\n",
    "for result in results_binary:\n",
    "    print(result)\n",
    "\n",
    "results_df = pd.DataFrame(results_binary)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Extraction of Overall Validity (Optional)\n",
    "\n",
    "After performing sequential binary classifications, we obtained a series of binary outcomes (`0` for invalid and `1` for valid) for each step in the cycle. Depending on the analysis goal, it may be useful to perform an additional step to assess the overall validity of each cycle.\n",
    "\n",
    "In the KidsReflect dataset, a cycle is considered **valid** only if **all** individual binary classifications are valid (`1`). If any of the classifications are invalid (`0`), the entire cycle is deemed **invalid**.\n",
    "\n",
    "In this step, we extract the overall validity of each cycle based on the results of the sequential binary classifications.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Extract global validity\n",
    "global_validity_df = extract_global_validity(\n",
    "    results_df,                  # Input DataFrame\n",
    "    id_column='Id',              # Column with subject/group identifiers\n",
    "    label_column='Label',        # Column with binary classification labels\n",
    "    global_validity_column='Global_Validity'  # Output column for validity\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Saving the Results\n",
    "\n",
    "This step provides an option to save the classification results to a file for future reference or further analysis."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Possibility to save the results\n",
    "\n",
    "# import datetime\n",
    "# # Define the save path\n",
    "# outputs_dir = os.path.join(data_dir, 'outputs')\n",
    "# os.makedirs(outputs_dir, exist_ok=True)\n",
    "# timestamp = datetime.now().strftime('%Y%m%d_%H%M%S')\n",
    "# save_path = os.path.join(\n",
    "#     outputs_dir, \n",
    "#     f\"experiment_{model_name.replace('/', '_')}_{timestamp}.xlsx\" \n",
    "# )\n",
    "\n",
    "# # Save the DataFrame to Excel\n",
    "# try:\n",
    "#     results_df.to_excel(save_path, index=False, sheet_name='Results')\n",
    "#     print(f\"Results successfully saved to: {save_path}\")\n",
    "# except Exception as e:\n",
    "#     print(f\"An error occurred while saving the Excel file: {e}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Evaluating Model Performance Against Human Annotations\n",
    "\n",
    "### Computing Cohen's Kappa\n",
    "\n",
    "In this step, we compare the model's classifications with human annotations using Cohen's Kappa.\n",
    "\n",
    "1. **Unweighted Kappa**:  \n",
    "   All disagreements are treated equally, regardless of the \"distance\" between the categories. For example, a mismatch between `0` and `1` is penalized the same as a mismatch between `0` and `2`.\n",
    "\n",
    "2. **Linear Weights**:  \n",
    "   With `weights='linear'`, disagreements are penalized proportionally to their distance. For instance:\n",
    "   - A mismatch between `0` and `1` incurs a smaller penalty than a mismatch between `0` and `2`.\n",
    "   - This approach is suitable when the categories have a natural ordering (e.g., `0 < 1 < 2`) but the differences are not exponential.\n",
    "\n",
    "3. **Quadratic Weights**:  \n",
    "   With `weights='quadratic'`, disagreements are penalized more heavily as the distance increases. This approach emphasizes larger disagreements even more than linear weights.\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "model vs human1: 0.00\n"
     ]
    }
   ],
   "source": [
    "# Assuming you have human annotations in the data\n",
    "model_coding = global_validity_df['Global_Validity'].tolist()\n",
    "model_coding = [int(x) for x in model_coding]\n",
    "\n",
    "human_annotations = {\n",
    "    'human1': [int(x) for x in data['corr_cycle1'].tolist()],\n",
    "}\n",
    "\n",
    "# Shorten data for demonstration if needed\n",
    "model_coding_short = model_coding[:subset]\n",
    "human_annotations_short = {rater: ann[:subset] for rater, ann in human_annotations.items()}\n",
    "\n",
    "# Compute Cohen's Kappa for all pairings and print results\n",
    "kappa_results = compute_all_kappas(\n",
    "    model_coding_short,\n",
    "    human_annotations_short,\n",
    "    labels=[0, 1],\n",
    "    weights='linear',\n",
    "    verbose=True\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Visualizing Agreement\n",
    "\n",
    "In this step, we plot confusion matrices to visualize the agreement between the model's classifications and human annotations, or between two human annotations.\n",
    "\n",
    "**Inputs**:\n",
    "- `model_coding_short`: The model's predictions for the subset of data.\n",
    "- `human_annotations_short`: The annotations from human raters for the same subset.\n",
    "\n",
    "**Labels**:\n",
    "- `[0, 1, 2]`: The three classification categories representing:\n",
    "   - `0`: The answer is explicitly in the text.\n",
    "   - `1`: The answer can be inferred from the text.\n",
    "   - `2`: The answer is not in the text at all.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAjAAAAHkCAYAAAAgthXLAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjkuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8hTgPZAAAACXBIWXMAAA9hAAAPYQGoP6dpAABFDklEQVR4nO3deXxN1/7/8fcJMhHElKi0xhJjTImaQmMeqij3atV01VBTiRqvWRWtWcRwL0VRWqVotYq6tKVCRQdjr6mUJGYiksjw+8PP+ToSnBMnOdn3vJ597MdD1l5Z63PSPpqPz1prb1NqamqqAAAADMTF0QEAAADYigQGAAAYDgkMAAAwHBIYAABgOCQwAADAcEhgAACA4ZDAAAAAwyGBAQAAhkMCAwAADIcEBniCLl26qFy5curUqdNj+wwZMkTlypXTyJEjn3m+/fv3q1y5ctq/f3+mfo89jBw5UiEhIVk6Z1ZISkrS3/72N82fP9/RoQB4AhIY4ClcXFx0+PBhRUVFpbkXFxenXbt2OSAqZIaEhAQNHTpUv/zyi6NDAfAUJDDAU1SoUEFubm765ptv0tzbtWuXPDw85OPj44DIYE8HDx5Ux44d9dNPPzk6FABWIIEBnsLT01MNGjRIN4HZunWrmjVrppw5c1q0JyQkaMGCBWrevLkqV66spk2basmSJUpJSbHot3btWjVr1kxVqlTRm2++qYsXL6aZ4+LFiwoNDVVQUJACAgLUrVs3HT161Or4x44dq7p16yo5OdmifcqUKapVq5bu3bun+Ph4TZgwQcHBwapUqZKaN2+upUuXWjX+hg0b1KxZM1WuXFlt2rTR7t27zffmz5+vcuXKpfmecuXKmZdoLly4oHLlyumbb75Rv379VLVqVdWpU0fh4eGKjY3V6NGjVaNGDdWpU0cffvihHn7/7IULFzR8+HDVq1dPFStWVO3atTV8+HBdv37d3CckJETz5s3T9OnTVadOHVWpUkU9e/bU2bNnLWJ6++239dxzz2nDhg1WfW4AjkUCA1ihZcuWaZaRYmNjtWfPHrVu3dqib2pqqvr27at///vf6tixoxYtWqTmzZtrzpw5Gj9+vLnfqlWrNH78eDVo0EDh4eEKCAjQ2LFjLca6du2aOnXqpCNHjmjs2LGaOXOmUlJS1LlzZ506dcqq2F999VVduXLFYo9MSkqKvv76a7Vq1Uq5cuXS+++/rz179mjEiBFaunSpGjVqpA8++ECff/75E8e+dOmSlixZonfeeUfz58+XyWTSoEGDdPXqVatie9iYMWNUtmxZLVy4ULVr19bcuXPVoUMHubu7KywsTE2bNtW///1vcyJ59+5dde3aVadOndL48eO1dOlSde3aVV999ZVmz55tMfbKlSt1+vRpTZ06Ve+9955+//13jRgxwqLPqlWrtGjRIhUrVszm2AFkvZxP7wKgYcOG8vDw0DfffKPu3btLkrZv366CBQuqRo0aFn337NmjvXv3atasWWrVqpUkqW7dunJ3d9fcuXPVtWtXlSlTRuHh4WrZsqVGjx4tSapXr55iY2O1du1a81grVqzQjRs39Mknn5h/sQYHB6tly5aaO3eu5s2b99TYa9SooWLFiunLL79UnTp1JN3f+Hv58mW9+uqrkqSIiAjVrVvXHG+tWrXk6empggULPnHslJQULViwQKVLl5Ykubm5qXv37jp8+LAaNWr01NgeVr9+fQ0ePFiS9OKLL+rLL79UwYIFNW7cOEnSSy+9pC1btujQoUNq0aKFzp49K19fX02fPl3PP/+8uc8vv/yiiIgIi7Hz5s2r8PBw5ciRQ5L0559/av78+bp+/bq8vb0lKd1KEYDsiwoMYAV3d3eFhIRYLCN99dVXatGihUwmk0XfiIgI5cyZU82bN7dob9Omjfn+6dOndfXqVb388ssWfVq0aGHx9b59+1S+fHn5+PgoKSlJSUlJcnFxUXBwsPbu3WtV7CaTSW3atNGOHTuUmJhojr1EiRIKCAiQdD9h+fTTT9WrVy+tWrVK58+fV//+/dWwYcMnju3t7W1OXiTJz89PknT79m2rYntYtWrVzH8uVKiQJKlKlSoWnyNfvnzmscuXL681a9aoWLFiOnv2rHbv3q2lS5fq9OnT5s/5QOXKlc3JiyT5+vpKul/FAWBMJDCAlVq0aGFeRrp+/br27dtnrlg87ObNm/L29rb4hSlJhQsXlnT/l/vNmzclyfy3/0f7PHDjxg0dPnxYFStWtLhWr16t27dvW/0L+NVXX9XNmzf1/fffKzExUd9++605oZKkf/7znxo8eLAuXLigyZMnq3HjxurUqZOOHz/+xHE9PT0tvn6QzD2618caefLkeer4j/roo49Uu3ZtNWvWTKNHj1ZERIQ8PDzS9Hu0zcXFJcNxAsgeWEICrBQcHKzcuXPrm2++kaenp/z8/FSpUqU0/fLly6fr168rOTnZIomJiYmRdD9peZC4PLpX5MaNGxZfe3l5KSgoSMOHD083JldXV6tiL1mypKpUqaKvv/5aLi4uunXrlkUC4+rqqrfffltvv/22Ll68qF27dik8PFxDhw7VV199ZdUc6XmQ0Dz8s7hz506Gx3vYli1bNG3aNA0bNkzt27dXgQIFJEnvvPOOfvvtN7vMASD7ogIDWMnV1VWNGzfWtm3bzBtg0xMUFKSkpKQ0p5Y2b94s6f6elBIlSqho0aJp+jz6TJmgoCCdOXNGJUuWVOXKlc3Xpk2btH79+jRVnid59dVX9f333+urr75S9erVzftG4uPj1axZMy1btkyS9Nxzz6lz585q1apVuqeibPGgqvLw5ueff/75mcZ8eJy8efPqrbfeMicvd+7c0c8//0xlBXACJDCADVq2bKnIyEjt37//sQlMcHCwatWqpTFjxmjZsmXau3ev5syZo/DwcLVr105lypSRyWTSu+++q127dmnMmDH64YcfFBYWpk8++cRirO7duyslJUXdu3fX1q1btW/fPo0dO1Yff/yxSpYsaXPsd+7c0datW82bd6X7+3sqVqyosLAwffzxx4qIiNC6deu0ceNGNWvWzPYf0kMaNGggSRo3bpz27t2rzz//XBMmTFDu3LmfaVzp/v6YW7duadq0adq/f7+2bNmizp0768qVK+xtAZwAS0iADerUqaO8efOqaNGiFptXH2YymbR48WLNmzdPy5cv17Vr1+Tn56fQ0FD16NHD3K9169ZycXFReHi4Nm3apLJly2rSpEkKDQ019/Hx8dHatWs1c+ZMTZgwQQkJCSpRooSmTJmiDh062BR7gQIFVK9ePf34449pNhhPmjRJc+bM0bJly3T58mUVLFhQHTp00DvvvGPTHI8qWbKkpk+froULF6p3794qXbq0Jk+erMmTJz/TuJLUrl07XbhwQZ9//rnWrFkjHx8fNWjQQG+88YbGjh2rU6dOPfbfEQDjM6U+/FQoAAAAA2AJCQAAGA4JDAAAMBwSGAAA8EyuXr2qYcOG6aWXXlK1atXUu3fvJ77u5Pr16xo6dKgCAwMVFBSkiRMn2rz5ngQGAAA8k/79++vcuXNasmSJ1q9fL3d3d3Xv3v2xScmgQYN07tw5LV++XHPnztXu3bs1YcIEm+YkgQEAABl28+ZNFStWTO+9956qVKmi0qVLq1+/foqJidEff/yRpn9kZKQiIiI0ffp081vkJ02apE2bNik6OtrqeUlgAABAhuXLl08zZ85U2bJlJUnXrl3T8uXL5evrqzJlyqTpf/DgQRUuXNjiMQdBQUEymUw2PeiS58AAAICnvkF+586dTx1j7Nix+vTTT+Xq6qqFCxem+z6z6OhoFS1a1KLN1dVV+fPn16VLl6yO12kSmPgkR0cAGM+XR6z/nwmA/9MhoOjTO9mBR7UBdhurToFnH6Nbt276+9//rtWrV6t///5as2aNKlasaNHn7t276b7Hzc3NTQkJCVbP5TQJDAAAeDxrKixP82DJaMqUKfrll1+0atUqTZ061aKPu7u7EhMT03xvQkLCU99A/zD2wAAAYFQmF/tdGXTt2jV99dVXSkr6v6UOFxcXlSlTRjExMWn6+/r6pmlPTEzUjRs3VKRIEavnJYEBAMCoTCb7XRl05coVhYaGat++fea2e/fu6ejRo+m+jywwMFBRUVE6d+6cuS0iIkKSVKNGDavnJYEBAAAZVrZsWQUHB+u9997TgQMHdPLkSY0cOVK3bt1S9+7dlZycrMuXLys+Pl6SFBAQoOrVq2vIkCH69ddf9dNPP2ncuHFq27atfHx8rJ6XBAYAAKPKBktIkjRr1izVrl1bQ4YMUceOHXXjxg2tXr1azz33nC5duqR69epp69at90M2mRQWFiY/Pz9169ZNgwcPVnBwsM0PsnOat1FzCgmwHaeQgIzJslNINYfYbay7B2fbbayswCkkAACM6hn2rhgdCQwAAEb1jEs/Rua8nxwAABgWFRgAAIyKJSQAAGA4LCEBAAAYBxUYAACMiiUkAABgOCwhAQAAGAcVGAAAjIolJAAAYDgsIQEAABgHFRgAAIyKJSQAAGA4LCEBAAAYBxUYAACMigoMAACAcVCBAQDAqFzYxAsAAIyGJSQAAADjoAIDAIBR8RwYAABgOCwhAQAAGAcVGAAAjIolJAAAYDgsIQEAABgHFRgAAIyKJSQAAGA4LCEBAAAYBxUYAACMiiUkAABgOCwhAQAAGAcVGAAAjMqJl5CowAAAAMOhAgMAgFE58R4YEhgAAIzKiRMY5/3kAADAsKjAAABgVE68iZcEBgAAo2IJCQAAwDiowAAAYFQsIQEAAMNhCQkAAMA4qMAAAGBULCEBAACjMTlxAsMSEgAAMBwqMAAAGJQzV2BIYAAAMCrnzV9YQgIAAMZDBQYAAINy5iUkKjAAAMBwqMAAAGBQzlyBIYEBAMCgnDmBYQkJAAAYDhUYAAAMKrtUYG7cuKFZs2bpP//5j2JjY1WuXDkNHTpUNWvWTLf/woULNWfOnDTtJ06csHpOEhgAAIwqe+QvCg0N1eXLlzVr1iwVLFhQH3/8sXr27KmNGzeqVKlSafqfOHFCr776qoYNG5bhOVlCAgAAGXbu3Dn9+OOPmjBhgmrWrKmSJUtq7NixKlKkiLZs2ZLu95w8eVIVKlRQ4cKFLS5bUIEBAMCgssMSkre3t5YsWaLKlSub20wmk0wmk27dupWmf2Jios6ePZtuZcYWJDAAABiUPROYRo0aPfH+zp07023PmzevGjRoYNG2bds2nTt3TqNHj07T/7///a+Sk5O1bds2TZkyRQkJCQoMDNSwYcNUpEgRq+NlCQkAANjNoUOHNGrUKDVt2lQNGzZMc//kyZOSJA8PD82dO1dTpkzR6dOn1bVrV8XHx1s9DxUYAAAMyp4VmMdVWGyxY8cOvfvuu6pevbpmzJiRbp+2bdsqODhYBQoUMLe9+OKLCg4O1nfffaeWLVtaNRcVGAAADOrBXhN7XM9q1apVGjhwoF5++WUtWrRIbm5uj+37cPIiSUWKFFH+/PkVFRVl9XwkMAAA4JmsWbNGkydPVufOnTVr1iy5uro+tu/s2bPVrFkzpaammtsuXLig69evq0yZMlbPSQIDAIBRmex4ZdCZM2f0/vvvq0mTJurTp4+uXLmiy5cv6/Lly7p9+7YSExN1+fJlJSYmSpKaNGmiv/76SxMmTNCZM2d04MABDRw4UNWrV1f9+vWtnpc9MAAAGFR2OEa9bds23bt3T9u3b9f27dst7rVr107t2rVT165dtXLlStWqVUuVKlXSv/71L82dO1ft27eXq6urGjVqpBEjRtj0eUypD9dw/ofFJzk6AsB4vjxyydEhAIbUIaBolsxTqPtau411ZXknu42VFajAAABgUNmhAuMo7IEBAACGQwUGAACDcuYKDAkMAABG5bz5C0tIAADAeKjAAABgUCwhAQAAw3HmBIYlJAAAYDhUYAAAMChnrsCQwAAAYFDOnMCwhAQAAAyHCgwAAEblvAUYEhgAAIyKJSQAAAADoQIDAIBBOXMFhgQGAACDcuYEhiUkAABgOFRgAAAwKuctwFCBAQAAxkMCA4fZ++MPeuNvr6lWjQC1aBqiFR8tVWpqqqPDAgzj5tUYTe7eSqePRDo6FDiIyWSy22U0LCHBIX795bAG9uurZi1aqP/AdxR56GfNnvmhkpKS1bNXb0eHB2R7N67EaPmUYYqPu+PoUOBARkw87IUEBg4RHjZf/uXL6/1pH0qS6tYP1r2kJC391yJ17tJV7u7uDo4QyJ5SUlJ0eM82ff3xQlGwhDNjCQlZLjExUQcP7FdIoyYW7U2aNtOdO3cUeehnB0UGZH9Rf57Spn/NUrXgZuo4YLSjw4GDOfMSEgkMstyF8+d17949FS9RwqL9hReKS5LOnjnjgKgAY8hfyEeh81arZbf+yuXm5uhw4GDOnMA4dAkpKSlJ3377rQ4cOKBLly4pMTFRHh4e8vHxUWBgoJo2baocOXI4MkRkgtjY25KkPHnyWLR75s4tSbpzJzbLYwKMwjNPXilPXkeHATicwyowFy5cUKtWrTR69GidOHFC7u7uKly4sHLlyqXjx49r1KhReuWVV3Tx4kVHhYhMkpKS8sT7JhOFQQCwismOl8E4rAIzadIk+fn5af369fLy8kpz/9atWxoyZIgmTZqkRYsWOSBCZJY8///f9507lqcn7sTer7x4eeVJ8z0AgLSMuPRjLw77q+6BAwc0fPjwdJMXScqbN6+GDRumAwcOZHFkyGzPP/+CcuTIofN/nrNo//PPPyVJJUuVdkRYAAADcVgC4+Xlpejo6Cf2uXjxIsdp/we5ubmpeo2a2rlju8WD63Zs3yYvLy9VqlzFgdEBgHE48yZehyUwHTp00MiRI7Vu3TqdO3dOiYmJku4fsT1//rw+//xz/fOf/1T79u0dFSIyUa8+b+u3X3/RsNB39MP3uxU2b45WfLRUPXv1kYeHh6PDAwBDMJnsdxmNw/bADBw4UC4uLvrggw8UFxeX5n7u3LnVuXNnvfPOOw6IDpmt1ku1NXPOfC1cME+DB/ZXER8fDXl3uLp1/4ejQwMAGIAp1cEvn7l3756OHTum6Oho3b17V+7u7vL19ZW/v79cXV3tNk98kt2GApzGl0cuOToEwJA6BBTNknleHPaN3cb648PmdhsrKzj8VQK5cuVSlSrseQAAwFZGXPqxFx64AQAADMfhFRgAAJAxRjw9ZC9UYAAAgOFQgQEAwKCcuABDAgMAgFG5uDhvBsMSEgAAMBwqMAAAGBRLSAAAwHA4hQQAAGAgVGAAADAoJy7AkMAAAGBULCEBAAAYCBUYAAAMypkrMCQwAAAYlBPnLywhAQAA46ECAwCAQTnzEhIVGAAAYDh2SWAuX76sI0eOKDk52R7DAQAAK5hM9ruMxuYEJjY2VqNGjdLq1aslSV9//bVefvlldejQQa1bt9alS5fsHiQAAEjLZDLZ7TIamxOYmTNnatu2bcqXL58kacaMGfL391dYWJhy5sypGTNm2D1IAACAh9m8iXfnzp0aOXKkWrdurd9//11//fWXhg8frkaNGikpKUnjx4/PjDgBAMAjDFg4sRubKzA3btxQqVKlJEm7d+9Wzpw5VbduXUlSvnz5lJCQYN8IAQBAurLLEtKNGzc0btw4BQcHq3r16nr99dd18ODBx/a/cOGC+vTpo+rVq6tevXqaM2eOzftobU5gihUrphMnTkiSduzYoapVqypPnjyS7ic0fn5+tg4JAAAMLDQ0VJGRkZo1a5Y+//xzlS9fXj179tTp06fT9L1375569uwpSVq7dq0mTJigTz75RAsWLLBpTpsTmE6dOmnatGlq2bKljh07pjfeeEOSNGDAAC1fvlydOnWydUgAAJAB2eEU0rlz5/Tjjz9qwoQJqlmzpkqWLKmxY8eqSJEi2rJlS5r+27Zt08WLF/XBBx+obNmyaty4sUJDQ7VixQolJiZaPa/Ne2C6deumggUL6sCBAxowYIBatmwpScqVK5cmTJigv//977YOCQAAMiA7nB7y9vbWkiVLVLlyZXPbg2WpW7dupel/8OBBVaxY0XwYSJJeeuklxcbG6tixYwoICLBq3gw9ibd169Zq3bq1Rdvs2bMzMhQAAMgGGjVq9MT7O3fuTLc9b968atCggUXbtm3bdO7cOY0ePTpN/6ioKPn6+lq0FSlSRJJ06dKlzE1gzpw5o927dysuLk4pKSkW90wmk/r375+RYQEAgA2yQQEmjUOHDmnUqFFq2rSpGjZsmOZ+fHy88ubNa9Hm5uYmSTYdBLI5gdm0aZNGjhyp1NTUdO+TwAAAkDXsuYT0uAqLLXbs2KF3331X1atXf+xz4dzd3dPsdXmQuHh6elo9l80JTHh4uOrUqaP33ntPvr6+2WL9DQAAONaqVas0ZcoUNW/eXNOnT5erq2u6/Xx9fXXy5EmLtpiYGEmSj4+P1fPZfArp4sWLeuutt1S0aFGSFwAAHCg7nEKSpDVr1mjy5Mnq3LmzZs2a9djkRZICAwN19OhRxcbGmtt++ukn5c6dW/7+/lbPaXMCU7JkSd53BABANpAdHmR35swZvf/++2rSpIn69OmjK1eu6PLly7p8+bJu376txMREXb582bxs1LhxYxUuXFiDBw/W8ePHtWPHDs2aNUv/+Mc/npj4PMrmBGbo0KEKDw/X/v37eeouAABObtu2bbp37562b9+uevXqWVxTpkxRZGSk6tWrp8jISEn3N+z++9//VkpKiv72t79p4sSJeuONN9SvXz+b5jWlPm437mM0bdpUly9fVnx8fPoDmkw6evSoTUFkhfgkR0cAGM+XR6i2AhnRIaBolsxTb8b3dhvrh3fr222srGDzJt42bdpkRhwAAABWszmBGTBgQGbEAQAAbOTMh2ky9CC7xMREff7554qIiNCtW7fk7e2tmjVrqm3btnJ3d7d3jAAAIB0kMDa4deuWunbtquPHj+u5555T4cKFdebMGX355ZdavXq11qxZIy8vr8yIFQAAQFIGEpiZM2cqKipKq1atUs2aNc3tBw8e1KBBgzR37lyNGTPGrkECAIC0nLgAY/sx6p07d2rw4MEWyYsk1axZU4MGDdK3335rt+AAAMDjZYfnwDiKzQnMnTt39Pzzz6d77/nnn9eNGzeeNSYAAIAnsjmBKVWqlHbt2pXuvV27dql48eLPHBQAAHi67PIqAUeweQ9Mz549NXToUCUnJ6tVq1YqVKiQrly5oi+//FKffvqpxo8fnxlxAgCARxhx6cdebE5gWrZsqbNnz2rRokVau3atJCk1NVWurq7q16+f/v73v9s9SAAAgIdl6Dkw/fr105tvvqnIyEjdunVL+fLlU0BAgPLly2fv+AAAwGM4cQEmYwmMJOXNm1cNGjSwZywAAMAGLk6cwViVwJQvX17r1q1TlSpV5O/v/8Q1t+z6MkcAAPC/w6oEpn///vLx8TH/2Zk3DQEAkF04869jqxKYh1/gOHDgwCf2jYqKeraIAACAVZy5oGDzc2DKly+vX3/9Nd17Bw8eVIsWLZ45KAAAgCexqgKzbNkyxcXFSbp/ZPqzzz7Tnj170vSLjIyUq6urfSMEAADpcnHeAox1CUxCQoLCwsIk3S9XffbZZ2n6uLi4yMvLS2+//bZ9IwQAAHiEVQnM22+/bU5M/P399emnn6pKlSqZGhgAAHgyZ94DY/NzYI4fP/7E+6mpqU79AwUAIKs486/bDD3IbuvWrYqIiFBiYqJSU1Ml3U9c4uLidPjw4XT3xwAAANiLzQlMWFiYwsLC5OXlpaSkJOXKlUs5c+bUtWvX5OLioo4dO2ZGnAAA4BEmOW8JxuZj1Bs3blTbtm0VERGh7t276+WXX9bevXu1fv165c+fXy+++GJmxAkAAB7hYrLfZTQ2JzDR0dF65ZVXZDKZVL58eUVGRkqSKlWqpL59+6Z7QgkAAMCebF5C8vT0NG/SLV68uC5cuKD4+Hi5u7urfPnyunDhgt2DBAAAaTnzoRmbKzCVK1fWF198IUkqWbKkcuTIoX379kmSTp06xYPsAADIIiaT/S6jsbkC07dvX/Xo0UO3bt3SokWL1KZNG40YMUK1atXSDz/8oMaNG2dGnAAAAGY2JzCBgYFav369Tpw4IUkaN26cXFxcdOjQITVv3lyjRo2ye5AAACAtFyOWTuzE5gTm4sWLKl26tPz9/SVJbm5umjx5sqT7rxw4cuSIqlevbt8oAQBAGk6cv9i+B6ZRo0Y6duxYuvd+/fVX9ejR45mDAgAAeBKrKjDTp0/XjRs3JN1/4m54eLi8vb3T9Dt27Ji8vLzsGiAAAEifM59CsiqBKVWqlBYuXCjp/g/r999/T3PaKEeOHPLy8mIPDAAAWcSJ8xfrEpiOHTuaXxEQEhKi8PBw8x4YAACArGbzJt7vvvvO/OdTp07p9u3b8vb2VvHixe0aGAAAeDJOIdnoyy+/1PTp03XlyhVzW6FChTR06FC1bdvWXrEBAACkK0MVmGHDhumll15SaGioChUqpJiYGG3evFmjRo1S/vz51bBhw0wIFQAAPMx56y8ZSGAWLlyo5s2ba/bs2Rbtr732moYMGaLFixeTwAAAkAWc+RSSzc+BOXnypNq1a5fuvXbt2un48ePPHBQAAMCT2FyB8fb21s2bN9O9d+PGDV7mCABAFnFx3gKM7RWY2rVrKywsTFFRURbtly5d0oIFC1S3bl27BQcAAB7PZDLZ7TIamyswoaGheu2119S0aVNVq1ZNhQoV0pUrVxQZGal8+fJp6NChmREnAACAmc0VmMKFC2vjxo3q0qWL7t69q99//113795Vly5dtHHjRhUrViwz4gQAAI8wmex3GU2GngNTsGBBDRs2zN6xAAAAGxhx6cdeMpTAnDlzRrt371ZcXJxSUlIs7plMJvXv398uwQEAAKTH5gRm06ZNGjlypFJTU9O9TwIDAEDWcOZTSDYnMOHh4apTp47ee+89+fr6OnX5CgAAR3Lm38E2b+K9ePGi3nrrLRUtWtSpf3AAAMBxbK7AlCxZUpcuXcqMWAAAgA2cuYxgcwVm6NChCg8P1/79+5WQkJAZMQEAACu4mEx2u4zG5grMlClTdPXqVXXv3j3d+yaTSUePHn3WuAAAAB7L5gSmTZs2mREHAACwkQELJ3ZjcwIzYMCAzIgDAADAahl6kF1qaqqOHTumuLi4dJ8HExgY+MyBAQCAJ3Pm08A2JzC//vqr3nnnHfPbqB8kMCaTSampqTKZTDp27Jh9owQAAGk4cf5iewIzdepU5cyZU1OnTpWvr69cXGw+yAQAAP5HLV68WD/88IM+/vjjx/bZvHlzuu9U3Llzp/z8/Kyax+YE5siRI5o1a5YaN25s67cCAAA7ym7Hn1evXq05c+aoZs2aT+x34sQJBQUFadasWRbtBQoUsHoumxOYggULKkeOHLZ+GwAAsLPskr9ER0dr/Pjx2r9/v0qUKPHU/idPnlS5cuVUuHDhDM9p8/rPG2+8ocWLFysuLi7DkwIAgP8dR44cUa5cubR582YFBAQ8tf+JEydUunTpZ5rT5grMuXPndOrUKdWtW1cvvvii3N3dLe6bTCatWLHimYICAABPZ89TSI0aNXri/Z07dz72XkhIiEJCQqya5+bNm4qOjtbBgwe1Zs0aXb9+XVWqVNGwYcNUsmRJq+PNUALj7+9v/vrRY9TpHasGYExduk9xdAiAIXWIDMuSeYx4jOaPP/6QdD9fmDp1quLj47Vw4UK98cYb2rJliwoVKmTVODYnME/aVfwgIAAAYCxPqrDYU82aNbVv3z55e3ubK0hhYWFq2LChNmzYoN69e1s1jt2St5iYGIWFhVldQgIAAM/GZDLZ7cpKBQoUsJjTw8NDfn5+io6OtnqMZ05gvv/+e/Xv318hISEKCwvjhBIAAFnExWS/K6usW7dOtWrVsjgMFBsbq7Nnz6pMmTJWj5OhBObatWtasmSJGjdurN69eysiIkLt2rXTqlWrtGPHjowMCQAA/gclJyfr8uXLio+PlyQFBwcrJSVFw4cP1x9//KHffvtNAwcOVIECBdS+fXurx7VpD8xPP/2kdevWaceOHUpOTlaNGjX0119/acGCBQoKCrLtEwEAgGeSlZWTjLp06ZIaNWqkqVOnqn379ipatKiWL1+umTNn6vXXX1dqaqrq1q2rlStXys3NzepxrUpgli9frnXr1unMmTMqXry4+vXrp3bt2snT01NBQUFO/TIpAAAcJTv+/p02bZrF135+fjpx4oRFW8WKFbVs2bJnmseqBGbatGkqV66cVq5caVFpuX379jNNDgAAkBFW7YFp1aqVzp07pz59+qhfv37avn27kpKSMjs2AADwBEbcxGsvVlVgZs6cqdjYWG3ZskUbNmzQwIED5e3trcaNGzvk+BUAAHBuVp9CypMnj15//XV99tln2rJli1599VV99913Sk1N1ejRozV37lz997//zcxYAQDAQ0wm+11GY0p9hkfnJiUladeuXfr888/1ww8/KDk5WS+++KI2b95szxjtIp4VL8Bm3oEDHB0CYEh3s+hVAiO3nrTbWNNalrXbWFnB5lcJWHxzzpxq0qSJmjRpoitXrmjjxo3auHGjvWIDAABIl91eJVCoUCH16tVLW7dutdeQAADgCVzseBnNM1VgAACA4xhx74q9GDHpAgAATo4KDAAABuXixCUYEhgAAAzKifMXlpAAAIDxWFWBOXDggE2DBgYGZigYAABgPSO+AsBerEpg+vXrp9jYWElSamrqY18d8ODesWPH7BchAABIF3tgnmLLli36xz/+oWvXrmn69Ony8PDI7LgAAAAey6oExtfXV0uXLlW7du30008/acSIEZkdFwAAeAonLsBYv4m3aNGiGj58uFavXq3o6OjMjAkAAFjBxWS/y2hsOkbdrl07VahQgSUkAADgUDYlMCaTSf7+/pkVCwAAsIFJBiyd2AnPgQEAAIZj85N44+PjtXDhQu3atUt3795VSkqKxX2TyaQdO3bYLUAAAJA+I+5dsRebE5gpU6Zo/fr1CgoKUvny5eXiQhEHAABHIIGxwbfffqshQ4aod+/emREPAADAU9mcwNy7d09VqlTJjFgAAIANHvdkfGdg8/pPvXr1tGfPnsyIBQAA2IDnwNigZcuWGj9+vK5du6aAgIB0nwnTtm1be8QGAACQLlNqamqqLd/wtOfAZNeXOcYnOToCwHi8Awc4OgTAkO5GhmXJPLP2nLbbWKHBpew2VlawuQKzc+fOzIgDAADYiLdR26BYsWJPvG9jQQcAAMBmNicwkrR161ZFREQoMTHRnLCkpqYqLi5Ohw8fZpMvAABZwIibb+3F5gQmLCxMYWFh8vLyUlJSknLlyqWcOXPq2rVrcnFxUceOHTMjTgAA8AgnXkGy/Rj1xo0b1bZtW0VERKh79+56+eWXtXfvXq1fv1758+fXiy++mBlxAgAAmNmcwERHR+uVV16RyWRS+fLlFRkZKUmqVKmS+vbtq88++8zuQQIAgLRcZLLbZTQ2JzCenp7mJ/8VL15cFy5cUHx8vCSpfPnyunDhgn0jBAAAeITNCUzlypX1xRdfSJJKliypHDlyaN++fZKkU6dOydXV1a4BAgCA9JlM9ruMxuZNvH379lWPHj1069YtLVq0SG3atNGIESNUq1Yt/fDDD2rcuHFmxAkAAB7BKSQbBAYGav369Tpx4oQkady4cXJxcdGhQ4fUvHlzjRw50u5BAgAAPCxDz4Hx9/c3v1LAzc1NkydPtmtQAADg6XgSr42io6P1+++/6/bt2+ne52WOAABkPifOX2xPYLZu3aqRI0cqMTEx3fsmk4kEBgAAZCqbE5g5c+aoSpUqGjVqlPLnz58JIQEAAGuwhGSDmJgYTZo0SRUrVsyMeAAAgJWcOH+x/TkwVatW1fHjxzMjFgAAAKvYXIEZP368+vbtq9jYWFWuXFmenp5p+gQGBtolOAAA8Hg2VyH+h9icwJw9e1ZXrlxRWFiYJJlfKyBJqampMplMOnbsmP0iBAAA6TI58RqSzQnM9OnT9cILL6hXr14qVKhQZsQEAADwRDYnMBcvXtSiRYtUp06dzIgHAABYyXnrLxlIYMqWLatLly5lRiwAAMAGHKO2wahRo/Tuu+8qOTlZVatWVZ48edL0ee655+wSHAAAQHpsTmB69OihpKQkjRs37rGbh9jECwBA5nPe+ksGEpgJEyY49a5nAADgeDYnMO3bt8+MOAAAgI2cuZ5gcwJz4MCBp/bhQXYAAGQ+Z14RsTmB6dKli0wmk1JTU81tj/4A2QMDAAAyk80JzMqVK9O0xcXF6eDBg9q0aZPmz59vl8AAAMCTZcdXCSxevFg//PCDPv7448f2uX79ut577z3t2bNHJpNJrVq10vDhw+Xh4WH1PDYnMEFBQem2N2zYUJ6enlq4cKEWL15s67AAAMBG2W0JafXq1ZozZ45q1qz5xH6DBg3S3bt3tXz5ct26dUv//Oc/FRcXp+nTp1s9l12Tt5o1ayoiIsKeQwIAgGwuOjpaffv21YwZM1SiRIkn9o2MjFRERISmT5+uihUrqnbt2po0aZI2bdqk6Ohoq+e0awLz3XffKXfu3PYcEgAAPIbJjtezOHLkiHLlyqXNmzcrICDgiX0PHjyowoULq3Tp0ua2oKAgmUwm/fzzz1bPafMSUteuXdO0paSkKCoqSn/99Zd69epl65AAACAD7LmE1KhRoyfe37lz52PvhYSEKCQkxKp5oqOjVbRoUYs2V1dX5c+f36ZXFdmcwDx8+ugBFxcXlS1bVn369NFrr71m65AAAMBJ3L17V66urmna3dzclJCQYPU4NicwT9pVDAAAso4994E8qcJiT+7u7kpMTEzTnpCQIE9PT6vHsTmBeeDOnTu6ffu2UlJS0tzjZY4AAGS+7HYKyRq+vr7asWOHRVtiYqJu3LihIkWKWD2OzQnMn3/+qdDQUB05cuSxfXiQHQAASE9gYKBmzJihc+fOqXjx4pJkPsFco0YNq8exOYGZOHGizp8/r759+8rPz08uLtnxMToAAPzvM0L9JTk5WdeuXZOXl5fc3d0VEBCg6tWra8iQIZowYYLi4uI0btw4tW3bVj4+PlaPa3MCc+jQIY0fP15t27a19VsBAIAdGWEF6dKlS2rUqJGmTp2q9u3by2QyKSwsTBMnTlS3bt3k5uam5s2ba9SoUTaNa3MCkzt3bhUuXNjWbwMAAE5g2rRpFl/7+fnpxIkTFm0FCxbUvHnznmkem9d/Xn31Va1cuVLJycnPNDEAAHg2LjLZ7TIaqyowD5d1kpKS9P3336tJkyaqUqVKmhcvmUwmvf/++/aNEgAA4CFWJTD79++3+NrX11eS9Ouvv6bpa8QjXQAAGJEz/8q1KoH57rvvMjsOAABgI5MBl37shTPQAADAcDL8JF4AAOBYLCEBAADDMeLpIXthCQkAABgOFRgAAAyKJSQAAGA4zpzAsIQEAAAMhwoMAAAG5czPgSGBAQDAoFycN39hCQkAABgPFRgAAAyKJSQAAGA4nEICAAAwECowAAAYlDMvIVGBAQAAhkMCA4fZ++MPeuNvr6lWjQC1aBqiFR8tVWpqqqPDArI1k8mkwV0a6bdN43Rt3yztXzdSnVrUdHRYcBAXk/0uo2EJCQ7x6y+HNbBfXzVr0UL9B76jyEM/a/bMD5WUlKyevXo7Ojwg2xr3diuFdm+syQu/0sEj59S8XkV99H53paSm6tNvfnZ0eMhizryERAIDhwgPmy//8uX1/rQPJUl16wfrXlKSlv5rkTp36Sp3d3cHRwhkPx7uuTSg88tasOY/mvHRdknSfyJOqlr5F9Tv9YYkMHAqLCEhyyUmJurggf0KadTEor1J02a6c+eOIg/xP2EgPQmJSXq5+0zN/XinRXvivSS5ufL3UWdkMtnvMhoSGGS5C+fP6969eypeooRF+wsvFJcknT1zxgFRAdlfSkqqfv/joqKv3pYkFSngpXd7NFFIrXJa8un3Do4OjmCy42U0pOzIcrGx9//nmydPHot2z9y5JUl37sRmeUyA0fyteQ2tmNpDkrR1z+/6ZOsBB0cEZC0qMMhyKSkpT7xvMvGfJfA0B34/p8Y9Z2vItE9Vu2opbV7Qz9EhwQFcTCa7XUZDBQZZLo+XlyTpzp07Fu13Yu9XXry88qT5HgCWzly4ojMXrujHQ6d06068lk7uqrrVS+vHQ6ccHRqykPHSDvtxaALTpUsXmazM+lauXJnJ0SCrPP/8C8qRI4fO/3nOov3PP/+UJJUsVdoRYQHZXiHvPGpat4K2/3hUl6//31Lr4WPnJUnPFc7voMiArOfQWn29evV08OBBXb16VcWKFXvihf8dbm5uql6jpnbu2G7x4Lod27fJy8tLlSpXcWB0QPbl4ZZLSyd3Vbd2dSzaG9cuL0n67Y+/HBEWHMmJd/E6tALTp08f5cmTRzNnztTixYvl5+fnyHCQhXr1eVt93uqhYaHvqG3713Q4MlIrPlqqd4YMlYeHh6PDA7Kl81HXtfyLvRrdq7mS7iXr8InzqlutjN7t0UQfbdyr46ejHB0ispgzP8jOlJoNnt3et29fubq6at68eZk2R3xSpg2NDNq5Y7sWLpins2fOqIiPj/7+emd16/4PR4eFh3gHDnB0CHhErpw5NKRbY735Si29UNRbF6JuaNmGHzV75U5exZGN3I0My5J59p+6abexapXOZ7exskK2SGBiYmJ05MgRvfzyy5k2BwkMYDsSGCBjsiqBiThtvwQmqJSxEphscQqpSJEiKlKkiKPDAADAUJx3AYnnwAAAAAPKFhUYAACQAU5cgqECAwAADIcKDAAABuXMx6hJYAAAMCgDvsLIblhCAgAAhkMFBgAAg3LiAgwJDAAAhuXEGQxLSAAAwHCowAAAYFCcQgIAAIbDKSQAAAADoQIDAIBBOXEBhgQGAADDcuIMhiUkAABgOFRgAAAwKE4hAQAAw+EUEgAAgIFQgQEAwKCcuABDBQYAABgPFRgAAIzKiUswVGAAADAokx3/eRYpKSmaN2+e6tevr6pVq6pXr146f/78Y/tv3rxZ5cqVS3NduHDB6jmpwAAAgGcSHh6uNWvWaNq0afL19dWHH36ot956S1u2bJGrq2ua/idOnFBQUJBmzZpl0V6gQAGr56QCAwCAQZlM9rsyKjExUcuWLdOgQYPUsGFD+fv7a/bs2YqKitK3336b7vecPHlS5cqVU+HChS2uHDlyWD0vCQwAAAZlsuOVUcePH9edO3dUu3Ztc1vevHlVoUIFHThwIN3vOXHihEqXLv0Ms7KEBAAAJDVq1OiJ93fu3Jlue1RUlCSpaNGiFu1FihQx33vYzZs3FR0drYMHD2rNmjW6fv26qlSpomHDhqlkyZJWx0sFBgAAo8oGJZi7d+9KUpq9Lm5ubkpISEjT/48//pAkpaamaurUqZozZ44SEhL0xhtv6MqVK1bPSwUGAACDsue7kB5XYXkad3d3Sff3wjz4syQlJCTIw8MjTf+aNWtq37598vb2lun/b74JCwtTw4YNtWHDBvXu3duqeanAAACADHuwdBQTE2PRHhMTIx8fn3S/p0CBAubkRZI8PDzk5+en6Ohoq+clgQEAwKCywykkf39/5cmTR/v37ze33bp1S0ePHlVgYGCa/uvWrVOtWrUUFxdnbouNjdXZs2dVpkwZq+clgQEAwKCywRYYubq66s0339SMGTO0c+dOHT9+XEOGDJGvr6+aNm2q5ORkXb58WfHx8ZKk4OBgpaSkaPjw4frjjz/022+/aeDAgSpQoIDat29v9bwkMAAA4JkMGjRIHTp00JgxY/T6668rR44cWrp0qXLlyqVLly6pXr162rp1q6T7S07Lly9XXFycXn/9dXXv3l1eXl5auXKl3NzcrJ7TlJqamppZHyg7iU9ydASA8XgHDnB0CIAh3Y0My5J5TkbHPb2Tlcr6eNptrKzAKSQAAAzKnqeQjIYlJAAAYDhUYAAAMKhnOT1kdFRgAACA4VCBAQDAoJy4AEMCAwCAYTlxBsMSEgAAMBwqMAAAGJQzH6MmgQEAwKA4hQQAAGAgVGAAADAoJy7AkMAAAGBYTpzBsIQEAAAMhwoMAAAGxSkkAABgOJxCAgAAMBAqMAAAGJQTF2BIYAAAMCqWkAAAAAyECgwAAIblvCUYKjAAAMBwqMAAAGBQzrwHhgQGAACDcuL8hSUkAABgPFRgAAAwKJaQAACA4Tjzu5BYQgIAAIZDBQYAAKNy3gIMCQwAAEblxPkLS0gAAMB4qMAAAGBQnEICAACGwykkAAAAA6ECAwCAUTlvAYYKDAAAMB4qMAAAGJQTF2BIYAAAMCpnPoXEEhIAADAcKjAAABiUMx+jJoEBAMCgWEICAAAwEBIYAABgOCwhAQBgUCwhAQAAGAgVGAAADIpTSAAAwHBYQgIAADAQKjAAABiUExdgSGAAADAsJ85gWEICAACGQwUGAACDcuZTSFRgAACA4VCBAQDAoJz5GDUJDAAABuXE+QtLSAAA4NmkpKRo3rx5ql+/vqpWrapevXrp/Pnzj+1//fp1DR06VIGBgQoKCtLEiRN19+5dm+YkgQEAwKhMdryeQXh4uNasWaPJkydr7dq1SklJ0VtvvaXExMR0+w8aNEjnzp3T8uXLNXfuXO3evVsTJkywaU4SGAAADMpkx38yKjExUcuWLdOgQYPUsGFD+fv7a/bs2YqKitK3336bpn9kZKQiIiI0ffp0VaxYUbVr19akSZO0adMmRUdHWz0vCQwAAMiw48eP686dO6pdu7a5LW/evKpQoYIOHDiQpv/BgwdVuHBhlS5d2twWFBQkk8mkn3/+2ep52cQLAIBB2fMUUqNGjZ54f+fOnem2R0VFSZKKFi1q0V6kSBHzvYdFR0en6evq6qr8+fPr0qVLVsfrNAmMu9N8UsB+7kaGOToEAE+QHX63Pdh86+rqatHu5uammzdvptv/0b4P+ickJFg9bzb46AAAwNEeV2F5Gnd3d0n398I8+LMkJSQkyMPDI93+6W3uTUhIkKenp9XzsgcGAABk2IPloJiYGIv2mJgY+fj4pOnv6+ubpm9iYqJu3LihIkWKWD0vCQwAAMgwf39/5cmTR/v37ze33bp1S0ePHlVgYGCa/oGBgYqKitK5c+fMbREREZKkGjVqWD0vS0gAACDDXF1d9eabb2rGjBkqUKCAihUrpg8//FC+vr5q2rSpkpOTde3aNXl5ecnd3V0BAQGqXr26hgwZogkTJiguLk7jxo1T27Zt063YPI4pNTU1NRM/FwAA+B+XnJysWbNmacOGDYqPj1dgYKDGjRsnPz8/XbhwQY0aNdLUqVPVvn17SdLVq1c1ceJEff/993Jzc1Pz5s01atQoubm5WT0nCQwAADAc9sAAAADDIYEBAACGQwIDAAAMhwQGAAAYDgkMAAAwHBIYAABgOCQwcIiUlBTNmzdP9evXV9WqVdWrVy+dP3/e0WEBhrJ48WJ16dLF0WEADkECA4cIDw/XmjVrNHnyZK1du1YpKSl666230n3BF4C0Vq9erTlz5jg6DMBhSGCQ5RITE7Vs2TINGjRIDRs2lL+/v2bPnq2oqCh9++23jg4PyNaio6PVt29fzZgxQyVKlHB0OIDDkMAgyx0/flx37txR7dq1zW158+ZVhQoVdODAAQdGBmR/R44cUa5cubR582YFBAQ4OhzAYXiZI7JcVFSUpP97BfsDRYoUMd8DkL6QkBCFhIQ4OgzA4ajAIMvdvXtX0v03mD7Mzc1NCQkJjggJAGAwJDDIcu7u7pKUZsNuQkKCPDw8HBESAMBgSGCQ5R4sHcXExFi0x8TEyMfHxxEhAQAMhgQGWc7f31958uTR/v37zW23bt3S0aNHFRgY6MDIAABGwSZeZDlXV1e9+eabmjFjhgoUKKBixYrpww8/lK+vr5o2bero8AAABkACA4cYNGiQkpKSNGbMGMXHxyswMFBLly5Vrly5HB0aAMAATKmpqamODgIAAMAW7IEBAACGQwIDAAAMhwQGAAAYDgkMAAAwHBIYAABgOCQwAADAcEhgAACA4ZDAAMj2eFwVgEeRwACP0aVLF5UrV06dOnV6bJ8hQ4aoXLlyGjly5DPPt3//fpUrV87iHVH2+J4Hn+Phq1KlSmrYsKEmTpyomzdvPnPsj7NhwwaVK1dOFy5ckCTNnz9f5cqVs/r7o6Ki1Lt3b/3111/mtpCQELv8vAEYG68SAJ7AxcVFhw8fVlRUlHx9fS3uxcXFadeuXQ6KzDYVKlTQ+PHjzV/fu3dPR44c0axZs3Ts2DF98sknMplMmR5Hx44dVb9+fav77927V7t377ZoCwsLU548eewdGgCDIYEBnqBChQr673//q2+++Ubdu3e3uLdr1y55eHgob968jgnOBnny5FHVqlUt2gIDA3Xnzh3NmzdPv/zyS5r7mcHX1zdNImirChUq2CkaAEbGEhLwBJ6enmrQoIG++eabNPe2bt2qZs2aKWdOy78HJCQkaMGCBWrevLkqV66spk2basmSJUpJSbHot3btWjVr1kxVqlTRm2++qYsXL6aZ4+LFiwoNDVVQUJACAgLUrVs3HT161G6fr1KlSuZ5pPvLTe+++64GDRqkqlWrqkePHubP9MEHH6hBgwaqVKmSXnnlFW3dutVirJSUFIWHh6thw4YKCAhQv3790ixPpbeE9MUXX6hdu3YKCAhQw4YNNXPmTCUmJmrDhg0aNWqUJKlRo0bmZaNHl5Bu376tqVOnqnHjxqpcubJat26t9evXW8wREhKiefPmafr06apTp46qVKminj176uzZs+Y+165d09ChQ1W3bl1VrlxZr776qr744osM/mQBZDYqMMBTtGzZUoMHD7ZYRoqNjdWePXv00Ucfac+ePea+qamp6tu3rw4fPqwBAwbI399f+/fv15w5c3T+/HlNnjxZkrRq1SpNnjxZ3bp1U3BwsPbt26exY8dazHvt2jV16tRJHh4eGjt2rDw8PLRixQp17txZ69evV+nSpZ/5s505c0aS9Pzzz5vbvv76a7Vp00YLFy5USkqKUlNT1b9/fx06dEiDBg1S6dKltX37dg0ZMkSJiYlq27atJOnDDz/UypUr9fbbbysgIEBff/21Zs6c+cT5V69erUmTJqljx44KDQ3V+fPn9cEHH+jmzZsaPHiw3n77bS1cuFBhYWHp7p2Jj4/XG2+8oatXr2rQoEEqVqyYduzYoX/+85+6cuWK+vbta+67cuVK1ahRQ1OnTtXNmzc1ZcoUjRgxQuvWrZMkDRs2TFevXtXEiROVJ08ebdq0SSNGjJCvr69eeumlZ/1RA7AzEhjgKRo2bCgPDw+LZaTt27erYMGCqlGjhkXfPXv2aO/evZo1a5ZatWolSapbt67c3d01d+5cde3aVWXKlFF4eLhatmyp0aNHS5Lq1aun2NhYrV271jzWihUrdOPGDX3yyScqVqyYJCk4OFgtW7bU3LlzNW/ePKs/Q2pqqpKSksxf37x5UxEREVq4cKGqVatmrsRIUq5cuTRx4kS5urpKkn788Ud9//33mj17tlq2bClJql+/vu7evasZM2aodevWiouL08cff6wePXpowIAB5j4xMTH6/vvv040pJSVFCxYsUOPGjfXee++Z2+/evauvvvpKXl5eeuGFFyRJ5cuXl5+fX5oxNmzYoJMnT2rt2rWqVq2aed6kpCSFh4erU6dOyp8/vyQpb968Cg8PV44cOSRJf/75p+bPn6/r16/L29tbERER6t+/vxo3bixJCgoKUv78+c0/BwDZC0tIwFO4u7srJCTEYhnpq6++UosWLdJsfI2IiFDOnDnVvHlzi/Y2bdqY758+fVpXr17Vyy+/bNGnRYsWFl/v27dP5cuXl4+Pj5KSkpSUlCQXFxcFBwdr7969Nn2GAwcOqGLFiuarTp06Cg0NVaVKlTRz5kyLz1GqVCmLX9r79u2TyWRSgwYNzHEkJSUpJCREly9f1h9//KHDhw/r3r17T/1MDztz5oyuXr2qJk2aWLT37NlTGzZsUK5cuZ76uSIiIlSsWDFz8vJAmzZtlJCQoF9++cXcVrlyZXPyIslcTbt7964kqVatWpo/f74GDRqkzz77TFeuXNGIESNUvXr1p8YBIOtRgQGs0KJFCw0YMEBRUVFyc3PTvn37NHjw4DT9bt68KW9vb4tflJJUuHBhSff3azzYF+Lt7Z1unwdu3Lihc+fOqWLFiunG9OAXrzUqVqyoiRMnSpJMJpPc3NxUtGjRdE/z5M6dO00cqampj/1FHhMTo1u3bkl6+md6dFxJKliwoNWf41E3b95Md45ChQpJkjkuSfLw8LDo4+Jy/+9vD/YmzZ49W4sWLdLXX3+tbdu2ycXFRXXq1NGkSZPMFTAA2QcJDGCF4OBg5c6dW9988408PT3l5+dnsezyQL58+XT9+nUlJydbJDExMTGS7v+Cf/BL/urVqxbf++AX+gNeXl4KCgrS8OHD043JlqWN3Llzq3Llylb3fzQOT09PrVy5Mt37xYsX16+//irp/mcqVaqU+d6jn+lhD05vXbt2zaL9+vXrOnr0aJqqSnry5cunc+fOpWm/fPmypLQJ1ZN4eXlp2LBhGjZsmE6fPq2dO3cqPDxcEydO1JIlS6weB0DWYAkJsIKrq6saN26sbdu26euvvzbvb3lUUFCQkpKS0pxa2rx5sySpRo0aKlGihIoWLZqmz6PPlAkKCtKZM2dUsmRJVa5c2Xxt2rRJ69evT1PlySxBQUGKi4tTamqqRRwnT57UggULlJSUpGrVqsnd3f2pn+lhpUqVkre3d5o+mzZtUu/evXXv3j1zleRxAgMD9ddffykyMtKiffPmzcqVK5eqVKli1Wf866+/LE6blSpVSr169VKdOnXSPR0GwPGowABWatmypfr06SMXFxeNGTMm3T7BwcGqVauWxowZo+joaPn7+ysiIkL/+te/1K5dO5UpU0aS9O6772ro0KEaM2aMmjdvrsOHD+uTTz6xGKt79+7atGmTunfvrn/84x/y9vbW1q1b9emnn5qPF2eFBg0aKDAwUP369VO/fv1UunRp/frrr5o3b57q16+vAgUKSJL69eunOXPmyMPDQy+99JJ27979xAQmR44cGjhwoCZNmqSCBQsqJCREZ86c0bx589S5c2fly5fPXKXZvn27goOD05y8at++vdasWaP+/ftr0KBB8vPz03fffafPP/9cAwYMsPoZPcWKFZOvr6/ee+89xcbG6oUXXtDvv/+u3bt3q0+fPhn8yQHITCQwgJXq1KmjvHnzqmjRoo89wmwymbR48WLNmzdPy5cv17Vr1+Tn56fQ0FDzM1UkqXXr1nJxcVF4eLg2bdqksmXLatKkSQoNDTX38fHx0dq1azVz5kxNmDBBCQkJKlGihKZMmaIOHTpk+ud9wMXFRUuWLNHcuXO1ePFiXb16VT4+PurRo4f69+9v7tenTx95enpqxYoVWrFihapVq6YRI0ZowoQJjx27c+fO8vT01NKlS7Vu3Tr5+vqqV69e6tWrl6T7G2vr1KmjmTNnat++fWmWcjw8PPTxxx9r5syZmjt3rmJjY1WqVKkM/YzCwsI0a9YszZ07V9evX1fRokU1YMAA9e7d26ZxAGQNUypvSQMAAAbDHhgAAGA4JDAAAMBwSGAAAIDhkMAAAADDIYEBAACGQwIDAAAMhwQGAAAYDgkMAAAwHBIYAABgOCQwAADAcEhgAACA4ZDAAAAAw/l/dD94D8BBoeYAAAAASUVORK5CYII=",
      "text/plain": [
       "<Figure size 600x500 with 2 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Plot confusion matrices for all comparisons\n",
    "plot_confusion_matrices(model_coding_short, human_annotations_short, labels=[0, 1])"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "gpt_rl",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.11.10"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}