examples/pyterrier_reranking_comparison.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "source": [
    "# Re-Ranking Comparison\n",
    "\n",
    "The notebook below showcases different axiomatic re-ranking approaches in `ir_axioms` using PyTerrier:\n",
    "\n",
    "1. KwikSort with manually combined axioms,\n",
    "2. KwikSort with an estimation of the ORACLE axiom, and\n",
    "3. aggregated axiom preferences as features for LamdaMART.\n",
    "\n",
    "We use run files and qrels from the passage retrieval task of the TREC Deep Learning track in 2019 and 2020 as example (using BM25 as a baseline).\n",
    "In this notebook, we evaluate nDCG@10, reciprocal rank, and average precision for the baseline and the re-ranked pipelines using PyTerrier's standard `Experiment` functionality."
   ],
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%% md\n"
    }
   }
  },
  {
   "cell_type": "markdown",
   "source": [
    "## Preparation\n",
    "\n",
    "Install the `ir_axioms` framework and [PyTerrier](https://github.com/terrier-org/pyterrier). In Google Colab, we do this automatically."
   ],
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%% md\n"
    }
   }
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "outputs": [],
   "source": [
    "from sys import modules\n",
    "\n",
    "if 'google.colab' in modules:\n",
    "    !pip install -q ir_axioms[examples] python-terrier"
   ],
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%%\n"
    }
   }
  },
  {
   "cell_type": "markdown",
   "source": [
    "We initialize PyTerrier and import all required libraries and load the data from [ir_datasets](https://ir-datasets.com/)."
   ],
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%% md\n"
    }
   }
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "PyTerrier 0.8.0 has loaded Terrier 5.6 (built by craigmacdonald on 2021-09-17 13:27)\n",
      "\n",
      "No etc/terrier.properties, using terrier.default.properties for bootstrap configuration.\n"
     ]
    }
   ],
   "source": [
    "from pyterrier import started, init\n",
    "\n",
    "if not started():\n",
    "    init(tqdm=\"auto\")"
   ],
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%%\n"
    }
   }
  },
  {
   "cell_type": "markdown",
   "source": [
    "## Datasets and Index\n",
    "Using PyTerrier's `get_dataset()`, we load the MS MARCO passage ranking dataset."
   ],
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%% md\n"
    }
   }
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "outputs": [],
   "source": [
    "from pyterrier.datasets import get_dataset, Dataset\n",
    "\n",
    "# Load dataset.\n",
    "dataset_name = \"msmarco-passage\"\n",
    "dataset: Dataset = get_dataset(f\"irds:{dataset_name}\")\n",
    "dataset_train: Dataset = get_dataset(f\"irds:{dataset_name}/trec-dl-2019/judged\")\n",
    "dataset_test: Dataset = get_dataset(f\"irds:{dataset_name}/trec-dl-2020/judged\")"
   ],
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%%\n"
    }
   }
  },
  {
   "cell_type": "markdown",
   "source": [
    "Now define paths where we will store temporary files, datasets, and the search index."
   ],
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%% md\n"
    }
   }
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "outputs": [],
   "source": [
    "from pathlib import Path\n",
    "\n",
    "cache_dir = Path(\"cache/\")\n",
    "index_dir = cache_dir / \"indices\" / dataset_name.split(\"/\")[0]\n",
    "results_dir = Path(\"results\")"
   ],
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%%\n"
    }
   }
  },
  {
   "cell_type": "markdown",
   "source": [
    "If the index is not ready yet, now is a good time to create it and index the MS MARCO passages.\n",
    "(Lean back and relax as this may take a while...)"
   ],
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%% md\n"
    }
   }
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "outputs": [],
   "source": [
    "from pyterrier.index import IterDictIndexer\n",
    "\n",
    "if not index_dir.exists():\n",
    "    indexer = IterDictIndexer(str(index_dir.absolute()))\n",
    "    indexer.index(\n",
    "        dataset.get_corpus_iter(),\n",
    "        fields=[\"text\"]\n",
    "    )"
   ],
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%%\n"
    }
   }
  },
  {
   "cell_type": "markdown",
   "source": [
    "## Baseline Run\n",
    "\n",
    "We use PyTerrier's `BatchRetrieve` to create a baseline search pipeline for retrieving with BM25 from the index we just created."
   ],
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%% md\n"
    }
   }
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "outputs": [],
   "source": [
    "from pyterrier.batchretrieve import BatchRetrieve\n",
    "\n",
    "bm25 = BatchRetrieve(str(index_dir.absolute()), wmodel=\"BM25\")"
   ],
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%%\n"
    }
   }
  },
  {
   "cell_type": "markdown",
   "source": [
    "## Combine and Import Axioms\n",
    "Here we're listing which axioms we want to use in our experiments."
   ],
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%% md\n"
    }
   }
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "outputs": [],
   "source": [
    "from ir_axioms.axiom import (\n",
    "    ArgUC, QTArg, QTPArg, aSL, PROX1, PROX2, PROX3, PROX4, PROX5, TFC1, TFC3, RS_TF, RS_TF_IDF, RS_BM25, RS_PL2, RS_QL,\n",
    "    AND, LEN_AND, M_AND, LEN_M_AND, DIV, LEN_DIV, M_TDC, LEN_M_TDC, STMC1, STMC1_f, STMC2, STMC2_f, LNC1, TF_LNC, LB1,\n",
    "    REG, ANTI_REG, REG_f, ANTI_REG_f, ASPECT_REG, ASPECT_REG_f, ORIG\n",
    ")"
   ],
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%%\n"
    }
   }
  },
  {
   "cell_type": "markdown",
   "source": [
    "First, we combine many of the axioms implemented in `ir_axioms` to form a majority voting.\n",
    "That is, we only want to keep preferences, where at least 50% (or 0.5) of the axioms agree.\n",
    "Because some axioms require API calls or are computationally expensive, we cache the voting result using the tilde operator (`~`).\n",
    "We are going to use that vote axiom in a `KwikSortReranker` later."
   ],
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%% md\n"
    }
   }
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "outputs": [],
   "source": [
    "from ir_axioms.axiom import VoteAxiom\n",
    "\n",
    "majority_vote_axiom = (\n",
    "        ~VoteAxiom([\n",
    "            ArgUC(), QTArg(), QTPArg(), aSL(),\n",
    "            LNC1(), TF_LNC(), LB1(),\n",
    "            PROX1(), PROX2(), PROX3(), PROX4(), PROX5(),\n",
    "            REG(), REG_f(), ANTI_REG(), ANTI_REG_f(), ASPECT_REG(), ASPECT_REG_f(),\n",
    "            AND(), LEN_AND(), M_AND(), LEN_M_AND(), DIV(), LEN_DIV(),\n",
    "            RS_TF(), RS_TF_IDF(), RS_BM25(), RS_PL2(), RS_QL(),\n",
    "            TFC1(), TFC3(), M_TDC(), LEN_M_TDC(),\n",
    "            STMC1(), STMC1_f(), STMC2(), STMC2_f(),\n",
    "        ], minimum_votes=0.5) | ORIG()\n",
    ")"
   ],
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%%\n"
    }
   }
  },
  {
   "cell_type": "markdown",
   "source": [
    "Then, for estimating the ORACLE axiom and for generating axiomatic features for learning to rank with LambdaMART, we define a list of all axioms that we want to use in our experiments.\n",
    "Again, we implement caching for the axioms (using `~`)."
   ],
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%% md\n"
    }
   }
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "outputs": [],
   "source": [
    "all_axioms = [\n",
    "    ~ArgUC(), ~QTArg(), ~QTPArg(), ~aSL(),\n",
    "    ~LNC1(), ~TF_LNC(), ~LB1(),\n",
    "    ~PROX1(), ~PROX2(), ~PROX3(), ~PROX4(), ~PROX5(),\n",
    "    ~REG(), ~REG_f(), ~ANTI_REG(), ~ANTI_REG_f(), ~ASPECT_REG(), ~ASPECT_REG_f(),\n",
    "    ~AND(), ~LEN_AND(), ~M_AND(), ~LEN_M_AND(), ~DIV(), ~LEN_DIV(),\n",
    "    ~RS_TF(), ~RS_TF_IDF(), ~RS_BM25(), ~RS_PL2(), ~RS_QL(),\n",
    "    ~TFC1(), ~TFC3(), ~M_TDC(), ~LEN_M_TDC(),\n",
    "    ~STMC1(), ~STMC1_f(), ~STMC2(), ~STMC2_f(),\n",
    "    ORIG()\n",
    "]"
   ],
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%%\n"
    }
   }
  },
  {
   "cell_type": "markdown",
   "source": [
    "## Re-ranking Approaches\n",
    "We will now compare the three different axiomatic re-ranking approaches.\n",
    "Please refer to the other notebooks in this repository for more detailed explanations of each of the approaches."
   ],
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%% md\n"
    }
   }
  },
  {
   "cell_type": "markdown",
   "source": [
    "### KwikSort Re-ranking\n",
    "For the first re-ranker, we re-rank the top-20 baseline results using the KwikSort algorithm, using our previously defined vote axiom."
   ],
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%% md\n"
    }
   }
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "outputs": [],
   "source": [
    "from ir_axioms.modules.pivot import MiddlePivotSelection\n",
    "from ir_axioms.backend.pyterrier.transformers import KwikSortReranker\n",
    "\n",
    "kwiksort = bm25 % 20 >> KwikSortReranker(\n",
    "    axiom=majority_vote_axiom,\n",
    "    index=index_dir,\n",
    "    dataset=dataset_name,\n",
    "    pivot_selection=MiddlePivotSelection(),\n",
    "    cache_dir=cache_dir,\n",
    "    verbose=True\n",
    ")"
   ],
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%%\n"
    }
   }
  },
  {
   "cell_type": "markdown",
   "source": [
    "### KwikSort Re-ranking with Estimating the ORACLE Axiom\n",
    "The second re-ranker works by estimating the ORACLE axiom using preferences from all reference axioms using a random forest classifier.\n",
    "The resulting output preferences are used with KwikSort to re-rank the top-20 baseline results."
   ],
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%% md\n"
    }
   }
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "outputs": [],
   "source": [
    "from sklearn.ensemble import RandomForestClassifier\n",
    "from ir_axioms.modules.pivot import MiddlePivotSelection\n",
    "from ir_axioms.backend.pyterrier.estimator import EstimatorKwikSortReranker\n",
    "\n",
    "random_forest = RandomForestClassifier(\n",
    "    max_depth=3,\n",
    ")\n",
    "kwiksort_random_forest = bm25 % 20 >> EstimatorKwikSortReranker(\n",
    "    axioms=all_axioms,\n",
    "    estimator=random_forest,\n",
    "    index=index_dir,\n",
    "    dataset=dataset_name,\n",
    "    pivot_selection=MiddlePivotSelection(),\n",
    "    cache_dir=cache_dir,\n",
    "    verbose=True,\n",
    ")"
   ],
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%%\n"
    }
   }
  },
  {
   "cell_type": "markdown",
   "source": [
    "We fit the estimator using preferences from the training dataset."
   ],
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%% md\n"
    }
   }
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "outputs": [
    {
     "data": {
      "text/plain": "Collecting axiom preferences:   0%|          | 0/699 [00:00<?, ?document pair/s]",
      "application/vnd.jupyter.widget-view+json": {
       "version_major": 2,
       "version_minor": 0,
       "model_id": "06dfe27470274744b3d9fd6bb32c85a7"
      }
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "kwiksort_random_forest.fit(\n",
    "    dataset_train.get_topics(),\n",
    "    dataset_train.get_qrels()\n",
    ")"
   ],
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%%\n"
    }
   }
  },
  {
   "cell_type": "markdown",
   "source": [
    "## Aggregating Axiomatic Features for LTR with LambdaMART\n",
    "For the third re-ranker, we aggregate axiomatic preferences into three features per axiom."
   ],
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%% md\n"
    }
   }
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "outputs": [],
   "source": [
    "from ir_axioms.backend.pyterrier.transformers import AggregatedAxiomaticPreferences\n",
    "\n",
    "aggregations = [\n",
    "    lambda prefs: sum(p >= 0 for p in prefs) / len(prefs),\n",
    "    lambda prefs: sum(p == 0 for p in prefs) / len(prefs),\n",
    "    lambda prefs: sum(p <= 0 for p in prefs) / len(prefs),\n",
    "]\n",
    "features = ~(bm25 % 20 >> AggregatedAxiomaticPreferences(\n",
    "    axioms=all_axioms,\n",
    "    index=index_dir,\n",
    "    aggregations=aggregations,\n",
    "    dataset=dataset_name,\n",
    "    cache_dir=cache_dir,\n",
    "    verbose=True,\n",
    "))"
   ],
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%%\n"
    }
   }
  },
  {
   "cell_type": "markdown",
   "source": [
    "After aggregating the preference features, we initialize a LambdaMART ranker for optimizing nDCG@10 and apply it to the top-20 baseline results."
   ],
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%% md\n"
    }
   }
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "outputs": [],
   "source": [
    "from lightgbm import LGBMRanker\n",
    "from pyterrier.ltr import apply_learned_model\n",
    "\n",
    "lambda_mart = LGBMRanker(\n",
    "    num_iterations=1000,\n",
    "    metric=\"ndcg\",\n",
    "    eval_at=[10],\n",
    "    importance_type=\"gain\",\n",
    ")\n",
    "ltr = features >> apply_learned_model(lambda_mart, form=\"ltr\")"
   ],
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%%\n"
    }
   }
  },
  {
   "cell_type": "markdown",
   "source": [
    "We also fit the re-ranker using preferences from the training dataset (using the last 5 topics as the validation dataset)."
   ],
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%% md\n"
    }
   }
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "outputs": [
    {
     "data": {
      "text/plain": "Aggregating query axiom preferences:   0%|          | 0/38 [00:00<?, ?query/s]",
      "application/vnd.jupyter.widget-view+json": {
       "version_major": 2,
       "version_minor": 0,
       "model_id": "1f7662107155467eb144711e68b94302"
      }
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/plain": "Aggregating query axiom preferences:   0%|          | 0/5 [00:00<?, ?query/s]",
      "application/vnd.jupyter.widget-view+json": {
       "version_major": 2,
       "version_minor": 0,
       "model_id": "94f50543fb0a4b049243a3f94602f3e0"
      }
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/home/heinrich/Development/webis/ir_axioms/venv/lib/python3.9/site-packages/lightgbm/sklearn.py:621: UserWarning: Found 'eval_at' in params. Will use it instead of 'eval_at' argument\n",
      "  _log_warning(f\"Found '{alias}' in params. Will use it instead of 'eval_at' argument\")\n",
      "/home/heinrich/Development/webis/ir_axioms/venv/lib/python3.9/site-packages/lightgbm/engine.py:177: UserWarning: Found `num_iterations` in params. Will use it instead of argument\n",
      "  _log_warning(f\"Found `{alias}` in params. Will use it instead of argument\")\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[1]\tvalid_0's ndcg@10: 0.499041\n",
      "[2]\tvalid_0's ndcg@10: 0.540851\n",
      "[3]\tvalid_0's ndcg@10: 0.56665\n",
      "[4]\tvalid_0's ndcg@10: 0.54274\n",
      "[5]\tvalid_0's ndcg@10: 0.554685\n",
      "[6]\tvalid_0's ndcg@10: 0.597619\n",
      "[7]\tvalid_0's ndcg@10: 0.567879\n",
      "[8]\tvalid_0's ndcg@10: 0.5714\n",
      "[9]\tvalid_0's ndcg@10: 0.562249\n",
      "[10]\tvalid_0's ndcg@10: 0.570762\n",
      "[11]\tvalid_0's ndcg@10: 0.589228\n",
      "[12]\tvalid_0's ndcg@10: 0.59746\n",
      "[13]\tvalid_0's ndcg@10: 0.633497\n",
      "[14]\tvalid_0's ndcg@10: 0.605597\n",
      "[15]\tvalid_0's ndcg@10: 0.601942\n",
      "[16]\tvalid_0's ndcg@10: 0.609102\n",
      "[17]\tvalid_0's ndcg@10: 0.608157\n",
      "[18]\tvalid_0's ndcg@10: 0.605237\n",
      "[19]\tvalid_0's ndcg@10: 0.616162\n",
      "[20]\tvalid_0's ndcg@10: 0.608126\n",
      "[21]\tvalid_0's ndcg@10: 0.581933\n",
      "[22]\tvalid_0's ndcg@10: 0.556621\n",
      "[23]\tvalid_0's ndcg@10: 0.547529\n",
      "[24]\tvalid_0's ndcg@10: 0.568303\n",
      "[25]\tvalid_0's ndcg@10: 0.566846\n",
      "[26]\tvalid_0's ndcg@10: 0.574184\n",
      "[27]\tvalid_0's ndcg@10: 0.589674\n",
      "[28]\tvalid_0's ndcg@10: 0.59666\n",
      "[29]\tvalid_0's ndcg@10: 0.609654\n",
      "[30]\tvalid_0's ndcg@10: 0.569011\n",
      "[31]\tvalid_0's ndcg@10: 0.577817\n",
      "[32]\tvalid_0's ndcg@10: 0.576675\n",
      "[33]\tvalid_0's ndcg@10: 0.568977\n",
      "[34]\tvalid_0's ndcg@10: 0.578097\n",
      "[35]\tvalid_0's ndcg@10: 0.580636\n",
      "[36]\tvalid_0's ndcg@10: 0.588973\n",
      "[37]\tvalid_0's ndcg@10: 0.584086\n",
      "[38]\tvalid_0's ndcg@10: 0.584686\n",
      "[39]\tvalid_0's ndcg@10: 0.598147\n",
      "[40]\tvalid_0's ndcg@10: 0.600644\n",
      "[41]\tvalid_0's ndcg@10: 0.595237\n",
      "[42]\tvalid_0's ndcg@10: 0.592427\n",
      "[43]\tvalid_0's ndcg@10: 0.594222\n",
      "[44]\tvalid_0's ndcg@10: 0.591437\n",
      "[45]\tvalid_0's ndcg@10: 0.602632\n",
      "[46]\tvalid_0's ndcg@10: 0.59881\n",
      "[47]\tvalid_0's ndcg@10: 0.600206\n",
      "[48]\tvalid_0's ndcg@10: 0.62777\n",
      "[49]\tvalid_0's ndcg@10: 0.603373\n",
      "[50]\tvalid_0's ndcg@10: 0.604893\n",
      "[51]\tvalid_0's ndcg@10: 0.604793\n",
      "[52]\tvalid_0's ndcg@10: 0.615026\n",
      "[53]\tvalid_0's ndcg@10: 0.614281\n",
      "[54]\tvalid_0's ndcg@10: 0.605717\n",
      "[55]\tvalid_0's ndcg@10: 0.593895\n",
      "[56]\tvalid_0's ndcg@10: 0.600675\n",
      "[57]\tvalid_0's ndcg@10: 0.59551\n",
      "[58]\tvalid_0's ndcg@10: 0.627196\n",
      "[59]\tvalid_0's ndcg@10: 0.589465\n",
      "[60]\tvalid_0's ndcg@10: 0.61717\n",
      "[61]\tvalid_0's ndcg@10: 0.620366\n",
      "[62]\tvalid_0's ndcg@10: 0.598995\n",
      "[63]\tvalid_0's ndcg@10: 0.583077\n",
      "[64]\tvalid_0's ndcg@10: 0.59631\n",
      "[65]\tvalid_0's ndcg@10: 0.594378\n",
      "[66]\tvalid_0's ndcg@10: 0.568743\n",
      "[67]\tvalid_0's ndcg@10: 0.571477\n",
      "[68]\tvalid_0's ndcg@10: 0.553437\n",
      "[69]\tvalid_0's ndcg@10: 0.551709\n",
      "[70]\tvalid_0's ndcg@10: 0.55451\n",
      "[71]\tvalid_0's ndcg@10: 0.553462\n",
      "[72]\tvalid_0's ndcg@10: 0.554788\n",
      "[73]\tvalid_0's ndcg@10: 0.552288\n",
      "[74]\tvalid_0's ndcg@10: 0.552181\n",
      "[75]\tvalid_0's ndcg@10: 0.552181\n",
      "[76]\tvalid_0's ndcg@10: 0.552781\n",
      "[77]\tvalid_0's ndcg@10: 0.551606\n",
      "[78]\tvalid_0's ndcg@10: 0.546469\n",
      "[79]\tvalid_0's ndcg@10: 0.545893\n",
      "[80]\tvalid_0's ndcg@10: 0.54503\n",
      "[81]\tvalid_0's ndcg@10: 0.546904\n",
      "[82]\tvalid_0's ndcg@10: 0.543964\n",
      "[83]\tvalid_0's ndcg@10: 0.537595\n",
      "[84]\tvalid_0's ndcg@10: 0.553026\n",
      "[85]\tvalid_0's ndcg@10: 0.537595\n",
      "[86]\tvalid_0's ndcg@10: 0.554667\n",
      "[87]\tvalid_0's ndcg@10: 0.555716\n",
      "[88]\tvalid_0's ndcg@10: 0.555316\n",
      "[89]\tvalid_0's ndcg@10: 0.555316\n",
      "[90]\tvalid_0's ndcg@10: 0.555052\n",
      "[91]\tvalid_0's ndcg@10: 0.559442\n",
      "[92]\tvalid_0's ndcg@10: 0.555453\n",
      "[93]\tvalid_0's ndcg@10: 0.555453\n",
      "[94]\tvalid_0's ndcg@10: 0.554795\n",
      "[95]\tvalid_0's ndcg@10: 0.557623\n",
      "[96]\tvalid_0's ndcg@10: 0.556035\n",
      "[97]\tvalid_0's ndcg@10: 0.556454\n",
      "[98]\tvalid_0's ndcg@10: 0.554476\n",
      "[99]\tvalid_0's ndcg@10: 0.555213\n",
      "[100]\tvalid_0's ndcg@10: 0.555931\n",
      "[101]\tvalid_0's ndcg@10: 0.555931\n",
      "[102]\tvalid_0's ndcg@10: 0.555546\n",
      "[103]\tvalid_0's ndcg@10: 0.556786\n",
      "[104]\tvalid_0's ndcg@10: 0.556084\n",
      "[105]\tvalid_0's ndcg@10: 0.573462\n",
      "[106]\tvalid_0's ndcg@10: 0.573144\n",
      "[107]\tvalid_0's ndcg@10: 0.573144\n",
      "[108]\tvalid_0's ndcg@10: 0.57288\n",
      "[109]\tvalid_0's ndcg@10: 0.538583\n",
      "[110]\tvalid_0's ndcg@10: 0.560883\n",
      "[111]\tvalid_0's ndcg@10: 0.55499\n",
      "[112]\tvalid_0's ndcg@10: 0.564356\n",
      "[113]\tvalid_0's ndcg@10: 0.558741\n",
      "[114]\tvalid_0's ndcg@10: 0.557457\n",
      "[115]\tvalid_0's ndcg@10: 0.558463\n",
      "[116]\tvalid_0's ndcg@10: 0.557726\n",
      "[117]\tvalid_0's ndcg@10: 0.558112\n",
      "[118]\tvalid_0's ndcg@10: 0.560173\n",
      "[119]\tvalid_0's ndcg@10: 0.560173\n",
      "[120]\tvalid_0's ndcg@10: 0.560436\n",
      "[121]\tvalid_0's ndcg@10: 0.565679\n",
      "[122]\tvalid_0's ndcg@10: 0.5598\n",
      "[123]\tvalid_0's ndcg@10: 0.587728\n",
      "[124]\tvalid_0's ndcg@10: 0.580994\n",
      "[125]\tvalid_0's ndcg@10: 0.581935\n",
      "[126]\tvalid_0's ndcg@10: 0.589874\n",
      "[127]\tvalid_0's ndcg@10: 0.585254\n",
      "[128]\tvalid_0's ndcg@10: 0.592657\n",
      "[129]\tvalid_0's ndcg@10: 0.592975\n",
      "[130]\tvalid_0's ndcg@10: 0.592808\n",
      "[131]\tvalid_0's ndcg@10: 0.592598\n",
      "[132]\tvalid_0's ndcg@10: 0.592205\n",
      "[133]\tvalid_0's ndcg@10: 0.591832\n",
      "[134]\tvalid_0's ndcg@10: 0.592226\n",
      "[135]\tvalid_0's ndcg@10: 0.592216\n",
      "[136]\tvalid_0's ndcg@10: 0.591504\n",
      "[137]\tvalid_0's ndcg@10: 0.591504\n",
      "[138]\tvalid_0's ndcg@10: 0.591822\n",
      "[139]\tvalid_0's ndcg@10: 0.592355\n",
      "[140]\tvalid_0's ndcg@10: 0.591618\n",
      "[141]\tvalid_0's ndcg@10: 0.591336\n",
      "[142]\tvalid_0's ndcg@10: 0.592037\n",
      "[143]\tvalid_0's ndcg@10: 0.591018\n",
      "[144]\tvalid_0's ndcg@10: 0.565836\n",
      "[145]\tvalid_0's ndcg@10: 0.565836\n",
      "[146]\tvalid_0's ndcg@10: 0.565836\n",
      "[147]\tvalid_0's ndcg@10: 0.565836\n",
      "[148]\tvalid_0's ndcg@10: 0.56723\n",
      "[149]\tvalid_0's ndcg@10: 0.569204\n",
      "[150]\tvalid_0's ndcg@10: 0.565414\n",
      "[151]\tvalid_0's ndcg@10: 0.568274\n",
      "[152]\tvalid_0's ndcg@10: 0.568643\n",
      "[153]\tvalid_0's ndcg@10: 0.569023\n",
      "[154]\tvalid_0's ndcg@10: 0.589489\n",
      "[155]\tvalid_0's ndcg@10: 0.565967\n",
      "[156]\tvalid_0's ndcg@10: 0.565649\n",
      "[157]\tvalid_0's ndcg@10: 0.565967\n",
      "[158]\tvalid_0's ndcg@10: 0.565967\n",
      "[159]\tvalid_0's ndcg@10: 0.566205\n",
      "[160]\tvalid_0's ndcg@10: 0.566205\n",
      "[161]\tvalid_0's ndcg@10: 0.566205\n",
      "[162]\tvalid_0's ndcg@10: 0.55574\n",
      "[163]\tvalid_0's ndcg@10: 0.538076\n",
      "[164]\tvalid_0's ndcg@10: 0.537758\n",
      "[165]\tvalid_0's ndcg@10: 0.536713\n",
      "[166]\tvalid_0's ndcg@10: 0.536248\n",
      "[167]\tvalid_0's ndcg@10: 0.536248\n",
      "[168]\tvalid_0's ndcg@10: 0.536359\n",
      "[169]\tvalid_0's ndcg@10: 0.564355\n",
      "[170]\tvalid_0's ndcg@10: 0.563619\n",
      "[171]\tvalid_0's ndcg@10: 0.56355\n",
      "[172]\tvalid_0's ndcg@10: 0.563412\n",
      "[173]\tvalid_0's ndcg@10: 0.563232\n",
      "[174]\tvalid_0's ndcg@10: 0.533082\n",
      "[175]\tvalid_0's ndcg@10: 0.53184\n",
      "[176]\tvalid_0's ndcg@10: 0.560644\n",
      "[177]\tvalid_0's ndcg@10: 0.559296\n",
      "[178]\tvalid_0's ndcg@10: 0.561148\n",
      "[179]\tvalid_0's ndcg@10: 0.533711\n",
      "[180]\tvalid_0's ndcg@10: 0.533711\n",
      "[181]\tvalid_0's ndcg@10: 0.533711\n",
      "[182]\tvalid_0's ndcg@10: 0.533207\n",
      "[183]\tvalid_0's ndcg@10: 0.532822\n",
      "[184]\tvalid_0's ndcg@10: 0.532822\n",
      "[185]\tvalid_0's ndcg@10: 0.532683\n",
      "[186]\tvalid_0's ndcg@10: 0.53314\n",
      "[187]\tvalid_0's ndcg@10: 0.551758\n",
      "[188]\tvalid_0's ndcg@10: 0.552334\n",
      "[189]\tvalid_0's ndcg@10: 0.549274\n",
      "[190]\tvalid_0's ndcg@10: 0.549107\n",
      "[191]\tvalid_0's ndcg@10: 0.547422\n",
      "[192]\tvalid_0's ndcg@10: 0.552824\n",
      "[193]\tvalid_0's ndcg@10: 0.553399\n",
      "[194]\tvalid_0's ndcg@10: 0.549274\n",
      "[195]\tvalid_0's ndcg@10: 0.553581\n",
      "[196]\tvalid_0's ndcg@10: 0.556293\n",
      "[197]\tvalid_0's ndcg@10: 0.556293\n",
      "[198]\tvalid_0's ndcg@10: 0.551758\n",
      "[199]\tvalid_0's ndcg@10: 0.557616\n",
      "[200]\tvalid_0's ndcg@10: 0.55805\n",
      "[201]\tvalid_0's ndcg@10: 0.55805\n",
      "[202]\tvalid_0's ndcg@10: 0.552824\n",
      "[203]\tvalid_0's ndcg@10: 0.557475\n",
      "[204]\tvalid_0's ndcg@10: 0.552824\n",
      "[205]\tvalid_0's ndcg@10: 0.553258\n",
      "[206]\tvalid_0's ndcg@10: 0.553258\n",
      "[207]\tvalid_0's ndcg@10: 0.553258\n",
      "[208]\tvalid_0's ndcg@10: 0.553258\n",
      "[209]\tvalid_0's ndcg@10: 0.553091\n",
      "[210]\tvalid_0's ndcg@10: 0.553258\n",
      "[211]\tvalid_0's ndcg@10: 0.553258\n",
      "[212]\tvalid_0's ndcg@10: 0.554565\n",
      "[213]\tvalid_0's ndcg@10: 0.553576\n",
      "[214]\tvalid_0's ndcg@10: 0.554414\n",
      "[215]\tvalid_0's ndcg@10: 0.554732\n",
      "[216]\tvalid_0's ndcg@10: 0.554131\n",
      "[217]\tvalid_0's ndcg@10: 0.558514\n",
      "[218]\tvalid_0's ndcg@10: 0.554298\n",
      "[219]\tvalid_0's ndcg@10: 0.563607\n",
      "[220]\tvalid_0's ndcg@10: 0.554131\n",
      "[221]\tvalid_0's ndcg@10: 0.563875\n",
      "[222]\tvalid_0's ndcg@10: 0.554565\n",
      "[223]\tvalid_0's ndcg@10: 0.554131\n",
      "[224]\tvalid_0's ndcg@10: 0.554131\n",
      "[225]\tvalid_0's ndcg@10: 0.554436\n",
      "[226]\tvalid_0's ndcg@10: 0.563607\n",
      "[227]\tvalid_0's ndcg@10: 0.567824\n",
      "[228]\tvalid_0's ndcg@10: 0.569107\n",
      "[229]\tvalid_0's ndcg@10: 0.569107\n",
      "[230]\tvalid_0's ndcg@10: 0.558949\n",
      "[231]\tvalid_0's ndcg@10: 0.557793\n",
      "[232]\tvalid_0's ndcg@10: 0.557793\n",
      "[233]\tvalid_0's ndcg@10: 0.557626\n",
      "[234]\tvalid_0's ndcg@10: 0.557793\n",
      "[235]\tvalid_0's ndcg@10: 0.539008\n",
      "[236]\tvalid_0's ndcg@10: 0.557626\n",
      "[237]\tvalid_0's ndcg@10: 0.557626\n",
      "[238]\tvalid_0's ndcg@10: 0.539008\n",
      "[239]\tvalid_0's ndcg@10: 0.539008\n",
      "[240]\tvalid_0's ndcg@10: 0.557626\n",
      "[241]\tvalid_0's ndcg@10: 0.539008\n",
      "[242]\tvalid_0's ndcg@10: 0.539008\n",
      "[243]\tvalid_0's ndcg@10: 0.539008\n",
      "[244]\tvalid_0's ndcg@10: 0.539856\n",
      "[245]\tvalid_0's ndcg@10: 0.539008\n",
      "[246]\tvalid_0's ndcg@10: 0.539008\n",
      "[247]\tvalid_0's ndcg@10: 0.539008\n",
      "[248]\tvalid_0's ndcg@10: 0.539008\n",
      "[249]\tvalid_0's ndcg@10: 0.538573\n",
      "[250]\tvalid_0's ndcg@10: 0.539008\n",
      "[251]\tvalid_0's ndcg@10: 0.538573\n",
      "[252]\tvalid_0's ndcg@10: 0.539583\n",
      "[253]\tvalid_0's ndcg@10: 0.539583\n",
      "[254]\tvalid_0's ndcg@10: 0.539583\n",
      "[255]\tvalid_0's ndcg@10: 0.539583\n",
      "[256]\tvalid_0's ndcg@10: 0.539583\n",
      "[257]\tvalid_0's ndcg@10: 0.539583\n",
      "[258]\tvalid_0's ndcg@10: 0.537296\n",
      "[259]\tvalid_0's ndcg@10: 0.53773\n",
      "[260]\tvalid_0's ndcg@10: 0.53773\n",
      "[261]\tvalid_0's ndcg@10: 0.53773\n",
      "[262]\tvalid_0's ndcg@10: 0.539583\n",
      "[263]\tvalid_0's ndcg@10: 0.53773\n",
      "[264]\tvalid_0's ndcg@10: 0.53773\n",
      "[265]\tvalid_0's ndcg@10: 0.53773\n",
      "[266]\tvalid_0's ndcg@10: 0.539583\n",
      "[267]\tvalid_0's ndcg@10: 0.539583\n",
      "[268]\tvalid_0's ndcg@10: 0.539583\n",
      "[269]\tvalid_0's ndcg@10: 0.539583\n",
      "[270]\tvalid_0's ndcg@10: 0.539583\n",
      "[271]\tvalid_0's ndcg@10: 0.539583\n",
      "[272]\tvalid_0's ndcg@10: 0.539583\n",
      "[273]\tvalid_0's ndcg@10: 0.539583\n",
      "[274]\tvalid_0's ndcg@10: 0.53773\n",
      "[275]\tvalid_0's ndcg@10: 0.53773\n",
      "[276]\tvalid_0's ndcg@10: 0.53773\n",
      "[277]\tvalid_0's ndcg@10: 0.53773\n",
      "[278]\tvalid_0's ndcg@10: 0.53773\n",
      "[279]\tvalid_0's ndcg@10: 0.53773\n",
      "[280]\tvalid_0's ndcg@10: 0.53773\n",
      "[281]\tvalid_0's ndcg@10: 0.53773\n",
      "[282]\tvalid_0's ndcg@10: 0.53773\n",
      "[283]\tvalid_0's ndcg@10: 0.53773\n",
      "[284]\tvalid_0's ndcg@10: 0.53773\n",
      "[285]\tvalid_0's ndcg@10: 0.53773\n",
      "[286]\tvalid_0's ndcg@10: 0.53773\n",
      "[287]\tvalid_0's ndcg@10: 0.539583\n",
      "[288]\tvalid_0's ndcg@10: 0.539583\n",
      "[289]\tvalid_0's ndcg@10: 0.538579\n",
      "[290]\tvalid_0's ndcg@10: 0.538579\n",
      "[291]\tvalid_0's ndcg@10: 0.538579\n",
      "[292]\tvalid_0's ndcg@10: 0.540432\n",
      "[293]\tvalid_0's ndcg@10: 0.540432\n",
      "[294]\tvalid_0's ndcg@10: 0.540432\n",
      "[295]\tvalid_0's ndcg@10: 0.540432\n",
      "[296]\tvalid_0's ndcg@10: 0.540432\n",
      "[297]\tvalid_0's ndcg@10: 0.540432\n",
      "[298]\tvalid_0's ndcg@10: 0.540432\n",
      "[299]\tvalid_0's ndcg@10: 0.538579\n",
      "[300]\tvalid_0's ndcg@10: 0.538579\n",
      "[301]\tvalid_0's ndcg@10: 0.538579\n",
      "[302]\tvalid_0's ndcg@10: 0.540432\n",
      "[303]\tvalid_0's ndcg@10: 0.540432\n",
      "[304]\tvalid_0's ndcg@10: 0.540432\n",
      "[305]\tvalid_0's ndcg@10: 0.540432\n",
      "[306]\tvalid_0's ndcg@10: 0.540432\n",
      "[307]\tvalid_0's ndcg@10: 0.540432\n",
      "[308]\tvalid_0's ndcg@10: 0.540432\n",
      "[309]\tvalid_0's ndcg@10: 0.541169\n",
      "[310]\tvalid_0's ndcg@10: 0.539316\n",
      "[311]\tvalid_0's ndcg@10: 0.539316\n",
      "[312]\tvalid_0's ndcg@10: 0.541169\n",
      "[313]\tvalid_0's ndcg@10: 0.541169\n",
      "[314]\tvalid_0's ndcg@10: 0.541169\n",
      "[315]\tvalid_0's ndcg@10: 0.541169\n",
      "[316]\tvalid_0's ndcg@10: 0.541169\n",
      "[317]\tvalid_0's ndcg@10: 0.541169\n",
      "[318]\tvalid_0's ndcg@10: 0.541169\n",
      "[319]\tvalid_0's ndcg@10: 0.541169\n",
      "[320]\tvalid_0's ndcg@10: 0.539316\n",
      "[321]\tvalid_0's ndcg@10: 0.541169\n",
      "[322]\tvalid_0's ndcg@10: 0.541169\n",
      "[323]\tvalid_0's ndcg@10: 0.541169\n",
      "[324]\tvalid_0's ndcg@10: 0.541169\n",
      "[325]\tvalid_0's ndcg@10: 0.541169\n",
      "[326]\tvalid_0's ndcg@10: 0.541169\n",
      "[327]\tvalid_0's ndcg@10: 0.541169\n",
      "[328]\tvalid_0's ndcg@10: 0.541169\n",
      "[329]\tvalid_0's ndcg@10: 0.541169\n",
      "[330]\tvalid_0's ndcg@10: 0.541169\n",
      "[331]\tvalid_0's ndcg@10: 0.541169\n",
      "[332]\tvalid_0's ndcg@10: 0.541169\n",
      "[333]\tvalid_0's ndcg@10: 0.541169\n",
      "[334]\tvalid_0's ndcg@10: 0.541169\n",
      "[335]\tvalid_0's ndcg@10: 0.541169\n",
      "[336]\tvalid_0's ndcg@10: 0.541169\n",
      "[337]\tvalid_0's ndcg@10: 0.541169\n",
      "[338]\tvalid_0's ndcg@10: 0.541169\n",
      "[339]\tvalid_0's ndcg@10: 0.541169\n",
      "[340]\tvalid_0's ndcg@10: 0.541169\n",
      "[341]\tvalid_0's ndcg@10: 0.541169\n",
      "[342]\tvalid_0's ndcg@10: 0.541169\n",
      "[343]\tvalid_0's ndcg@10: 0.541169\n",
      "[344]\tvalid_0's ndcg@10: 0.541169\n",
      "[345]\tvalid_0's ndcg@10: 0.540734\n",
      "[346]\tvalid_0's ndcg@10: 0.540734\n",
      "[347]\tvalid_0's ndcg@10: 0.540734\n",
      "[348]\tvalid_0's ndcg@10: 0.540734\n",
      "[349]\tvalid_0's ndcg@10: 0.540734\n",
      "[350]\tvalid_0's ndcg@10: 0.540734\n",
      "[351]\tvalid_0's ndcg@10: 0.540734\n",
      "[352]\tvalid_0's ndcg@10: 0.540734\n",
      "[353]\tvalid_0's ndcg@10: 0.540734\n",
      "[354]\tvalid_0's ndcg@10: 0.540734\n",
      "[355]\tvalid_0's ndcg@10: 0.540734\n",
      "[356]\tvalid_0's ndcg@10: 0.540734\n",
      "[357]\tvalid_0's ndcg@10: 0.540734\n",
      "[358]\tvalid_0's ndcg@10: 0.540734\n",
      "[359]\tvalid_0's ndcg@10: 0.540901\n",
      "[360]\tvalid_0's ndcg@10: 0.540734\n",
      "[361]\tvalid_0's ndcg@10: 0.540734\n",
      "[362]\tvalid_0's ndcg@10: 0.540734\n",
      "[363]\tvalid_0's ndcg@10: 0.540734\n",
      "[364]\tvalid_0's ndcg@10: 0.540734\n",
      "[365]\tvalid_0's ndcg@10: 0.540734\n",
      "[366]\tvalid_0's ndcg@10: 0.540734\n",
      "[367]\tvalid_0's ndcg@10: 0.540734\n",
      "[368]\tvalid_0's ndcg@10: 0.540734\n",
      "[369]\tvalid_0's ndcg@10: 0.540734\n",
      "[370]\tvalid_0's ndcg@10: 0.540734\n",
      "[371]\tvalid_0's ndcg@10: 0.540734\n",
      "[372]\tvalid_0's ndcg@10: 0.540734\n",
      "[373]\tvalid_0's ndcg@10: 0.540734\n",
      "[374]\tvalid_0's ndcg@10: 0.540734\n",
      "[375]\tvalid_0's ndcg@10: 0.540734\n",
      "[376]\tvalid_0's ndcg@10: 0.540734\n",
      "[377]\tvalid_0's ndcg@10: 0.540734\n",
      "[378]\tvalid_0's ndcg@10: 0.540734\n",
      "[379]\tvalid_0's ndcg@10: 0.540734\n",
      "[380]\tvalid_0's ndcg@10: 0.540734\n",
      "[381]\tvalid_0's ndcg@10: 0.540734\n",
      "[382]\tvalid_0's ndcg@10: 0.540734\n",
      "[383]\tvalid_0's ndcg@10: 0.540734\n",
      "[384]\tvalid_0's ndcg@10: 0.540734\n",
      "[385]\tvalid_0's ndcg@10: 0.540734\n",
      "[386]\tvalid_0's ndcg@10: 0.540734\n",
      "[387]\tvalid_0's ndcg@10: 0.543216\n",
      "[388]\tvalid_0's ndcg@10: 0.543216\n",
      "[389]\tvalid_0's ndcg@10: 0.543216\n",
      "[390]\tvalid_0's ndcg@10: 0.543216\n",
      "[391]\tvalid_0's ndcg@10: 0.540734\n",
      "[392]\tvalid_0's ndcg@10: 0.540734\n",
      "[393]\tvalid_0's ndcg@10: 0.540734\n",
      "[394]\tvalid_0's ndcg@10: 0.540734\n",
      "[395]\tvalid_0's ndcg@10: 0.543216\n",
      "[396]\tvalid_0's ndcg@10: 0.543216\n",
      "[397]\tvalid_0's ndcg@10: 0.543216\n",
      "[398]\tvalid_0's ndcg@10: 0.543216\n",
      "[399]\tvalid_0's ndcg@10: 0.543602\n",
      "[400]\tvalid_0's ndcg@10: 0.543216\n",
      "[401]\tvalid_0's ndcg@10: 0.543602\n",
      "[402]\tvalid_0's ndcg@10: 0.543602\n",
      "[403]\tvalid_0's ndcg@10: 0.543216\n",
      "[404]\tvalid_0's ndcg@10: 0.541119\n",
      "[405]\tvalid_0's ndcg@10: 0.541119\n",
      "[406]\tvalid_0's ndcg@10: 0.541119\n",
      "[407]\tvalid_0's ndcg@10: 0.541119\n",
      "[408]\tvalid_0's ndcg@10: 0.541119\n",
      "[409]\tvalid_0's ndcg@10: 0.541119\n",
      "[410]\tvalid_0's ndcg@10: 0.541119\n",
      "[411]\tvalid_0's ndcg@10: 0.541119\n",
      "[412]\tvalid_0's ndcg@10: 0.541119\n",
      "[413]\tvalid_0's ndcg@10: 0.541119\n",
      "[414]\tvalid_0's ndcg@10: 0.543602\n",
      "[415]\tvalid_0's ndcg@10: 0.543602\n",
      "[416]\tvalid_0's ndcg@10: 0.543602\n",
      "[417]\tvalid_0's ndcg@10: 0.543602\n",
      "[418]\tvalid_0's ndcg@10: 0.543602\n",
      "[419]\tvalid_0's ndcg@10: 0.54388\n",
      "[420]\tvalid_0's ndcg@10: 0.54388\n",
      "[421]\tvalid_0's ndcg@10: 0.54388\n",
      "[422]\tvalid_0's ndcg@10: 0.543602\n",
      "[423]\tvalid_0's ndcg@10: 0.54388\n",
      "[424]\tvalid_0's ndcg@10: 0.54388\n",
      "[425]\tvalid_0's ndcg@10: 0.54388\n",
      "[426]\tvalid_0's ndcg@10: 0.54388\n",
      "[427]\tvalid_0's ndcg@10: 0.54388\n",
      "[428]\tvalid_0's ndcg@10: 0.543602\n",
      "[429]\tvalid_0's ndcg@10: 0.543602\n",
      "[430]\tvalid_0's ndcg@10: 0.543602\n",
      "[431]\tvalid_0's ndcg@10: 0.543602\n",
      "[432]\tvalid_0's ndcg@10: 0.543602\n",
      "[433]\tvalid_0's ndcg@10: 0.543602\n",
      "[434]\tvalid_0's ndcg@10: 0.543602\n",
      "[435]\tvalid_0's ndcg@10: 0.543602\n",
      "[436]\tvalid_0's ndcg@10: 0.543602\n",
      "[437]\tvalid_0's ndcg@10: 0.543602\n",
      "[438]\tvalid_0's ndcg@10: 0.543602\n",
      "[439]\tvalid_0's ndcg@10: 0.543602\n",
      "[440]\tvalid_0's ndcg@10: 0.543602\n",
      "[441]\tvalid_0's ndcg@10: 0.543602\n",
      "[442]\tvalid_0's ndcg@10: 0.543602\n",
      "[443]\tvalid_0's ndcg@10: 0.543602\n",
      "[444]\tvalid_0's ndcg@10: 0.543602\n",
      "[445]\tvalid_0's ndcg@10: 0.543602\n",
      "[446]\tvalid_0's ndcg@10: 0.543602\n",
      "[447]\tvalid_0's ndcg@10: 0.543602\n",
      "[448]\tvalid_0's ndcg@10: 0.543602\n",
      "[449]\tvalid_0's ndcg@10: 0.543602\n",
      "[450]\tvalid_0's ndcg@10: 0.543602\n",
      "[451]\tvalid_0's ndcg@10: 0.543602\n",
      "[452]\tvalid_0's ndcg@10: 0.543602\n",
      "[453]\tvalid_0's ndcg@10: 0.543602\n",
      "[454]\tvalid_0's ndcg@10: 0.543602\n",
      "[455]\tvalid_0's ndcg@10: 0.543602\n",
      "[456]\tvalid_0's ndcg@10: 0.543602\n",
      "[457]\tvalid_0's ndcg@10: 0.543602\n",
      "[458]\tvalid_0's ndcg@10: 0.543602\n",
      "[459]\tvalid_0's ndcg@10: 0.543602\n",
      "[460]\tvalid_0's ndcg@10: 0.543602\n",
      "[461]\tvalid_0's ndcg@10: 0.543602\n",
      "[462]\tvalid_0's ndcg@10: 0.543602\n",
      "[463]\tvalid_0's ndcg@10: 0.543602\n",
      "[464]\tvalid_0's ndcg@10: 0.543602\n",
      "[465]\tvalid_0's ndcg@10: 0.543602\n",
      "[466]\tvalid_0's ndcg@10: 0.543602\n",
      "[467]\tvalid_0's ndcg@10: 0.543602\n",
      "[468]\tvalid_0's ndcg@10: 0.543602\n",
      "[469]\tvalid_0's ndcg@10: 0.543602\n",
      "[470]\tvalid_0's ndcg@10: 0.543602\n",
      "[471]\tvalid_0's ndcg@10: 0.543602\n",
      "[472]\tvalid_0's ndcg@10: 0.543602\n",
      "[473]\tvalid_0's ndcg@10: 0.543602\n",
      "[474]\tvalid_0's ndcg@10: 0.543602\n",
      "[475]\tvalid_0's ndcg@10: 0.543602\n",
      "[476]\tvalid_0's ndcg@10: 0.543602\n",
      "[477]\tvalid_0's ndcg@10: 0.543602\n",
      "[478]\tvalid_0's ndcg@10: 0.543602\n",
      "[479]\tvalid_0's ndcg@10: 0.543602\n",
      "[480]\tvalid_0's ndcg@10: 0.543602\n",
      "[481]\tvalid_0's ndcg@10: 0.543602\n",
      "[482]\tvalid_0's ndcg@10: 0.543602\n",
      "[483]\tvalid_0's ndcg@10: 0.543602\n",
      "[484]\tvalid_0's ndcg@10: 0.543602\n",
      "[485]\tvalid_0's ndcg@10: 0.543602\n",
      "[486]\tvalid_0's ndcg@10: 0.543602\n",
      "[487]\tvalid_0's ndcg@10: 0.543602\n",
      "[488]\tvalid_0's ndcg@10: 0.543602\n",
      "[489]\tvalid_0's ndcg@10: 0.543602\n",
      "[490]\tvalid_0's ndcg@10: 0.543602\n",
      "[491]\tvalid_0's ndcg@10: 0.543602\n",
      "[492]\tvalid_0's ndcg@10: 0.542583\n",
      "[493]\tvalid_0's ndcg@10: 0.542583\n",
      "[494]\tvalid_0's ndcg@10: 0.542583\n",
      "[495]\tvalid_0's ndcg@10: 0.542583\n",
      "[496]\tvalid_0's ndcg@10: 0.542583\n",
      "[497]\tvalid_0's ndcg@10: 0.542583\n",
      "[498]\tvalid_0's ndcg@10: 0.542583\n",
      "[499]\tvalid_0's ndcg@10: 0.542583\n",
      "[500]\tvalid_0's ndcg@10: 0.542583\n",
      "[501]\tvalid_0's ndcg@10: 0.542583\n",
      "[502]\tvalid_0's ndcg@10: 0.542583\n",
      "[503]\tvalid_0's ndcg@10: 0.542079\n",
      "[504]\tvalid_0's ndcg@10: 0.543602\n",
      "[505]\tvalid_0's ndcg@10: 0.543602\n",
      "[506]\tvalid_0's ndcg@10: 0.542583\n",
      "[507]\tvalid_0's ndcg@10: 0.542583\n",
      "[508]\tvalid_0's ndcg@10: 0.542198\n",
      "[509]\tvalid_0's ndcg@10: 0.542198\n",
      "[510]\tvalid_0's ndcg@10: 0.542198\n",
      "[511]\tvalid_0's ndcg@10: 0.542198\n",
      "[512]\tvalid_0's ndcg@10: 0.542198\n",
      "[513]\tvalid_0's ndcg@10: 0.542198\n",
      "[514]\tvalid_0's ndcg@10: 0.542198\n",
      "[515]\tvalid_0's ndcg@10: 0.542198\n",
      "[516]\tvalid_0's ndcg@10: 0.542198\n",
      "[517]\tvalid_0's ndcg@10: 0.542198\n",
      "[518]\tvalid_0's ndcg@10: 0.542198\n",
      "[519]\tvalid_0's ndcg@10: 0.542198\n",
      "[520]\tvalid_0's ndcg@10: 0.542198\n",
      "[521]\tvalid_0's ndcg@10: 0.542198\n",
      "[522]\tvalid_0's ndcg@10: 0.542198\n",
      "[523]\tvalid_0's ndcg@10: 0.542198\n",
      "[524]\tvalid_0's ndcg@10: 0.542198\n",
      "[525]\tvalid_0's ndcg@10: 0.542198\n",
      "[526]\tvalid_0's ndcg@10: 0.542198\n",
      "[527]\tvalid_0's ndcg@10: 0.542198\n",
      "[528]\tvalid_0's ndcg@10: 0.542476\n",
      "[529]\tvalid_0's ndcg@10: 0.542476\n",
      "[530]\tvalid_0's ndcg@10: 0.54299\n",
      "[531]\tvalid_0's ndcg@10: 0.542476\n",
      "[532]\tvalid_0's ndcg@10: 0.542198\n",
      "[533]\tvalid_0's ndcg@10: 0.541694\n",
      "[534]\tvalid_0's ndcg@10: 0.541694\n",
      "[535]\tvalid_0's ndcg@10: 0.541694\n",
      "[536]\tvalid_0's ndcg@10: 0.541694\n",
      "[537]\tvalid_0's ndcg@10: 0.541694\n",
      "[538]\tvalid_0's ndcg@10: 0.541972\n",
      "[539]\tvalid_0's ndcg@10: 0.541972\n",
      "[540]\tvalid_0's ndcg@10: 0.541972\n",
      "[541]\tvalid_0's ndcg@10: 0.54299\n",
      "[542]\tvalid_0's ndcg@10: 0.54299\n",
      "[543]\tvalid_0's ndcg@10: 0.54299\n",
      "[544]\tvalid_0's ndcg@10: 0.54299\n",
      "[545]\tvalid_0's ndcg@10: 0.54299\n",
      "[546]\tvalid_0's ndcg@10: 0.54299\n",
      "[547]\tvalid_0's ndcg@10: 0.54299\n",
      "[548]\tvalid_0's ndcg@10: 0.54299\n",
      "[549]\tvalid_0's ndcg@10: 0.54299\n",
      "[550]\tvalid_0's ndcg@10: 0.54299\n",
      "[551]\tvalid_0's ndcg@10: 0.54299\n",
      "[552]\tvalid_0's ndcg@10: 0.54299\n",
      "[553]\tvalid_0's ndcg@10: 0.548605\n",
      "[554]\tvalid_0's ndcg@10: 0.548605\n",
      "[555]\tvalid_0's ndcg@10: 0.547587\n",
      "[556]\tvalid_0's ndcg@10: 0.548605\n",
      "[557]\tvalid_0's ndcg@10: 0.541972\n",
      "[558]\tvalid_0's ndcg@10: 0.541972\n",
      "[559]\tvalid_0's ndcg@10: 0.547587\n",
      "[560]\tvalid_0's ndcg@10: 0.547587\n",
      "[561]\tvalid_0's ndcg@10: 0.547587\n",
      "[562]\tvalid_0's ndcg@10: 0.547587\n",
      "[563]\tvalid_0's ndcg@10: 0.547587\n",
      "[564]\tvalid_0's ndcg@10: 0.547587\n",
      "[565]\tvalid_0's ndcg@10: 0.547587\n",
      "[566]\tvalid_0's ndcg@10: 0.547587\n",
      "[567]\tvalid_0's ndcg@10: 0.547587\n",
      "[568]\tvalid_0's ndcg@10: 0.547587\n",
      "[569]\tvalid_0's ndcg@10: 0.547587\n",
      "[570]\tvalid_0's ndcg@10: 0.547587\n",
      "[571]\tvalid_0's ndcg@10: 0.547587\n",
      "[572]\tvalid_0's ndcg@10: 0.547587\n",
      "[573]\tvalid_0's ndcg@10: 0.547587\n",
      "[574]\tvalid_0's ndcg@10: 0.547587\n",
      "[575]\tvalid_0's ndcg@10: 0.547587\n",
      "[576]\tvalid_0's ndcg@10: 0.547587\n",
      "[577]\tvalid_0's ndcg@10: 0.547587\n",
      "[578]\tvalid_0's ndcg@10: 0.547587\n",
      "[579]\tvalid_0's ndcg@10: 0.547587\n",
      "[580]\tvalid_0's ndcg@10: 0.547587\n",
      "[581]\tvalid_0's ndcg@10: 0.547587\n",
      "[582]\tvalid_0's ndcg@10: 0.547587\n",
      "[583]\tvalid_0's ndcg@10: 0.547587\n",
      "[584]\tvalid_0's ndcg@10: 0.547587\n",
      "[585]\tvalid_0's ndcg@10: 0.547587\n",
      "[586]\tvalid_0's ndcg@10: 0.547587\n",
      "[587]\tvalid_0's ndcg@10: 0.547587\n",
      "[588]\tvalid_0's ndcg@10: 0.547587\n",
      "[589]\tvalid_0's ndcg@10: 0.547587\n",
      "[590]\tvalid_0's ndcg@10: 0.547587\n",
      "[591]\tvalid_0's ndcg@10: 0.547587\n",
      "[592]\tvalid_0's ndcg@10: 0.547587\n",
      "[593]\tvalid_0's ndcg@10: 0.547587\n",
      "[594]\tvalid_0's ndcg@10: 0.547587\n",
      "[595]\tvalid_0's ndcg@10: 0.547587\n",
      "[596]\tvalid_0's ndcg@10: 0.547587\n",
      "[597]\tvalid_0's ndcg@10: 0.547587\n",
      "[598]\tvalid_0's ndcg@10: 0.547587\n",
      "[599]\tvalid_0's ndcg@10: 0.547587\n",
      "[600]\tvalid_0's ndcg@10: 0.547587\n",
      "[601]\tvalid_0's ndcg@10: 0.547587\n",
      "[602]\tvalid_0's ndcg@10: 0.547587\n",
      "[603]\tvalid_0's ndcg@10: 0.548239\n",
      "[604]\tvalid_0's ndcg@10: 0.547587\n",
      "[605]\tvalid_0's ndcg@10: 0.548239\n",
      "[606]\tvalid_0's ndcg@10: 0.548239\n",
      "[607]\tvalid_0's ndcg@10: 0.547587\n",
      "[608]\tvalid_0's ndcg@10: 0.548239\n",
      "[609]\tvalid_0's ndcg@10: 0.548239\n",
      "[610]\tvalid_0's ndcg@10: 0.548239\n",
      "[611]\tvalid_0's ndcg@10: 0.548239\n",
      "[612]\tvalid_0's ndcg@10: 0.548239\n",
      "[613]\tvalid_0's ndcg@10: 0.548239\n",
      "[614]\tvalid_0's ndcg@10: 0.548239\n",
      "[615]\tvalid_0's ndcg@10: 0.548239\n",
      "[616]\tvalid_0's ndcg@10: 0.548239\n",
      "[617]\tvalid_0's ndcg@10: 0.548239\n",
      "[618]\tvalid_0's ndcg@10: 0.548239\n",
      "[619]\tvalid_0's ndcg@10: 0.548239\n",
      "[620]\tvalid_0's ndcg@10: 0.548239\n",
      "[621]\tvalid_0's ndcg@10: 0.548239\n",
      "[622]\tvalid_0's ndcg@10: 0.548239\n",
      "[623]\tvalid_0's ndcg@10: 0.548239\n",
      "[624]\tvalid_0's ndcg@10: 0.548239\n",
      "[625]\tvalid_0's ndcg@10: 0.548239\n",
      "[626]\tvalid_0's ndcg@10: 0.548239\n",
      "[627]\tvalid_0's ndcg@10: 0.548239\n",
      "[628]\tvalid_0's ndcg@10: 0.548239\n",
      "[629]\tvalid_0's ndcg@10: 0.548239\n",
      "[630]\tvalid_0's ndcg@10: 0.548239\n",
      "[631]\tvalid_0's ndcg@10: 0.548239\n",
      "[632]\tvalid_0's ndcg@10: 0.548239\n",
      "[633]\tvalid_0's ndcg@10: 0.548239\n",
      "[634]\tvalid_0's ndcg@10: 0.548239\n",
      "[635]\tvalid_0's ndcg@10: 0.548239\n",
      "[636]\tvalid_0's ndcg@10: 0.548239\n",
      "[637]\tvalid_0's ndcg@10: 0.548239\n",
      "[638]\tvalid_0's ndcg@10: 0.548239\n",
      "[639]\tvalid_0's ndcg@10: 0.548239\n",
      "[640]\tvalid_0's ndcg@10: 0.548239\n",
      "[641]\tvalid_0's ndcg@10: 0.548239\n",
      "[642]\tvalid_0's ndcg@10: 0.548239\n",
      "[643]\tvalid_0's ndcg@10: 0.56105\n",
      "[644]\tvalid_0's ndcg@10: 0.56105\n",
      "[645]\tvalid_0's ndcg@10: 0.56105\n",
      "[646]\tvalid_0's ndcg@10: 0.56105\n",
      "[647]\tvalid_0's ndcg@10: 0.56105\n",
      "[648]\tvalid_0's ndcg@10: 0.56105\n",
      "[649]\tvalid_0's ndcg@10: 0.548239\n",
      "[650]\tvalid_0's ndcg@10: 0.548239\n",
      "[651]\tvalid_0's ndcg@10: 0.548239\n",
      "[652]\tvalid_0's ndcg@10: 0.548239\n",
      "[653]\tvalid_0's ndcg@10: 0.548239\n",
      "[654]\tvalid_0's ndcg@10: 0.548239\n",
      "[655]\tvalid_0's ndcg@10: 0.56105\n",
      "[656]\tvalid_0's ndcg@10: 0.56105\n",
      "[657]\tvalid_0's ndcg@10: 0.56105\n",
      "[658]\tvalid_0's ndcg@10: 0.56105\n",
      "[659]\tvalid_0's ndcg@10: 0.56105\n",
      "[660]\tvalid_0's ndcg@10: 0.56105\n",
      "[661]\tvalid_0's ndcg@10: 0.537278\n",
      "[662]\tvalid_0's ndcg@10: 0.537278\n",
      "[663]\tvalid_0's ndcg@10: 0.537278\n",
      "[664]\tvalid_0's ndcg@10: 0.524467\n",
      "[665]\tvalid_0's ndcg@10: 0.524467\n",
      "[666]\tvalid_0's ndcg@10: 0.524467\n",
      "[667]\tvalid_0's ndcg@10: 0.524467\n",
      "[668]\tvalid_0's ndcg@10: 0.524467\n",
      "[669]\tvalid_0's ndcg@10: 0.524467\n",
      "[670]\tvalid_0's ndcg@10: 0.537278\n",
      "[671]\tvalid_0's ndcg@10: 0.524467\n",
      "[672]\tvalid_0's ndcg@10: 0.525485\n",
      "[673]\tvalid_0's ndcg@10: 0.525485\n",
      "[674]\tvalid_0's ndcg@10: 0.525485\n",
      "[675]\tvalid_0's ndcg@10: 0.525485\n",
      "[676]\tvalid_0's ndcg@10: 0.524467\n",
      "[677]\tvalid_0's ndcg@10: 0.524467\n",
      "[678]\tvalid_0's ndcg@10: 0.524467\n",
      "[679]\tvalid_0's ndcg@10: 0.525101\n",
      "[680]\tvalid_0's ndcg@10: 0.525101\n",
      "[681]\tvalid_0's ndcg@10: 0.525101\n",
      "[682]\tvalid_0's ndcg@10: 0.525101\n",
      "[683]\tvalid_0's ndcg@10: 0.525101\n",
      "[684]\tvalid_0's ndcg@10: 0.525101\n",
      "[685]\tvalid_0's ndcg@10: 0.525101\n",
      "[686]\tvalid_0's ndcg@10: 0.525101\n",
      "[687]\tvalid_0's ndcg@10: 0.525101\n",
      "[688]\tvalid_0's ndcg@10: 0.525101\n",
      "[689]\tvalid_0's ndcg@10: 0.525101\n",
      "[690]\tvalid_0's ndcg@10: 0.525101\n",
      "[691]\tvalid_0's ndcg@10: 0.525101\n",
      "[692]\tvalid_0's ndcg@10: 0.525101\n",
      "[693]\tvalid_0's ndcg@10: 0.537912\n",
      "[694]\tvalid_0's ndcg@10: 0.525101\n",
      "[695]\tvalid_0's ndcg@10: 0.525101\n",
      "[696]\tvalid_0's ndcg@10: 0.525101\n",
      "[697]\tvalid_0's ndcg@10: 0.525101\n",
      "[698]\tvalid_0's ndcg@10: 0.537912\n",
      "[699]\tvalid_0's ndcg@10: 0.525101\n",
      "[700]\tvalid_0's ndcg@10: 0.525101\n",
      "[701]\tvalid_0's ndcg@10: 0.525101\n",
      "[702]\tvalid_0's ndcg@10: 0.525101\n",
      "[703]\tvalid_0's ndcg@10: 0.525101\n",
      "[704]\tvalid_0's ndcg@10: 0.525101\n",
      "[705]\tvalid_0's ndcg@10: 0.525101\n",
      "[706]\tvalid_0's ndcg@10: 0.525101\n",
      "[707]\tvalid_0's ndcg@10: 0.525101\n",
      "[708]\tvalid_0's ndcg@10: 0.537912\n",
      "[709]\tvalid_0's ndcg@10: 0.537912\n",
      "[710]\tvalid_0's ndcg@10: 0.537912\n",
      "[711]\tvalid_0's ndcg@10: 0.537912\n",
      "[712]\tvalid_0's ndcg@10: 0.537912\n",
      "[713]\tvalid_0's ndcg@10: 0.537912\n",
      "[714]\tvalid_0's ndcg@10: 0.537912\n",
      "[715]\tvalid_0's ndcg@10: 0.537912\n",
      "[716]\tvalid_0's ndcg@10: 0.537912\n",
      "[717]\tvalid_0's ndcg@10: 0.537912\n",
      "[718]\tvalid_0's ndcg@10: 0.537912\n",
      "[719]\tvalid_0's ndcg@10: 0.537912\n",
      "[720]\tvalid_0's ndcg@10: 0.525101\n",
      "[721]\tvalid_0's ndcg@10: 0.525101\n",
      "[722]\tvalid_0's ndcg@10: 0.525101\n",
      "[723]\tvalid_0's ndcg@10: 0.525101\n",
      "[724]\tvalid_0's ndcg@10: 0.525101\n",
      "[725]\tvalid_0's ndcg@10: 0.525101\n",
      "[726]\tvalid_0's ndcg@10: 0.525101\n",
      "[727]\tvalid_0's ndcg@10: 0.525101\n",
      "[728]\tvalid_0's ndcg@10: 0.525101\n",
      "[729]\tvalid_0's ndcg@10: 0.525101\n",
      "[730]\tvalid_0's ndcg@10: 0.525101\n",
      "[731]\tvalid_0's ndcg@10: 0.525101\n",
      "[732]\tvalid_0's ndcg@10: 0.525101\n",
      "[733]\tvalid_0's ndcg@10: 0.525101\n",
      "[734]\tvalid_0's ndcg@10: 0.525101\n",
      "[735]\tvalid_0's ndcg@10: 0.525101\n",
      "[736]\tvalid_0's ndcg@10: 0.525101\n",
      "[737]\tvalid_0's ndcg@10: 0.525101\n",
      "[738]\tvalid_0's ndcg@10: 0.525101\n",
      "[739]\tvalid_0's ndcg@10: 0.525101\n",
      "[740]\tvalid_0's ndcg@10: 0.525101\n",
      "[741]\tvalid_0's ndcg@10: 0.525101\n",
      "[742]\tvalid_0's ndcg@10: 0.525101\n",
      "[743]\tvalid_0's ndcg@10: 0.525101\n",
      "[744]\tvalid_0's ndcg@10: 0.525101\n",
      "[745]\tvalid_0's ndcg@10: 0.525101\n",
      "[746]\tvalid_0's ndcg@10: 0.525101\n",
      "[747]\tvalid_0's ndcg@10: 0.525101\n",
      "[748]\tvalid_0's ndcg@10: 0.525101\n",
      "[749]\tvalid_0's ndcg@10: 0.525101\n",
      "[750]\tvalid_0's ndcg@10: 0.525101\n",
      "[751]\tvalid_0's ndcg@10: 0.525101\n",
      "[752]\tvalid_0's ndcg@10: 0.525101\n",
      "[753]\tvalid_0's ndcg@10: 0.525101\n",
      "[754]\tvalid_0's ndcg@10: 0.525101\n",
      "[755]\tvalid_0's ndcg@10: 0.525101\n",
      "[756]\tvalid_0's ndcg@10: 0.525101\n",
      "[757]\tvalid_0's ndcg@10: 0.525101\n",
      "[758]\tvalid_0's ndcg@10: 0.525101\n",
      "[759]\tvalid_0's ndcg@10: 0.525101\n",
      "[760]\tvalid_0's ndcg@10: 0.525101\n",
      "[761]\tvalid_0's ndcg@10: 0.525101\n",
      "[762]\tvalid_0's ndcg@10: 0.525101\n",
      "[763]\tvalid_0's ndcg@10: 0.525657\n",
      "[764]\tvalid_0's ndcg@10: 0.525657\n",
      "[765]\tvalid_0's ndcg@10: 0.525657\n",
      "[766]\tvalid_0's ndcg@10: 0.525657\n",
      "[767]\tvalid_0's ndcg@10: 0.525657\n",
      "[768]\tvalid_0's ndcg@10: 0.525657\n",
      "[769]\tvalid_0's ndcg@10: 0.525657\n",
      "[770]\tvalid_0's ndcg@10: 0.525657\n",
      "[771]\tvalid_0's ndcg@10: 0.525657\n",
      "[772]\tvalid_0's ndcg@10: 0.525657\n",
      "[773]\tvalid_0's ndcg@10: 0.538468\n",
      "[774]\tvalid_0's ndcg@10: 0.538468\n",
      "[775]\tvalid_0's ndcg@10: 0.538468\n",
      "[776]\tvalid_0's ndcg@10: 0.525657\n",
      "[777]\tvalid_0's ndcg@10: 0.525657\n",
      "[778]\tvalid_0's ndcg@10: 0.525657\n",
      "[779]\tvalid_0's ndcg@10: 0.525657\n",
      "[780]\tvalid_0's ndcg@10: 0.525657\n",
      "[781]\tvalid_0's ndcg@10: 0.525657\n",
      "[782]\tvalid_0's ndcg@10: 0.525657\n",
      "[783]\tvalid_0's ndcg@10: 0.525657\n",
      "[784]\tvalid_0's ndcg@10: 0.525657\n",
      "[785]\tvalid_0's ndcg@10: 0.525657\n",
      "[786]\tvalid_0's ndcg@10: 0.525657\n",
      "[787]\tvalid_0's ndcg@10: 0.538468\n",
      "[788]\tvalid_0's ndcg@10: 0.538468\n",
      "[789]\tvalid_0's ndcg@10: 0.538468\n",
      "[790]\tvalid_0's ndcg@10: 0.538468\n",
      "[791]\tvalid_0's ndcg@10: 0.538468\n",
      "[792]\tvalid_0's ndcg@10: 0.538468\n",
      "[793]\tvalid_0's ndcg@10: 0.538468\n",
      "[794]\tvalid_0's ndcg@10: 0.538468\n",
      "[795]\tvalid_0's ndcg@10: 0.538468\n",
      "[796]\tvalid_0's ndcg@10: 0.538468\n",
      "[797]\tvalid_0's ndcg@10: 0.538468\n",
      "[798]\tvalid_0's ndcg@10: 0.538468\n",
      "[799]\tvalid_0's ndcg@10: 0.538468\n",
      "[800]\tvalid_0's ndcg@10: 0.538468\n",
      "[801]\tvalid_0's ndcg@10: 0.538468\n",
      "[802]\tvalid_0's ndcg@10: 0.538468\n",
      "[803]\tvalid_0's ndcg@10: 0.537912\n",
      "[804]\tvalid_0's ndcg@10: 0.537912\n",
      "[805]\tvalid_0's ndcg@10: 0.525101\n",
      "[806]\tvalid_0's ndcg@10: 0.537912\n",
      "[807]\tvalid_0's ndcg@10: 0.537912\n",
      "[808]\tvalid_0's ndcg@10: 0.525101\n",
      "[809]\tvalid_0's ndcg@10: 0.537912\n",
      "[810]\tvalid_0's ndcg@10: 0.537912\n",
      "[811]\tvalid_0's ndcg@10: 0.537912\n",
      "[812]\tvalid_0's ndcg@10: 0.537912\n",
      "[813]\tvalid_0's ndcg@10: 0.525101\n",
      "[814]\tvalid_0's ndcg@10: 0.525101\n",
      "[815]\tvalid_0's ndcg@10: 0.525101\n",
      "[816]\tvalid_0's ndcg@10: 0.525101\n",
      "[817]\tvalid_0's ndcg@10: 0.525101\n",
      "[818]\tvalid_0's ndcg@10: 0.525657\n",
      "[819]\tvalid_0's ndcg@10: 0.525657\n",
      "[820]\tvalid_0's ndcg@10: 0.538468\n",
      "[821]\tvalid_0's ndcg@10: 0.537912\n",
      "[822]\tvalid_0's ndcg@10: 0.537912\n",
      "[823]\tvalid_0's ndcg@10: 0.537912\n",
      "[824]\tvalid_0's ndcg@10: 0.537912\n",
      "[825]\tvalid_0's ndcg@10: 0.537912\n",
      "[826]\tvalid_0's ndcg@10: 0.537912\n",
      "[827]\tvalid_0's ndcg@10: 0.537912\n",
      "[828]\tvalid_0's ndcg@10: 0.537912\n",
      "[829]\tvalid_0's ndcg@10: 0.537912\n",
      "[830]\tvalid_0's ndcg@10: 0.537912\n",
      "[831]\tvalid_0's ndcg@10: 0.537912\n",
      "[832]\tvalid_0's ndcg@10: 0.537912\n",
      "[833]\tvalid_0's ndcg@10: 0.537912\n",
      "[834]\tvalid_0's ndcg@10: 0.537912\n",
      "[835]\tvalid_0's ndcg@10: 0.537912\n",
      "[836]\tvalid_0's ndcg@10: 0.537912\n",
      "[837]\tvalid_0's ndcg@10: 0.537912\n",
      "[838]\tvalid_0's ndcg@10: 0.537912\n",
      "[839]\tvalid_0's ndcg@10: 0.537912\n",
      "[840]\tvalid_0's ndcg@10: 0.537912\n",
      "[841]\tvalid_0's ndcg@10: 0.537912\n",
      "[842]\tvalid_0's ndcg@10: 0.537912\n",
      "[843]\tvalid_0's ndcg@10: 0.537912\n",
      "[844]\tvalid_0's ndcg@10: 0.537912\n",
      "[845]\tvalid_0's ndcg@10: 0.537912\n",
      "[846]\tvalid_0's ndcg@10: 0.537912\n",
      "[847]\tvalid_0's ndcg@10: 0.537912\n",
      "[848]\tvalid_0's ndcg@10: 0.537912\n",
      "[849]\tvalid_0's ndcg@10: 0.537912\n",
      "[850]\tvalid_0's ndcg@10: 0.537912\n",
      "[851]\tvalid_0's ndcg@10: 0.537912\n",
      "[852]\tvalid_0's ndcg@10: 0.525101\n",
      "[853]\tvalid_0's ndcg@10: 0.525101\n",
      "[854]\tvalid_0's ndcg@10: 0.525101\n",
      "[855]\tvalid_0's ndcg@10: 0.525101\n",
      "[856]\tvalid_0's ndcg@10: 0.525101\n",
      "[857]\tvalid_0's ndcg@10: 0.525101\n",
      "[858]\tvalid_0's ndcg@10: 0.525101\n",
      "[859]\tvalid_0's ndcg@10: 0.525101\n",
      "[860]\tvalid_0's ndcg@10: 0.525101\n",
      "[861]\tvalid_0's ndcg@10: 0.525101\n",
      "[862]\tvalid_0's ndcg@10: 0.525101\n",
      "[863]\tvalid_0's ndcg@10: 0.525101\n",
      "[864]\tvalid_0's ndcg@10: 0.525101\n",
      "[865]\tvalid_0's ndcg@10: 0.525101\n",
      "[866]\tvalid_0's ndcg@10: 0.525101\n",
      "[867]\tvalid_0's ndcg@10: 0.525101\n",
      "[868]\tvalid_0's ndcg@10: 0.525101\n",
      "[869]\tvalid_0's ndcg@10: 0.525101\n",
      "[870]\tvalid_0's ndcg@10: 0.525101\n",
      "[871]\tvalid_0's ndcg@10: 0.525101\n",
      "[872]\tvalid_0's ndcg@10: 0.525101\n",
      "[873]\tvalid_0's ndcg@10: 0.525101\n",
      "[874]\tvalid_0's ndcg@10: 0.525101\n",
      "[875]\tvalid_0's ndcg@10: 0.525101\n",
      "[876]\tvalid_0's ndcg@10: 0.525101\n",
      "[877]\tvalid_0's ndcg@10: 0.525101\n",
      "[878]\tvalid_0's ndcg@10: 0.525101\n",
      "[879]\tvalid_0's ndcg@10: 0.525101\n",
      "[880]\tvalid_0's ndcg@10: 0.525101\n",
      "[881]\tvalid_0's ndcg@10: 0.525657\n",
      "[882]\tvalid_0's ndcg@10: 0.525657\n",
      "[883]\tvalid_0's ndcg@10: 0.525657\n",
      "[884]\tvalid_0's ndcg@10: 0.525657\n",
      "[885]\tvalid_0's ndcg@10: 0.525101\n",
      "[886]\tvalid_0's ndcg@10: 0.525101\n",
      "[887]\tvalid_0's ndcg@10: 0.525101\n",
      "[888]\tvalid_0's ndcg@10: 0.525101\n",
      "[889]\tvalid_0's ndcg@10: 0.525101\n",
      "[890]\tvalid_0's ndcg@10: 0.525101\n",
      "[891]\tvalid_0's ndcg@10: 0.525101\n",
      "[892]\tvalid_0's ndcg@10: 0.525101\n",
      "[893]\tvalid_0's ndcg@10: 0.525101\n",
      "[894]\tvalid_0's ndcg@10: 0.525101\n",
      "[895]\tvalid_0's ndcg@10: 0.525101\n",
      "[896]\tvalid_0's ndcg@10: 0.525101\n",
      "[897]\tvalid_0's ndcg@10: 0.525101\n",
      "[898]\tvalid_0's ndcg@10: 0.525101\n",
      "[899]\tvalid_0's ndcg@10: 0.525101\n",
      "[900]\tvalid_0's ndcg@10: 0.525101\n",
      "[901]\tvalid_0's ndcg@10: 0.525101\n",
      "[902]\tvalid_0's ndcg@10: 0.525101\n",
      "[903]\tvalid_0's ndcg@10: 0.525101\n",
      "[904]\tvalid_0's ndcg@10: 0.525101\n",
      "[905]\tvalid_0's ndcg@10: 0.525101\n",
      "[906]\tvalid_0's ndcg@10: 0.525101\n",
      "[907]\tvalid_0's ndcg@10: 0.524853\n",
      "[908]\tvalid_0's ndcg@10: 0.524853\n",
      "[909]\tvalid_0's ndcg@10: 0.524853\n",
      "[910]\tvalid_0's ndcg@10: 0.524853\n",
      "[911]\tvalid_0's ndcg@10: 0.524853\n",
      "[912]\tvalid_0's ndcg@10: 0.524853\n",
      "[913]\tvalid_0's ndcg@10: 0.524853\n",
      "[914]\tvalid_0's ndcg@10: 0.524853\n",
      "[915]\tvalid_0's ndcg@10: 0.524853\n",
      "[916]\tvalid_0's ndcg@10: 0.524853\n",
      "[917]\tvalid_0's ndcg@10: 0.524853\n",
      "[918]\tvalid_0's ndcg@10: 0.524853\n",
      "[919]\tvalid_0's ndcg@10: 0.524853\n",
      "[920]\tvalid_0's ndcg@10: 0.524853\n",
      "[921]\tvalid_0's ndcg@10: 0.524853\n",
      "[922]\tvalid_0's ndcg@10: 0.524853\n",
      "[923]\tvalid_0's ndcg@10: 0.524853\n",
      "[924]\tvalid_0's ndcg@10: 0.524853\n",
      "[925]\tvalid_0's ndcg@10: 0.524853\n",
      "[926]\tvalid_0's ndcg@10: 0.524853\n",
      "[927]\tvalid_0's ndcg@10: 0.524853\n",
      "[928]\tvalid_0's ndcg@10: 0.524853\n",
      "[929]\tvalid_0's ndcg@10: 0.524853\n",
      "[930]\tvalid_0's ndcg@10: 0.524853\n",
      "[931]\tvalid_0's ndcg@10: 0.524853\n",
      "[932]\tvalid_0's ndcg@10: 0.524853\n",
      "[933]\tvalid_0's ndcg@10: 0.524853\n",
      "[934]\tvalid_0's ndcg@10: 0.524853\n",
      "[935]\tvalid_0's ndcg@10: 0.524853\n",
      "[936]\tvalid_0's ndcg@10: 0.524853\n",
      "[937]\tvalid_0's ndcg@10: 0.524853\n",
      "[938]\tvalid_0's ndcg@10: 0.537665\n",
      "[939]\tvalid_0's ndcg@10: 0.524853\n",
      "[940]\tvalid_0's ndcg@10: 0.524853\n",
      "[941]\tvalid_0's ndcg@10: 0.524853\n",
      "[942]\tvalid_0's ndcg@10: 0.524853\n",
      "[943]\tvalid_0's ndcg@10: 0.524853\n",
      "[944]\tvalid_0's ndcg@10: 0.524853\n",
      "[945]\tvalid_0's ndcg@10: 0.524853\n",
      "[946]\tvalid_0's ndcg@10: 0.524853\n",
      "[947]\tvalid_0's ndcg@10: 0.524853\n",
      "[948]\tvalid_0's ndcg@10: 0.524853\n",
      "[949]\tvalid_0's ndcg@10: 0.524853\n",
      "[950]\tvalid_0's ndcg@10: 0.524853\n",
      "[951]\tvalid_0's ndcg@10: 0.524853\n",
      "[952]\tvalid_0's ndcg@10: 0.524853\n",
      "[953]\tvalid_0's ndcg@10: 0.524853\n",
      "[954]\tvalid_0's ndcg@10: 0.524853\n",
      "[955]\tvalid_0's ndcg@10: 0.524853\n",
      "[956]\tvalid_0's ndcg@10: 0.524853\n",
      "[957]\tvalid_0's ndcg@10: 0.524853\n",
      "[958]\tvalid_0's ndcg@10: 0.524853\n",
      "[959]\tvalid_0's ndcg@10: 0.524853\n",
      "[960]\tvalid_0's ndcg@10: 0.524853\n",
      "[961]\tvalid_0's ndcg@10: 0.524853\n",
      "[962]\tvalid_0's ndcg@10: 0.524853\n",
      "[963]\tvalid_0's ndcg@10: 0.524853\n",
      "[964]\tvalid_0's ndcg@10: 0.524853\n",
      "[965]\tvalid_0's ndcg@10: 0.524853\n",
      "[966]\tvalid_0's ndcg@10: 0.524853\n",
      "[967]\tvalid_0's ndcg@10: 0.524853\n",
      "[968]\tvalid_0's ndcg@10: 0.524853\n",
      "[969]\tvalid_0's ndcg@10: 0.524853\n",
      "[970]\tvalid_0's ndcg@10: 0.524853\n",
      "[971]\tvalid_0's ndcg@10: 0.524853\n",
      "[972]\tvalid_0's ndcg@10: 0.524853\n",
      "[973]\tvalid_0's ndcg@10: 0.524853\n",
      "[974]\tvalid_0's ndcg@10: 0.524853\n",
      "[975]\tvalid_0's ndcg@10: 0.524853\n",
      "[976]\tvalid_0's ndcg@10: 0.524853\n",
      "[977]\tvalid_0's ndcg@10: 0.524853\n",
      "[978]\tvalid_0's ndcg@10: 0.524853\n",
      "[979]\tvalid_0's ndcg@10: 0.524853\n",
      "[980]\tvalid_0's ndcg@10: 0.524853\n",
      "[981]\tvalid_0's ndcg@10: 0.524853\n",
      "[982]\tvalid_0's ndcg@10: 0.524853\n",
      "[983]\tvalid_0's ndcg@10: 0.524853\n",
      "[984]\tvalid_0's ndcg@10: 0.524853\n",
      "[985]\tvalid_0's ndcg@10: 0.524853\n",
      "[986]\tvalid_0's ndcg@10: 0.524853\n",
      "[987]\tvalid_0's ndcg@10: 0.524853\n",
      "[988]\tvalid_0's ndcg@10: 0.524853\n",
      "[989]\tvalid_0's ndcg@10: 0.524853\n",
      "[990]\tvalid_0's ndcg@10: 0.524853\n",
      "[991]\tvalid_0's ndcg@10: 0.524853\n",
      "[992]\tvalid_0's ndcg@10: 0.524853\n",
      "[993]\tvalid_0's ndcg@10: 0.524853\n",
      "[994]\tvalid_0's ndcg@10: 0.524853\n",
      "[995]\tvalid_0's ndcg@10: 0.524853\n",
      "[996]\tvalid_0's ndcg@10: 0.524853\n",
      "[997]\tvalid_0's ndcg@10: 0.524853\n",
      "[998]\tvalid_0's ndcg@10: 0.524853\n",
      "[999]\tvalid_0's ndcg@10: 0.524853\n",
      "[1000]\tvalid_0's ndcg@10: 0.524853\n"
     ]
    }
   ],
   "source": [
    "ltr.fit(\n",
    "    dataset_train.get_topics()[:-5],\n",
    "    dataset_train.get_qrels(),\n",
    "    dataset_train.get_topics()[-5:],\n",
    "    dataset_train.get_qrels()\n",
    ")"
   ],
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%%\n"
    }
   }
  },
  {
   "cell_type": "markdown",
   "source": [
    "## Experimental Evaluation\n",
    "Because our axiomatic re-rankers are PyTerrier modules, we can now use PyTerrier's `Experiment` interface to evaluate various metrics and to compare our new approaches to the BM25 baseline ranking.\n",
    "Refer to the PyTerrier [documentation](https://pyterrier.readthedocs.io/en/latest/experiments.html) to learn more about running experiments.\n",
    "(We concatenate results from the Baseline ranking for the ranking positions after the top-20 using the `^` operator.)"
   ],
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%% md\n"
    }
   }
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "outputs": [
    {
     "data": {
      "text/plain": "pt.Experiment:   0%|          | 0/4 [00:00<?, ?system/s]",
      "application/vnd.jupyter.widget-view+json": {
       "version_major": 2,
       "version_minor": 0,
       "model_id": "239d8faca3d24bdf89b58f71181c26c3"
      }
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/plain": "Reranking query axiomatically:   0%|          | 0/54 [00:00<?, ?query/s]",
      "application/vnd.jupyter.widget-view+json": {
       "version_major": 2,
       "version_minor": 0,
       "model_id": "111730074173468ba8cdeb027e79e7ff"
      }
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/plain": "Reranking query axiomatically:   0%|          | 0/54 [00:00<?, ?query/s]",
      "application/vnd.jupyter.widget-view+json": {
       "version_major": 2,
       "version_minor": 0,
       "model_id": "505b24edc6c04de3a102c470b2af802d"
      }
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/plain": "Aggregating query axiom preferences:   0%|          | 0/54 [00:00<?, ?query/s]",
      "application/vnd.jupyter.widget-view+json": {
       "version_major": 2,
       "version_minor": 0,
       "model_id": "97ccd23856ed4f2fa34c9d02ad09fee4"
      }
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "from pyterrier.pipelines import Experiment\n",
    "from ir_measures import nDCG, MAP, RR\n",
    "\n",
    "results_dir.mkdir(exist_ok=True)\n",
    "experiment = Experiment(\n",
    "    [\n",
    "        bm25,\n",
    "        kwiksort ^ bm25,\n",
    "        kwiksort_random_forest ^ bm25,\n",
    "        ltr ^ bm25\n",
    "    ],\n",
    "    dataset_test.get_topics(),\n",
    "    dataset_test.get_qrels(),\n",
    "    [nDCG @ 10, RR, MAP],\n",
    "    [\n",
    "        \"BM25\",\n",
    "        \"KwikSort\",\n",
    "        \"KwikSort Random Forest\",\n",
    "        \"Axiomatic LTR\"\n",
    "    ],\n",
    "    verbose=True,\n",
    ")\n",
    "experiment.sort_values(by=\"nDCG@10\", ascending=False, inplace=True)"
   ],
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%%\n"
    }
   }
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "outputs": [
    {
     "data": {
      "text/plain": "                     name   nDCG@10        RR        AP\n3           Axiomatic LTR  0.498449  0.831643  0.363394\n2  KwikSort Random Forest  0.494419  0.825317  0.361909\n0                    BM25  0.493627  0.802359  0.358724\n1                KwikSort  0.491858  0.802102  0.358587",
      "text/html": "<div>\n<style scoped>\n    .dataframe tbody tr th:only-of-type {\n        vertical-align: middle;\n    }\n\n    .dataframe tbody tr th {\n        vertical-align: top;\n    }\n\n    .dataframe thead th {\n        text-align: right;\n    }\n</style>\n<table border=\"1\" class=\"dataframe\">\n  <thead>\n    <tr style=\"text-align: right;\">\n      <th></th>\n      <th>name</th>\n      <th>nDCG@10</th>\n      <th>RR</th>\n      <th>AP</th>\n    </tr>\n  </thead>\n  <tbody>\n    <tr>\n      <th>3</th>\n      <td>Axiomatic LTR</td>\n      <td>0.498449</td>\n      <td>0.831643</td>\n      <td>0.363394</td>\n    </tr>\n    <tr>\n      <th>2</th>\n      <td>KwikSort Random Forest</td>\n      <td>0.494419</td>\n      <td>0.825317</td>\n      <td>0.361909</td>\n    </tr>\n    <tr>\n      <th>0</th>\n      <td>BM25</td>\n      <td>0.493627</td>\n      <td>0.802359</td>\n      <td>0.358724</td>\n    </tr>\n    <tr>\n      <th>1</th>\n      <td>KwikSort</td>\n      <td>0.491858</td>\n      <td>0.802102</td>\n      <td>0.358587</td>\n    </tr>\n  </tbody>\n</table>\n</div>"
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "experiment"
   ],
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%%\n"
    }
   }
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 2
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython2",
   "version": "2.7.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 0
}