Foundations_of_Predictive_Analytics_in_Python_Part 2.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Foundations of Predictive Analytics in Python (Part 2)\n",
    "\n",
    "Building good models only succeeds if you have a decent base table to start with. In this course you will learn how to construct a good base table, create variables and prepare your data for modeling. We finish with advanced topics on the matter. \n",
    "\n",
    "## Crucial base table concepts\n",
    "\n",
    "In this chapter you will learn how to construct the foundations of your base table, namely the population and the target.\n",
    "\n",
    "A predictive model can be use the predict an event.\n",
    "\n",
    "All information needed to make these predections are stored in the basetable. There are three important concept in the basetable. <br>\n",
    "**1- Population** is the group of people or object you want to make a prediction for.<br>\n",
    "**2- Candidate Predictors** describe the object in the populations(age, gender, etc...)<br>\n",
    "**3- target** has information the event the predict itself. It is one if the events occurs, and zero otherwise.\n",
    "\n",
    "**Draw a timeline** When biulding a baseline table for predictive modelling, the first thingyou should do is draw a timeline, on this timeline, you could depict the situation in which you want to use the predictive model. \n",
    "\n",
    "\n",
    "We want to construct a model that predicts `which donors are most likely to donate more than 50 Euro in April 2018`. To build the predictive model, we reconstruct the timeline one year back in time, so the target period of the basetable is April 2017.\n",
    "\n",
    "Assume that you want to construct a model that predicts whether someone will donate in a certain year. The timeline to construct the basetable has 2017 as target period, this means that the target is based on donations made in 2017, and that the predictive variables are based on donations made before 2017. All donations are given in a pandas dataframe gifts with three columns: the donor id, the donation date and the amount donated. In this exercise you will learn to construct a new pandas dataframe that excludes donations made in 2017 or later.\n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "#load the libraries\n",
    "import numpy as np\n",
    "import pandas as pd\n",
    "from datetime import datetime\n",
    "import warnings\n",
    "warnings.filterwarnings('ignore')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 62,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "   id        date  amount\n",
      "0   1  2015-10-16    75.0\n",
      "1   1  2014-02-11   111.0\n",
      "2   1  2012-03-28    93.0\n",
      "3   1  2013-12-13   113.0\n",
      "4   1  2012-01-10    93.0\n",
      "<class 'str'>\n"
     ]
    }
   ],
   "source": [
    "#import the data\n",
    "gift = pd.read_csv('data/gifts.csv', index_col=0)\n",
    "print(gift.head())\n",
    "print(type(gift['date'][0]))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 112,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<class 'pandas._libs.tslibs.timestamps.Timestamp'>\n",
      "   id       date  amount\n",
      "0   1 2015-10-16    75.0\n",
      "1   1 2014-02-11   111.0\n",
      "2   1 2012-03-28    93.0\n",
      "3   1 2013-12-13   113.0\n",
      "4   1 2012-01-10    93.0\n",
      "5   1 2015-04-22    85.0\n",
      "6   1 2014-03-05   109.0\n",
      "7   2 2015-04-30    53.0\n",
      "8   2 2012-06-05   101.0\n",
      "9   2 2014-01-18    86.0\n"
     ]
    }
   ],
   "source": [
    "#convert the date column to datetime \n",
    "gift['date'] = pd.to_datetime(gift['date'])\n",
    "print(type(gift['date'][0]))\n",
    "print(gift.head(10))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 64,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "id        145045\n",
      "date      145045\n",
      "amount    145045\n",
      "dtype: int64\n"
     ]
    }
   ],
   "source": [
    "# Start of the target is January 1st 2017\n",
    "start_date = datetime(year=2017, month=1, day=1)\n",
    "# Select gifts made before start_target\n",
    "gift_before_2017 = gift[gift['date'] < start_date]\n",
    "print(gift_before_2017.count())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Timeline violation\n",
    "To illustrate the importance of the timeline, consider an example where you violate the timeline and use information from the target period to construct the predictive variables.\n",
    "\n",
    "Let's crete a base table which will have two columns in the pandas dataframe basetable: \"amount_2017\" is the total amount of donations in 2017, and \"target\" is 1 if this amount is larger than 30 and 0 else.\n",
    "\n",
    "Construct a logistic regression model that uses \"amount_2017\" as single predictive variable to predict the target, and calculate the AUC"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "metadata": {},
   "outputs": [],
   "source": [
    "start_date = datetime(year=2017, month=1, day=1)\n",
    "end_date = datetime(year=2017, month=12, day=30)\n",
    "\n",
    "#create a data frame which only have the information from year 2017\n",
    "year_2017 = gift[(gift['date']> start_date) & (gift['date']<end_date)]\n",
    "#year_2017\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "metadata": {},
   "outputs": [],
   "source": [
    "#create target varibale which only have 30 euro or more donation made \n",
    "year_2017['target'] = np.where(year_2017['amount']>=30, 1,0)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 50,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "1.0\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "'Great job! As you can see, the model makes perfect predictions, \\n but is not realistic as target, the amount given, is not available if the gift has not been made yet.'"
      ]
     },
     "execution_count": 50,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Select the relevant predictors and the target\n",
    "X_year_2017 = np.array(year_2017[['amount']])\n",
    "#we need to rehsape the data for single future information\n",
    "X_year_2017 = X_year_2017.reshape(-1,1)\n",
    "y_year_2017 = year_2017['target']\n",
    "\n",
    "\n",
    "# Build the logistic regression model\n",
    "from sklearn import linear_model\n",
    "logreg = linear_model.LogisticRegression()\n",
    "#fit the model\n",
    "logreg.fit(X_year_2017,y_year_2017)\n",
    "# Make predictions for X\n",
    "predictions = logreg.predict_proba(X_year_2017)[:,1]\n",
    "# Calculate and print the AUC value\n",
    "from sklearn.metrics import roc_auc_score\n",
    "auc_year =roc_auc_score(y_year_2017, predictions)\n",
    "print(round(auc_year,2))\n",
    "\"\"\"Great job! As you can see, the model makes perfect predictions, \n",
    " but is not realistic as target, the amount given, is not available if the gift has not been made yet.\"\"\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## The population\n",
    "The population should be eligible for beging target\n",
    "* addrees should be avaible\n",
    "* privacy setting should be allow sending a letter \n",
    "\n",
    "*Time population: if you want to target younger than 25, you should make sure your target is youger than specific timeline that you are modelling for predicting.*\n",
    "\n",
    " \n",
    "Assume that you want to construct a basetable for a predictive model that predicts whether donors will donate in 2018. The timeline indicates that the population should contain `all donors that donated at least once since January 1st 2013`, `but made no donations after January 1st 2017`. Given is a pandas dataframe gifts with all the donations made since 2010. In this exercise, you will construct a set with the donor ids of all donors in the population.\n",
    "* find the donors made a donation after 2013 and later\n",
    "* find the donor made the donation after 2017 and later\n",
    "* exclude the 2017 and later donors from 2013 and later for our specific timeline \n",
    "* then we find the donor since January 1st 2013 but not in 2017 or later\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 114,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "27362 3843\n",
      "Population of the donor is:  23519\n"
     ]
    }
   ],
   "source": [
    "# Gifts made in 2013 or later\n",
    "gift_include = gift[gift['date'].dt.year >=2013]\n",
    "\n",
    "# Gifts made in 2017 or later\n",
    "gift_exclude =  gift[gift['date'].dt.year >=2017]\n",
    "\n",
    "#using the set() function to find unique donor's id \n",
    "donors_include = set(gift_include['id'])\n",
    "donors_exclude = set(gift_exclude['id'])\n",
    "print(len(donors_include), len(donors_exclude))\n",
    "#population is set of unique donor's id since donation 2013 but not in 2017\n",
    "population = donors_include.difference(donors_exclude)\n",
    "print('Population of the donor is: ',len(population))\n",
    "#print(gift_exclude)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Removing duplicate objects\n",
    "Assume that you want to construct a predictive model in order to select donors that are most likely to respond on a letter. The population of the basetable should contain donors that have an adress available, and that have privacy settings that allow to send them a letter. All candidate donors are given in a dataframe donors with three columns: the donor_id, a flag address that is 1 if the address is available and 0 otherwise, and a flag letter_allowed that is 1 if one can send this donor a letter and 0 otherwise. In this exercise you will construct a set with the donors that should go in the population."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "   id        date  amount  address  letter_allowed\n",
      "0   1  2015-10-16    75.0        1               1\n",
      "1   1  2014-02-11   111.0        1               1\n",
      "2   1  2012-03-28    93.0        0               1\n",
      "3   1  2013-12-13   113.0        0               1\n",
      "4   1  2012-01-10    93.0        1               1\n",
      "Lenght of the original data 150000\n"
     ]
    }
   ],
   "source": [
    "#load the dataframe with information address and letter allowed\n",
    "donors = pd.read_csv('data/gift_address_letter.csv', index_col=0)\n",
    "print(donors.head())\n",
    "print('Lenght of the original data',len(donors))\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Selected population  20698\n"
     ]
    }
   ],
   "source": [
    "#donor_population which has adress information and letter allowed\n",
    "donor_population = donors[(donors['address'] == 1) & (donors['letter_allowed'] == 1)]\n",
    "population_list = list(donor_population['id'])\n",
    "unique_population  = set(population_list)\n",
    "print('Selected population ', len(unique_population))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## The Target\n",
    "\n",
    "Once the timeline is set and the population in place, you are ready to add the target to the basetable. The target is a special column in the basetable, namely the value, zero or one, that you want to predict. In predicting modeling target id equel to the one if a certain events happends during the target period for the observations and zero otherwise.\n",
    "\n",
    "it is an unknown events that you want to predic. \n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "#load the doonor id unformayion\n",
    "donor_id = pd.read_csv('data/basetable.csv')\n",
    "donot_id = list(donor_id)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Calculate an event target\n",
    "======= You are organising a charity event and want to predict which donors are most likely to attend this event. You organized a similar event in the past, so you can use that information to construct a predictive model. Given is a list population with unique donor ids for this basetable and a list attend_event with donors in the population that attended this past event. In this exercise you will construct a basetable with two columns: the donor_id and the target, which is 1 if the donor attended the event and 0 otherwise.\n",
    "```python\n",
    "# Basetable with one column: donor_id\n",
    "basetable = pd.DataFrame(population, columns=[\"donor_id\"])\n",
    "# Add target to the basetable\n",
    "basetable[\"target\"] = pd.Series([1 if donor_id in donor_id else 0 for donor_id in basetable[\"donor_id\"]])\n",
    "\n",
    "# Calculate and print the target incidence\n",
    "print(round(basetable[\"target\"].sum() / len(basetable['target']), 2))\n",
    "\n",
    "```\n",
    "Assume you want to construct a predictive model that predicts which donors are most likely to donate more than 50 euro in a certain month.\n",
    "\n",
    "Given is a basetable basetable that already has one row for each donor in the population, the column donor_id represents the donor. The timeline indicates that the target should be 1 if the donor has donated more than 50 euro in January 2017 and 0 else.\n",
    "\n",
    "\n",
    "```python\n",
    "# Sum of donations for each donor in gifts_201701\n",
    "gifts_summed = gifts_201701.groupby(\"id\")[\"amount\"].sum().reset_index()\n",
    "\n",
    "# List with targets\n",
    "targets = list(gifts_summed[\"id\"][gifts_summed[\"amount\"] > 50])\n",
    "\n",
    "# Add targets to the basetable\n",
    "basetable[\"target\"] = pd.Series([1 if donor_id in targets else 0 for donor_id in basetable[\"donor_id\"]])\n",
    "\n",
    "# Calculate and print the target incidence\n",
    "print(round(basetable[\"target\"].sum() / len(basetable), 2))\n",
    "```\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## The basetable timeline\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Adding the donor segment\n",
    "Besides age, you also want to add the segment of a donor to the basetable. A selected group of donors that has made many donations in the past is assigned a segment: bronze, silver or gold. Given is an early stage basetable and a pandas dataframe segments that contains the segments for a selected group of the donors in the basetable. In this exercise you will add the segment to the basetable.\n",
    "\n",
    "You can left join two pandas dataframes using the following code:\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.2"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}