5.AB-Test II.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true,
    "deletable": true,
    "editable": true
   },
   "source": [
    "# Does a new ad on the webpage drive more success for both the company and customers?\n",
    "In the Online Education Website, they made the second change that is a more career focused description(ad) on a course overview page in the hope that this change may encourage more users to enroll and complete this course. In this experiment, we’re going to analyze the following **metrics**: \n",
    "\n",
    "1. **Enrollment Rate:** Click through rate for the Enroll button the course overview page\n",
    "2. **Average Reading Duration:** Average number of seconds spent on the course overview page\n",
    "3. **Average Classroom Time:** Average number of days spent in the classroom for students enrolled in the course\n",
    "4. **Completion Rate:** Course completion rate for students enrolled in the course\n",
    "\n",
    "First, let's determine if the difference observed for each metric is statistically significant individually."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": true,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import pandas as pd\n",
    "import matplotlib.pyplot as plt\n",
    "% matplotlib inline\n",
    "\n",
    "np.random.seed(42)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "### Metric 1. Enrollment Rate"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>timestamp</th>\n",
       "      <th>id</th>\n",
       "      <th>group</th>\n",
       "      <th>action</th>\n",
       "      <th>duration</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>2016-09-24 17:14:52.012145</td>\n",
       "      <td>261869</td>\n",
       "      <td>experiment</td>\n",
       "      <td>view</td>\n",
       "      <td>130.545004</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>2016-09-24 18:45:09.645857</td>\n",
       "      <td>226546</td>\n",
       "      <td>experiment</td>\n",
       "      <td>view</td>\n",
       "      <td>159.862440</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>2016-09-24 19:16:21.002533</td>\n",
       "      <td>286353</td>\n",
       "      <td>experiment</td>\n",
       "      <td>view</td>\n",
       "      <td>79.349315</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>2016-09-24 19:43:06.927785</td>\n",
       "      <td>842279</td>\n",
       "      <td>experiment</td>\n",
       "      <td>view</td>\n",
       "      <td>55.536126</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>2016-09-24 21:08:22.790333</td>\n",
       "      <td>781883</td>\n",
       "      <td>experiment</td>\n",
       "      <td>view</td>\n",
       "      <td>204.322437</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                    timestamp      id       group action    duration\n",
       "0  2016-09-24 17:14:52.012145  261869  experiment   view  130.545004\n",
       "1  2016-09-24 18:45:09.645857  226546  experiment   view  159.862440\n",
       "2  2016-09-24 19:16:21.002533  286353  experiment   view   79.349315\n",
       "3  2016-09-24 19:43:06.927785  842279  experiment   view   55.536126\n",
       "4  2016-09-24 21:08:22.790333  781883  experiment   view  204.322437"
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df = pd.read_csv('C:/Users/Minkun/Desktop/classes_1/NanoDeg/1.Data_AN/L4/data/course_page_actions.csv')\n",
    "df.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "df.info()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "df.action.unique()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.2364438839848676"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Get dataframe with all records from control group\n",
    "control_df = df.query('group == \"control\"')\n",
    "\n",
    "# Compute click through rate for control group\n",
    "control_ctr = control_df.query('action == \"enroll\"').id.nunique() / control_df.query('action == \"view\"').id.nunique()\n",
    "\n",
    "# Display click through rate\n",
    "control_ctr"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.2668693009118541"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Get dataframe with all records from experiment group\n",
    "experiment_df = df.query('group == \"experiment\"')\n",
    "\n",
    "# Compute click through rate for experiment group\n",
    "experiment_ctr = experiment_df.query('action == \"enroll\"').id.nunique() / experiment_df.query('action == \"view\"').id.nunique()\n",
    "\n",
    "# Display click through rate\n",
    "experiment_ctr"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.030425416926986526"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Compute the observed difference in click through rates\n",
    "obs_diff = experiment_ctr - control_ctr\n",
    "\n",
    "# Display observed difference\n",
    "obs_diff"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "collapsed": true,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "# Create a sampling distribution of the difference in proportions with bootstrapping\n",
    "diffs = []\n",
    "for i in range(10000):\n",
    "    b_samp = df.sample(df.shape[0], replace =True)\n",
    "    ctrl_df = b_samp.query('group == \"control\"')\n",
    "    exp_df = b_samp.query('group == \"experiment\"')\n",
    "    ctrl_ctr = ctrl_df.query('action == \"enroll\"').id.nunique() / ctrl_df.query('action == \"view\"').id.nunique()\n",
    "    exp_ctr = exp_df.query('action == \"enroll\"').id.nunique() / exp_df.query('action == \"view\"').id.nunique()\n",
    "    diffs.append(exp_ctr - ctrl_ctr)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(array([   11.,    80.,   520.,  1702.,  3145.,  2744.,  1368.,   367.,\n",
       "           57.,     6.]),\n",
       " array([-0.02943202, -0.01728909, -0.00514617,  0.00699675,  0.01913967,\n",
       "         0.03128259,  0.04342551,  0.05556843,  0.06771136,  0.07985428,\n",
       "         0.0919972 ]),\n",
       " <a list of 10 Patch objects>)"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYEAAAD8CAYAAACRkhiPAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAEhdJREFUeJzt3X+snNV95/H3p4YQNgkFlruWYztrIrkrGaSaYHnZtlpl\nl27xQlVTVYocqYWqWdwKNkq6lVam/aPpH5ZI01+LtLDrJlnMNg2ymqRYAbolVqS2UoFeshRjExYn\nwOJbg29bdUn3Dyo73/4xh3a4udd37q8ZO+f9kh7NmfOc8zznMBd/5vkxM6kqJEl9+p5JD0CSNDmG\ngCR1zBCQpI4ZApLUMUNAkjpmCEhSxwwBSeqYISBJHTMEJKljF016AIu56qqrasuWLZMehiRdUJ5+\n+um/rKqpxdqd9yGwZcsWpqenJz0MSbqgJHlllHaeDpKkjhkCktQxQ0CSOmYISFLHDAFJ6pghIEkd\nMwQkqWOGgCR1zBCQpI6d958YlhazZd8jE9nvy/fcMpH9SqvJIwFJ6pghIEkdMwQkqWOGgCR1zBCQ\npI4tGgJJ3pnkqSR/nuRYkl9p9VcmeTzJi+3xiqE+dyc5keSFJDcN1V+f5Ghbd2+SrM20JEmjGOVI\n4E3g31bV9wPbgV1JbgD2AUeqaitwpD0nyTZgD3ANsAu4L8m6tq37gTuArW3ZtYpzkSQt0aIhUAN/\n255e3JYCdgMHW/1B4NZW3g08VFVvVtVLwAlgZ5INwGVV9URVFfDgUB9J0gSMdE0gybokzwCngcer\n6klgfVWdak1eA9a38kbg1aHuJ1vdxlaeWz/f/vYmmU4yPTs7O/JkJElLM1IIVNXZqtoObGLwrv7a\nOeuLwdHBqqiqA1W1o6p2TE0t+jvJkqRlWtLdQVX1N8BXGZzLf72d4qE9nm7NZoDNQ902tbqZVp5b\nL0makFHuDppKcnkrXwr8O+DrwGHg9tbsduDhVj4M7ElySZKrGVwAfqqdOnojyQ3trqDbhvpIkiZg\nlC+Q2wAcbHf4fA9wqKq+nORPgUNJPgK8AnwIoKqOJTkEHAfOAHdV1dm2rTuBB4BLgcfaIkmakEVD\noKqeBa6bp/6vgBsX6LMf2D9P/TRw7Xf2kCRNgp8YlqSOGQKS1DFDQJI6ZghIUscMAUnqmCEgSR0z\nBCSpY4aAJHXMEJCkjhkCktQxQ0CSOjbKF8hJmseWfY9MbN8v33PLxPat7y4eCUhSxwwBSeqYISBJ\nHTMEJKljhoAkdcwQkKSOGQKS1DFDQJI6ZghIUscMAUnqmCEgSR0zBCSpY4uGQJLNSb6a5HiSY0k+\n1uo/kWQmyTNtuXmoz91JTiR5IclNQ/XXJzna1t2bJGszLUnSKEb5FtEzwC9U1deSvAd4Osnjbd1v\nVtWvDTdOsg3YA1wDvBf4SpLvq6qzwP3AHcCTwKPALuCx1ZmKJGmpFj0SqKpTVfW1Vv4W8Dyw8Rxd\ndgMPVdWbVfUScALYmWQDcFlVPVFVBTwI3LriGUiSlm1J1wSSbAGuY/BOHuCjSZ5N8tkkV7S6jcCr\nQ91OtrqNrTy3fr797E0ynWR6dnZ2KUOUJC3ByCGQ5N3AF4CPV9UbDE7tvB/YDpwCfn21BlVVB6pq\nR1XtmJqaWq3NSpLmGCkEklzMIAA+V1VfBKiq16vqbFV9G/htYGdrPgNsHuq+qdXNtPLceknShIxy\nd1CAzwDPV9VvDNVvGGr248BzrXwY2JPkkiRXA1uBp6rqFPBGkhvaNm8DHl6leUiSlmGUu4N+EPgp\n4GiSZ1rdLwIfTrIdKOBl4GcBqupYkkPAcQZ3Ft3V7gwCuBN4ALiUwV1B3hkkSRO0aAhU1Z8A893P\n/+g5+uwH9s9TPw1cu5QBSpLWjp8YlqSOGQKS1DFDQJI6ZghIUscMAUnqmCEgSR0zBCSpY4aAJHXM\nEJCkjhkCktQxQ0CSOmYISFLHDAFJ6pghIEkdMwQkqWOGgCR1zBCQpI4ZApLUMUNAkjpmCEhSxwwB\nSeqYISBJHTMEJKlji4ZAks1JvprkeJJjST7W6q9M8niSF9vjFUN97k5yIskLSW4aqr8+ydG27t4k\nWZtpSZJGMcqRwBngF6pqG3ADcFeSbcA+4EhVbQWOtOe0dXuAa4BdwH1J1rVt3Q/cAWxty65VnIsk\naYkWDYGqOlVVX2vlbwHPAxuB3cDB1uwgcGsr7wYeqqo3q+ol4ASwM8kG4LKqeqKqCnhwqI8kaQKW\ndE0gyRbgOuBJYH1VnWqrXgPWt/JG4NWhbidb3cZWnlsvSZqQkUMgybuBLwAfr6o3hte1d/a1WoNK\nsjfJdJLp2dnZ1dqsJGmOkUIgycUMAuBzVfXFVv16O8VDezzd6meAzUPdN7W6mVaeW/8dqupAVe2o\nqh1TU1OjzkWStESj3B0U4DPA81X1G0OrDgO3t/LtwMND9XuSXJLkagYXgJ9qp47eSHJD2+ZtQ30k\nSRNw0QhtfhD4KeBokmda3S8C9wCHknwEeAX4EEBVHUtyCDjO4M6iu6rqbOt3J/AAcCnwWFskSROy\naAhU1Z8AC93Pf+MCffYD++epnwauXcoAJUlrx08MS1LHDAFJ6pghIEkdMwQkqWOGgCR1zBCQpI4Z\nApLUMUNAkjpmCEhSxwwBSeqYISBJHTMEJKljhoAkdcwQkKSOGQKS1LFRflRGWtSWfY9MegiSlsEj\nAUnqmCEgSR0zBCSpY4aAJHXMEJCkjhkCktQxQ0CSOmYISFLHFg2BJJ9NcjrJc0N1n0gyk+SZttw8\ntO7uJCeSvJDkpqH665McbevuTZLVn44kaSlGORJ4ANg1T/1vVtX2tjwKkGQbsAe4pvW5L8m61v5+\n4A5ga1vm26YkaYwWDYGq+iPgr0fc3m7goap6s6peAk4AO5NsAC6rqieqqoAHgVuXO2hJ0upYyTWB\njyZ5tp0uuqLVbQReHWpzstVtbOW59ZKkCVpuCNwPvB/YDpwCfn3VRgQk2ZtkOsn07Ozsam5akjRk\nWSFQVa9X1dmq+jbw28DOtmoG2DzUdFOrm2nlufULbf9AVe2oqh1TU1PLGaIkaQTLCoF2jv8tPw68\ndefQYWBPkkuSXM3gAvBTVXUKeCPJDe2uoNuAh1cwbknSKlj09wSSfB74IHBVkpPALwMfTLIdKOBl\n4GcBqupYkkPAceAMcFdVnW2bupPBnUaXAo+1RZI0QYuGQFV9eJ7qz5yj/X5g/zz108C1SxqdJGlN\n+YlhSeqYISBJHTMEJKljhoAkdWzRC8OSzj9b9j0ykf2+fM8tE9mv1o5HApLUMUNAkjpmCEhSxwwB\nSeqYISBJHTMEJKljhoAkdcwQkKSOGQKS1DFDQJI6ZghIUscMAUnqmCEgSR0zBCSpY4aAJHXMEJCk\njhkCktQxQ0CSOmYISFLHFg2BJJ9NcjrJc0N1VyZ5PMmL7fGKoXV3JzmR5IUkNw3VX5/kaFt3b5Ks\n/nQkSUsxypHAA8CuOXX7gCNVtRU40p6TZBuwB7im9bkvybrW537gDmBrW+ZuU5I0ZouGQFX9EfDX\nc6p3Awdb+SBw61D9Q1X1ZlW9BJwAdibZAFxWVU9UVQEPDvWRJE3Icq8JrK+qU638GrC+lTcCrw61\nO9nqNrby3HpJ0gSt+MJwe2dfqzCWf5Bkb5LpJNOzs7OruWlJ0pDlhsDr7RQP7fF0q58BNg+129Tq\nZlp5bv28qupAVe2oqh1TU1PLHKIkaTHLDYHDwO2tfDvw8FD9niSXJLmawQXgp9qpozeS3NDuCrpt\nqI8kaUIuWqxBks8DHwSuSnIS+GXgHuBQko8ArwAfAqiqY0kOAceBM8BdVXW2bepOBncaXQo81hZJ\n0gQtGgJV9eEFVt24QPv9wP556qeBa5c0OknSmvITw5LUMUNAkjpmCEhSxwwBSeqYISBJHTMEJKlj\nhoAkdcwQkKSOGQKS1DFDQJI6ZghIUscMAUnqmCEgSR0zBCSpY4aAJHXMEJCkjhkCktQxQ0CSOmYI\nSFLHDAFJ6pghIEkdMwQkqWOGgCR1zBCQpI6tKASSvJzkaJJnkky3uiuTPJ7kxfZ4xVD7u5OcSPJC\nkptWOnhJ0sqsxpHAv6mq7VW1oz3fBxypqq3AkfacJNuAPcA1wC7gviTrVmH/kqRlWovTQbuBg618\nELh1qP6hqnqzql4CTgA712D/kqQRXbTC/gV8JclZ4L9X1QFgfVWdautfA9a38kbgiaG+J1udVtGW\nfY9MegiSLiArDYEfqqqZJP8MeDzJ14dXVlUlqaVuNMleYC/A+973vhUOUZK0kBWdDqqqmfZ4GvgS\ng9M7ryfZANAeT7fmM8Dmoe6bWt182z1QVTuqasfU1NRKhihJOodlh0CSdyV5z1tl4EeA54DDwO2t\n2e3Aw618GNiT5JIkVwNbgaeWu39J0sqt5HTQeuBLSd7azu9W1R8k+TPgUJKPAK8AHwKoqmNJDgHH\ngTPAXVV1dkWjlyStyLJDoKq+CXz/PPV/Bdy4QJ/9wP7l7lOStLr8xLAkdcwQkKSOGQKS1DFDQJI6\nttIPi0nqyKQ+kf7yPbdMZL898EhAkjpmCEhSxwwBSeqYISBJHTMEJKljhoAkdcwQkKSOGQKS1DFD\nQJI6ZghIUscMAUnqmCEgSR0zBCSpY4aAJHXMEJCkjhkCktQxQ0CSOmYISFLH/HnJNTCpn+CTpKUa\newgk2QX8F2Ad8OmqumfcY5B0YZnkG6vv9t83HuvpoCTrgP8K/HtgG/DhJNvGOQZJ0j8a9zWBncCJ\nqvpmVf0d8BCwe8xjkCQ14z4dtBF4dej5SeBfrtXOPDcvaaUm9e/IuE5DnZcXhpPsBfa2p3+b5IVJ\njuccrgL+ctKDWAXO4/ziPM4vE5lHPrniTfzzURqNOwRmgM1Dzze1urepqgPAgXENarmSTFfVjkmP\nY6Wcx/nFeZxfvlvmsZBxXxP4M2BrkquTvAPYAxwe8xgkSc1YjwSq6kyS/wj8Lwa3iH62qo6NcwyS\npH809msCVfUo8Oi497tGzvtTViNyHucX53F++W6Zx7xSVZMegyRpQvzuIEnqmCFwDkmuTPJ4khfb\n4xULtNuV5IUkJ5LsG6r/VJKvJ3k2yZeSXD6+0S88rqH1SXJvW/9skg+M2necljuPJJuTfDXJ8STH\nknxs/KN/2ziX/Xq09euS/O8kXx7fqOe3wr+ty5P8Xvt/4/kk/2q8o3/bOFcyj59vf1fPJfl8kneO\nd/SrpKpcFliAXwX2tfI+4JPztFkHfAN4P/AO4M+BbW3djwAXtfIn5+u/hmNfcFxDbW4GHgMC3AA8\nOWrfC2QeG4APtPJ7gP9zIc5jaP1/An4X+PIk5rBacwEOAv+hld8BXH6hzYPBB19fAi5tzw8BPz3J\n12W5i0cC57abwR8s7fHWedos+FUYVfWHVXWmtXuCwecixmWUr+jYDTxYA08AlyfZMGLfcVn2PKrq\nVFV9DaCqvgU8z+B/3klYyetBkk3ALcCnxznoBSx7Lkm+F/jXwGcAqurvqupvxjn4ISt6TRjcWHNp\nkouAfwL8xbgGvpoMgXNbX1WnWvk1YP08beb7Koz5/qH5GQbvKMZllHEt1GbUOY3DSubxD5JsAa4D\nnlz1EY5mpfP4LeA/A99eqwEuwUrmcjUwC/yPdmrr00netZaDPYdlz6OqZoBfA/4vcAr4f1X1h2s4\n1jXTfQgk+Uo7pzd3eds7ghoc8y3rVqokvwScAT63CkPWEiV5N/AF4ONV9cakx7NUSX4UOF1VT096\nLKvgIuADwP1VdR3w/xmcar2gtOuDuxmE2nuBdyX5ycmOannOy+8OGqeq+uGF1iV5/a3TCu0Q8PQ8\nzc75VRhJfhr4UeDGFiTjMspXdCzU5uIR+o7LSuZBkosZBMDnquqLazjOxaxkHj8B/FiSm4F3Apcl\n+Z2qmtQ/OiuZSwEnq+qtI7LfY3IhsJJ5/DDwUlXNAiT5IvADwO+s2WjXyqQvSpzPC/Ap3n5h+Ffn\naXMR8E0G7wjeurh0TVu3CzgOTE1g7AuOa6jNLbz9otdTo/a9QOYR4EHgt86Dv6Vlz2NOmw8y+QvD\nK5oL8MfAv2jlTwCfutDmweDbj48xuBYQBtcMPzrpv7Nl/XeY9ADO5wX4p8AR4EXgK8CVrf69wKND\n7W5mcOfJN4BfGqo/weB84jNt+W9jHv93jAv4OeDnWjkMfuTnG8BRYMdic5rQ67CseQA/xOCd57ND\nr8HNF9o85mxj4iGwCn9b24Hp9rr8PnDFBTqPXwG+DjwH/E/gkkm/LstZ/MSwJHWs+wvDktQzQ0CS\nOmYISFLHDAFJ6pghIEkdMwQkqWOGgCR1zBCQpI79PZQRyS+X/nqDAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x28e516a5470>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Convert to numpy array\n",
    "diffs = np.array(diffs)\n",
    "\n",
    "# Plot sampling distribution\n",
    "plt.hist(diffs)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "((10000,), 10000, 0.02986836562405594, 0.014890736730544262)"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "diffs.shape, diffs.size, diffs.mean(), diffs.std()\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(array([   10.,    88.,   485.,  1605.,  2831.,  2844.,  1543.,   498.,\n",
       "           88.,     8.]),\n",
       " array([ -5.84074295e-02,  -4.67202293e-02,  -3.50330292e-02,\n",
       "         -2.33458290e-02,  -1.16586289e-02,   2.85712647e-05,\n",
       "          1.17157714e-02,   2.34029716e-02,   3.50901717e-02,\n",
       "          4.67773719e-02,   5.84645720e-02]),\n",
       " <a list of 10 Patch objects>)"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYEAAAD8CAYAAACRkhiPAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAENlJREFUeJzt3X+MpVV9x/H3p6xSq1KgrNt12XSxWZssJEVZtzTaBmuj\nFEwX08QsSYWmVrRSqqlJs2gaTZNNUOuPmlQaVMoSUbJRLESgFoip9Q/AgSKwi5RVlrCbBcaaBvsP\nhvXbP+5ZvQ6zzJ25M3NnOO9XcnPPPc85z3POPLPzuc+PezdVhSSpT7806QFIkibHEJCkjhkCktQx\nQ0CSOmYISFLHDAFJ6pghIEkdMwQkqWOGgCR1bM2kBzCXU045pTZt2jTpYUjSqnLPPff8sKrWztVu\nxYfApk2bmJqamvQwJGlVSfLYKO08HSRJHTMEJKljhoAkdcwQkKSOGQKS1DFDQJI6ZghIUscMAUnq\nmCEgSR1b8Z8YllaqTTtvnti2D1xx/sS2rRcWQ0Cr3iT/GEurnaeDJKljhoAkdcwQkKSOGQKS1DFD\nQJI6ZghIUscMAUnqmCEgSR0zBCSpY4aAJHXMEJCkjhkCktQxQ0CSOmYISFLHDAFJ6pghIEkdMwQk\nqWOGgCR1zBCQpI7NGQJJNib5ZpJ9SfYmeV+r/0iSQ0nua4/zhvpcnmR/koeTvGWo/qwkD7Rln0mS\npZmWJGkUo/xH888CH6iqe5O8HLgnyW1t2aeq6h+GGyfZAuwATgdeCdye5NVVdQS4EngXcBdwC3Au\ncOviTEWSNF9zHglU1eGqureVfww8BGx4ni7bgeur6pmqehTYD2xLsh44oarurKoCrgUuGHsGkqQF\nm9c1gSSbgNcweCcPcFmS+5NcneSkVrcBeHyo28FWt6GVZ9ZLkiZk5BBI8jLgq8D7q+ppBqd2XgWc\nCRwGPrFYg0pySZKpJFPT09OLtVpJ0gwjhUCSFzEIgOuq6gaAqnqyqo5U1U+BzwHbWvNDwMah7qe2\nukOtPLP+OarqqqraWlVb165dO5/5SJLmYZS7gwJ8AXioqj45VL9+qNnbgAdb+SZgR5Ljk5wGbAbu\nrqrDwNNJzm7rvAi4cZHmIUlagFHuDno98A7ggST3tboPAhcmORMo4ADwboCq2ptkD7CPwZ1Fl7Y7\ngwDeC1wDvITBXUHeGSRJEzRnCFTVt4HZ7ue/5Xn67AJ2zVI/BZwxnwFKkpaOnxiWpI4ZApLUMUNA\nkjpmCEhSxwwBSeqYISBJHTMEJKljhoAkdcwQkKSOGQKS1DFDQJI6ZghIUscMAUnqmCEgSR0zBCSp\nY4aAJHXMEJCkjhkCktQxQ0CSOmYISFLHDAFJ6pghIEkdMwQkqWOGgCR1zBCQpI4ZApLUMUNAkjpm\nCEhSx+YMgSQbk3wzyb4ke5O8r9WfnOS2JI+055OG+lyeZH+Sh5O8Zaj+rCQPtGWfSZKlmZYkaRSj\nHAk8C3ygqrYAZwOXJtkC7ATuqKrNwB3tNW3ZDuB04Fzgs0mOa+u6EngXsLk9zl3EuUiS5mnOEKiq\nw1V1byv/GHgI2ABsB3a3ZruBC1p5O3B9VT1TVY8C+4FtSdYDJ1TVnVVVwLVDfSRJEzCvawJJNgGv\nAe4C1lXV4bboCWBdK28AHh/qdrDVbWjlmfWzbeeSJFNJpqanp+czREnSPIwcAkleBnwVeH9VPT28\nrL2zr8UaVFVdVVVbq2rr2rVrF2u1kqQZRgqBJC9iEADXVdUNrfrJdoqH9vxUqz8EbBzqfmqrO9TK\nM+slSRMyyt1BAb4APFRVnxxadBNwcStfDNw4VL8jyfFJTmNwAfjuduro6SRnt3VeNNRHkjQBa0Zo\n83rgHcADSe5rdR8ErgD2JHkn8BjwdoCq2ptkD7CPwZ1Fl1bVkdbvvcA1wEuAW9tDkjQhc4ZAVX0b\nONb9/G86Rp9dwK5Z6qeAM+YzQEnS0vETw5LUMUNAkjpmCEhSxwwBSeqYISBJHTMEJKljhoAkdcwQ\nkKSOGQKS1DFDQJI6ZghIUscMAUnqmCEgSR0zBCSpY4aAJHXMEJCkjhkCktQxQ0CSOmYISFLHDAFJ\n6pghIEkdMwQkqWOGgCR1bM2kB6AXhk07b570ECQtgEcCktQxQ0CSOmYISFLH5rwmkORq4K3AU1V1\nRqv7CPAuYLo1+2BV3dKWXQ68EzgC/HVVfaPVnwVcA7wEuAV4X1XVYk5G6sWkrsEcuOL8iWxXS2eU\nI4FrgHNnqf9UVZ3ZHkcDYAuwAzi99flskuNa+ysZBMfm9phtnZKkZTRnCFTVt4Afjbi+7cD1VfVM\nVT0K7Ae2JVkPnFBVd7Z3/9cCFyx00JKkxTHONYHLktyf5OokJ7W6DcDjQ20OtroNrTyzXpI0QQsN\ngSuBVwFnAoeBTyzaiIAklySZSjI1PT09dwdJ0oIsKASq6smqOlJVPwU+B2xriw4BG4eantrqDrXy\nzPpjrf+qqtpaVVvXrl27kCFKkkawoBBo5/iPehvwYCvfBOxIcnyS0xhcAL67qg4DTyc5O0mAi4Ab\nxxi3JGkRjHKL6JeBc4BTkhwEPgyck+RMoIADwLsBqmpvkj3APuBZ4NKqOtJW9V5+fovore0hSZqg\nOUOgqi6cpfoLz9N+F7Brlvop4Ix5jU6StKT8xLAkdcwQkKSOGQKS1DFDQJI6ZghIUscMAUnqmCEg\nSR0zBCSpY4aAJHXMEJCkjhkCktQxQ0CSOmYISFLHDAFJ6pghIEkdMwQkqWOGgCR1zBCQpI4ZApLU\nMUNAkjpmCEhSxwwBSeqYISBJHTMEJKljhoAkdcwQkKSOGQKS1LE5QyDJ1UmeSvLgUN3JSW5L8kh7\nPmlo2eVJ9id5OMlbhurPSvJAW/aZJFn86UiS5mOUI4FrgHNn1O0E7qiqzcAd7TVJtgA7gNNbn88m\nOa71uRJ4F7C5PWauU5K0zOYMgar6FvCjGdXbgd2tvBu4YKj++qp6pqoeBfYD25KsB06oqjurqoBr\nh/pIkiZkodcE1lXV4VZ+AljXyhuAx4faHWx1G1p5Zr0kaYLGvjDc3tnXIozlZ5JckmQqydT09PRi\nrlqSNGShIfBkO8VDe36q1R8CNg61O7XVHWrlmfWzqqqrqmprVW1du3btAocoSZrLQkPgJuDiVr4Y\nuHGofkeS45OcxuAC8N3t1NHTSc5udwVdNNRHkjQha+ZqkOTLwDnAKUkOAh8GrgD2JHkn8BjwdoCq\n2ptkD7APeBa4tKqOtFW9l8GdRi8Bbm0PSdIEzRkCVXXhMRa96RjtdwG7ZqmfAs6Y1+gkSUvKTwxL\nUscMAUnqmCEgSR0zBCSpY4aAJHXMEJCkjhkCktQxQ0CSOmYISFLHDAFJ6pghIEkdMwQkqWOGgCR1\nzBCQpI4ZApLUMUNAkjpmCEhSxwwBSeqYISBJHTMEJKljhoAkdcwQkKSOGQKS1DFDQJI6ZghIUscM\nAUnqmCEgSR0zBCSpY2OFQJIDSR5Icl+SqVZ3cpLbkjzSnk8aan95kv1JHk7ylnEHL0kaz2IcCbyx\nqs6sqq3t9U7gjqraDNzRXpNkC7ADOB04F/hskuMWYfuSpAVaitNB24HdrbwbuGCo/vqqeqaqHgX2\nA9uWYPuSpBGNGwIF3J7kniSXtLp1VXW4lZ8A1rXyBuDxob4HW91zJLkkyVSSqenp6TGHKEk6ljVj\n9n9DVR1K8grgtiTfG15YVZWk5rvSqroKuApg69at8+7fs007b570EPQCNqnfrwNXnD+R7fZgrCOB\nqjrUnp8Cvsbg9M6TSdYDtOenWvNDwMah7qe2OknShCw4BJK8NMnLj5aBNwMPAjcBF7dmFwM3tvJN\nwI4kxyc5DdgM3L3Q7UuSxjfO6aB1wNeSHF3Pl6rq35J8B9iT5J3AY8DbAapqb5I9wD7gWeDSqjoy\n1uglSWNZcAhU1Q+A356l/n+ANx2jzy5g10K3KUlaXH5iWJI6ZghIUscMAUnqmCEgSR0zBCSpY4aA\nJHXMEJCkjhkCktQxQ0CSOmYISFLHDAFJ6pghIEkdMwQkqWOGgCR1zBCQpI4ZApLUMUNAkjpmCEhS\nxwwBSeqYISBJHTMEJKljhoAkdcwQkKSOGQKS1LE1kx7AC9GmnTdPegjSC8ok/00duOL8iW17OXgk\nIEkdMwQkqWPLHgJJzk3ycJL9SXYu9/YlST+3rCGQ5Djgn4A/ArYAFybZspxjkCT93HIfCWwD9lfV\nD6rqJ8D1wPZlHoMkqVnuu4M2AI8PvT4I/M5Sbcy7dCSNa1J/R5brrqQVeYtokkuAS9rL/0vy8Ihd\nTwF+uDSjmgjns7I5n5VtVc8nH31O1Xzn8xujNFruEDgEbBx6fWqr+wVVdRVw1XxXnmSqqrYufHgr\ni/NZ2ZzPyuZ8RrPc1wS+A2xOclqSFwM7gJuWeQySpGZZjwSq6tkkfwV8AzgOuLqq9i7nGCRJP7fs\n1wSq6hbgliVa/bxPIa1wzmdlcz4rm/MZQapqKdYrSVoF/NoISerYqguBJCcnuS3JI+35pGO0O+bX\nUyS5LMn3kuxN8rHlGfnsFmM+bfkHklSSU5Z+1Mc27nySfLztm/uTfC3Jics3+ucf29DyJPlMW35/\nkteO2ndSFjqnJBuTfDPJvvbv5X3LP/rnGmcfteXHJfmvJF9fvlEf25i/cycm+Ur7d/NQkt+d18ar\nalU9gI8BO1t5J/DRWdocB3wfeBXwYuC7wJa27I3A7cDx7fUrVvN82vKNDC62PwacsprnA7wZWNPK\nH52t/xKP/3l/1q3NecCtQICzgbtG7TuhfTLOnNYDr23llwP/Pek5jTOfoeV/A3wJ+Ppq3j9t2W7g\nL1r5xcCJ89n+qjsSYPA1E7tbeTdwwSxtnu/rKf4SuKKqngGoqqeWeLxzGXc+AJ8C/hZYCRd4xppP\nVf17VT3b2t3J4LMky2mUrzbZDlxbA3cCJyZZP2LfSVjwnKrqcFXdC1BVPwYeYvDJ/0kaZx+R5FTg\nfODzyzno57Hg+ST5VeD3gS8AVNVPqup/57Px1RgC66rqcCs/Aaybpc1sX09x9Bf31cDvJbkryX8k\ned3SDXUkY80nyXbgUFV9d0lHObpx98+wP2fw7mc5jTK2Y7UZdV7LbZw5/UySTcBrgLsWfYTzM+58\nPs3gTdNPl2qA8zTOfE4DpoF/aae3Pp/kpfPZ+Er92ojbgV+fZdGHhl9UVSWZ77vfNcDJDA6pXgfs\nSfKqasdSS2Gp5pPkV4APMjiFsmyWeP8c3caHgGeB6xbSX4srycuArwLvr6qnJz2ehUryVuCpqron\nyTmTHs8iWAO8Frisqu5K8o8MTsP+3XxWsOJU1R8ea1mSJ48eprbDu9lO5zzf11McBG5of/TvTvJT\nBt/JMb04o3+uJZzPbzJ4J/DdJEfr702yraqeWLQJzLDE+4ckfwa8FXjTUobzMYzy1SbHavOiEfpO\nwjhzIsmLGATAdVV1wxKOc1TjzOdPgD9Och7wy8AJSb5YVX+6hOOdyzjzKeBgVR09OvsKgxAY3aQv\nisz3AXycX7zw+LFZ2qwBfsDgD+TRCy2nt2XvAf6+lV/N4BArq3U+M9odYPIXhsfdP+cC+4C1Exr/\nnD9rBueThy/S3T2f/bTK5hTgWuDTk57HYsxnRptzWBkXhseaD/CfwG+18keAj89r+5P+ASzgB/Zr\nwB3AIwzu8jm51b8SuGWo3XkM7mT4PvChofoXA18EHgTuBf5gNc9nxroOMPkQGHf/7GcQzPe1xz9P\nYA7PGRuDNw/vaeUw+M+Rvg88AGydz36a0H5Z0JyANzB4t3n/0D45b7XOZ8Y6zmEFhMAi/M6dCUy1\nffSvwEnz2bafGJakjq3Gu4MkSYvEEJCkjhkCktQxQ0CSOmYISFLHDAFJ6pghIEkdMwQkqWP/D06k\nIJ8rQE20AAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x28e516e92b0>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Simulate distribution under the null hypothesis\n",
    "# why.. diffs.std() != np.std(diffs) ??????????????\n",
    "null_vals = np.random.normal(0, diffs.std(), diffs.size)\n",
    "\n",
    "# Plot the null distribution\n",
    "plt.hist(null_vals)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<matplotlib.lines.Line2D at 0x28e5187ccc0>"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYEAAAD8CAYAAACRkhiPAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAEQRJREFUeJzt3X+s3XV9x/Hna1SZUxkwaoelWWGpJoVkKLXD6BacizIg\nK2aJqcmEZc7qZAwzk6VgFs2SJqDzx0wmC05GiShpFAdRmAOic/4BeGEItMioUkKbQq+aBfcPBnzv\nj/OpHC+3vefec+89LZ/nIzk5n/P5fj7f7+dzv7f3db4/zmmqCklSn35l0gOQJE2OISBJHTMEJKlj\nhoAkdcwQkKSOGQKS1DFDQJI6ZghIUscMAUnq2IpJD2AuJ510Uq1du3bSw5C0WB55ZPD82tdOdhwv\ncvfee++PqmrlXO2O+BBYu3YtU1NTkx6GpMVyzjmD5299a5KjeNFL8vgo7TwdJEkdMwQkqWOGgCR1\nzBCQpI4ZApLUMUNAkjpmCEhSxwwBSeqYISBJHTviPzEsHanWbv36xLa958rzJ7ZtvbgYAjrqTfKP\nsXS083SQJHXMEJCkjhkCktQxQ0CSOmYISFLHDAFJ6pghIEkdMwQkqWOGgCR1zBCQpI4ZApLUMUNA\nkjpmCEhSxwwBSeqYISBJHTMEJKljhoAkdcwQkKSOGQKS1LE5QyDJmiTfTLIryc4kl7X6jybZl+T+\n9jhvqM/lSXYneSTJ24fqz0ryYFv2mSRZmmlJkkYxyn80/yzwoaq6L8krgXuT3N6Wfaqq/mG4cZL1\nwGbgdODVwB1JXlNVzwFXA+8F7gZuBc4FblucqUiS5mvOI4Gq2l9V97XyT4GHgdWH6bIJuLGqnqmq\nx4DdwMYkJwPHVdVdVVXA9cCFY89AkrRg87omkGQt8DoG7+QBLk3yQJJrk5zQ6lYDTwx129vqVrfy\nzHpJ0oSMHAJJXgF8BfhgVT3N4NTOacCZwH7gE4s1qCRbkkwlmZqenl6s1UqSZhgpBJK8hEEA3FBV\nNwFU1VNV9VxV/Rz4HLCxNd8HrBnqfkqr29fKM+tfoKquqaoNVbVh5cqV85mPJGkeRrk7KMDngYer\n6pND9ScPNXsH8FAr3wJsTnJsklOBdcA9VbUfeDrJ2W2dFwE3L9I8JEkLMMrdQW8C3g08mOT+VncF\n8K4kZwIF7AHeB1BVO5PsAHYxuLPoknZnEMAHgOuAlzG4K8g7gyRpguYMgar6DjDb/fy3HqbPNmDb\nLPVTwBnzGaAkaen4iWFJ6pghIEkdMwQkqWOGgCR1zBCQpI4ZApLUMUNAkjpmCEhSxwwBSeqYISBJ\nHTMEJKljhoAkdcwQkKSOGQKS1DFDQJI6ZghIUscMAUnqmCEgSR0zBCSpY4aAJHXMEJCkjhkCktQx\nQ0CSOmYISFLHDAFJ6pghIEkdMwQkqWOGgCR1bM4QSLImyTeT7EqyM8llrf7EJLcnebQ9nzDU5/Ik\nu5M8kuTtQ/VnJXmwLftMkizNtCRJoxjlSOBZ4ENVtR44G7gkyXpgK3BnVa0D7myvacs2A6cD5wKf\nTXJMW9fVwHuBde1x7iLORZI0T3OGQFXtr6r7WvmnwMPAamATsL012w5c2MqbgBur6pmqegzYDWxM\ncjJwXFXdVVUFXD/UR5I0AfO6JpBkLfA64G5gVVXtb4ueBFa18mrgiaFue1vd6laeWT/bdrYkmUoy\nNT09PZ8hSpLmYeQQSPIK4CvAB6vq6eFl7Z19LdagquqaqtpQVRtWrly5WKuVJM0wUggkeQmDALih\nqm5q1U+1Uzy05wOtfh+wZqj7Ka1uXyvPrJckTcgodwcF+DzwcFV9cmjRLcDFrXwxcPNQ/eYkxyY5\nlcEF4HvaqaOnk5zd1nnRUB9J0gSsGKHNm4B3Aw8mub/VXQFcCexI8h7gceCdAFW1M8kOYBeDO4su\nqarnWr8PANcBLwNuaw9J0oTMGQJV9R3gUPfzv/UQfbYB22apnwLOmM8AJUlLx08MS1LHDAFJ6pgh\nIEkdMwQkqWOGgCR1zBCQpI4ZApLUMUNAkjpmCEhSxwwBSeqYISBJHTMEJKljhoAkdcwQkKSOGQKS\n1DFDQJI6ZghIUscMAUnqmCEgSR0zBCSpY4aAJHXMEJCkjhkCktSxFZMegF4c1m79+qSHIGkBPBKQ\npI4ZApLUMUNAkjo25zWBJNcCFwAHquqMVvdR4L3AdGt2RVXd2pZdDrwHeA7466r6Rqs/C7gOeBlw\nK3BZVdViTkbqxaSuwey58vyJbFdLZ5QjgeuAc2ep/1RVndkeBwNgPbAZOL31+WySY1r7qxkEx7r2\nmG2dkqRlNGcIVNW3gZ+MuL5NwI1V9UxVPQbsBjYmORk4rqruau/+rwcuXOigJUmLY5xrApcmeSDJ\ntUlOaHWrgSeG2uxtdatbeWa9JGmCFhoCVwOnAWcC+4FPLNqIgCRbkkwlmZqenp67gyRpQRYUAlX1\nVFU9V1U/Bz4HbGyL9gFrhpqe0ur2tfLM+kOt/5qq2lBVG1auXLmQIUqSRrCgEGjn+A96B/BQK98C\nbE5ybJJTGVwAvqeq9gNPJzk7SYCLgJvHGLckaRGMcovol4BzgJOS7AU+ApyT5EyggD3A+wCqameS\nHcAu4Fngkqp6rq3qAzx/i+ht7SFJmqA5Q6Cq3jVL9ecP034bsG2W+ingjHmNTpK0pPzEsCR1zBCQ\npI4ZApLUMUNAkjpmCEhSxwwBSeqYISBJHTMEJKljhoAkdcwQkKSOGQKS1DFDQJI6ZghIUscMAUnq\nmCEgSR0zBCSpY4aAJHXMEJCkjhkCktQxQ0CSOmYISFLHDAFJ6pghIEkdMwQkqWOGgCR1zBCQpI4Z\nApLUsTlDIMm1SQ4keWio7sQktyd5tD2fMLTs8iS7kzyS5O1D9WclebAt+0ySLP50JEnzMcqRwHXA\nuTPqtgJ3VtU64M72miTrgc3A6a3PZ5Mc0/pcDbwXWNceM9cpSVpmc4ZAVX0b+MmM6k3A9lbeDlw4\nVH9jVT1TVY8Bu4GNSU4Gjququ6qqgOuH+kiSJmSh1wRWVdX+Vn4SWNXKq4EnhtrtbXWrW3lmvSRp\ngsa+MNze2dcijOUXkmxJMpVkanp6ejFXLUkastAQeKqd4qE9H2j1+4A1Q+1OaXX7Wnlm/ayq6pqq\n2lBVG1auXLnAIUqS5rLQELgFuLiVLwZuHqrfnOTYJKcyuAB8Tzt19HSSs9tdQRcN9ZEkTciKuRok\n+RJwDnBSkr3AR4ArgR1J3gM8DrwToKp2JtkB7AKeBS6pqufaqj7A4E6jlwG3tYckaYLmDIGqetch\nFr31EO23AdtmqZ8CzpjX6CRJS8pPDEtSxwwBSeqYISBJHTMEJKljhoAkdcwQkKSOGQKS1DFDQJI6\nZghIUscMAUnqmCEgSR0zBCSpY4aAJHXMEJCkjhkCktQxQ0CSOmYISFLHDAFJ6pghIEkdMwQkqWOG\ngCR1zBCQpI4ZApLUMUNAkjpmCEhSxwwBSeqYISBJHTMEJKljY4VAkj1JHkxyf5KpVndiktuTPNqe\nTxhqf3mS3UkeSfL2cQcvSRrPYhwJvKWqzqyqDe31VuDOqloH3Nlek2Q9sBk4HTgX+GySYxZh+5Kk\nBVqK00GbgO2tvB24cKj+xqp6pqoeA3YDG5dg+5KkEY0bAgXckeTeJFta3aqq2t/KTwKrWnk18MRQ\n372t7gWSbEkylWRqenp6zCFKkg5lxZj931xV+5K8Crg9yfeHF1ZVJan5rrSqrgGuAdiwYcO8+/ds\n7davT3oIehFbjN+vG3/4YwA2z2Nde648f+ztanZjHQlU1b72fAD4KoPTO08lORmgPR9ozfcBa4a6\nn9LqJEkTsuAQSPLyJK88WAbeBjwE3AJc3JpdDNzcyrcAm5Mcm+RUYB1wz0K3L0ka3zing1YBX01y\ncD1frKp/T/JdYEeS9wCPA+8EqKqdSXYAu4BngUuq6rmxRi9JGsuCQ6Cqfgj8ziz1Pwbeeog+24Bt\nC92mJGlx+YlhSeqYISBJHTMEJKljhoAkdcwQkKSOGQKS1DFDQJI6ZghIUscMAUnqmCEgSR0zBCSp\nY4aAJHXMEJCkjhkCktQxQ0CSOmYISFLHDAFJ6pghIEkdMwQkqWOGgCR1zBCQpI4ZApLUMUNAkjpm\nCEhSx1ZMegAvRmu3fn3SQ5BeVCb5b2rPledPbNvLwSMBSeqYISBJHVv2EEhybpJHkuxOsnW5ty9J\net6yhkCSY4B/Av4IWA+8K8n65RyDJOl5y30ksBHYXVU/rKqfATcCm5Z5DJKkZrnvDloNPDH0ei/w\nu0u1Me/SkTSuSf0dWa67ko7IW0STbAG2tJf/l+SREbueBPxoaUY1Ec7nyOZ8FuCNBwtXXbDUmzqq\n90+uekHVfOfzW6M0Wu4Q2AesGXp9Sqv7JVV1DXDNfFeeZKqqNix8eEcW53Nkcz5HNuczmuW+JvBd\nYF2SU5O8FNgM3LLMY5AkNct6JFBVzyb5K+AbwDHAtVW1cznHIEl63rJfE6iqW4Fbl2j18z6FdIRz\nPkc253Nkcz4jSFUtxXolSUcBvzZCkjp21IVAkhOT3J7k0fZ8wiHaHfLrKZJcmuT7SXYm+djyjHx2\nizGftvxDSSrJSUs/6kMbdz5JPt72zQNJvprk+OUb/eHHNrQ8ST7Tlj+Q5PWj9p2Uhc4pyZok30yy\nq/17uWz5R/9C4+yjtvyYJP+d5GvLN+pDG/N37vgkX27/bh5O8saZ/Q+rqo6qB/AxYGsrbwWumqXN\nMcAPgNOAlwLfA9a3ZW8B7gCOba9fdTTPpy1fw+Bi++PASUfzfIC3ASta+arZ+i/x+A/7s25tzgNu\nAwKcDdw9at8J7ZNx5nQy8PpWfiXwP5Oe0zjzGVr+N8AXga8dzfunLdsO/EUrvxQ4fj7bP+qOBBh8\nzcT2Vt4OXDhLm8N9PcVfAldW1TMAVXVgicc7l3HnA/Ap4G+BI+ECz1jzqar/qKpnW7u7GHyWZDmN\n8tUmm4Dra+Au4PgkJ4/YdxIWPKeq2l9V9wFU1U+Bhxl88n+SxtlHJDkFOB/4l+Uc9GEseD5Jfh34\nfeDzAFX1s6r63/ls/GgMgVVVtb+VnwRWzdJmtq+nOPiL+xrg95LcneQ/k7xh6YY6krHmk2QTsK+q\nvrekoxzduPtn2J8zePeznEYZ26HajDqv5TbOnH4hyVrgdcDdiz7C+Rl3Pp9m8Kbp50s1wHkaZz6n\nAtPAv7bTW/+S5OXz2fiR+rURdwC/OcuiDw+/qKpKMt93vyuAExkcUr0B2JHktGrHUkthqeaT5NeA\nKxicQlk2S7x/Dm7jw8CzwA0L6a/FleQVwFeAD1bV05Mez0IluQA4UFX3Jjln0uNZBCuA1wOXVtXd\nSf6RwWnYv5vPCo44VfWHh1qW5KmDh6nt8G620zmH+3qKvcBN7Y/+PUl+zuA7OaYXZ/QvtITz+W0G\n7wS+l+Rg/X1JNlbVk4s2gRmWeP+Q5M+AC4C3LmU4H8IoX21yqDYvGaHvJIwzJ5K8hEEA3FBVNy3h\nOEc1znz+BPjjJOcBvwocl+QLVfWnSzjeuYwznwL2VtXBo7MvMwiB0U36osh8H8DH+eULjx+bpc0K\n4IcM/kAevNByelv2fuDvW/k1DA6xcrTOZ0a7PUz+wvC4++dcYBewckLjn/NnzeB88vBFunvms5+O\nsjkFuB749KTnsRjzmdHmHI6MC8NjzQf4L+C1rfxR4OPz2v6kfwAL+IH9BnAn8CiDu3xObPWvBm4d\nancegzsZfgB8eKj+pcAXgIeA+4A/OJrnM2Nde5h8CIy7f3YzCOb72+OfJzCHF4yNwZuH97dyGPzn\nSD8AHgQ2zGc/TWi/LGhOwJsZvNt8YGifnHe0zmfGOs7hCAiBRfidOxOYavvo34AT5rNtPzEsSR07\nGu8OkiQtEkNAkjpmCEhSxwwBSeqYISBJHTMEJKljhoAkdcwQkKSO/T88eSmlGx8o8AAAAABJRU5E\nrkJggg==\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x28e5187c5f8>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Plot observed statistic with the null distibution\n",
    "plt.hist(null_vals)\n",
    "plt.axvline(obs_diff, c='red')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.021600000000000001"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Compute p-value\n",
    "(null_vals > obs_diff).mean()\n",
    "\n",
    "#the difference is significant! the possibility of the No difference is 0.022.. \n",
    "# With a type I error rate of 0.05, we have evidence that the enrollment rate for this course increases when using the \n",
    "#experimental description on its overview page? YES. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true,
    "deletable": true,
    "editable": true
   },
   "source": [
    "### Metric 2. Average Reading Duration"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(115.40710650582038, 130.93220512539477)"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# compute average reading durations for each group\n",
    "control_mean = df.query('group == \"control\"').duration.mean()\n",
    "experiment_mean = df.query('group == \"experiment\"').duration.mean()\n",
    "control_mean, experiment_mean"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "15.525098619574393"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# compute observed difference in means\n",
    "obs_diff = experiment_mean - control_mean\n",
    "obs_diff"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "collapsed": true,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "# simulate sampling distribution for the difference in means\n",
    "diffs = []\n",
    "for _ in range(10000):\n",
    "    b_samp = df.sample(df.shape[0], replace=True)\n",
    "    control_mean = b_samp.query('group == \"control\"').duration.mean()\n",
    "    experiment_mean = b_samp.query('group == \"experiment\"').duration.mean()\n",
    "    diffs.append(experiment_mean - control_mean)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "collapsed": true,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "# convert to numpy array\n",
    "diffs = np.array(diffs)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "# plot sampling distribution\n",
    "plt.hist(diffs);"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "collapsed": true,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "# simulate the distribution under the null hypothesis\n",
    "null_vals = np.random.normal(0, diffs.std(), diffs.size)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "# plot null distribution\n",
    "plt.hist(null_vals);"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYEAAAD8CAYAAACRkhiPAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAEDNJREFUeJzt3X+onuV9x/H3p7Fz0lameBayJF0sZIUoNMVDJrQMO7ea\n1UHsPxL/qIGJKZh1LfSf2H/aMQJurC0IM5BSSYS2LtCKoWqLhhYpzNpjccbEBkONmENM0nbD9h+H\n6Xd/nCvz2emJ5+dzntNc7xfcPNfzve/req6bw+Fz7h/PfVJVSJL69K5RT0CSNDqGgCR1zBCQpI4Z\nApLUMUNAkjpmCEhSxwwBSeqYISBJHTMEJKljl416ArO55pprasOGDaOehiQtn+PHp14/+MEFD/Hc\nc8/9oqrGZttuxYfAhg0bmJiYGPU0JGn53HTT1OsPf7jgIZK8OpftPB0kSR0zBCSpY4aAJHXMEJCk\njhkCktQxQ0CSOmYISFLHDAFJ6pghIEkdW/HfGNbvnw27Hxvq+Cfvu3Wo40s98UhAkjpmCEhSxwwB\nSeqYISBJHTMEJKljhoAkdcwQkKSOGQKS1DFDQJI6NmsIJPnDJM8m+c8kR5P8Y6tfneTJJC+316sG\n+tyb5ESS40luGajfkORIW3d/kgxntyRJczGXI4E3gb+sqg8Bm4GtSW4EdgOHq2ojcLi9J8kmYDtw\nHbAVeCDJqjbWXuBuYGNbti7hvkiS5mnWEKgpv2lv392WArYBB1r9AHBba28DHq6qN6vqFeAEsCXJ\nGuDKqnqmqgp4aKCPJGkE5nRNIMmqJM8DZ4Enq+rHwOqqOt02eR1Y3dprgdcGup9qtbWtPb0uSRqR\nOYVAVZ2vqs3AOqb+qr9+2vpi6uhgSSTZmWQiycS5c+eWalhJ0jTzujuoqv4b+AFT5/LPtFM8tNez\nbbNJYP1At3WtNtna0+szfc6+qhqvqvGxsbH5TFGSNA9zuTtoLMkftfYVwF8DPwMOATvaZjuAR1v7\nELA9yeVJrmXqAvCz7dTRG0lubHcF3TnQR5I0AnP5pzJrgAPtDp93AQer6rtJ/gM4mOQu4FXgdoCq\nOprkIHAMeAvYVVXn21j3APuBK4An2iJJGpFZQ6CqXgA+PEP9l8DNF+mzB9gzQ30CuP53e0iSRsFv\nDEtSxwwBSeqYISBJHTMEJKljhoAkdcwQkKSOGQKS1DFDQJI6ZghIUscMAUnqmCEgSR0zBCSpY4aA\nJHXMEJCkjhkCktQxQ0CSOmYISFLHDAFJ6pghIEkdMwQkqWOGgCR1zBCQpI4ZApLUMUNAkjo2awgk\nWZ/kB0mOJTma5LOt/qUkk0meb8snBvrcm+REkuNJbhmo35DkSFt3f5IMZ7ckSXNx2Ry2eQv4fFX9\nNMn7gOeSPNnWfbWq/nVw4ySbgO3AdcCfAE8l+bOqOg/sBe4Gfgw8DmwFnliaXZEkzdesRwJVdbqq\nftravwZeAta+Q5dtwMNV9WZVvQKcALYkWQNcWVXPVFUBDwG3LXoPJEkLNq9rAkk2AB9m6i95gM8k\neSHJg0muarW1wGsD3U612trWnl6f6XN2JplIMnHu3Ln5TFGSNA9zOR0EQJL3At8GPldVbyTZC/wT\nUO31y8DfLcWkqmofsA9gfHy8lmJMXTo27H5saGOfvO/WoY0trURzOhJI8m6mAuAbVfUdgKo6U1Xn\nq+q3wNeALW3zSWD9QPd1rTbZ2tPrkqQRmcvdQQG+DrxUVV8ZqK8Z2OyTwIutfQjYnuTyJNcCG4Fn\nq+o08EaSG9uYdwKPLtF+SJIWYC6ngz4CfAo4kuT5VvsCcEeSzUydDjoJfBqgqo4mOQgcY+rOol3t\nziCAe4D9wBVM3RXknUGSNEKzhkBV/QiY6X7+x9+hzx5gzwz1CeD6+UxQkjQ8fmNYkjpmCEhSxwwB\nSeqYISBJHTMEJKljhoAkdcwQkKSOGQKS1DFDQJI6ZghIUscMAUnqmCEgSR0zBCSpY4aAJHXMEJCk\njhkCktSxOf+jeV1ahvnP2iX9/vBIQJI6ZghIUscMAUnqmCEgSR0zBCSpY4aAJHVs1hBIsj7JD5Ic\nS3I0yWdb/eokTyZ5ub1eNdDn3iQnkhxPcstA/YYkR9q6+5NkOLslSZqLuRwJvAV8vqo2ATcCu5Js\nAnYDh6tqI3C4vaet2w5cB2wFHkiyqo21F7gb2NiWrUu4L5KkeZo1BKrqdFX9tLV/DbwErAW2AQfa\nZgeA21p7G/BwVb1ZVa8AJ4AtSdYAV1bVM1VVwEMDfSRJIzCvawJJNgAfBn4MrK6q023V68Dq1l4L\nvDbQ7VSrrW3t6fWZPmdnkokkE+fOnZvPFCVJ8zDnEEjyXuDbwOeq6o3Bde0v+1qqSVXVvqoar6rx\nsbGxpRpWkjTNnEIgybuZCoBvVNV3WvlMO8VDez3b6pPA+oHu61ptsrWn1yVJIzKXu4MCfB14qaq+\nMrDqELCjtXcAjw7Utye5PMm1TF0AfradOnojyY1tzDsH+kiSRmAuTxH9CPAp4EiS51vtC8B9wMEk\ndwGvArcDVNXRJAeBY0zdWbSrqs63fvcA+4ErgCfaIkkakVlDoKp+BFzsfv6bL9JnD7BnhvoEcP18\nJihJGh6/MSxJHTMEJKljhoAkdcwQkKSOGQKS1DFDQJI6ZghIUscMAUnqmCEgSR0zBCSpY4aAJHXM\nEJCkjhkCktQxQ0CSOmYISFLHDAFJ6pghIEkdMwQkqWOGgCR1zBCQpI4ZApLUMUNAkjpmCEhSx2YN\ngSQPJjmb5MWB2peSTCZ5vi2fGFh3b5ITSY4nuWWgfkOSI23d/Umy9LsjSZqPuRwJ7Ae2zlD/alVt\nbsvjAEk2AduB61qfB5KsatvvBe4GNrZlpjElScto1hCoqqeBX81xvG3Aw1X1ZlW9ApwAtiRZA1xZ\nVc9UVQEPAbctdNKSpKWxmGsCn0nyQjtddFWrrQVeG9jmVKutbe3p9Rkl2ZlkIsnEuXPnFjFFSdI7\nWWgI7AU+AGwGTgNfXrIZAVW1r6rGq2p8bGxsKYeWJA1YUAhU1ZmqOl9VvwW+BmxpqyaB9QObrmu1\nydaeXpckjdCCQqCd47/gk8CFO4cOAduTXJ7kWqYuAD9bVaeBN5Lc2O4KuhN4dBHzliQtgctm2yDJ\nt4CbgGuSnAK+CNyUZDNQwEng0wBVdTTJQeAY8Bawq6rOt6HuYepOoyuAJ9oiSRqhWUOgqu6Yofz1\nd9h+D7BnhvoEcP28ZidJGiq/MSxJHTMEJKljhoAkdcwQkKSOGQKS1DFDQJI6ZghIUscMAUnqmCEg\nSR0zBCSpY4aAJHXMEJCkjhkCktQxQ0CSOmYISFLHDAFJ6pghIEkdMwQkqWOGgCR1zBCQpI4ZApLU\nMUNAkjpmCEhSxy6bbYMkDwJ/C5ytqutb7Wrg34ENwEng9qr6r7buXuAu4DzwD1X1/Va/AdgPXAE8\nDny2qmppd0danA27Hxvq+Cfvu3Wo40vzNZcjgf3A1mm13cDhqtoIHG7vSbIJ2A5c1/o8kGRV67MX\nuBvY2JbpY0qSltmsIVBVTwO/mlbeBhxo7QPAbQP1h6vqzap6BTgBbEmyBriyqp5pf/0/NNBHkjQi\nC70msLqqTrf268Dq1l4LvDaw3alWW9va0+uSpBFa9IXh9pf9kp7bT7IzyUSSiXPnzi3l0JKkAQsN\ngTPtFA/t9WyrTwLrB7Zb12qTrT29PqOq2ldV41U1PjY2tsApSpJms9AQOATsaO0dwKMD9e1JLk9y\nLVMXgJ9tp47eSHJjkgB3DvSRJI3IXG4R/RZwE3BNklPAF4H7gINJ7gJeBW4HqKqjSQ4Cx4C3gF1V\ndb4NdQ9v3yL6RFskSSM0awhU1R0XWXXzRbbfA+yZoT4BXD+v2UmShspvDEtSxwwBSeqYISBJHTME\nJKljhoAkdcwQkKSOGQKS1DFDQJI6ZghIUscMAUnqmCEgSR0zBCSpY4aAJHXMEJCkjhkCktSxWf+f\ngEZjw+7HRj0FSR3wSECSOmYISFLHDAFJ6pghIEkdMwQkqWOGgCR1zBCQpI4tKgSSnExyJMnzSSZa\n7eokTyZ5ub1eNbD9vUlOJDme5JbFTl6StDhLcSTwsaraXFXj7f1u4HBVbQQOt/ck2QRsB64DtgIP\nJFm1BJ8vSVqgYZwO2gYcaO0DwG0D9Yer6s2qegU4AWwZwudLkuZosSFQwFNJnkuys9VWV9Xp1n4d\nWN3aa4HXBvqeajVJ0ogs9tlBH62qySR/DDyZ5GeDK6uqktR8B22BshPg/e9//yKnKEm6mEUdCVTV\nZHs9CzzC1OmdM0nWALTXs23zSWD9QPd1rTbTuPuqaryqxsfGxhYzRUnSO1hwCCR5T5L3XWgDHwde\nBA4BO9pmO4BHW/sQsD3J5UmuBTYCzy708yVJi7eY00GrgUeSXBjnm1X1vSQ/AQ4muQt4FbgdoKqO\nJjkIHAPeAnZV1flFzV6StCgLDoGq+jnwoRnqvwRuvkifPcCehX6mJGlp+Y1hSeqYISBJHTMEJKlj\nhoAkdcwQkKSOGQKS1DFDQJI6ZghIUscMAUnqmCEgSR0zBCSpY4aAJHXMEJCkji32P4tJmocNux8b\n2tgn77t1aGPr0uWRgCR1zBCQpI4ZApLUMUNAkjrmheEFGuYFPklaLh4JSFLHDAFJ6pghIEkdMwQk\nqWPLHgJJtiY5nuREkt3L/fmSpLctawgkWQX8G/A3wCbgjiSblnMOkqS3LfctoluAE1X1c4AkDwPb\ngGPLPA/pkjPs25Z9NtGlablDYC3w2sD7U8CfD+vDvJdfkt7ZivyyWJKdwM729jdJjo9yPiNwDfCL\nUU9iRNz3FSr/PNThV/S+D9nF9z1ZzLh/OpeNljsEJoH1A+/Xtdr/U1X7gH3LNamVJslEVY2Peh6j\n4L67770Z9b4v991BPwE2Jrk2yR8A24FDyzwHSVKzrEcCVfVWkr8Hvg+sAh6sqqPLOQdJ0tuW/ZpA\nVT0OPL7cn/t7pttTYbjvvXLfRyRVNcrPlySNkI+NkKSOGQIrVJIvJZlM8nxbPjHqOQ1Tz48TSXIy\nyZH2c54Y9XyGLcmDSc4meXGgdnWSJ5O83F6vGuUch+Ei+z3y33NDYGX7alVtbsslex3Fx4kA8LH2\nc+7hNsn9wNZptd3A4araCBxu7y81+/nd/YYR/54bAloJ/u9xIlX1P8CFx4noElRVTwO/mlbeBhxo\n7QPAbcs6qWVwkf0eOUNgZftMkhfaYeQld3g8YKbHiawd0VxGoYCnkjzXvi3fo9VVdbq1XwdWj3Iy\ny2ykv+eGwAgleSrJizMs24C9wAeAzcBp4MsjnayG6aNVtZmp02G7kvzFqCc0SjV1y2Ivty2O/Pd8\nRT47qBdV9Vdz2S7J14DvDnk6ozSnx4lcqqpqsr2eTfIIU6fHnh7trJbdmSRrqup0kjXA2VFPaDlU\n1ZkL7VH9nnsksEK1X4QLPgm8eLFtLwHdPk4kyXuSvO9CG/g4l/bP+mIOATtaewfw6AjnsmxWwu+5\nRwIr178k2czUYfFJ4NOjnc7wdP44kdXAI5l6WuRlwDer6nujndJwJfkWcBNwTZJTwBeB+4CDSe4C\nXgVuH90Mh+Mi+33TqH/P/cawJHXM00GS1DFDQJI6ZghIUscMAUnqmCEgSR0zBCSpY4aAJHXMEJCk\njv0vrNxpOkMAFasAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x28e51a0c160>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# plot null distribution and where our observed statistic falls\n",
    "plt.hist(null_vals)\n",
    "plt.axvline(x=obs_diff, color='red');"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.0"
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# compute p-value\n",
    "(null_vals > obs_diff).mean()\n",
    "\n",
    "# the difference is very significant. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true,
    "deletable": true,
    "editable": true
   },
   "source": [
    "### Metric 3. Average Classroom Time"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>timestamp</th>\n",
       "      <th>id</th>\n",
       "      <th>group</th>\n",
       "      <th>total_days</th>\n",
       "      <th>completed</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>2015-08-10 17:06:01.032740</td>\n",
       "      <td>610019</td>\n",
       "      <td>experiment</td>\n",
       "      <td>97</td>\n",
       "      <td>True</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>2015-08-10 17:15:28.950975</td>\n",
       "      <td>690224</td>\n",
       "      <td>control</td>\n",
       "      <td>75</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>2015-08-10 17:34:40.920384</td>\n",
       "      <td>564994</td>\n",
       "      <td>experiment</td>\n",
       "      <td>128</td>\n",
       "      <td>True</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>2015-08-10 17:50:39.847374</td>\n",
       "      <td>849588</td>\n",
       "      <td>experiment</td>\n",
       "      <td>66</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>2015-08-10 19:10:40.650599</td>\n",
       "      <td>849826</td>\n",
       "      <td>experiment</td>\n",
       "      <td>34</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                    timestamp      id       group  total_days completed\n",
       "0  2015-08-10 17:06:01.032740  610019  experiment          97      True\n",
       "1  2015-08-10 17:15:28.950975  690224     control          75     False\n",
       "2  2015-08-10 17:34:40.920384  564994  experiment         128      True\n",
       "3  2015-08-10 17:50:39.847374  849588  experiment          66     False\n",
       "4  2015-08-10 19:10:40.650599  849826  experiment          34     False"
      ]
     },
     "execution_count": 19,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df = pd.read_csv('C:/Users/Minkun/Desktop/classes_1/NanoDeg/1.Data_AN/L4/data/classroom_actions.csv')\n",
    "df.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "df.info()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(73.36899038461539, 74.6715935334873)"
      ]
     },
     "execution_count": 20,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# get the average classroom time for control group\n",
    "control_mean = df.query('group == \"control\"').total_days.mean()\n",
    "\n",
    "# get the average classroom time for experiment group\n",
    "experiment_mean = df.query('group == \"experiment\"').total_days.mean()\n",
    "\n",
    "# display average classroom time for each group\n",
    "control_mean, experiment_mean"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "1.3026031488719099"
      ]
     },
     "execution_count": 21,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# compute observed difference in classroom time\n",
    "obs_diff = experiment_mean - control_mean\n",
    "\n",
    "# display observed difference\n",
    "obs_diff"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {
    "collapsed": true,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "# create sampling distribution of difference in average classroom times\n",
    "# with boostrapping\n",
    "diffs = []\n",
    "size = df.shape[0]\n",
    "for _ in range(10000):\n",
    "    b_samp = df.sample(size, replace=True)\n",
    "    control_mean = b_samp.query('group == \"control\"').total_days.mean()\n",
    "    experiment_mean = b_samp.query('group == \"experiment\"').total_days.mean()\n",
    "    diffs.append(experiment_mean - control_mean)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {
    "collapsed": true,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "# convert to numpy array\n",
    "diffs = np.array(diffs)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "# plot sampling distribution\n",
    "plt.hist(diffs);"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {
    "collapsed": true,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "# simulate distribution under the null hypothesis\n",
    "null_vals = np.random.normal(0, diffs.std(), diffs.size)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<matplotlib.lines.Line2D at 0x28e51d97048>"
      ]
     },
     "execution_count": 25,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYEAAAD8CAYAAACRkhiPAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAADqZJREFUeJzt3W+onvV9x/H3Z9qJrJVZPMtcTHcsZEKUzWLIHN0DN8fM\nalnsYBIfVMfEFMxKC4UR2wftk4Blawuy6UhRVHANgbYYUNel0iKFWXuU0JikWUONmENq0vZBOgaO\npN89OD/Xe6cnOf/vK+b3fsHN/bu/1/W77u+Fmk+uP/dlqgpJUp9+begGJEnDMQQkqWOGgCR1zBCQ\npI4ZApLUMUNAkjpmCEhSxwwBSeqYISBJHbt06Abmc9VVV9Xk5OTQbUhaqiNHZt6vu27YPjrz8ssv\n/6SqJuZb74IPgcnJSaampoZuQ9JS3XLLzPu3vz1kF91J8vpC1vN0kCR1zBCQpI4ZApLUMUNAkjpm\nCEhSxwwBSeqYISBJHTMEJKljhoAkdeyC/8WwNJ/JHc8M3cLYHXvw9qFb0EXCIwFJ6pghIEkdMwQk\nqWOGgCR1zBCQpI4ZApLUMUNAkjo2bwgkWZfkW0kOJTmY5BOt/rkk00n2t9eHRuY8kORokiNJbhup\n35TkQFv2UJKszm5JkhZiIT8WOwN8qqpeSfIe4OUk+9qyL1XVP46unGQDsBW4Hvgd4JtJfq+qzgKP\nAPcB3wWeBTYDz63MrkiSFmveI4GqOlFVr7Txz4HDwNrzTNkC7K6qt6rqNeAosCnJ1cAVVfViVRXw\nJHDHsvdAkrRki7omkGQS+AAzf5MH+HiS7yd5LMmVrbYWeGNk2vFWW9vGs+uSpIEsOASSvBv4KvDJ\nqjrNzKmd9wM3AieAL6xUU0m2JZlKMnXq1KmV2qwkaZYFhUCSdzETAE9V1dcAqurNqjpbVb8Avgxs\naqtPA+tGpl/TatNtPLv+K6pqV1VtrKqNExMTi9kfSdIiLOTuoACPAoer6osj9atHVvsI8Gob7wW2\nJrksybXAeuClqjoBnE5yc9vm3cDTK7QfkqQlWMjdQR8EPgocSLK/1T4N3JXkRqCAY8DHAKrqYJI9\nwCFm7iza3u4MArgfeBy4nJm7grwzSJIGNG8IVNV3gLnu53/2PHN2AjvnqE8BNyymQUnS6vEXw5LU\nMUNAkjpmCEhSxwwBSeqYISBJHTMEJKljhoAkdcwQkKSOGQKS1DFDQJI6ZghIUscMAUnqmCEgSR0z\nBCSpY4aAJHXMEJCkjhkCktQxQ0CSOmYISFLHDAFJ6pghIEkdMwQkqWOGgCR1zBCQpI4ZApLUMUNA\nkjpmCEhSxwwBSeqYISBJHTMEJKlj84ZAknVJvpXkUJKDST7R6u9Nsi/JD9v7lSNzHkhyNMmRJLeN\n1G9KcqAteyhJVme3JEkLsZAjgTPAp6pqA3AzsD3JBmAH8HxVrQeeb59py7YC1wObgYeTXNK29Qhw\nH7C+vTav4L5IkhZp3hCoqhNV9Uob/xw4DKwFtgBPtNWeAO5o4y3A7qp6q6peA44Cm5JcDVxRVS9W\nVQFPjsyRJA1gUdcEkkwCHwC+C6ypqhNt0Y+BNW28FnhjZNrxVlvbxrPrkqSBLDgEkrwb+Crwyao6\nPbqs/c2+VqqpJNuSTCWZOnXq1EptVpI0y4JCIMm7mAmAp6rqa638ZjvFQ3s/2erTwLqR6de02nQb\nz67/iqraVVUbq2rjxMTEQvdFkrRIC7k7KMCjwOGq+uLIor3APW18D/D0SH1rksuSXMvMBeCX2qmj\n00lubtu8e2SOJGkAly5gnQ8CHwUOJNnfap8GHgT2JLkXeB24E6CqDibZAxxi5s6i7VV1ts27H3gc\nuBx4rr0kSQOZNwSq6jvAue7nv/Ucc3YCO+eoTwE3LKZBSdLq8RfDktQxQ0CSOmYISFLHDAFJ6pgh\nIEkdMwQkqWOGgCR1zBCQpI4ZApLUMUNAkjpmCEhSxwwBSeqYISBJHTMEJKljhoAkdcwQkKSOLeT/\nLCbNa3LHM0O3IGkJPBKQpI4ZApLUMUNAkjpmCEhSxwwBSeqYISBJHTMEJKljhoAkdcwQkKSOGQKS\n1DFDQJI6ZghIUsd8gJz0DjTUA/uOPXj7IN+r1TPvkUCSx5KcTPLqSO1zSaaT7G+vD40seyDJ0SRH\nktw2Ur8pyYG27KEkWfndkSQtxkJOBz0ObJ6j/qWqurG9ngVIsgHYClzf5jyc5JK2/iPAfcD69ppr\nm5KkMZo3BKrqBeBnC9zeFmB3Vb1VVa8BR4FNSa4GrqiqF6uqgCeBO5batCRpZSznwvDHk3y/nS66\nstXWAm+MrHO81da28ey6JGlASw2BR4D3AzcCJ4AvrFhHQJJtSaaSTJ06dWolNy1JGrGkEKiqN6vq\nbFX9AvgysKktmgbWjax6TatNt/Hs+rm2v6uqNlbVxomJiaW0KElagCWFQDvH/7aPAG/fObQX2Jrk\nsiTXMnMB+KWqOgGcTnJzuyvobuDpZfQtSVoB8/5OIMlXgFuAq5IcBz4L3JLkRqCAY8DHAKrqYJI9\nwCHgDLC9qs62Td3PzJ1GlwPPtZckaUDzhkBV3TVH+dHzrL8T2DlHfQq4YVHdSZJWlY+NkKSOGQKS\n1DFDQJI6ZghIUscMAUnqmCEgSR0zBCSpY4aAJHXMEJCkjhkCktQxQ0CSOmYISFLHDAFJ6pghIEkd\nMwQkqWOGgCR1zBCQpI4ZApLUMUNAkjpmCEhSxwwBSeqYISBJHTMEJKljhoAkdcwQkKSOGQKS1DFD\nQJI6ZghIUscMAUnqmCEgSR2bNwSSPJbkZJJXR2rvTbIvyQ/b+5Ujyx5IcjTJkSS3jdRvSnKgLXso\nSVZ+dyRJi7GQI4HHgc2zajuA56tqPfB8+0ySDcBW4Po25+Ekl7Q5jwD3Aevba/Y2JUljNm8IVNUL\nwM9mlbcAT7TxE8AdI/XdVfVWVb0GHAU2JbkauKKqXqyqAp4cmSNJGshSrwmsqaoTbfxjYE0brwXe\nGFnveKutbePZdUnSgJZ9Ybj9zb5WoJf/k2RbkqkkU6dOnVrJTUuSRiw1BN5sp3ho7ydbfRpYN7Le\nNa023caz63Oqql1VtbGqNk5MTCyxRUnSfJYaAnuBe9r4HuDpkfrWJJcluZaZC8AvtVNHp5Pc3O4K\nuntkjiRpIJfOt0KSrwC3AFclOQ58FngQ2JPkXuB14E6AqjqYZA9wCDgDbK+qs21T9zNzp9HlwHPt\nJUka0LwhUFV3nWPRredYfyewc476FHDDorqTJK0qfzEsSR0zBCSpY4aAJHXMEJCkjhkCktQxQ0CS\nOmYISFLHDAFJ6pghIEkdMwQkqWOGgCR1zBCQpI4ZApLUMUNAkjpmCEhSxwwBSeqYISBJHTMEJKlj\nhoAkdcwQkKSOGQKS1DFDQJI6ZghIUscMAUnqmCEgSR0zBCSpY4aAJHXs0qEb0Mqa3PHM0C1Iegfx\nSECSOuaRgKQFW8qR5u4f/RSArcs4Sj324O1LnqvzW9aRQJJjSQ4k2Z9kqtXem2Rfkh+29ytH1n8g\nydEkR5LcttzmJUnLsxKng/6kqm6sqo3t8w7g+apaDzzfPpNkA7AVuB7YDDyc5JIV+H5J0hKtxjWB\nLcATbfwEcMdIfXdVvVVVrwFHgU2r8P2SpAVabggU8M0kLyfZ1mprqupEG/8YWNPGa4E3RuYebzVJ\n0kCWe2H4j6tqOslvAfuS/GB0YVVVklrsRlugbAN43/vet8wWJUnnsqwjgaqabu8nga8zc3rnzSRX\nA7T3k231aWDdyPRrWm2u7e6qqo1VtXFiYmI5LUqSzmPJIZDkN5K85+0x8OfAq8Be4J622j3A0228\nF9ia5LIk1wLrgZeW+v2SpOVbzumgNcDXk7y9nX+tqn9L8j1gT5J7gdeBOwGq6mCSPcAh4AywvarO\nLqt7SdKyLDkEqupHwB/MUf8pcOs55uwEdi71OyVJK8vHRkhSxwwBSeqYISBJHTMEJKljhoAkdcwQ\nkKSOGQKS1DFDQJI6ZghIUscMAUnqmCEgSR0zBCSpY4aAJHXMEJCkjhkCktQxQ0CSOmYISFLHDAFJ\n6pghIEkdMwQkqWOGgCR1zBCQpI4ZApLUMUNAkjp26dANXIwmdzwzdAuStCAeCUhSxwwBSeqYp4Mk\nXfCGPMV67MHbB/vucfBIQJI6ZghIUsfGHgJJNic5kuRokh3j/n5J0i+NNQSSXAL8M/AXwAbgriQb\nxtmDJOmXxn0ksAk4WlU/qqr/AXYDW8bcgySpGffdQWuBN0Y+Hwf+cLW+zB9tSVquof4cGdddSRfk\nLaJJtgHb2sf/SnJkyH5W0FXAT4ZuYgx62M8e9hFWYD//6O3B5z+87GZW0QX3zzOfX/YmfnchK407\nBKaBdSOfr2m1/6eqdgG7xtXUuCSZqqqNQ/ex2nrYzx72EdzPHoz7msD3gPVJrk3y68BWYO+Ye5Ak\nNWM9EqiqM0n+DvgGcAnwWFUdHGcPkqRfGvs1gap6Fnh23N97gbjoTnGdQw/72cM+gvt50UtVDd2D\nJGkgPjZCkjpmCIxZkn9I8oMk30/y9SS/OXRPKy3JXyc5mOQXSS66Oy56ePRJkseSnEzy6tC9rJYk\n65J8K8mh9u/rJ4buaQiGwPjtA26oqt8H/hN4YOB+VsOrwF8BLwzdyErr6NEnjwObh25ilZ0BPlVV\nG4Cbge0X6T/L8zIExqyq/r2qzrSPLzLzW4mLSlUdrqqL5Qd+s3Xx6JOqegH42dB9rKaqOlFVr7Tx\nz4HDzDzVoCuGwLD+Fnhu6Ca0KHM9+qS7PzguNkkmgQ8A3x22k/G7IB8b8U6X5JvAb8+x6DNV9XRb\n5zPMHI4+Nc7eVspC9lF6J0jybuCrwCer6vTQ/YybIbAKqurPzrc8yd8AHwZurXfoPbrz7eNFbEGP\nPtE7Q5J3MRMAT1XV14buZwieDhqzJJuBvwf+sqr+e+h+tGg++uQikSTAo8Dhqvri0P0MxRAYv38C\n3gPsS7I/yb8M3dBKS/KRJMeZeYDkM0m+MXRPK6Vd1H/70SeHgT0X46NPknwF+A/guiTHk9w7dE+r\n4IPAR4E/bf8t7k/yoaGbGjd/MSxJHfNIQJI6ZghIUscMAUnqmCEgSR0zBCSpY4aAJHXMEJCkjhkC\nktSx/wVygq3YqdlQkwAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x28e51d97c50>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# plot null distribution\n",
    "plt.hist(null_vals)\n",
    "\n",
    "# plot line for observed statistic\n",
    "plt.axvline(obs_diff, c='red')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.032199999999999999"
      ]
     },
     "execution_count": 26,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# compute p value\n",
    "(null_vals > obs_diff).mean()\n",
    "\n",
    "# we have evidence that users spend more time in the classroom after seeing the experimental description in the course overview \n",
    "#page...YES. \n",
    "\n",
    "# But Even if the increase in classroom time is statistically significant, engaging students for 1.3 more days in the classroom,\n",
    "#when they average around 74 days in total, doesn't seem to indicate a large enough value to launch this change from a practical\n",
    "#perspective for the company...."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "### Metric 4. Completion Rate"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.3719951923076923"
      ]
     },
     "execution_count": 27,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Create dataframe with all control records\n",
    "control_df = df.query('group == \"control\"')\n",
    "\n",
    "# Compute completion rate\n",
    "control_ctr = control_df.query('completed == True').id.nunique() / control_df.id.nunique()\n",
    "\n",
    "# Display completion rate\n",
    "control_ctr"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.3935334872979215"
      ]
     },
     "execution_count": 28,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Create dataframe with all experiment records\n",
    "experiment_df = df.query('group == \"experiment\"')\n",
    "\n",
    "# Compute completion rate\n",
    "experiment_ctr = experiment_df.query('completed == True').id.nunique() / experiment_df.id.nunique()\n",
    "\n",
    "# Display completion rate\n",
    "experiment_ctr"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.02153829499022919"
      ]
     },
     "execution_count": 29,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Compute observed difference in completion rates\n",
    "obs_diff = experiment_ctr - control_ctr\n",
    "\n",
    "# Display observed difference in completion rates\n",
    "obs_diff"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {
    "collapsed": true,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "# Create sampling distribution for difference in completion rates\n",
    "# with boostrapping\n",
    "diffs = []\n",
    "size = df.shape[0]\n",
    "for _ in range(10000):\n",
    "    b_samp = df.sample(size, replace=True)\n",
    "    control_df = b_samp.query('group == \"control\"')\n",
    "    experiment_df = b_samp.query('group == \"experiment\"')\n",
    "    control_ctr = control_df.query('completed == True').id.nunique() / control_df.id.nunique()\n",
    "    experiment_ctr = experiment_df.query('completed == True').id.nunique() / experiment_df.id.nunique()\n",
    "    diffs.append(experiment_ctr - control_ctr)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {
    "collapsed": true,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "# convert to numpy array\n",
    "diffs = np.array(diffs)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "# plot distribution\n",
    "plt.hist(diffs);"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {
    "collapsed": true,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "# create distribution under the null hypothesis\n",
    "null_vals = np.random.normal(0, diffs.std(), diffs.size)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYEAAAD8CAYAAACRkhiPAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAEY9JREFUeJzt3W+MXddd7vHvgxNCoI2akMEY28UpMkhOJFJimSB4kXsD\nxCTVdRBS5UqQIKoYlIBaUQk57QvghaWUf0WRbnJlaBVHFCJLbRWrTS6kVlFViSRMQhrXTk3cJlE8\ncmJThFLeBNn98eIst4fpOHNmzsw5467vR9o666y91t5r+Uh+zv5z9qSqkCT16fumPQBJ0vQYApLU\nMUNAkjpmCEhSxwwBSeqYISBJHTMEJKljhoAkdcwQkKSOXTLtASzm6quvri1btkx7GJJW0vHjg9ef\n+qnpjuN72DPPPPNvVTWzWLs1HwJbtmxhdnZ22sOQtJJuumnw+o//OM1RfE9L8soo7TwdJEkdMwQk\nqWOGgCR1zBCQpI4ZApLUMUNAkjpmCEhSxwwBSeqYISBJHVvzvxiW1qotez83tX2/fN9tU9u3vrd4\nJCBJHTMEJKljhoAkdcwQkKSOGQKS1LFFQyDJDyR5OsmXkxxN8set/qokTyR5sb1eOdTn3iQnkhxP\ncstQ/Q1JjrR19yfJ6kxLkjSKUY4E3gT+d1X9NHA9sDPJjcBe4HBVbQUOt/ck2QbsBq4FdgIPJFnX\ntvUgcBewtS07V3AukqQlWjQEauA/29tL21LALuBAqz8A3N7Ku4BHqurNqnoJOAHsSLIBuKKqnqyq\nAh4e6iNJmoKRrgkkWZfkOeA08ERVPQWsr6pTrclrwPpW3gi8OtT9ZKvb2Mrz6xfa354ks0lmz5w5\nM/JkJElLM1IIVNW5qroe2MTgW/1189YXg6ODFVFV+6tqe1Vtn5lZ9O8kS5KWaUl3B1XVfwBfYHAu\n//V2iof2ero1mwM2D3Xb1OrmWnl+vSRpSka5O2gmyTta+XLgl4CvAoeAO1uzO4FHW/kQsDvJZUmu\nYXAB+Ol26uiNJDe2u4LuGOojSZqCUR4gtwE40O7w+T7gYFV9Nsk/AQeTvB94BXgvQFUdTXIQOAac\nBe6pqnNtW3cDDwGXA4+3RZI0JYuGQFU9D7x7gfpvADdfoM8+YN8C9bPAdd/dQ5I0Df5iWJI6ZghI\nUscMAUnqmCEgSR0zBCSpY4aAJHXMEJCkjhkCktQxQ0CSOmYISFLHDAFJ6pghIEkdMwQkqWOGgCR1\nzBCQpI4ZApLUMUNAkjpmCEhSxwwBSeqYISBJHTMEJKljhoAkdcwQkKSOGQKS1LFFQyDJ5iRfSHIs\nydEkH2j1f5RkLslzbbl1qM+9SU4kOZ7klqH6G5IcaevuT5LVmZYkaRSXjNDmLPChqno2yduBZ5I8\n0dZ9rKr+bLhxkm3AbuBa4MeAzyf5yao6BzwI3AU8BTwG7AQeX5mpSJKWatEjgao6VVXPtvI3gReA\njW/RZRfwSFW9WVUvASeAHUk2AFdU1ZNVVcDDwO1jz0CStGxLuiaQZAvwbgbf5AF+L8nzST6R5MpW\ntxF4dajbyVa3sZXn1y+0nz1JZpPMnjlzZilDlCQtwcghkORtwKeAD1bVGwxO7bwLuB44Bfz5Sg2q\nqvZX1faq2j4zM7NSm5UkzTNSCCS5lEEAfLKqPg1QVa9X1bmq+hbwV8CO1nwO2DzUfVOrm2vl+fWS\npCkZ5e6gAB8HXqiqvxiq3zDU7FeBr7TyIWB3ksuSXANsBZ6uqlPAG0lubNu8A3h0heYhSVqGUe4O\n+nngN4AjSZ5rdR8G3pfkeqCAl4HfBqiqo0kOAscY3Fl0T7szCOBu4CHgcgZ3BXlnkCRN0aIhUFVf\nAha6n/+xt+izD9i3QP0scN1SBihJWj2jHAlIa9qWvZ+b9hCki5aPjZCkjhkCktQxQ0CSOmYISFLH\nDAFJ6pghIEkdMwQkqWOGgCR1zBCQpI4ZApLUMUNAkjpmCEhSx3yAnHQRmtZD816+77ap7FerxyMB\nSeqYISBJHTMEJKljhoAkdcwQkKSOGQKS1DFDQJI6ZghIUscMAUnqmCEgSR1bNASSbE7yhSTHkhxN\n8oFWf1WSJ5K82F6vHOpzb5ITSY4nuWWo/oYkR9q6+5NkdaYlSRrFKEcCZ4EPVdU24EbgniTbgL3A\n4araChxu72nrdgPXAjuBB5Ksa9t6ELgL2NqWnSs4F0nSEi0aAlV1qqqebeVvAi8AG4FdwIHW7ABw\neyvvAh6pqjer6iXgBLAjyQbgiqp6sqoKeHiojyRpCpZ0TSDJFuDdwFPA+qo61Va9Bqxv5Y3Aq0Pd\nTra6ja08v36h/exJMptk9syZM0sZoiRpCUYOgSRvAz4FfLCq3hhe177Z10oNqqr2V9X2qto+MzOz\nUpuVJM0zUggkuZRBAHyyqj7dql9vp3hor6db/Ryweaj7plY318rz6yVJUzLK3UEBPg68UFV/MbTq\nEHBnK98JPDpUvzvJZUmuYXAB+Ol26uiNJDe2bd4x1EeSNAWj/GWxnwd+AziS5LlW92HgPuBgkvcD\nrwDvBaiqo0kOAscY3Fl0T1Wda/3uBh4CLgceb4skaUoWDYGq+hJwofv5b75An33AvgXqZ4HrljJA\nSdLq8RfDktQxQ0CSOmYISFLHDAFJ6pghIEkdMwQkqWOGgCR1zBCQpI4ZApLUMUNAkjpmCEhSxwwB\nSeqYISBJHTMEJKljhoAkdcwQkKSOGQKS1DFDQJI6ZghIUscMAUnqmCEgSR0zBCSpY4aAJHVs0RBI\n8okkp5N8Zajuj5LMJXmuLbcOrbs3yYkkx5PcMlR/Q5Ijbd39SbLy05EkLcUoRwIPATsXqP9YVV3f\nlscAkmwDdgPXtj4PJFnX2j8I3AVsbctC25QkTdCiIVBVXwT+fcTt7QIeqao3q+ol4ASwI8kG4Iqq\nerKqCngYuH25g5YkrYxxrgn8XpLn2+miK1vdRuDVoTYnW93GVp5fL0maouWGwIPAu4DrgVPAn6/Y\niIAke5LMJpk9c+bMSm5akjRkWSFQVa9X1bmq+hbwV8COtmoO2DzUdFOrm2vl+fUX2v7+qtpeVdtn\nZmaWM0RJ0giWFQLtHP95vwqcv3PoELA7yWVJrmFwAfjpqjoFvJHkxnZX0B3Ao2OMW5K0Ai5ZrEGS\nvwNuAq5OchL4Q+CmJNcDBbwM/DZAVR1NchA4BpwF7qmqc21TdzO40+hy4PG2SJKmaNEQqKr3LVD9\n8bdovw/Yt0D9LHDdkkYnSVpV/mJYkjpmCEhSxwwBSeqYISBJHTMEJKljhoAkdcwQkKSOGQKS1DFD\nQJI6ZghIUscMAUnqmCEgSR0zBCSpY4aAJHXMEJCkjhkCktQxQ0CSOmYISFLHDAFJ6pghIEkdMwQk\nqWOGgCR1zBCQpI4ZApLUsUVDIMknkpxO8pWhuquSPJHkxfZ65dC6e5OcSHI8yS1D9TckOdLW3Z8k\nKz8dSdJSjHIk8BCwc17dXuBwVW0FDrf3JNkG7AaubX0eSLKu9XkQuAvY2pb525QkTdiiIVBVXwT+\nfV71LuBAKx8Abh+qf6Sq3qyql4ATwI4kG4ArqurJqirg4aE+kqQpWe41gfVVdaqVXwPWt/JG4NWh\ndidb3cZWnl8vSZqisS8Mt2/2tQJj+bYke5LMJpk9c+bMSm5akjTkkmX2ez3Jhqo61U71nG71c8Dm\noXabWt1cK8+vX1BV7Qf2A2zfvn1FA0arY8vez017CJKWYblHAoeAO1v5TuDRofrdSS5Lcg2DC8BP\nt1NHbyS5sd0VdMdQH0nSlCx6JJDk74CbgKuTnAT+ELgPOJjk/cArwHsBqupokoPAMeAscE9VnWub\nupvBnUaXA4+3RZI0RYuGQFW97wKrbr5A+33AvgXqZ4HrljQ6SdKq8hfDktQxQ0CSOmYISFLHDAFJ\n6pghIEkdMwQkqWPL/cWwpA6t1C/DH/n6NwDYPeL2Xr7vthXZr76bRwKS1DFDQJI6ZghIUscMAUnq\nmCEgSR0zBCSpY4aAJHXMEJCkjhkCktQxQ0CSOmYISFLHDAFJ6pghIEkdMwQkqWOGgCR1zBCQpI4Z\nApLUsbFCIMnLSY4keS7JbKu7KskTSV5sr1cOtb83yYkkx5PcMu7gJUnjWYkjgf9VVddX1fb2fi9w\nuKq2Aofbe5JsA3YD1wI7gQeSrFuB/UuSlmk1TgftAg608gHg9qH6R6rqzap6CTgB7FiF/UuSRjRu\nCBTw+STPJNnT6tZX1alWfg1Y38obgVeH+p5sdZKkKblkzP6/UFVzSX4EeCLJV4dXVlUlqaVutAXK\nHoB3vvOdYw5RknQhYx0JVNVcez0NfIbB6Z3Xk2wAaK+nW/M5YPNQ902tbqHt7q+q7VW1fWZmZpwh\nSpLewrJDIMkPJXn7+TLwy8BXgEPAna3ZncCjrXwI2J3ksiTXAFuBp5e7f0nS+MY5HbQe+EyS89v5\n26r6/0n+GTiY5P3AK8B7AarqaJKDwDHgLHBPVZ0ba/SSpLEsOwSq6uvATy9Q/w3g5gv02QfsW+4+\nJUkry18MS1LHDAFJ6pghIEkdMwQkqWOGgCR1zBCQpI4ZApLUMUNAkjpmCEhSxwwBSeqYISBJHTME\nJKljhoAkdcwQkKSOGQKS1LFx/8aw1pgtez837SFIuogYApLWvGl+uXn5vtumtu9J8HSQJHXMEJCk\njhkCktQxQ0CSOmYISFLHDAFJ6pghIEkdm3gIJNmZ5HiSE0n2Tnr/kqTvmGgIJFkH/F/gV4BtwPuS\nbJvkGCRJ3zHpXwzvAE5U1dcBkjwC7AKOTXgcq8pHN0i6WEw6BDYCrw69Pwn87ITHIEkjm9aXukk9\nrmJNPjsoyR5gT3v7n0mOT3M8E3I18G/THsQU9Djv7uf8c+cLH33PVAYzIWN9zvno2Pv/8VEaTToE\n5oDNQ+83tbr/oar2A/snNai1IMlsVW2f9jgmrcd5O+c+XCxznvTdQf8MbE1yTZLvB3YDhyY8BklS\nM9Ejgao6m+R3gb8H1gGfqKqjkxyDJOk7Jn5NoKoeAx6b9H4vAl2d/hrS47ydcx8uijmnqqY9BknS\nlPjYCEnqmCEwQUmuSvJEkhfb65UXaPeWj9ZI8qEkleTq1R/1eMadc5I/TfLVJM8n+UySd0xu9Esz\nwueWJPe39c8n+ZlR+65Vy51zks1JvpDkWJKjST4w+dEv3zifdVu/Lsm/JPns5EZ9AVXlMqEF+BNg\nbyvvBT66QJt1wNeAdwHfD3wZ2Da0fjODC+uvAFdPe06rPWfgl4FLWvmjC/VfC8tin1trcyvwOBDg\nRuCpUfuuxWXMOW8AfqaV3w7868Uw53HnPbT+94G/BT477fl4JDBZu4ADrXwAuH2BNt9+tEZV/Rdw\n/tEa530M+APgYrmYM9acq+ofqupsa/ckg9+WrEWLfW609w/XwJPAO5JsGLHvWrTsOVfVqap6FqCq\nvgm8wOCJAheDcT5rkmwCbgP+epKDvhBDYLLWV9WpVn4NWL9Am4UerbERIMkuYK6qvryqo1xZY815\nnt9i8O1qLRplDhdqM+r815px5vxtSbYA7waeWvERro5x5/2XDL7IfWu1BrgUa/KxERezJJ8HfnSB\nVR8ZflNVlWTkb/NJfhD4MIPTI2vKas153j4+ApwFPrmc/lqbkrwN+BTwwap6Y9rjWW1J3gOcrqpn\nktw07fGAIbDiquoXL7QuyevnD4XboeHpBZpd6NEaPwFcA3w5yfn6Z5PsqKrXVmwCy7CKcz6/jd8E\n3gPcXO2E6ho0yiNRLtTm0hH6rkXjzJkklzIIgE9W1adXcZwrbZx5/xrwf5LcCvwAcEWSv6mqX1/F\n8b61aV+U6GkB/pT/eZH0TxZocwnwdQb/4Z+/6HTtAu1e5uK4MDzWnIGdDB41PjPtuSwyz0U/Nwbn\ngYcvFj69lM98rS1jzjnAw8BfTnsek5z3vDY3sQYuDE/9H7SnBfhh4DDwIvB54KpW/2PAY0PtbmVw\nt8TXgI9cYFsXSwiMNWfgBINzq8+15f9Ne05vMdfvmgPwO8DvtHIY/FGlrwFHgO1L+czX4rLcOQO/\nwODmhueHPttbpz2fSXzWQ9tYEyHgL4YlqWPeHSRJHTMEJKljhoAkdcwQkKSOGQKS1DFDQJI6ZghI\nUscMAUnq2H8DmhAUlQWDIQcAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x28e51daacc0>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# plot null distribution\n",
    "plt.hist(null_vals);\n",
    "\n",
    "# plot line for observed statistic\n",
    "plt.axvline(obs_diff, c='red');"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.038699999999999998"
      ]
     },
     "execution_count": 34,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# compute p value\n",
    "(null_vals > obs_diff).mean()\n",
    "\n",
    "# We have evidence that this course increases in completion rate when using the experimental description on its course overview \n",
    "#page.."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "## Analyzing Multiple Metrics\n",
    "\n",
    "The more metrics you evaluate, the more likely you are to observe significant differences just by chance - similar to what you saw in previous lessons with multiple tests. Luckily, this multiple comparisons problem can be handled in several ways. The probability of any `false Positive` (alpha) increases as we increase the number of metrics - Bonferroni Correction. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "### Here are the p-values computed for the four metrics in this experiment\n",
    "\n",
    "- 1.Enrollment Rate: 0.0159\n",
    "- 2.Average Reading Duration: 0\n",
    "- 3.Average Classroom Time: 0.0384\n",
    "- 4.Completion Rate: 0.0363\n",
    "\n",
    "If our original alpha value was 0.05, Bonferroni corrected alpha value is 0.0125. so.. With the Bonferroni corrected alpha value, statistically significant metrics are...only Average Reading Duration...well..Bonferroni value seems overly conservative in this case. \n",
    "\n",
    "Since the Bonferroni method is too conservative when we expect correlation among metrics, we can better approach this problem with more sophisticated methods, such as:\n",
    "- the closed testing procedure\n",
    "- Boole-Bonferroni bound\n",
    "- the Holm-Bonferroni method\n",
    "\n",
    "These are less conservative and take this correlation into account. If you do choose to use a less conservative method, just make sure the assumptions of that method are truly met in your situation, and that you're not just trying to cheat on a p-value. Choosing a poorly suited test just to get significant results will only lead to misguided decisions that harm your company's performance in the long run."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "### REALITY: \n",
    "What they don't know, is that although the enrollment rate appears to increase with the new description, the results from this A/B test are unreliable and largely due to chance, because fewer than 40 out of thousands of visitors enrolled during this experiment. This makes even one new student for the course substantially impact the results and potentially even the conclusion.\n",
    "\n",
    "In this case, we needed to have more time and traffic to the page to see if the change had a significant impact."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "When designing an A/B test and drawing conclusions based on its results, there are some common ones to consider.\n",
    "- Novelty effect and change aversion when existing users first experience a change\n",
    "- Sufficient traffic and conversions to have significant and repeatable results\n",
    "- Best metric choice for making the ultimate decision (eg. measuring revenue vs. clicks)\n",
    "- Long enough run time for the experiment to account for changes in behavior based on time of day/week or seasonal events.\n",
    "- Practical significance of a conversion rate (the cost of launching a new feature vs. the gain from the increase in conversion)\n",
    "- Consistency among test subjects in the control and experiment group (imbalance in the population represented in each group can lead to situations like Simpson's Paradox)\n",
    "\n",
    "\n"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.5.2"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}