Dato Basic Training.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# GraphLab Create Training Day\n",
    "\n",
    "## Installation & preparation\n",
    "\n",
    "1. # for a [free academic license here](https://dato.com/download/academic.html). Collect your GraphLab Create license key.\n",
    "\n",
    "2. [Download the Dato Launcher from here](https://dato.com/download/install-graphlab-create.html) and follow the installation instructions. We have installers for Windows, Linux and Mac (64 bit). \n",
    "\n",
    "3. Please download the following input files and save them locally:\n",
    "<ul>\n",
    "<li>`business.csv.gz`: https://s3.amazonaws.com/dato-datasets/dato-training/business.csv.gz (640.2 KB)\n",
    "<li>`review.csv.gz`: https://s3.amazonaws.com/dato-datasets/dato-training/review.csv.gz (80.4 MB)\n",
    "<li>`user.csv.gz`: https://s3.amazonaws.com/dato-datasets/dato-training/user.csv.gz (1.1 MB)\n",
    "</ul>\n",
    "\n",
    "You can unzip these files, or not - GraphLab can read from gzipped files as well."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "import os\n",
    "from urllib import urlretrieve\n",
    "url_template = 'https://s3.amazonaws.com/dato-datasets/dato-training/%s'\n",
    "targets = [\"business.csv.gz\", \"review.csv.gz\", \"user.csv.gz\"]\n",
    "\n",
    "for target in targets:\n",
    "    if not os.path.exists(target):\n",
    "        urlretrieve(url_template % target, target)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Testing your setup\n",
    "\n",
    "If you installed via Dato Launcher, open `ipython notebook`.\n",
    "\n",
    "Otherwise if you installed using pip, run the command `ipython notebook` in a terminal window.\n",
    "\n",
    "Create a new notebook (pressing \"New notebook\").\n",
    "\n",
    "In the command box write: `import graphlab` and then click the \"run\" button and make sure it runs without error.\n",
    "\n",
    "Run the following commands and verify the output is as seen in the picture."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "import graphlab\n",
    "graphlab.canvas.set_target('ipynb')\n",
    "graphlab.SArray([\"1\",\"2\",\"3\",\"4\"]).show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Expected Output:\n",
    "<table>\n",
    "<img src=\"images/graphlab.SArray1234.show.png\" width=\"50%\"></img>\n",
    "\n",
    "\n",
    "### Technical support\n",
    "For any questions, please email <a href=\"guy@dato.com\">guy@dato.com</a>, <a href=\"alon@dato.com\">alon@dato.com</a> or <a href=\"bickson@dato.com\">bickson@dato.com</a>.\n",
    "\n",
    "\n",
    "### One last note...\n",
    "Isn't `graphlab` a bit too long to type? In many notebooks you will see this shorthand alias:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "import graphlab as gl"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Getting Started with GraphLab Create: 5-Lines Recommender\n",
    "\n",
    "Let's [build a recommender system with 5 lines of Python code](https://dato.com/learn/gallery/notebooks/five_line_recommender.html).\n",
    "\n",
    "### Loading a dataset\n",
    "Load reviews dataset (a table stored in a gzipped CSV file) into a GraphLab SFrame. Since we're only demonstrting how to get started, we'll load just 100 rows:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "review = graphlab.SFrame.read_csv('review.csv.gz', nrows=100)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Visualizing a dataset\n",
    "Get some statistics using the GraphLab Canvas:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "gl.canvas.set_target('browser')\n",
    "review.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Or directly look at the beginning of the data using `user.head()`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "review.head(3)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "model = graphlab.recommender.create(review, user_id=\"user_id\", item_id=\"business_id\", target=\"stars\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# You can now make recommendations for all the users you've just trained on\n",
    "results = model.recommend(users=None, k=5)\n",
    "results.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# Save the model for later use\n",
    "model.save(\"model_100rows\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Exercise A - Data Engineering\n",
    "\n",
    "We'll do the data engineering exercise over the users data. Let's load it into an SFrame:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "user = graphlab.SFrame.read_csv('user.csv.gz')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Accessing a column\n",
    "Use `user['name']` to access the column `name`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "user['name']"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Computing statistics of a column\n",
    "Calculating the mean average_star rating:\n",
    "Use `user['average_stars'].mean()` to compute the mean star rating."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "user['average_stars'].mean()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Before you continue: a primer on operator overloading in Python\n",
    "**Intermediate-level Python programmers can skip this section**.\n",
    "\n",
    "You are about to discover the SFrame, the basic data structure used by GraphLab Create. If you are also new to Python, some of the syntax you are about to see may look weird to you. Don't worry! It is just a result of a really cool feature in Python called **operator overloading**. While Python is not the only language that has this feature, it certainly made programming operator overloading super-easy, so we'll be using it a lot to write shorter, more beautiful code.\n",
    "\n",
    "What is operator overloading? An operator is just what you know from your math class -\n",
    "```\n",
    "a + b ==> the + operator applied to a and b ==> +(a, b)\n",
    "```\n",
    "\n",
    "Since Python is an object-oriented language, when we see `object1 + object2`, what happens behind the scenes is:\n",
    "```\n",
    "a.__add__(b) # __add__ stands for +\n",
    "```\n",
    "\n",
    "So whenever you see some unfamiliar syntax, think: perhaps it's only operator overloading, and I should look for the documentation of that operator?\n",
    "\n",
    "Here's a quick example:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "class Pacman():\n",
    "    \"\"\"An object with some customized operators in it.\"\"\"\n",
    "    \n",
    "    def __init__(self):\n",
    "        \"\"\"Constructor\"\"\"\n",
    "        self.stomach = []\n",
    "    \n",
    "    def eat(self, food):\n",
    "        \"\"\"Method\"\"\"\n",
    "        self.stomach.append(food)\n",
    "        return self\n",
    "    \n",
    "    def __add__(self, food):\n",
    "        \"\"\"+ operator\"\"\"\n",
    "        return self.eat(food)\n",
    "    \n",
    "    def __str__(self):\n",
    "        \"\"\"print statement operator\"\"\"\n",
    "        return \"(%s<)\" % (\"\".join(map(str, self.stomach)))\n",
    "    \n",
    "    def __repr__(self):\n",
    "        \"\"\"REPL printout operator\"\"\"\n",
    "        return \"Pacman(%s)\" % (str(self.stomach))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "pac = Pacman()\n",
    "print pac"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "pac.eat(1)\n",
    "pac.eat('*')\n",
    "print pac"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "pac = pac + 3\n",
    "print pac"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "pac"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Even what's printed on the screen may get manipulated by objects... *You have been warned!*\n",
    "\n",
    "And now...\n",
    "\n",
    "## Back to the Exercise"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Accessing a row\n",
    "Print out the first 3 rows:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "user.head(3) # or user[:3]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Print out the last 3 rows:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "user.tail(3) # or user[-3:]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Accessing a row column combination\n",
    "Print the name and average_stars column of the first 10 rows:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "user[0:10][['name','average_stars']]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Logical operators\n",
    "Test for all users name Jim:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "selection = user['name'] == 'Jim'"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Select all data rows for users named Jim:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "user[selection]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Shorthand notation (AKA one-liner):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "user[user['name'] == 'Jim']"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### How many user named Jim  have more than 2 avg_stars:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "user[(user['name'] == 'Jim') & (user['average_stars'] > 2)]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### SQL primitives\n",
    "\n",
    "Count users with the same name:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "user.groupby(['name'], graphlab.aggregate.COUNT())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Count users with the same name and number of votes_funny:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "user.groupby(['name', 'votes_funny'], graphlab.aggregate.COUNT())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Calculate the average number of reviews per name:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "user.groupby(['name'], graphlab.aggregate.AVG('review_count'))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Transformations\n",
    "Add 12 reviews to all users. This does not modify the SFrame, and the result is not stored:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "user['review_count'].apply(lambda x: x + 12)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can also use Python libraries and your own user-defined functions with the `apply()` method."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "from string import capitalize\n",
    "\n",
    "def cap_string(s):\n",
    "    \"\"\"First letter gets uppercased, the rest get lowercased.\"\"\"\n",
    "    return capitalize(s)\n",
    "\n",
    "user['user_id'].apply(capitalize)[:3]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Generate a new column which is the sum of votes_cool and votes_funny"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "user['funny_cool'] = user['votes_cool'] + user['votes_funny']"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "By storing the resulting column into a new column name, the SFrame is modified.\n",
    "\n",
    "### Load business and review data:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "business = graphlab.SFrame.read_csv('business.csv.gz')\n",
    "review = graphlab.SFrame.read_csv('review.csv.gz')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Visualize the business data:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "business.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Show reviews with the max stars rating:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "review.topk(\"stars\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Remove all the fields in the business table with has only a single unique value:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "business.remove_columns([\"type\"])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Find all reviews of all users named Jim:\n",
    "\n",
    "Review writers are identified by their `user_id` in the `review` SFrame, but their names are in a different SFrame (`user`). We'll join the two SFrames in order to find Jim's reviews (and demonstrate `join()`, of course)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "joined = review.join(user, on='user_id')\n",
    "joined[joined['name'] == 'Jim']"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Exercise B: Graph Analytics\n",
    "\n",
    "### Construct a graph out of the 500 first rows with edges between year and name:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "joined['year'] = joined['year'].astype(str) # vertices should have the same data type\n",
    "graph = graphlab.SGraph().add_edges(joined[:500], src_field='year', dst_field='name')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Visualize the graph:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "highlight_color = [0.69, 0.0, 0.498]\n",
    "highlight = {year: highlight_color for year in joined['year'].unique()}\n",
    "graph.show(vlabel_hover=True, vlabel='__id', highlight=highlight)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Calculate PageRank:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "pr = graphlab.pagerank.create(graph) "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Find the 10 most important nodes in the graph:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "print pr['pagerank'].topk('pagerank')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Add the pagerank information into the graph vertices:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "graph = graph.add_vertices(pr['pagerank'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Show the pagerank values on the graph:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "graph.show(vlabel_hover=True, vlabel='pagerank')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Define a new label for the visualization"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "def give_label(row):\n",
    "    label = \"%s (%f)\" % (row[\"__id\"], row[\"pagerank\"])\n",
    "    return label\n",
    "    \n",
    "graph.vertices['label'] = graph.vertices.apply(give_label)\n",
    "graph.vertices"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "graph.show(vlabel_hover=True, vlabel='label', highlight=highlight)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Exercise C: Recommenders (User modeling)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Split the data into 90% train and  10% test sets:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "train, test = review.random_split(0.9)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Build a matrix factorization recommender to predict star rating:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "model = graphlab.factorization_recommender.create(train, target='stars',\n",
    "                                                  item_id='business_id',\n",
    "                                                  side_data_factorization=False)\n",
    "print 'Training RMSE', model.get('training_rmse')\n",
    "print 'Test RMSE', graphlab.evaluation.rmse(test['stars'], model.predict(test))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Now we will improve the model using business side features:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "model_with_side_data_factorization = graphlab.factorization_recommender.create(train,\n",
    "                                                                               target='stars',\n",
    "                                                                               item_id='business_id',\n",
    "                                                                               side_data_factorization=True,\n",
    "                                                                               user_data=user, \n",
    "                                                                               item_data=business)\n",
    "\n",
    "print 'Training RMSE', model_with_side_data_factorization.get('training_rmse')\n",
    "print 'Test RMSE', graphlab.evaluation.rmse(test['stars'], model_with_side_data_factorization.predict(test))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Is there an improvement?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "print \"Test RMSE without side features: %f\" % graphlab.evaluation.rmse(test['stars'], model.predict(test))\n",
    "print \"Test RMSE with side features: %f\" % graphlab.evaluation.rmse(test['stars'], model_with_side_data_factorization.predict(test))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "print \"Max error without side features: %f\" % graphlab.evaluation.max_error(test['stars'], model.predict(test))\n",
    "print \"Max error with side features: %f\" % graphlab.evaluation.max_error(test['stars'], model_with_side_data_factorization.predict(test))"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 2",
   "language": "python",
   "name": "python2"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 2
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython2",
   "version": "2.7.10"
  },
  "widgets": {
   "state": {},
   "version": "1.1.2"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 0
}