demo/Inline_Demo.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This notebook demonstrates [vaquero](https://github.com/jbn/vaquero), as both a library and data cleaning pattern."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": false,
    "run_control": {
     "frozen": false,
     "read_only": false
    }
   },
   "outputs": [],
   "source": [
    "from vaquero import Vaquero, callables_from"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Task\n",
    "\n",
    "Say you think you have pairs of numbers serialized as comma separated values in a file. You want to extract the pair from each line, then sum over the result (per line). "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Sample Data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "collapsed": true,
    "run_control": {
     "frozen": false,
     "read_only": false
    }
   },
   "outputs": [],
   "source": [
    "lines = [\"1, 1.0\",  # An errant float\n",
    "         \"1, $\",    # A bad number\n",
    "         \"1,-1\",    # A good line\n",
    "         \"10\"]      # Missing the second value"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Initial Implementation"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "collapsed": true,
    "run_control": {
     "frozen": false,
     "read_only": false
    }
   },
   "outputs": [],
   "source": [
    "def extract_pairs(s):\n",
    "    return s.split(\",\")\n",
    "    \n",
    "    \n",
    "def to_int(items):\n",
    "    return [int(item) for item in items]\n",
    "\n",
    "\n",
    "def sum_pair(items):\n",
    "    return items[0], items[1]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Iteration 1\n",
    "\n",
    "First, instantiate a vaquero instance. Here, I've set the maximum number of failures allowed to 5. After that many failures, the `Vaquero` object raises a `VaqueroException`. Generally, you want it to be large enough to collect a lot of unexpected failures. But, you don't want it to be so large you exhaust memory. This is an iterative process.\n",
    "\n",
    "Also, as a tip, always instantiate the `Vaquero` object in its own cell. This way, you get to inspect it in your notebook even if it raises a `VaqueroException`.\n",
    "\n",
    "I also registered all functions (well, callables) in this notebook with `vaquero`. The error capturing machinery only operates on the registered functions. And, it always ignores a `KeyboardInterrupt`. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "collapsed": true,
    "run_control": {
     "frozen": false,
     "read_only": false
    }
   },
   "outputs": [],
   "source": [
    "vaquero = Vaquero(max_failures=5)\n",
    "vaquero.register_targets(callables_from(globals()))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Just to be sure, I'll check the registered functions. It does matching by name, which is a bit naive. But, it's also surprisingly robust given vaquero usage patterns. Looking, you can see some things that don't belong. But, again, it mostly works well. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "collapsed": false,
    "run_control": {
     "frozen": false,
     "read_only": false
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'Vaquero',\n",
       " 'callables_from',\n",
       " 'exit',\n",
       " 'extract_pairs',\n",
       " 'get_ipython',\n",
       " 'quit',\n",
       " 'sum_pair',\n",
       " 'to_int'}"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "vaquero.target_funcs"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now, run my trivial examples over the initial implementation."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "collapsed": false,
    "run_control": {
     "frozen": false,
     "read_only": false
    }
   },
   "outputs": [],
   "source": [
    "results = []\n",
    "\n",
    "for s in lines:\n",
    "    with vaquero.on_input(s):\n",
    "        results.append(sum_pair(to_int(extract_pairs(s))))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "It was not successful."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "collapsed": false,
    "run_control": {
     "frozen": false,
     "read_only": false
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "False"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "vaquero.was_successful"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "So, look at the failures. There were two functions, and both had failures."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "collapsed": false,
    "run_control": {
     "frozen": false,
     "read_only": false
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'failures': 3,\n",
       " 'failures_by': {'sum_pair': 1, 'to_int': 2},\n",
       " 'ignored': 0,\n",
       " 'successes': 1}"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "vaquero.stats()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To get a sense of what happened, examine the failing functions.\n",
    "\n",
    "You can do this by calling `examine` with the name of the function (or the function object). It returns the captured invocations and errors.\n",
    "\n",
    "Here you can see that the `to_int` function from cell `In [3]` failed with a `ValueError` exception."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "collapsed": false,
    "run_control": {
     "frozen": false,
     "read_only": false
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[{'call_args': [['1', ' 1.0']],\n",
       "  'exc_type': 'ValueError',\n",
       "  'exc_value': \"invalid literal for int() with base 10: ' 1.0'\",\n",
       "  'filename': 'In [3]',\n",
       "  'lineno': 6,\n",
       "  'name': 'to_int'},\n",
       " {'call_args': [['1', ' $']],\n",
       "  'exc_type': 'ValueError',\n",
       "  'exc_value': \"invalid literal for int() with base 10: ' $'\",\n",
       "  'filename': 'In [3]',\n",
       "  'lineno': 6,\n",
       "  'name': 'to_int'}]"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "vaquero.examine('to_int')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Often though, we want to query only parts of the capture for a specific function. To do so, you can use [JMESPath](http://jmespath.org/), specifying the selector as an argument to `exam`. Also, you can say, show me only the set applied to the selected result (assuming it's hashable), to simplify things. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "collapsed": false,
    "run_control": {
     "frozen": false,
     "read_only": false
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{\"invalid literal for int() with base 10: ' $'\",\n",
       " \"invalid literal for int() with base 10: ' 1.0'\"}"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "vaquero.examine('to_int', '[*].exc_value', as_set=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "And, for `sum_pair`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "collapsed": false,
    "run_control": {
     "frozen": false,
     "read_only": false
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[{'call_args': [[10]],\n",
       "  'exc_type': 'IndexError',\n",
       "  'exc_value': 'list index out of range',\n",
       "  'filename': 'In [3]',\n",
       "  'lineno': 10,\n",
       "  'name': 'sum_pair'}]"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "vaquero.examine('sum_pair')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Iteration 2\n",
    "\n",
    "We know know that there are some ints encoded as doubles. But, we know from our data source, it can only be an int. So, in `to_ints`, let's parse the strings first as `float`s, then create an `int` from it. It's robust.\n",
    "\n",
    "Also, we know that some lines don't have two components. Those are just bad lines. Let's assert there are two parts as  post condition of `extract_pairs`. \n",
    "\n",
    "Finally, after a bit of digging, we found that `$` means `NA`. After cursing for a minute -- because that's crazy -- you decide to ignore those entries. Instead of adding this to an existing function, you write an `assert_no_missing_data` function."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "collapsed": true,
    "run_control": {
     "frozen": false,
     "read_only": false
    }
   },
   "outputs": [],
   "source": [
    "def no_missing_data(s):\n",
    "    assert '$' not in s, \"'{}' has missing data\".format(s)\n",
    "    \n",
    "    \n",
    "def extract_pairs(s):\n",
    "    parts = s.split(\",\")\n",
    "    assert len(parts) == 2, \"'{}' not in 2 parts\".format(s)\n",
    "    return tuple(parts)\n",
    "    \n",
    "    \n",
    "def to_int(items):\n",
    "    return [int(float(item)) for item in items]\n",
    "\n",
    "\n",
    "def sum_pair(items):\n",
    "    assert len(items) == 2, \"Line is improperly formatted\"\n",
    "    return items[0] + items[1]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "collapsed": true,
    "run_control": {
     "frozen": false,
     "read_only": false
    }
   },
   "outputs": [],
   "source": [
    "vaquero.reset()  # Clear logged errors, mostly.\n",
    "vaquero.register_targets(globals())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "collapsed": false,
    "run_control": {
     "frozen": false,
     "read_only": false
    }
   },
   "outputs": [],
   "source": [
    "results = []\n",
    "\n",
    "for s in lines:\n",
    "    with vaquero.on_input(s):\n",
    "        no_missing_data(s)\n",
    "        results.append(sum_pair(to_int(extract_pairs(s))))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now, we have one more success, but still two failures. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "collapsed": false,
    "run_control": {
     "frozen": false,
     "read_only": false
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'failures': 2,\n",
       " 'failures_by': {'extract_pairs': 1, 'no_missing_data': 1},\n",
       " 'ignored': 0,\n",
       " 'successes': 2}"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "vaquero.stats()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's quickly examine."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "collapsed": false,
    "run_control": {
     "frozen": false,
     "read_only": false
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[{'call_args': ['10'],\n",
       "  'exc_type': 'AssertionError',\n",
       "  'exc_value': \"'10' not in 2 parts\",\n",
       "  'filename': 'In [12]',\n",
       "  'lineno': 7,\n",
       "  'name': 'extract_pairs'}]"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "vaquero.examine('extract_pairs')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {
    "collapsed": false,
    "run_control": {
     "frozen": false,
     "read_only": false
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[{'call_args': ['1, $'],\n",
       "  'exc_type': 'AssertionError',\n",
       "  'exc_value': \"'1, $' has missing data\",\n",
       "  'filename': 'In [12]',\n",
       "  'lineno': 2,\n",
       "  'name': 'no_missing_data'}]"
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "vaquero.examine('no_missing_data')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Both these exceptions are bad data. We want to ignore them."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {
    "collapsed": false,
    "run_control": {
     "frozen": false,
     "read_only": false
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'failures': 0, 'failures_by': {}, 'ignored': 2, 'successes': 2}"
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "vaquero.stats_ignoring('AssertionError')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Looking at the results accumulated,"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {
    "collapsed": false,
    "run_control": {
     "frozen": false,
     "read_only": false
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[2, 0]"
      ]
     },
     "execution_count": 19,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "results"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Things look good."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now that we have something that works, we can use Vaquero in a more production-oriented mode. That is, we allow for unlimited errors, but we don't capture anything. That is, we note the failure, but otherwise ignore it since we won't be post-processing."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {
    "collapsed": false,
    "run_control": {
     "frozen": false,
     "read_only": false
    }
   },
   "outputs": [],
   "source": [
    "vaquero.reset(turn_off_error_capturing=True) \n",
    "# Or, Vaquero(capture_error_invocations=False)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {
    "collapsed": false,
    "run_control": {
     "frozen": false,
     "read_only": false
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[2, 0]"
      ]
     },
     "execution_count": 21,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "results = []\n",
    "\n",
    "for s in lines:\n",
    "    with vaquero.on_input(s):\n",
    "        no_missing_data(s)\n",
    "        results.append(sum_pair(to_int(extract_pairs(s))))\n",
    "        \n",
    "results"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "They still show up as failures, but it doesn't waste memory storing the captures."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {
    "collapsed": false,
    "run_control": {
     "frozen": false,
     "read_only": false
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'failures': 2, 'failures_by': {}, 'ignored': 0, 'successes': 2}"
      ]
     },
     "execution_count": 22,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "vaquero.stats()"
   ]
  }
 ],
 "metadata": {
  "anaconda-cloud": {},
  "kernelspec": {
   "display_name": "Python [default]",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.5.2"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 0
}