2_Data Analysis and Visualization.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Data Analysis and Visualization\n",
    "\n",
    "Analyzing and Visualizing data using Pandas\n",
    "\n",
    "`import pandas as pd`\n",
    "\n",
    "`from matplotlib import pyplot as plt`\n",
    "\n",
    "`%matplotlib inline`\n",
    "\n",
    "`from zipfile import ZipFile`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Create a DataFrame\n",
    "\n",
    "Create a dataframe from San Francisco 311 case reports from 2019. Set the index column to `\"CaseID\"`\n",
    "\n",
    "`cases = pd.read_csv(\"311_Cases2019.csv\", index_col=\"CaseID\")`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "with ZipFile(\"./data/311_Cases2019.csv.zip\", \"r\") as zip:\n",
    "    zip.extractall(\"./data/\")\n",
    "    \n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Selecting Data by Column\n",
    "\n",
    "From `cases` select all rows from columns `\"Opened\"` through `\"Source\"` *column indicies 0-17*\n",
    "\n",
    "`cases = cases.loc[:, \"Opened\":\"Source\"]`\n",
    "\n",
    "or\n",
    "\n",
    "`cases = cases.iloc[:, 0:18]`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Sorting Data\n",
    "\n",
    "Use `sort_values` to sort on a specific column. Use `axis=0`. Use `ascending=False` for reverse sorting\n",
    "\n",
    "`cases = cases.sort_values(\"Request Type\", axis=0, ascending=True)`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Replacing NaN Values\n",
    "\n",
    "Use the `fillna` function to replace NaN values with another value\n",
    "\n",
    "`cases = cases.fillna(\"None\")`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Value Counts\n",
    "\n",
    "Use `value_counts()` to find the 5 most common categories\n",
    "\n",
    "`cat_counts = cases[\"Category\"].value_counts()[:5]`\n",
    "\n",
    "`nhood_counts = cases[\"Neighborhood\"].value_counts()`\n",
    "\n",
    "`nhood_counts`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Grouping Data\n",
    "\n",
    "Using the `groupby` function\n",
    "\n",
    "`gc = cases.groupby(\"Category\")`\n",
    "\n",
    "`gc.get_group(\"Street and Sidewalk Cleaning\").head()`\n",
    "\n",
    "`gc = cases.groupby(\"Category\").size()`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Analyzing Case Data by Weekday\n",
    "\n",
    "Copy `cases` into a new dataframe called `weekdays`. Use the `to_datetime` function to create an index from the `\"Opened\"` column.\n",
    "\n",
    "`weekdays = cases[[\"Opened\", \"Category\", \"Neighborhood\",\"Request Type\"]].copy()`\n",
    "\n",
    "`date_index = pd.DatetimeIndex(pd.to_datetime(weekdays[\"Opened\"]))`\n",
    "\n",
    "Create a new column for `Weekday` and add the value from `date_index` using the attribute `weekday_name`\n",
    "\n",
    "`weekdays.loc[:,'Weekday'] = date_index.weekday_name`\n",
    "\n",
    "`weekdays.loc[:,\"Opened\"] = date_index`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Visualizing Data\n",
    "\n",
    "Plot a bar graph of category counts\n",
    "\n",
    "`cat_counts.plot(kind=\"bar\")`\n",
    "\n",
    "`plt.suptitle(\"SF 311 Cases (2019)\")`\n",
    "\n",
    "`plt.xlabel(\"Category\")`\n",
    "\n",
    "`plt.ylabel(\"Cases\")`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Plotting Data on a Map\n",
    "\n",
    "Use Latitude and Longitude columns to plot locations on a map. View the latitude and longitude data using `describe()`. \n",
    "\n",
    "`cases[['Latitude', 'Longitude']].describe()`\n",
    "\n",
    "Note that some min and max values are 0."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Create a subset that ignores bad values\n",
    "\n",
    "`valid_latlon = cases[(cases.Latitude > 30) & (cases.Longitude < -120)]`\n",
    "\n",
    "`valid_latlon[['Latitude', 'Longitude']].describe()`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Get all cases from the `\"Noise Report\"` Category:\n",
    "\n",
    "`categories = valid_latlon.groupby(\"Category\")`\n",
    "\n",
    "`noise = categories.get_group(\"Noise Report\")`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Plot the data on a map"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.7"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}