3. Hard Challenge.ipynb

{
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": "### Challenge questions\n\nEasy questions:\n\n 1. How many total pings are in the Ocearch shark data?\n 2. How many unique species of sharks are in the data set?\n 3. What is the name, weight, and species of the heaviest shark(s)?\n 4. When and where was the very first ping?\n 5. Excluding results with 0 distance traveled: what's the minimum, average, and maximum travel distances?\n \nIntermediate questions:\n\n 1. Which shark had the most pings??\n 2. Which shark has been pinging the longest, and how long has that been?\n 3. Which shark species has the most individual sharks tagged?\n 4. What is the average length and weight of each shark species?\n 5. Which shark has the biggest geographic box (largest distance from min lat/lon to max lat/lon, not dist_traveled)?\n \nHard questions:\n 1. Use folium to plot the first ping, last ping, and a line connecting each ping for the Tiger shark Emma.  Make the first ping marker a 'play' icon, and last ping icon a 'stop' icon.\n 2. Resample Emma data to have a daily lat/lon average, and interpolate missing results.  Plot a marker for each day, and color them blue for hard data, green for interpolated lat/lons\n 3. Resample all shark data for daily lat/lon averages, and interpolate missing results\n 4. Calculate distance between Emma and other sharks on a daily basis\n 5. Identify the shark that has the shortest average distance to Emma per day (minimum 50 days of pings with Emma)\n 6. Plot Emma and her closest buddy: interpolated results for each in green, Emma as circle icons and her buddy as square icons",
      "id": "461f94b4"
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "### Load data"
      ],
      "id": "3f2d399e"
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "#### Query Ocearch API"
      ],
      "id": "aebc8750"
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "ExecuteTime": {
          "end_time": "2019-02-23T02:33:00.784133Z",
          "start_time": "2019-02-23T02:32:56.020412Z"
        }
      },
      "outputs": [],
      "source": [
        "import requests\n",
        "url = 'http://www.ocearch.org/tracker/ajax/filter-sharks'\n",
        "\n",
        "resp = requests.get(url)\n",
        "resp"
      ],
      "id": "db997a1a"
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "#### Transform data"
      ],
      "id": "8bcf701c"
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "noteable": {
          "output_collection_id": "5c15d5a9-8355-4c25-b740-e2681fe0542e"
        },
        "ExecuteTime": {
          "end_time": "2023-04-25T19:23:52.957660+00:00",
          "start_time": "2023-04-25T19:23:50.698652+00:00"
        },
        "datalink": {
          "93b8783d-9a17-47e9-a9dd-d044bae766d7": {
            "applied_filters": [],
            "dataframe_info": {
              "default_index_used": true,
              "orig_num_cols": 12,
              "orig_num_rows": 65793,
              "orig_size_bytes": 6316256,
              "truncated_num_cols": 12,
              "truncated_num_rows": 50000,
              "truncated_size_bytes": 5200000,
              "truncated_string_columns": []
            },
            "display_id": "93b8783d-9a17-47e9-a9dd-d044bae766d7",
            "dx_settings": {
              "ALLOW_NOTEABLE_ATTRS": true,
              "COLUMN_SAMPLING_METHOD": "outer",
              "DB_LOCATION": ":memory:",
              "DEV_MODE": false,
              "DISPLAY_MAX_COLUMNS": 100,
              "DISPLAY_MAX_ROWS": 50000,
              "DISPLAY_MODE": "simple",
              "ENABLE_ASSIGNMENT": true,
              "ENABLE_DATALINK": true,
              "FLATTEN_COLUMN_VALUES": true,
              "FLATTEN_INDEX_VALUES": false,
              "GENERATE_DEX_METADATA": false,
              "HTML_TABLE_SCHEMA": false,
              "LOG_LEVEL": 30,
              "MAX_RENDER_SIZE_BYTES": 104857600,
              "MAX_STRING_LENGTH": 250,
              "NUM_PAST_SAMPLES_TRACKED": 3,
              "RANDOM_STATE": 12648430,
              "RESET_INDEX_VALUES": false,
              "ROW_SAMPLING_METHOD": "random",
              "SAMPLING_FACTOR": 0.1,
              "SAMPLING_METHOD": "random",
              "STRINGIFY_COLUMN_VALUES": true,
              "STRINGIFY_INDEX_VALUES": false
            },
            "sample_history": [],
            "sampling_time": "2023-04-25T19:23:52.409057",
            "user_variable_name": null,
            "variable_name": "unk_dataframe_1c8ddf4a7287455c8441480ae8aacd6f"
          }
        }
      },
      "outputs": [],
      "source": "pd.read_csv('data/sharks.csv')",
      "id": "b714eb74"
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "### Explore data"
      ],
      "id": "9d8b65cd"
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {},
      "outputs": [],
      "source": [],
      "id": "8eea33f0"
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "### Challenge Questions"
      ],
      "id": "f9cf8525"
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "#### Easy Questions\n",
        " 1. How many total pings are in the Ocearch shark data?\n",
        " 2. How many unique species of sharks are in the data set?\n",
        " 3. What is the name, weight, and species of the heaviest shark(s)?\n",
        " 4. When and where was the very first ping?\n",
        " 5. Excluding results with 0 distance traveled: what's the minimum, average, and maximum travel distances?"
      ],
      "id": "52d13eff"
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "##### Total pings\n",
        "How many total pings are in the Ocearch shark data?"
      ],
      "id": "7f95a6ec"
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "ExecuteTime": {
          "end_time": "2019-02-23T02:33:20.857761Z",
          "start_time": "2019-02-23T02:33:20.744835Z"
        }
      },
      "outputs": [],
      "source": [],
      "id": "cc9031a4"
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "##### Unique species\n",
        "How many unique species of sharks are in the data set?"
      ],
      "id": "9bb07f03"
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "ExecuteTime": {
          "end_time": "2019-02-23T02:33:20.985760Z",
          "start_time": "2019-02-23T02:33:20.862738Z"
        }
      },
      "outputs": [],
      "source": [],
      "id": "a1fd8097"
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "##### Heaviest shark(s)\n",
        "What is the name, weight, and species of the heaviest shark(s)?"
      ],
      "id": "d72f3dec"
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "ExecuteTime": {
          "end_time": "2019-02-23T02:33:21.107187Z",
          "start_time": "2019-02-23T02:33:20.986759Z"
        }
      },
      "outputs": [],
      "source": [],
      "id": "2a35a56c"
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "##### First ping\n",
        "When and where was the very first ping?"
      ],
      "id": "4f6a0af0"
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "ExecuteTime": {
          "end_time": "2019-02-23T02:33:21.622987Z",
          "start_time": "2019-02-23T02:33:21.372026Z"
        }
      },
      "outputs": [],
      "source": [],
      "id": "c847d9a5"
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "##### Distance travelled\n",
        "Excluding results with 0 distance traveled: what's the minimum, average, and maximum travel distances?"
      ],
      "id": "3d03a5e8"
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "ExecuteTime": {
          "end_time": "2019-02-23T02:33:21.637981Z",
          "start_time": "2019-02-23T02:33:21.624986Z"
        }
      },
      "outputs": [],
      "source": [],
      "id": "759a0e87"
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "#### Intermediate questions\n",
        "\n",
        " 1. Which shark had the most pings?\n",
        " 2. Which shark has been pinging the longest, and how long has that been?\n",
        " 3. Which shark species has the most individual sharks tagged?\n",
        " 4. What is the average length and weight of each shark species?\n",
        " 5. Which shark has the biggest geographic box (largest distance from min lat/lon to max lat/lon, not dist_traveled)?"
      ],
      "id": "17c9685e"
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "##### Most pings\n",
        "Which shark had the most pings?"
      ],
      "id": "231d76e3"
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "ExecuteTime": {
          "end_time": "2019-02-23T02:33:21.838513Z",
          "start_time": "2019-02-23T02:33:21.638982Z"
        }
      },
      "outputs": [],
      "source": [],
      "id": "d8d379ed"
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "##### Longest duration pinger\n",
        "Which shark has been pinging the longest, and how long has that been?"
      ],
      "id": "cf7df508"
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "ExecuteTime": {
          "end_time": "2019-02-23T02:33:21.901753Z",
          "start_time": "2019-02-23T02:33:21.839513Z"
        }
      },
      "outputs": [],
      "source": [],
      "id": "e30e5a37"
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "##### Individual count by species\n",
        "Which shark species has the most individual sharks tagged?"
      ],
      "id": "b988a61f"
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "ExecuteTime": {
          "end_time": "2019-02-23T02:33:22.063151Z",
          "start_time": "2019-02-23T02:33:21.902750Z"
        }
      },
      "outputs": [],
      "source": [],
      "id": "dad7a3ba"
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "##### Average length/weight by species\n",
        "What is the average length and weight of each shark species?"
      ],
      "id": "e0a109fc"
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "ExecuteTime": {
          "end_time": "2019-02-23T02:33:22.164115Z",
          "start_time": "2019-02-23T02:33:22.065137Z"
        }
      },
      "outputs": [],
      "source": [],
      "id": "4b9c2df1"
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "##### Biggest geographic box\n",
        "Which shark has the biggest geographic box (largest area calculated from min lat/lon to max lat/lon, not dist_traveled)?"
      ],
      "id": "a0b82e6c"
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "ExecuteTime": {
          "end_time": "2019-02-23T02:33:22.325111Z",
          "start_time": "2019-02-23T02:33:22.165108Z"
        }
      },
      "outputs": [],
      "source": [],
      "id": "3680ba39"
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "#### Hard questions\n",
        "\n",
        " 1. Use folium to plot the first ping, last ping, and a line connecting each ping for the Tiger shark Emma.  Make the first ping marker a 'play' icon, and last ping icon a 'stop' icon.\n",
        " 2. Resample Emma data to have a daily lat/lon average, and interpolate missing results.  Plot a marker for each day, and color them blue for hard data, green for interpolated lat/lons\n",
        " 3. Resample all shark data for daily lat/lon averages, and interpolate missing results\n",
        " 4. Calculate distance between Emma and other sharks on a daily basis\n",
        " 5. Identify the shark that has the shortest average distance to Emma per day (minimum 50 days of pings with Emma)\n",
        " 6. Plot Emma and her closest buddy: interpolated results for each in green, Emma as circle icons and her buddy as square icons"
      ],
      "id": "beae018a"
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "##### Plot Emma locations\n",
        "Plot the ping locations for the shark named Emma as a `PolyLine` in folium.  Include the first and last ping location as markers."
      ],
      "id": "07b7c24a"
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "ExecuteTime": {
          "end_time": "2019-02-23T02:33:22.565497Z",
          "start_time": "2019-02-23T02:33:22.436186Z"
        }
      },
      "outputs": [],
      "source": [],
      "id": "0eb98d2d"
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "##### Plot interpolated locs\n",
        "Resample the Emma locations on a per-day basis and interpolate missing locations.  Then, plot the daily markes in folium along with a `PolyLine`."
      ],
      "id": "c7d154b6"
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "ExecuteTime": {
          "end_time": "2019-02-23T02:33:23.286310Z",
          "start_time": "2019-02-23T02:33:23.277313Z"
        }
      },
      "outputs": [],
      "source": [],
      "id": "263aa061"
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "##### Resample all shark data\n",
        "Resample all shark data for daily lat/lon averages, and interpolate missing results"
      ],
      "id": "1a38cb0f"
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "ExecuteTime": {
          "end_time": "2019-02-23T02:33:26.298171Z",
          "start_time": "2019-02-23T02:33:23.913944Z"
        }
      },
      "outputs": [],
      "source": [],
      "id": "f2685493"
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "##### Distance to Emma\n",
        "Identify the shark that has the shortest average distance to Emma per day"
      ],
      "id": "9b07dc54"
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "ExecuteTime": {
          "end_time": "2019-02-23T02:33:26.495109Z",
          "start_time": "2019-02-23T02:33:26.476115Z"
        }
      },
      "outputs": [],
      "source": [],
      "id": "281adc07"
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "##### Emma's buddy\n",
        "Identify the shark that has the shortest average distance to Emma per day (minimum 50 days of pings with Emma)"
      ],
      "id": "97d8b68d"
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "ExecuteTime": {
          "end_time": "2019-02-23T02:33:27.338157Z",
          "start_time": "2019-02-23T02:33:27.189078Z"
        }
      },
      "outputs": [],
      "source": [],
      "id": "ee394bc6"
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "##### Plot Emma and Buddy\n",
        "Plot Emma and her closest buddy on folium.  Emma should be blue/green (known/interpolated) and her buddy should be red/black (known/interpolated)."
      ],
      "id": "305ebf04"
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "ExecuteTime": {
          "end_time": "2019-02-23T02:33:27.685365Z",
          "start_time": "2019-02-23T02:33:27.558416Z"
        }
      },
      "outputs": [],
      "source": [],
      "id": "5bcfc544"
    }
  ],
  "metadata": {
    "kernelspec": {
      "display_name": "Python3.6",
      "language": "python",
      "name": "py36"
    },
    "language_info": {
      "codemirror_mode": {
        "name": "ipython",
        "version": 3
      },
      "file_extension": ".py",
      "mimetype": "text/x-python",
      "name": "python",
      "nbconvert_exporter": "python",
      "pygments_lexer": "ipython3",
      "version": "3.7.2"
    },
    "toc": {
      "base_numbering": 1,
      "nav_menu": {},
      "number_sections": true,
      "sideBar": true,
      "skip_h1_title": false,
      "title_cell": "Table of Contents",
      "title_sidebar": "Contents",
      "toc_cell": false,
      "toc_position": {
        "height": "calc(100% - 180px)",
        "left": "10px",
        "top": "150px",
        "width": "256px"
      },
      "toc_section_display": true,
      "toc_window_display": true
    },
    "noteable": {
      "last_transaction_id": "dead25c4-a059-425a-800d-9440c4cfab3a"
    },
    "selected_hardware_size": "small"
  },
  "nbformat": 4,
  "nbformat_minor": 5
}