01b_workflow.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "4b0c4908-c142-4be8-ae18-3d91ddb2a33a",
   "metadata": {},
   "source": [
    "# Designing A Web App for Exploring Land-Ocean Temperature Data\n",
    "\n",
    "### Goal of this Notebook\n",
    "\n",
    "This notebook is designed to provide a review of the data and the smoothing algorithms we will use in the web app being developed as a part of this tutorial.\n",
    "\n",
    "### Steps You Will Take In This Notebook\n",
    "In this notebook you will:\n",
    "1. Review how to load data from a CSV file into a `pandas` DataFrame.\n",
    "2. Plot the data to visualize the temperature trends using the `matplotlib` library.\n",
    "3. Review the smoothing algorithms we will use in the web app and add smoothed data to the DataFrame (and plot).\n",
    "4. Revise how to export a selected subset of the data to a new CSV file."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ea7e3b45",
   "metadata": {},
   "source": [
    "## Describing the Elements of our Dashboard\n",
    "\n",
    "Across all the notebooks in this tutorial, we will be developing a small web app that will provide a graphical user interface (GUI) for exploring a dataset that contains global land-ocean temperature data. The data is stored in a CSV file.\n",
    "\n",
    "We will build the web app using a Jupyter notebook where our workflow will:\n",
    "- **load a data file** with global land-ocean temperature data and\n",
    "- **smooth the data with an algorithm with different smoothing parameters**.\n",
    "- In the GUI, you should be able to **select a particular range of years** and\n",
    "- **create a new data file** containing the smoothed data for desired time range.\n",
    "\n",
    "Before building the web app, let's explore the data and the smoothing algorithms we will use."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "d8180eff-3c0e-4b77-b3d0-51632faedd91",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Load the required libraries\n",
    "import pandas as pd\n",
    "import os"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "90084ed2-dcc5-4a98-a697-02b36bdba826",
   "metadata": {},
   "source": [
    "## Load data from file\n",
    "\n",
    "We have a CSV data file containing Global Annual Mean Surface Air Temperature Change data from 1880 to 2023 [downloaded from NASA](https://data.giss.nasa.gov/gistemp/graphs/).  This is the data that we would ultimately like to visualize using a web app. We will start by loading the data for this tutorial from the CSV file into a `pandas` DataFrame."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "3a345487-7730-4d80-9e49-2da04e70d9df",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Load data into memory from file\n",
    "DATA_DIR = 'data'\n",
    "DATA_FILE = 'land-ocean-temp-index.csv'\n",
    "\n",
    "df = pd.read_csv(os.path.join(DATA_DIR, DATA_FILE), escapechar='#')\n",
    "df"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "36d691eb-4284-4624-bf57-0f1cdf80f5ab",
   "metadata": {},
   "source": [
    "## Plotting the Raw Data\n",
    "\n",
    "We will plot the raw data to visualize the temperature trends."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "67e1839e-22ff-4bab-b42d-f962a6e2babb",
   "metadata": {},
   "outputs": [],
   "source": [
    "from matplotlib import pyplot as plt\n",
    "\n",
    "plt.xlabel('Year')\n",
    "plt.ylabel('Temperature Anomalies over Land w.r.t. 1951-80 (˚C)')\n",
    "plt.title('Global Annual Mean Surface Air Temperature Change')\n",
    "plt.plot(df['Year'], df['Temperature'], label='Raw Data')\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a4922ab5-9a59-4be1-913b-a4935764bee6",
   "metadata": {},
   "source": [
    "## Adding Columns for Filtered Data\n",
    "\n",
    "Let's implement the [Savitzky-Golay filter](https://en.wikipedia.org/wiki/Savitzky–Golay_filter) and add a new column to the data frame with the smoothed data.  This filter is a type of low-pass filter that can be used for smoothing noisy data. The \"moving average\" filter common in financial data analysis is a special case of the Savitzky-Golay filter with polynomial order 1."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "d7c81ed6-a450-4c15-b4a8-e92213ff8537",
   "metadata": {},
   "outputs": [],
   "source": [
    "from scipy.signal import savgol_filter\n",
    "\n",
    "# Set the window size and polynomial order for the Savitzky-Golay filter\n",
    "window_size = 7\n",
    "poly_order = 5\n",
    "\n",
    "moving_avg_col = f'Moving_Average_{window_size}'\n",
    "SG_col = f'Savitzky_Golay{poly_order}_{window_size}'\n",
    "\n",
    "# Apply Savitzky-Golay smoothing to the Temperature column (rounding to 3 decimal\n",
    "# places)\n",
    "df[moving_avg_col] = savgol_filter(df['Temperature'], window_size, 1).round(decimals=3)\n",
    "df[SG_col] = savgol_filter(df['Temperature'], window_size, poly_order).round(decimals=3)\n",
    "df"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "323a4f58-801c-4da6-88b3-e92ac19a9cdc",
   "metadata": {},
   "outputs": [],
   "source": [
    "plt.xlabel('Year')\n",
    "plt.ylabel('Temperature Anomalies over Land w.r.t. 1951-80 (˚C)')\n",
    "plt.title('Global Annual Mean Surface Air Temperature Change')\n",
    "plt.plot(df['Year'], df['Temperature'], label='Raw Data')\n",
    "plt.plot(df['Year'], df[SG_col], label='Savitzky-Golay Filtered')\n",
    "plt.plot(df['Year'], df[moving_avg_col], label='Moving Average Filtered')\n",
    "plt.legend()\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2ae0c6d8-a9ca-421d-98c5-e736103caa36",
   "metadata": {},
   "source": [
    "## Selecting a Subset of Data to Export\n",
    "\n",
    "`pandas` makes it easy to select a subset of the data. We will select a subset of the data covering the 60 year period from 1920 to 1980, plot it, and then export it to a new CSV file."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "dfc10dc3-ee0d-42a4-afda-d063499b8d2d",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Create a new pandas DataFrame containing only the selected range of years\n",
    "from_year = 1920\n",
    "to_year = 1980\n",
    "subset_df = df[(df['Year'] >= from_year) & (df['Year'] <= to_year)]\n",
    "subset_df"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "7b21668d-a050-406c-9179-2311c547cdf3",
   "metadata": {},
   "outputs": [],
   "source": [
    "plt.xlabel('Year')\n",
    "plt.ylabel('Temperature Anomalies over Land w.r.t. 1951-80 (˚C)')\n",
    "plt.title('Global Annual Mean Surface Air Temperature Change')\n",
    "plt.plot(subset_df['Year'], subset_df['Temperature'], label='Raw Data')\n",
    "plt.plot(subset_df['Year'], subset_df[SG_col], label='Savitzky-Golay Filtered')\n",
    "plt.plot(subset_df['Year'], subset_df[moving_avg_col], label='Moving Average Filtered')\n",
    "plt.legend()\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "695f7de6-db67-4cac-ab24-f1f2151bc68e",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Save the subset DataFrame to a new CSV file\n",
    "OUTPUT_FILE = 'output.csv'\n",
    "\n",
    "subset_df.to_csv(os.path.join(DATA_DIR, OUTPUT_FILE), index=False)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "python3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.11.9"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}