code/03-data-analysis-with-pandas/14-dataframe-comparisons-and-iterations.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "[back](./13-dataframe-operations.ipynb)\n",
    "\n",
    "---\n",
    "## `DataFrame Comparisons and Iterations`\n",
    "\n",
    "- [Comparing entire DataFrames](#comparing-entire-dataframes)\n",
    "- [Comparing DataFrame rows and columns](#comparing-dataframe-rows-and-columns)\n",
    "- [Iterating through DataFrames](#iterating-through-dataframes)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### `Initial Setup`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Importing Pandas\n",
    "\n",
    "import pandas as pd\n",
    "import numpy as np\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Original DataFrame 1:\n",
      "      col1  col2\n",
      "row1   1.0   4.0\n",
      "row2   1.0   NaN\n",
      "row3   3.0   9.0\n",
      "row4   NaN   6.0\n",
      "--------------------------------------------------------------------------------\n",
      "Original DataFrame 2:\n",
      "      col1  col2\n",
      "row1   2.0   4.0\n",
      "row2   5.0   NaN\n",
      "row3   1.0   8.0\n",
      "row4   NaN   7.0\n",
      "--------------------------------------------------------------------------------\n"
     ]
    }
   ],
   "source": [
    "# Data set-up\n",
    "\n",
    "df1 = pd.DataFrame({\n",
    "    'col1': {'row1': 1, 'row2': 1, 'row3': 3},\n",
    "    'col2': {'row1': 4, 'row3': 9, 'row4': 6}\n",
    "})\n",
    "\n",
    "df2 = pd.DataFrame({\n",
    "    'col1': {'row1': 2, 'row2': 5, 'row3': 1},\n",
    "    'col2': {'row1': 4, 'row3': 8, 'row4': 7}\n",
    "})\n",
    "\n",
    "def reset_df1():\n",
    "  global df1\n",
    "  df1 = pd.DataFrame({\n",
    "      'col1': {'row1': 1, 'row2': 1, 'row3': 3},\n",
    "      'col2': {'row1': 4, 'row3': 9, 'row4': 6}\n",
    "  })\n",
    "  print_df1()\n",
    "\n",
    "def reset_df2():\n",
    "  global df2\n",
    "  df2 = pd.DataFrame({\n",
    "      'col1': {'row1': 2, 'row2': 5, 'row3': 1},\n",
    "      'col2': {'row1': 4, 'row3': 8, 'row4': 7}\n",
    "  })\n",
    "  print_df2()\n",
    "\n",
    "\n",
    "def print_df1():\n",
    "  print('Original DataFrame 1:')\n",
    "  print(df1)\n",
    "  divider()\n",
    "\n",
    "def print_df2():\n",
    "  print('Original DataFrame 2:')\n",
    "  print(df2)\n",
    "  divider()\n",
    "\n",
    "def divider():\n",
    "  print('-'*80)\n",
    "\n",
    "\n",
    "print_df1()\n",
    "print_df2()\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### `Comparing entire DataFrames`"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The simplest for of comparisons are using the basic comparison operators like `<`, `>`, `==`, `!=`, `<=`, `>=`\n",
    "\n",
    "Much like with Series, these will return a new DataFrame with `True` or `False` values"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Original DataFrame 1:\n",
      "      col1  col2\n",
      "row1   1.0   4.0\n",
      "row2   1.0   NaN\n",
      "row3   3.0   9.0\n",
      "row4   NaN   6.0\n",
      "--------------------------------------------------------------------------------\n",
      "Original DataFrame 2:\n",
      "      col1  col2\n",
      "row1   2.0   4.0\n",
      "row2   5.0   NaN\n",
      "row3   1.0   8.0\n",
      "row4   NaN   7.0\n",
      "--------------------------------------------------------------------------------\n",
      "Comparing if df1 > df2:\n",
      "       col1   col2\n",
      "row1  False  False\n",
      "row2  False  False\n",
      "row3   True   True\n",
      "row4  False  False\n"
     ]
    }
   ],
   "source": [
    "print_df1()\n",
    "print_df2()\n",
    "\n",
    "print('Comparing if df1 > df2:')\n",
    "print(df1 > df2)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "---\n",
    "We can also use functions instead of the basic operators like `lt`, `gt`, `eq`, `ne`, `le`, `ge`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Original DataFrame 1:\n",
      "      col1  col2\n",
      "row1   1.0   4.0\n",
      "row2   1.0   NaN\n",
      "row3   3.0   9.0\n",
      "row4   NaN   6.0\n",
      "--------------------------------------------------------------------------------\n",
      "Original DataFrame 2:\n",
      "      col1  col2\n",
      "row1   2.0   4.0\n",
      "row2   5.0   NaN\n",
      "row3   1.0   8.0\n",
      "row4   NaN   7.0\n",
      "--------------------------------------------------------------------------------\n",
      "Comparing if df1.lt(df2):\n",
      "       col1   col2\n",
      "row1   True  False\n",
      "row2   True  False\n",
      "row3  False  False\n",
      "row4  False   True\n"
     ]
    }
   ],
   "source": [
    "print_df1()\n",
    "print_df2()\n",
    "\n",
    "print('Comparing if df1.lt(df2):')\n",
    "print(df1.lt(df2))\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "---\n",
    "**NOTE:** This is different from the `.equals()`, where this functions just return a single value, `True` if all the elements are equal, `False` otherwise"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Original DataFrame 1:\n",
      "      col1  col2\n",
      "row1   1.0   4.0\n",
      "row2   1.0   NaN\n",
      "row3   3.0   9.0\n",
      "row4   NaN   6.0\n",
      "--------------------------------------------------------------------------------\n",
      "Original DataFrame 2:\n",
      "      col1  col2\n",
      "row1   2.0   4.0\n",
      "row2   5.0   NaN\n",
      "row3   1.0   8.0\n",
      "row4   NaN   7.0\n",
      "--------------------------------------------------------------------------------\n",
      "df1.equals(df2):\n",
      "False\n"
     ]
    }
   ],
   "source": [
    "print_df1()\n",
    "print_df2()\n",
    "\n",
    "print('df1.equals(df2):')\n",
    "print(df1.equals(df2))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### `Comparing DataFrame rows and columns`"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Comparing rows and columns are similar to Series"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Original DataFrame 1:\n",
      "      col1  col2\n",
      "row1   1.0   4.0\n",
      "row2   1.0   NaN\n",
      "row3   3.0   9.0\n",
      "row4   NaN   6.0\n",
      "--------------------------------------------------------------------------------\n",
      "Original DataFrame 2:\n",
      "      col1  col2\n",
      "row1   2.0   4.0\n",
      "row2   5.0   NaN\n",
      "row3   1.0   8.0\n",
      "row4   NaN   7.0\n",
      "--------------------------------------------------------------------------------\n",
      "If df1['col1'] > df2['col1']:\n",
      "row1    False\n",
      "row2    False\n",
      "row3     True\n",
      "row4    False\n",
      "Name: col1, dtype: bool\n"
     ]
    }
   ],
   "source": [
    "print_df1()\n",
    "print_df2()\n",
    "\n",
    "# Using basic operator\n",
    "\n",
    "print('If df1[\\'col1\\'] > df2[\\'col1\\']:')\n",
    "print(df1['col1'] > df2['col1'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Original DataFrame 1:\n",
      "      col1  col2\n",
      "row1   1.0   4.0\n",
      "row2   1.0   NaN\n",
      "row3   3.0   9.0\n",
      "row4   NaN   6.0\n",
      "--------------------------------------------------------------------------------\n",
      "Original DataFrame 2:\n",
      "      col1  col2\n",
      "row1   2.0   4.0\n",
      "row2   5.0   NaN\n",
      "row3   1.0   8.0\n",
      "row4   NaN   7.0\n",
      "--------------------------------------------------------------------------------\n",
      "If df1['col1'].gt(df2['col1']):\n",
      "row1    False\n",
      "row2    False\n",
      "row3     True\n",
      "row4    False\n",
      "Name: col1, dtype: bool\n"
     ]
    }
   ],
   "source": [
    "print_df1()\n",
    "print_df2()\n",
    "\n",
    "# Using inbuilt function\n",
    "\n",
    "print('If df1[\\'col1\\'].gt(df2[\\'col1\\']):')\n",
    "print(df1['col1'].gt(df2['col1']))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Original DataFrame 1:\n",
      "      col1  col2\n",
      "row1   1.0   4.0\n",
      "row2   1.0   NaN\n",
      "row3   3.0   9.0\n",
      "row4   NaN   6.0\n",
      "--------------------------------------------------------------------------------\n",
      "Original DataFrame 2:\n",
      "      col1  col2\n",
      "row1   2.0   4.0\n",
      "row2   5.0   NaN\n",
      "row3   1.0   8.0\n",
      "row4   NaN   7.0\n",
      "--------------------------------------------------------------------------------\n",
      "If df1.loc['row1'] < df2.loc['row1']:\n",
      "col1     True\n",
      "col2    False\n",
      "Name: row1, dtype: bool\n"
     ]
    }
   ],
   "source": [
    "print_df1()\n",
    "print_df2()\n",
    "\n",
    "# Comparing across rows, using default operators\n",
    "\n",
    "print('If df1.loc[\\'row1\\'] < df2.loc[\\'row1\\']:')\n",
    "print(df1.loc['row1'] < df2.loc['row1'])\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Original DataFrame 1:\n",
      "      col1  col2\n",
      "row1   1.0   4.0\n",
      "row2   1.0   NaN\n",
      "row3   3.0   9.0\n",
      "row4   NaN   6.0\n",
      "--------------------------------------------------------------------------------\n",
      "Original DataFrame 2:\n",
      "      col1  col2\n",
      "row1   2.0   4.0\n",
      "row2   5.0   NaN\n",
      "row3   1.0   8.0\n",
      "row4   NaN   7.0\n",
      "--------------------------------------------------------------------------------\n",
      "If df1.loc['row1'].gt(df2.loc['row1']):\n",
      "col1    False\n",
      "col2    False\n",
      "Name: row1, dtype: bool\n"
     ]
    }
   ],
   "source": [
    "print_df1()\n",
    "print_df2()\n",
    "\n",
    "# Comparing across rows, using inbuilt function\n",
    "\n",
    "print('If df1.loc[\\'row1\\'].gt(df2.loc[\\'row1\\']):')\n",
    "print(df1.loc['row1'].gt(df2.loc['row1']))\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### `Iterating through DataFrames`"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "There are a lot of ways in which we can iterate through a DataFrame\n",
    "\n",
    "- Columns\n",
    "- Indexes\n",
    "- Items (will help us get the column labels and index values)\n",
    "- Rows (will help us get the row labels and column values)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Original DataFrame 1:\n",
      "      col1  col2\n",
      "row1   1.0   4.0\n",
      "row2   1.0   NaN\n",
      "row3   3.0   9.0\n",
      "row4   NaN   6.0\n",
      "--------------------------------------------------------------------------------\n",
      "col1\n",
      "col2\n",
      "--------------------------------------------------------------------------------\n"
     ]
    }
   ],
   "source": [
    "print_df1()\n",
    "\n",
    "# Columns (labels)\n",
    "for col in df1:\n",
    "  print(col)\n",
    "divider()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Original DataFrame 1:\n",
      "      col1  col2\n",
      "row1   1.0   4.0\n",
      "row2   1.0   NaN\n",
      "row3   3.0   9.0\n",
      "row4   NaN   6.0\n",
      "--------------------------------------------------------------------------------\n",
      "row1\n",
      "row2\n",
      "row3\n",
      "row4\n",
      "--------------------------------------------------------------------------------\n"
     ]
    }
   ],
   "source": [
    "print_df1()\n",
    "\n",
    "# Row (labels)\n",
    "for row in df1.index:\n",
    "  print(row)\n",
    "divider()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Original DataFrame 1:\n",
      "      col1  col2\n",
      "row1   1.0   4.0\n",
      "row2   1.0   NaN\n",
      "row3   3.0   9.0\n",
      "row4   NaN   6.0\n",
      "--------------------------------------------------------------------------------\n",
      "Iterating based on column:\n",
      "--------------------------------------------------------------------------------\n",
      "col1\n",
      "row1    1.0\n",
      "row2    1.0\n",
      "row3    3.0\n",
      "row4    NaN\n",
      "Name: col1, dtype: float64\n",
      "<class 'pandas.core.series.Series'>\n",
      "--------------------------------------------------------------------------------\n",
      "col2\n",
      "row1    4.0\n",
      "row2    NaN\n",
      "row3    9.0\n",
      "row4    6.0\n",
      "Name: col2, dtype: float64\n",
      "<class 'pandas.core.series.Series'>\n",
      "--------------------------------------------------------------------------------\n"
     ]
    }
   ],
   "source": [
    "print_df1()\n",
    "\n",
    "# Iterate and print individual elements in a DataFrame\n",
    "print('Iterating based on column:')\n",
    "divider()\n",
    "for col_name, row_val in df1.items():\n",
    "  print(col_name)\n",
    "  print(row_val)\n",
    "  print(type(row_val))\n",
    "  divider()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Original DataFrame 1:\n",
      "      col1  col2\n",
      "row1   1.0   4.0\n",
      "row2   1.0   NaN\n",
      "row3   3.0   9.0\n",
      "row4   NaN   6.0\n",
      "--------------------------------------------------------------------------------\n",
      "Iterating based on rows:\n",
      "--------------------------------------------------------------------------------\n",
      "row1\n",
      "col1    1.0\n",
      "col2    4.0\n",
      "Name: row1, dtype: float64\n",
      "--------------------------------------------------------------------------------\n",
      "row2\n",
      "col1    1.0\n",
      "col2    NaN\n",
      "Name: row2, dtype: float64\n",
      "--------------------------------------------------------------------------------\n",
      "row3\n",
      "col1    3.0\n",
      "col2    9.0\n",
      "Name: row3, dtype: float64\n",
      "--------------------------------------------------------------------------------\n",
      "row4\n",
      "col1    NaN\n",
      "col2    6.0\n",
      "Name: row4, dtype: float64\n",
      "--------------------------------------------------------------------------------\n"
     ]
    }
   ],
   "source": [
    "print_df1()\n",
    "\n",
    "# Iterate and print individual elements in a DataFrame\n",
    "print('Iterating based on rows:')\n",
    "divider()\n",
    "for row_name, col_val in df1.iterrows():\n",
    "  print(row_name)\n",
    "  print(col_val)\n",
    "  divider()\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "---\n",
    "[next](./15-reading-csv-into-dataframe.ipynb)"
   ]
  }
 ],
 "metadata": {
  "interpreter": {
   "hash": "0b4484ae063bf9ef0d5115063857dd0e382a2c73ed49a11a000ca56af03302b6"
  },
  "kernelspec": {
   "display_name": "Python 3.9.7 64-bit ('base': conda)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.7"
  },
  "orig_nbformat": 4
 },
 "nbformat": 4,
 "nbformat_minor": 2
}