notebooks/02.2_create_sample_data.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Notebook 02.2: Create Sample Data\n",
    "\n",
    "### Introduction\n",
    "\n",
    "In this notebook, we will create a stratified sample of the clean password dataset obtained in the previous notebook. The stratified sample will ensure that we have representative samples from different password strength levels, allowing us to perform accurate analysis and modeling.\n",
    "\n",
    "### Setup\n",
    "\n",
    "Let's start by importing the necessary libraries.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "###  Data Loading\n",
    "Let's load the clean password dataset from the previous notebook."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>password</th>\n",
       "      <th>strength</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>123456</td>\n",
       "      <td>0.172331</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>12345</td>\n",
       "      <td>0.128996</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>123456789</td>\n",
       "      <td>0.316992</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>password</td>\n",
       "      <td>0.249543</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>iloveyou</td>\n",
       "      <td>0.249543</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "    password  strength\n",
       "0     123456  0.172331\n",
       "1      12345  0.128996\n",
       "2  123456789  0.316992\n",
       "3   password  0.249543\n",
       "4   iloveyou  0.249543"
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df = pd.read_parquet(\"./data/clean_passwords.gzip\", engine=\"pyarrow\")\n",
    "df.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "###  Stratified Sampling\n",
    "To create a stratified sample, we will divide the dataset into several strength level bins and sample a fixed number of passwords from each bin."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>password</th>\n",
       "      <th>strength</th>\n",
       "      <th>bin</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>8901913</th>\n",
       "      <td>csillik</td>\n",
       "      <td>0.180594</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7505062</th>\n",
       "      <td>huniihuu</td>\n",
       "      <td>0.177778</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1931018</th>\n",
       "      <td>chaipy</td>\n",
       "      <td>0.172331</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>11728518</th>\n",
       "      <td>876876b</td>\n",
       "      <td>0.155556</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5548757</th>\n",
       "      <td>miiwhy</td>\n",
       "      <td>0.154795</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "          password  strength  bin\n",
       "8901913    csillik  0.180594    0\n",
       "7505062   huniihuu  0.177778    0\n",
       "1931018     chaipy  0.172331    0\n",
       "11728518   876876b  0.155556    0\n",
       "5548757     miiwhy  0.154795    0"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "num_bins = 5\n",
    "sample_size = 2_000\n",
    "\n",
    "df[\"bin\"] = pd.cut(df[\"strength\"], num_bins, labels=False)\n",
    "sample_df = df.groupby(\"bin\", group_keys=False).apply(\n",
    "    lambda x: x.sample(min(len(x), sample_size), random_state=24)\n",
    ")\n",
    "\n",
    "sample_df.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "###  Dropping 'bin' Column\n",
    "Since we have created the stratified sample, we can drop the 'bin' column from the sample DataFrame."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>password</th>\n",
       "      <th>strength</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>8901913</th>\n",
       "      <td>csillik</td>\n",
       "      <td>0.180594</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7505062</th>\n",
       "      <td>huniihuu</td>\n",
       "      <td>0.177778</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1931018</th>\n",
       "      <td>chaipy</td>\n",
       "      <td>0.172331</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>11728518</th>\n",
       "      <td>876876b</td>\n",
       "      <td>0.155556</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5548757</th>\n",
       "      <td>miiwhy</td>\n",
       "      <td>0.154795</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "          password  strength\n",
       "8901913    csillik  0.180594\n",
       "7505062   huniihuu  0.177778\n",
       "1931018     chaipy  0.172331\n",
       "11728518   876876b  0.155556\n",
       "5548757     miiwhy  0.154795"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "sample_df.drop([\"bin\"], axis=1, inplace=True)\n",
    "sample_df.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "###  Sample Data Information\n",
    "Let's examine the information of the stratified sample."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<class 'pandas.core.frame.DataFrame'>\n",
      "Int64Index: 10000 entries, 8901913 to 5783544\n",
      "Data columns (total 2 columns):\n",
      " #   Column    Non-Null Count  Dtype  \n",
      "---  ------    --------------  -----  \n",
      " 0   password  10000 non-null  object \n",
      " 1   strength  10000 non-null  float64\n",
      "dtypes: float64(1), object(1)\n",
      "memory usage: 234.4+ KB\n"
     ]
    }
   ],
   "source": [
    "sample_df.info()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "###  Saving Stratified Sample Data\n",
    "Finally, we will save the stratified sample data as a CSV file for further analysis and modeling."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "sample_df.to_csv(\"./data/stratified_sample_data.csv\", index=False)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "🎉Congratulations! We have successfully created a stratified sample of the clean password dataset. This stratified sample will allow us to perform in-depth analysis and build accurate models for password strength prediction.\n",
    "\n",
    "Next, let's move on to the next notebook to apply feature engineering to the sample data."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "env",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.6"
  },
  "orig_nbformat": 4
 },
 "nbformat": 4,
 "nbformat_minor": 2
}