Programming Assignment_Unsupervised Learning.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This is the third assignment for the Coursera course \"Advanced Machine Learning and Signal Processing\"\n",
    "\n",
    "Just execute all cells one after the other and you are done - just note that in the last one you must update your email address (the one you've used for coursera) and obtain a submission token, you get this from the programming assignment directly on coursera.\n",
    "\n",
    "Please fill in the sections labelled with \"###YOUR_CODE_GOES_HERE###\"\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This notebook is designed to run in a IBM Watson Studio default runtime (NOT the Watson Studio Apache Spark Runtime as the default runtime with 1 vCPU is free of charge). Therefore, we install Apache Spark in local mode for test purposes only. Please don't use it in production.\n",
    "\n",
    "In case you are facing issues, please read the following two documents first:\n",
    "\n",
    "https://github.com/IBM/skillsnetwork/wiki/Environment-Setup\n",
    "\n",
    "https://github.com/IBM/skillsnetwork/wiki/FAQ\n",
    "\n",
    "Then, please feel free to ask:\n",
    "\n",
    "https://coursera.org/learn/machine-learning-big-data-apache-spark/discussions/all\n",
    "\n",
    "Please make sure to follow the guidelines before asking a question:\n",
    "\n",
    "https://github.com/IBM/skillsnetwork/wiki/FAQ#im-feeling-lost-and-confused-please-help-me\n",
    "\n",
    "\n",
    "If running outside Watson Studio, this should work as well. In case you are running in an Apache Spark context outside Watson Studio, please remove the Apache Spark setup in the first notebook cells."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/markdown": [
       "# <span style=\"color:red\"><<<<<!!!!! It seems that you are running in a IBM Watson Studio Apache Spark Notebook. Please run it in an IBM Watson Studio Default Runtime (without Apache Spark) !!!!!>>>>></span>"
      ],
      "text/plain": [
       "<IPython.core.display.Markdown object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "from IPython.display import Markdown, display\n",
    "def printmd(string):\n",
    "    display(Markdown('# <span style=\"color:red\">'+string+'</span>'))\n",
    "\n",
    "\n",
    "if ('sc' in locals() or 'sc' in globals()):\n",
    "    printmd('<<<<<!!!!! It seems that you are running in a IBM Watson Studio Apache Spark Notebook. Please run it in an IBM Watson Studio Default Runtime (without Apache Spark) !!!!!>>>>>')\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "#!pip install pyspark==2.4.5"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "try:\n",
    "    from pyspark import SparkContext, SparkConf\n",
    "    from pyspark.sql import SparkSession\n",
    "except ImportError as e:\n",
    "    printmd('<<<<<!!!!! Please restart your kernel after installing Apache Spark !!!!!>>>>>')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "sc = SparkContext.getOrCreate(SparkConf().setMaster(\"local[*]\"))\n",
    "\n",
    "spark = SparkSession \\\n",
    "    .builder \\\n",
    "    .getOrCreate()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "--2023-02-21 16:27:20--  https://github.com/IBM/coursera/raw/master/coursera_ml/a2.parquet\n",
      "Resolving github.com (github.com)... 140.82.114.4\n",
      "Connecting to github.com (github.com)|140.82.114.4|:443... connected.\n",
      "HTTP request sent, awaiting response... 301 Moved Permanently\n",
      "Location: https://github.com/claimed-framework/component-library/raw/master/coursera_ml/a2.parquet [following]\n",
      "--2023-02-21 16:27:20--  https://github.com/claimed-framework/component-library/raw/master/coursera_ml/a2.parquet\n",
      "Reusing existing connection to github.com:443.\n",
      "HTTP request sent, awaiting response... 302 Found\n",
      "Location: https://raw.githubusercontent.com/claimed-framework/component-library/master/coursera_ml/a2.parquet [following]\n",
      "--2023-02-21 16:27:20--  https://raw.githubusercontent.com/claimed-framework/component-library/master/coursera_ml/a2.parquet\n",
      "Resolving raw.githubusercontent.com (raw.githubusercontent.com)... 185.199.110.133, 185.199.108.133, 185.199.111.133, ...\n",
      "Connecting to raw.githubusercontent.com (raw.githubusercontent.com)|185.199.110.133|:443... connected.\n",
      "HTTP request sent, awaiting response... 200 OK\n",
      "Length: 59032 (58K) [application/octet-stream]\n",
      "Saving to: ‘a2.parquet.1’\n",
      "\n",
      "a2.parquet.1        100%[===================>]  57.65K  --.-KB/s    in 0.001s  \n",
      "\n",
      "2023-02-21 16:27:20 (55.1 MB/s) - ‘a2.parquet.1’ saved [59032/59032]\n",
      "\n"
     ]
    }
   ],
   "source": [
    "!wget https://github.com/IBM/coursera/raw/master/coursera_ml/a2.parquet"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now it’s time to have a look at the recorded sensor data. You should see data similar to the one exemplified below….\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "+-----+-----------+-------------------+-------------------+-------------------+\n",
      "|CLASS|   SENSORID|                  X|                  Y|                  Z|\n",
      "+-----+-----------+-------------------+-------------------+-------------------+\n",
      "|    0|         26| 380.66434005495194| -139.3470983812975|-247.93697521077704|\n",
      "|    0|         29| 104.74324299209692| -32.27421440203938|-25.105013725863852|\n",
      "|    0| 8589934658| 118.11469236129976| 45.916682927433534| -87.97203782706572|\n",
      "|    0|34359738398| 246.55394030642543|-0.6122810693132044|-398.18662513951506|\n",
      "|    0|17179869241|-190.32584900181487|  234.7849657520335|-206.34483804019288|\n",
      "|    0|25769803830| 178.62396382387422| -47.07529438881511|  84.38310769821979|\n",
      "|    0|25769803831|  85.03128805189493|-4.3024316644854546|-1.1841857567516714|\n",
      "|    0|34359738411| 26.786262674736566| -46.33193951911338| 20.880756008396055|\n",
      "|    0| 8589934592|-16.203752396859194| 51.080957032176954| -96.80526656416971|\n",
      "|    0|25769803852|   47.2048142440404|  -78.2950899652916| 181.99604091494786|\n",
      "|    0|34359738369| 15.608872398939273| -79.90322809181754|  69.62150711098005|\n",
      "|    0|         19|-4.8281721129789315| -67.38050508399905| 221.24876396496404|\n",
      "|    0|         54| -98.40725712852762|-19.989364074314732|  -302.695196085276|\n",
      "|    0|17179869313| 22.835845394816594|   17.1633660118843| 32.877914832011385|\n",
      "|    0|34359738454|  84.20178070080324| -32.81572075916947| -48.63517643958031|\n",
      "|    0|          0|  56.54732521345129| -7.980106018032676|  95.05162719436447|\n",
      "|    0|17179869201|  -57.6008655247749|  5.135393798773895| 236.99158698947267|\n",
      "|    0|17179869308| -65.59264738389012| -48.92660057215126| -61.58970715383383|\n",
      "|    0|25769803790|  34.82337351291005|  9.483542084393937|  197.6066372962772|\n",
      "|    0|25769803825|  39.80573823439121|-0.7955236412785212| -79.66652640650325|\n",
      "+-----+-----------+-------------------+-------------------+-------------------+\n",
      "only showing top 20 rows\n",
      "\n"
     ]
    }
   ],
   "source": [
    "df=spark.read.load('a2.parquet')\n",
    "\n",
    "df.createOrReplaceTempView(\"df\")\n",
    "spark.sql(\"SELECT * from df\").show()\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let’s check if we have balanced classes – this means that we have roughly the same number of examples for each class we want to predict. This is important for classification but also helpful for clustering"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "+------------+-----+\n",
      "|count(class)|class|\n",
      "+------------+-----+\n",
      "|        1416|    1|\n",
      "|        1626|    0|\n",
      "+------------+-----+\n",
      "\n"
     ]
    }
   ],
   "source": [
    "spark.sql(\"SELECT count(class), class from df group by class\").show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's create a VectorAssembler which consumes columns X, Y and Z and produces a column “features”\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "from pyspark.ml.feature import VectorAssembler\n",
    "vectorAssembler = VectorAssembler(inputCols=[\"X\",\"Y\",\"Z\"],\n",
    "                                  outputCol=\"features\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Please insatiate a clustering algorithm from the SparkML package and assign it to the clust variable. Here we don’t need to take care of the “CLASS” column since we are in unsupervised learning mode – so let’s pretend to not even have the “CLASS” column for now – but it will become very handy later in assessing the clustering performance. PLEASE NOTE – IN REAL-WORLD SCENARIOS THERE IS NO CLASS COLUMN – THEREFORE YOU CAN’T ASSESS CLASSIFICATION PERFORMANCE USING THIS COLUMN \n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "from pyspark.ml.clustering import KMeans\n",
    "\n",
    "clust = KMeans().setK(3).setSeed(1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let’s train...\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "from pyspark.ml import Pipeline\n",
    "pipeline = Pipeline(stages=[vectorAssembler, clust])\n",
    "model = pipeline.fit(df)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "...and evaluate..."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "+-----+-----------+-------------------+-------------------+-------------------+--------------------+----------+\n",
      "|CLASS|   SENSORID|                  X|                  Y|                  Z|            features|prediction|\n",
      "+-----+-----------+-------------------+-------------------+-------------------+--------------------+----------+\n",
      "|    0|         26| 380.66434005495194| -139.3470983812975|-247.93697521077704|[380.664340054951...|         1|\n",
      "|    0|         29| 104.74324299209692| -32.27421440203938|-25.105013725863852|[104.743242992096...|         0|\n",
      "|    0| 8589934658| 118.11469236129976| 45.916682927433534| -87.97203782706572|[118.114692361299...|         0|\n",
      "|    0|34359738398| 246.55394030642543|-0.6122810693132044|-398.18662513951506|[246.553940306425...|         1|\n",
      "|    0|17179869241|-190.32584900181487|  234.7849657520335|-206.34483804019288|[-190.32584900181...|         1|\n",
      "|    0|25769803830| 178.62396382387422| -47.07529438881511|  84.38310769821979|[178.623963823874...|         0|\n",
      "|    0|25769803831|  85.03128805189493|-4.3024316644854546|-1.1841857567516714|[85.0312880518949...|         0|\n",
      "|    0|34359738411| 26.786262674736566| -46.33193951911338| 20.880756008396055|[26.7862626747365...|         0|\n",
      "|    0| 8589934592|-16.203752396859194| 51.080957032176954| -96.80526656416971|[-16.203752396859...|         0|\n",
      "|    0|25769803852|   47.2048142440404|  -78.2950899652916| 181.99604091494786|[47.2048142440404...|         2|\n",
      "|    0|34359738369| 15.608872398939273| -79.90322809181754|  69.62150711098005|[15.6088723989392...|         0|\n",
      "|    0|         19|-4.8281721129789315| -67.38050508399905| 221.24876396496404|[-4.8281721129789...|         2|\n",
      "|    0|         54| -98.40725712852762|-19.989364074314732|  -302.695196085276|[-98.407257128527...|         1|\n",
      "|    0|17179869313| 22.835845394816594|   17.1633660118843| 32.877914832011385|[22.8358453948165...|         0|\n",
      "|    0|34359738454|  84.20178070080324| -32.81572075916947| -48.63517643958031|[84.2017807008032...|         0|\n",
      "|    0|          0|  56.54732521345129| -7.980106018032676|  95.05162719436447|[56.5473252134512...|         0|\n",
      "|    0|17179869201|  -57.6008655247749|  5.135393798773895| 236.99158698947267|[-57.600865524774...|         2|\n",
      "|    0|17179869308| -65.59264738389012| -48.92660057215126| -61.58970715383383|[-65.592647383890...|         0|\n",
      "|    0|25769803790|  34.82337351291005|  9.483542084393937|  197.6066372962772|[34.8233735129100...|         2|\n",
      "|    0|25769803825|  39.80573823439121|-0.7955236412785212| -79.66652640650325|[39.8057382343912...|         0|\n",
      "+-----+-----------+-------------------+-------------------+-------------------+--------------------+----------+\n",
      "only showing top 20 rows\n",
      "\n"
     ]
    }
   ],
   "source": [
    "prediction = model.transform(df)\n",
    "prediction.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.7685733070348455"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "prediction.createOrReplaceTempView('prediction')\n",
    "spark.sql('''\n",
    "select max(correct)/max(total) as accuracy from (\n",
    "\n",
    "    select sum(correct) as correct, count(correct) as total from (\n",
    "        select case when class != prediction then 1 else 0 end as correct from prediction \n",
    "    ) \n",
    "    \n",
    "    union\n",
    "    \n",
    "    select sum(correct) as correct, count(correct) as total from (\n",
    "        select case when class = prediction then 1 else 0 end as correct from prediction \n",
    "    ) \n",
    ")\n",
    "''').rdd.map(lambda row: row.accuracy).collect()[0]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If you reached at least 55% of accuracy you are fine to submit your predictions to the grader. Otherwise please experiment with parameters setting to your clustering algorithm, use a different algorithm or just re-record your data and try to obtain. In case you are stuck, please use the Coursera Discussion Forum. Please note again – in a real-world scenario there is no way in doing this – since there is no class label in your data. Please have a look at this further reading on clustering performance evaluation https://en.wikipedia.org/wiki/Cluster_analysis#Evaluation_and_assessment\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "--2023-02-21 16:28:25--  https://raw.githubusercontent.com/IBM/coursera/master/rklib.py\n",
      "Resolving raw.githubusercontent.com (raw.githubusercontent.com)... 185.199.110.133, 185.199.108.133, 185.199.109.133, ...\n",
      "Connecting to raw.githubusercontent.com (raw.githubusercontent.com)|185.199.110.133|:443... connected.\n",
      "HTTP request sent, awaiting response... 200 OK\n",
      "Length: 2540 (2.5K) [text/plain]\n",
      "Saving to: ‘rklib.py’\n",
      "\n",
      "rklib.py            100%[===================>]   2.48K  --.-KB/s    in 0s      \n",
      "\n",
      "2023-02-21 16:28:25 (41.5 MB/s) - ‘rklib.py’ saved [2540/2540]\n",
      "\n"
     ]
    }
   ],
   "source": [
    "!rm -f rklib.py\n",
    "!wget https://raw.githubusercontent.com/IBM/coursera/master/rklib.py"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [],
   "source": [
    "!rm -Rf a2_m3.json"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [],
   "source": [
    "prediction= prediction.repartition(1)\n",
    "prediction.write.json('a2_m3.json')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [],
   "source": [
    "import zipfile\n",
    "import os\n",
    "\n",
    "def zipdir(path, ziph):\n",
    "    for root, dirs, files in os.walk(path):\n",
    "        for file in files:\n",
    "            ziph.write(os.path.join(root, file))\n",
    "\n",
    "zipf = zipfile.ZipFile('a2_m3.json.zip', 'w', zipfile.ZIP_DEFLATED)\n",
    "zipdir('a2_m3.json', zipf)\n",
    "zipf.close()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [],
   "source": [
    "!base64 a2_m3.json.zip > a2_m3.json.zip.base64"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Submission successful, please check on the coursera grader page for the status\n",
      "-------------------------\n",
      "{\"elements\":[{\"itemId\":\"Cu6KW\",\"id\":\"f_F-qCtuEei_fRLwaVDk3g~Cu6KW~xrphFLIEEe2VphICt6NGUQ\",\"courseId\":\"f_F-qCtuEei_fRLwaVDk3g\"}],\"paging\":{},\"linked\":{}}\n",
      "-------------------------\n"
     ]
    }
   ],
   "source": [
    "from rklib import submit\n",
    "key = \"pPfm62VXEeiJOBL0dhxPkA\"\n",
    "part = \"EOTMs\"\n",
    "email = \"mdanasmondol43@gmail.com\"\n",
    "token = \"eAFs3baPmEuOQ0pf\" # (have a look here if you need more information on how to obtain the token https://youtu.be/GcDo0Rwe06U?t=276)\n",
    "\n",
    "\n",
    "with open('a2_m3.json.zip.base64', 'r') as myfile:\n",
    "    data=myfile.read()\n",
    "submit(email, token, key, part, [part], data)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3.9 with Spark",
   "language": "python3",
   "name": "python39"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.13"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}