City/Courseworks/Spam Detection with PySpark.ipynb

{
    "nbformat_minor": 1, 
    "cells": [
        {
            "source": "# Coursework Part 1: Detecting Spam with Spark\n\nThis is the final submission for IN432 Big Data coursework 2018, part 1. ", 
            "cell_type": "markdown", 
            "metadata": {}
        }, 
        {
            "source": "## Load and prepare the data\n\nWe will use the lingspam dataset in this coursework (see [http://csmining.org/index.php/ling-spam-datasets.html](http://csmining.org/index.php/ling-spam-datasets.html) for more information).", 
            "cell_type": "markdown", 
            "metadata": {}
        }, 
        {
            "execution_count": 4, 
            "cell_type": "code", 
            "metadata": {}, 
            "outputs": [
                {
                    "output_type": "stream", 
                    "name": "stdout", 
                    "text": "/gpfs/global_fs01/sym_shared/YPProdSpark/user/s8bb-4c006897e4ec94-b51c55267e90/notebook/work\nfatal: destination path 'City-Data-Science' already exists and is not an empty directory.\n/gpfs/global_fs01/sym_shared/YPProdSpark/user/s8bb-4c006897e4ec94-b51c55267e90/notebook/work/City-Data-Science\nAlready up-to-date.\n/gpfs/global_fs01/sym_shared/YPProdSpark/user/s8bb-4c006897e4ec94-b51c55267e90/notebook/work/City-Data-Science/datasets\n"
                }
            ], 
            "source": "# Version control\n%cd ~/notebook/work/\n!git clone https://github.com/tweyde/City-Data-Science.git\n%cd ~/notebook/work/City-Data-Science/\n!git pull\n%cd ./datasets/ "
        }, 
        {
            "execution_count": 5, 
            "cell_type": "code", 
            "metadata": {}, 
            "outputs": [
                {
                    "output_type": "stream", 
                    "name": "stdout", 
                    "text": ">>> Extracting the ling_spam dataset, this can take a moment.\n>>> Unzipping finished.\n/gpfs/global_fs01/sym_shared/YPProdSpark/user/s8bb-4c006897e4ec94-b51c55267e90/notebook/work/City-Data-Science/datasets/lingspam_public/bare\n>>> pwd \n/gpfs/global_fs01/sym_shared/YPProdSpark/user/s8bb-4c006897e4ec94-b51c55267e90/notebook/work/City-Data-Science/datasets/lingspam_public/bare\n>>> ls \npart1  part10  part2  part3  part4  part5  part6  part7  part8\tpart9\n/gpfs/global_fs01/sym_shared/YPProdSpark/user/s8bb-4c006897e4ec94-b51c55267e90/notebook/work/City-Data-Science/datasets/lingspam_public\n"
                }
            ], 
            "source": "# Extracting the dataset\nprint(\">>> Extracting the ling_spam dataset, this can take a moment.\")\n!tar -xf lingspam_public02.tar.gz\nprint(\">>> Unzipping finished.\")\n\n# We now have a new dataset in directory 'bare'.\n%cd lingspam_public/bare \nprint(\">>> pwd \")\n!pwd\nprint(\">>> ls \")\n!ls\n# the line before last of output should show \"part1 part10 part2  part3  part4  part5  part6  part7 part8 part9\"\n%cd .."
        }, 
        {
            "source": "### Troubleshooting", 
            "cell_type": "markdown", 
            "metadata": {}
        }, 
        {
            "execution_count": 6, 
            "cell_type": "code", 
            "metadata": {}, 
            "outputs": [], 
            "source": "# Set print options\nimport numpy as np\nnp.set_printoptions(precision=3)"
        }, 
        {
            "execution_count": 7, 
            "cell_type": "code", 
            "metadata": {}, 
            "outputs": [], 
            "source": "from pyspark import SparkContext\nsc = spark.sparkContext"
        }, 
        {
            "execution_count": 8, 
            "cell_type": "code", 
            "metadata": {}, 
            "outputs": [], 
            "source": "# Fix \"error while instantiating when multiple notebooks are opened\"\n!rm -Rf /gpfs/global_fs01/sym_shared/YPProdSpark/user/s8bb-4c006897e4ec94-b51c55267e90/notebook/jupyter-rt/kernel-8d2bd250-95e0-4bc0-be9f-99d2d60b63f0-20180311_223518/metastore_db"
        }, 
        {
            "source": "## Task 1) Read the dataset and create RDDs \na) Start by reading the directory with text files from the file system (`~/notebook/work/City-Data-Science/datasets/bare`). Load all text files per directory (part1,part2, ... ,part10).\n\nb) Split data into training set and test set.\n\nb) Remove the path and extension from the filename.\n\n**Note**: If the filename starts with 'spmsg' it is spam, otherwise it is not", 
            "cell_type": "markdown", 
            "metadata": {}
        }, 
        {
            "execution_count": 9, 
            "cell_type": "code", 
            "metadata": {}, 
            "outputs": [
                {
                    "output_type": "stream", 
                    "name": "stdout", 
                    "text": "/gpfs/global_fs01/sym_shared/YPProdSpark/user/s8bb-4c006897e4ec94-b51c55267e90/notebook/work/City-Data-Science/datasets/lingspam_public\nbare  lemm  lemm_stop  readme.txt  stop\n[PosixPath('bare/part5'), PosixPath('bare/part8'), PosixPath('bare/part2'), PosixPath('bare/part9'), PosixPath('bare/part3'), PosixPath('bare/part6'), PosixPath('bare/part1'), PosixPath('bare/part4'), PosixPath('bare/part7'), PosixPath('bare/part10')]\nlen(rddList) 10\ncreated the RDDs\ntestRDD.count():  291\ntestRDD.getNumPartitions() 2\ntestRDD.getStorageLevel() Serialized 1x Replicated\ntestRDD.take(1):  [('9-66msg1', \"Subject: xth conference of nordic and general ling .\\n\\nthe tenth conference of nordic and general linguistics will be held in reykjavik , iceland , from saturday june 6 , to monday june 8 , 1998 . it is organized by the institute of linguistics , university of iceland . the deadline for pre-registration at a reduced price is january 31 , 1998 . pre - registration forms and further information can be found on our web site ( http : / / www . rhi . hi . is / ~ nordconf ) and can also be mailed or e-mailed upon request . papers on any linguistic topic are invited , especially papers on synchronic and diachronic aspects of the nordic languages . invited speakers : anders holmberg , tromsoe ( syntax ) tomas riad , stockholm ( phonology ) inge lise pedersen , copenhagen ( dialectology ) interest groups can ask to arrange special sessions ( workshops ) and at present the following are planned ( names of the workshop organizers in parentheses ) : comparative semantics for nordic languages ( elisabet engdahl ) optimality theory and nordic languages ( kersti boerjars ) the time allotted to each paper ( except for the invited talks ) is 30 minutes ( including discussion ) . participants who want to present a paper are requested to submit an abstract no later than january 31 , 1998 . abstracts may not exceed 2 pages with at least a 1 inch margin on all four sides and should employ a font not smaller than 12 pt . they should be sent anonymously in five copies , accompanied by a camera-ready original with the author 's name , address and affiliation . abstracts sent by e-mail will not be accepted . the authors will be notified about the acceptance of their papers by february 28 , 1998 . those interested in presenting papers in the special sessions ( workshops ) should consult the organizers of these ( engdahl @ ling . gu . se , k . borjars @ man . ac . uk ) . otherwise , correspondence should be addressed to : the xth conference of nordic and general linguistics institute of linguistics university of iceland arnagardi vid sudurgoetu \\\\ 193rnagar \\\\ 240i vi \\\\ 240 su \\\\ 240urg \\\\ 246tu 101 reykjavik , iceland tel . + 354 525 4408 fax + 354 525 4242 e - mail : nordconf @ rhi . hi . is url : http : / / www . rhi . hi . is / ~ nordconf - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - h \\\\ 246skuldur \\\\ 222r \\\\ 225insson hoskuldur thrainsson heimspekideild department of icelandic h \\\\ 225sk \\\\ 243la \\\\ 205slands university of iceland \\\\ 193rnagar \\\\ 240i v . su \\\\ 240urg \\\\ 246tu arnagardi v . sudurgoetu 101 reykjavik 101 reykjavik , iceland netfang : hoski @ rhi . hi . is e-mail hoski @ rhi . hi . is simi : 525-4420 ( i vinnu ) phone : ( 354 ) 525-4420 ( office ) 566-7141 ( heima ) ( 354 ) 566-7141 ( home ) br \\\\ 233fsimi : 525-4242 ( i vinnu ) fax : ( 354 ) 525-4242 ( work ) 566-8141 ( heima - ( 354 ) 566-8141 ( home - ef l \\\\ 225ti \\\\ 240 er vita fyrirfram ) if notified in advance )\\n\")]\n"
                }
            ], 
            "source": "from pathlib import Path\nimport re\n\ndef makeTestTrainRDDs(pathString):\n    \n    p = Path(pathString) # gets a path object representing the current directory path.\n    dirs = list(p.iterdir()) # get the directories part1 ... part10. \n    print(dirs) # Print to check that the directory is correct\n    \n    rddList = [] # create a list for the RDDs\n    \n    # Create an RDD for each 'part' directory and add them to rddList\n    for d in dirs: # iterate through the directories\n        rdd = sc.wholeTextFiles(str(d.absolute())) #>>> # read the files in the directory \n        rddList.append(rdd) #>>> append the RDD to the rddList\n        \n    print('len(rddList)',len(rddList))  # we should now have 10 RDDs in the list # just for testing\n    \n    # print(rddList[1].take(1)) # just for testing, comment out if worked.\n    \n    testRDD1 = rddList[9] # set the test set\n    trainRDD1 = rddList[0] # start the training set from 0 and \n    \n    # Create a union of the remaining RDDs\n    for i in range(1,9):\n        trainRDD1 = trainRDD1.union(rddList[i]) \n            \n    # Remove the paths and extensions from the filename. \n    testRDD2 = testRDD1.map(lambda fn_txt: (re.split('[/\\.]',fn_txt[0])[-2],fn_txt[1]))    \n    trainRDD2 = trainRDD1.map(lambda fn_txt: (re.split('[/\\.]', fn_txt[0])[-2], fn_txt[1]))\n    \n    return (trainRDD2,testRDD2)\n\n# this makes sure we are in the right directory\n%cd ~/notebook/work/City-Data-Science/datasets/lingspam_public/\n\n# this should show \"bare  lemm  lemm_stop  readme.txt  stop\"\n!ls\n\n# Testing the function makeTestTrainRDDs\ntrainRDD_testRDD = makeTestTrainRDDs('bare') # read from the 'bare' directory - this takes a bit of time\n(trainRDD,testRDD) = trainRDD_testRDD # unpack the returned tuple\n\nprint('created the RDDs') # notify the user, so that we can figure out where things went wrong if they do.\nprint('testRDD.count(): ',testRDD.count()) # should be ~291 \n\n#print('trainRDD.count(): ',trainRDD.count()) # should be ~2602 - commented out to save time\n\nprint('testRDD.getNumPartitions()',testRDD.getNumPartitions()) # normally 2 on DSX\nprint('testRDD.getStorageLevel()',testRDD.getStorageLevel()) # Serialized 1x Replicated on DSX\nprint('testRDD.take(1): ',testRDD.take(1)) # should be (filename,[tokens]) \n\nrdd1 = testRDD # use this for developemnt in the next tasks "
        }, 
        {
            "source": "## Task 2) Tokenize and remove punctuation\n\nWe will use the Python [Natural Language Toolkit](http://www.nltk.org) *NLTK* to do the tokenization (rather than splitting ourselves, as these specialist tools usually do that we can ourselves). We use the NLTK function word_tokenize.\n\nThen we will remove punctuation. There is no specific funtion for this, so we use a regular expression in a list comprehension.\n\nWe separate keys and values of the RDD, using the RDD functions `keys()` and `values()`, which yield each a new RDD. Then we process the values and *zip* them together with the keys again. We wrap the whole sequence into one function `prepareTokenRDD` for later use.", 
            "cell_type": "markdown", 
            "metadata": {}
        }, 
        {
            "execution_count": 6, 
            "cell_type": "code", 
            "metadata": {
                "scrolled": false
            }, 
            "outputs": [
                {
                    "output_type": "stream", 
                    "name": "stdout", 
                    "text": "[('9-66msg1', ['Subject', 'xth', 'conference', 'of', 'nordic', 'and', 'general', 'ling', 'the', 'tenth', 'conference', 'of', 'nordic', 'and', 'general', 'linguistics', 'will', 'be', 'held', 'in', 'reykjavik', 'iceland', 'from', 'saturday', 'june', '6', 'to', 'monday', 'june', '8', '1998', 'it', 'is', 'organized', 'by', 'the', 'institute', 'of', 'linguistics', 'university', 'of', 'iceland', 'the', 'deadline', 'for', 'preregistration', 'at', 'a', 'reduced', 'price', 'is', 'january', '31', '1998', 'pre', 'registration', 'forms', 'and', 'further', 'information', 'can', 'be', 'found', 'on', 'our', 'web', 'site', 'http', 'www', 'rhi', 'hi', 'is', 'nordconf', 'and', 'can', 'also', 'be', 'mailed', 'or', 'emailed', 'upon', 'request', 'papers', 'on', 'any', 'linguistic', 'topic', 'are', 'invited', 'especially', 'papers', 'on', 'synchronic', 'and', 'diachronic', 'aspects', 'of', 'the', 'nordic', 'languages', 'invited', 'speakers', 'anders', 'holmberg', 'tromsoe', 'syntax', 'tomas', 'riad', 'stockholm', 'phonology', 'inge', 'lise', 'pedersen', 'copenhagen', 'dialectology', 'interest', 'groups', 'can', 'ask', 'to', 'arrange', 'special', 'sessions', 'workshops', 'and', 'at', 'present', 'the', 'following', 'are', 'planned', 'names', 'of', 'the', 'workshop', 'organizers', 'in', 'parentheses', 'comparative', 'semantics', 'for', 'nordic', 'languages', 'elisabet', 'engdahl', 'optimality', 'theory', 'and', 'nordic', 'languages', 'kersti', 'boerjars', 'the', 'time', 'allotted', 'to', 'each', 'paper', 'except', 'for', 'the', 'invited', 'talks', 'is', '30', 'minutes', 'including', 'discussion', 'participants', 'who', 'want', 'to', 'present', 'a', 'paper', 'are', 'requested', 'to', 'submit', 'an', 'abstract', 'no', 'later', 'than', 'january', '31', '1998', 'abstracts', 'may', 'not', 'exceed', '2', 'pages', 'with', 'at', 'least', 'a', '1', 'inch', 'margin', 'on', 'all', 'four', 'sides', 'and', 'should', 'employ', 'a', 'font', 'not', 'smaller', 'than', '12', 'pt', 'they', 'should', 'be', 'sent', 'anonymously', 'in', 'five', 'copies', 'accompanied', 'by', 'a', 'cameraready', 'original', 'with', 'the', 'author', 's', 'name', 'address', 'and', 'affiliation', 'abstracts', 'sent', 'by', 'email', 'will', 'not', 'be', 'accepted', 'the', 'authors', 'will', 'be', 'notified', 'about', 'the', 'acceptance', 'of', 'their', 'papers', 'by', 'february', '28', '1998', 'those', 'interested', 'in', 'presenting', 'papers', 'in', 'the', 'special', 'sessions', 'workshops', 'should', 'consult', 'the', 'organizers', 'of', 'these', 'engdahl', 'ling', 'gu', 'se', 'k', 'borjars', 'man', 'ac', 'uk', 'otherwise', 'correspondence', 'should', 'be', 'addressed', 'to', 'the', 'xth', 'conference', 'of', 'nordic', 'and', 'general', 'linguistics', 'institute', 'of', 'linguistics', 'university', 'of', 'iceland', 'arnagardi', 'vid', 'sudurgoetu', '193rnagar', '240i', 'vi', '240', 'su', '240urg', '246tu', '101', 'reykjavik', 'iceland', 'tel', '354', '525', '4408', 'fax', '354', '525', '4242', 'e', 'mail', 'nordconf', 'rhi', 'hi', 'is', 'url', 'http', 'www', 'rhi', 'hi', 'is', 'nordconf', 'h', '246skuldur', '222r', '225insson', 'hoskuldur', 'thrainsson', 'heimspekideild', 'department', 'of', 'icelandic', 'h', '225sk', '243la', '205slands', 'university', 'of', 'iceland', '193rnagar', '240i', 'v', 'su', '240urg', '246tu', 'arnagardi', 'v', 'sudurgoetu', '101', 'reykjavik', '101', 'reykjavik', 'iceland', 'netfang', 'hoski', 'rhi', 'hi', 'is', 'email', 'hoski', 'rhi', 'hi', 'is', 'simi', '5254420', 'i', 'vinnu', 'phone', '354', '5254420', 'office', '5667141', 'heima', '354', '5667141', 'home', 'br', '233fsimi', '5254242', 'i', 'vinnu', 'fax', '354', '5254242', 'work', '5668141', 'heima', '354', '5668141', 'home', 'ef', 'l', '225ti', '240', 'er', 'vita', 'fyrirfram', 'if', 'notified', 'in', 'advance'])]\n"
                }
            ], 
            "source": "import nltk\nimport re\nfrom nltk.corpus import stopwords\n\ndef tokenize(text):\n    nltk.download('punkt') # this loads the standard NLTK tokenizer model \n    return nltk.word_tokenize(text) # use the nltk function word_tokenize\n    \ndef removePunctuation(tokens):\n    tokens2 =  [re.sub('\\W','',s) for s in tokens] # use a list comprehension to remove punctuaton \n    return tokens2\n    \ndef prepareTokenRDD(fn_txt_RDD):\n    rdd_vals2 = fn_txt_RDD.values() # It's convenient to process only the values. \n    rdd_vals3 = rdd_vals2.map(tokenize) # Create a tokenised version of the values by mapping\n    rdd_vals4 = rdd_vals3.map(removePunctuation) # remove punctuation from the values\n    rdd4 = fn_txt_RDD.keys().zip(rdd_vals4) # Zip the two RDDs together \n    \n    # remove any empty value strings (i.e. length 0) that we may have created by removing punctiation.\n    rdd5 = rdd4.map(lambda x: (x[0], [y for y in x[1] if len(y)>0])) # remove empty strings/\n    rdd6 =  rdd5.filter(lambda x: len(x[1]) > 0) # remove items without tokens.\n    \n    return rdd6 \n\nrdd2 = prepareTokenRDD(rdd1) # Use the test set for now, because it is smaller\nprint(rdd2.take(1)) # For checking result of task 2. "
        }, 
        {
            "source": "## <font color='green'>Question: why should this be filtering done after zipping the keys and values together?</font>\n**Answer:** To avoid the mismatch between key and values. If this had been done before, empty values are discarded while their corresponding keys are not. This will cause a mismatch between key and values, and a difference between the number of keys and the number of values.", 
            "cell_type": "markdown", 
            "metadata": {}
        }, 
        {
            "source": "## Task 3) Creating normalised TF.IDF vectors of defined dimensionality, measure the effect of caching.\n\nWe use the hashing trick to create fixed size TF vectors directly from the word list now.\n\nThen we'll use the IDF and Normalizer functions provided by Spark.\n\nWe want control of the dimensionality in the `normTFIDF` function, so we introduce an argument into our functions that enables us to vary dimensionalty later. Here is also an opportunity to benefit from caching.", 
            "cell_type": "markdown", 
            "metadata": {}
        }, 
        {
            "execution_count": 7, 
            "cell_type": "code", 
            "metadata": {}, 
            "outputs": [
                {
                    "output_type": "stream", 
                    "name": "stdout", 
                    "text": "[('9-66msg1', DenseVector([0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 1.0, 0.0, 0.0, 0.0]))]\n"
                }
            ], 
            "source": "# use the hashing trick to create a fixed-size vector from a word list\n\ndef hashing_vectorize(text,N): # arguments: the list and the size of the output vector\n    v = [0] * N  # create vector of 0s\n    for word in text: # iterate through the words \n        h = hash(word) # get the hash value \n        v [h % N] += 1 #  add 1 at the hashed address \n    return v # return hashed word vector\n\nfrom pyspark.mllib.feature import IDF, Normalizer\n\ndef normTFIDF(fn_tokens_RDD, vecDim, caching=True):\n    keysRDD = fn_tokens_RDD.keys()\n    tokensRDD = fn_tokens_RDD.values()\n    tfVecRDD = tokensRDD.map(lambda tokens: hashing_vectorize(tokens,vecDim)) #>>> passing the vecDim value.\n    \n    if caching:\n        tfVecRDD.persist(StorageLevel.MEMORY_ONLY) # since we will read more than once, caching in Memory will make things quicker.\n    idf = IDF() # create IDF object\n    idfModel = idf.fit(tfVecRDD) # calculate IDF values\n    tfIdfRDD = idfModel.transform(tfVecRDD)\n    \n    norm = Normalizer(float('inf')) # create a Normalizer object like in the example linked above\n    normTfIdfRDD = norm.transform(tfIdfRDD) # and apply it to the tfIdfRDD \n    zippedRDD =  keysRDD.zip(normTfIdfRDD) # zip the keys and values together\n    \n    return zippedRDD\n\ntestDim = 10 # testing\nrdd3 = normTFIDF(rdd2, testDim, True) # testing\nprint(rdd3.take(1)) # we should now have tuples with ('filename',[N-dim vector])"
        }, 
        {
            "source": "### Task 3a) Caching experiment\n\nThe normTFIDF let's us switch caching on or off. Write a bit of code that measures the effect of caching by takes the time for both options.\n\nAdd a short comment on the result (why is there an effect, why of the size that it is?)", 
            "cell_type": "markdown", 
            "metadata": {}
        }, 
        {
            "execution_count": 8, 
            "cell_type": "code", 
            "metadata": {}, 
            "outputs": [
                {
                    "output_type": "stream", 
                    "name": "stdout", 
                    "text": "Creating TF.IDF vectors, 3 trials - mean time with caching:  15.7701662381 , mean time without caching:  18.1433038712\n"
                }
            ], 
            "source": "#run a small experiment with caching set to True or False, 3 times each\n\nfrom time import time\n\nresCaching = [] # for storing results\nresNoCache = [] # for storing results\nfor i in range(3): # 3 samples\n    \n    startTime = time()  # start timer\n    testRDD1 = normTFIDF(rdd2, testDim, True) # \n    testRDD1.count() # call an action on the RDD to force execution\n    endTime = time()  # end timer\n    resCaching.append( endTime - startTime ) # calculate the difference\n    \n    startTime = time()  # start timer\n    testRDD2 = normTFIDF(rdd2, testDim, False) \n    testRDD2.count() # call an action to force execution   \n    endTime = time()  # end timer\n    resNoCache.append( endTime - startTime )\n    \nmeanTimeCaching = np.mean(resCaching)# calculate average times\nmeanTimeNoCache = np.mean(resNoCache)# calculate average times\n\nprint('Creating TF.IDF vectors, 3 trials - mean time with caching: ', meanTimeCaching, ', mean time without caching: ', meanTimeNoCache)"
        }, 
        {
            "source": "## <font color='green'>Results </font>\n\nmean time with caching:  15.7701662381 , mean time without caching:  18.1433038712. Processing time without caching is slower than with caching by approximately 20%.\n\n## <font color='green'>Comments</font>\n\n**Why is there an effect**: Caching keeps the data for later use, which means that the data don't need to be reloaded in each iteration. This helps speed up operations where specific RDDs need to be accessed multiple times. \n\n**Why of the size that it is:** The effect is noticeable as the count of each RDD is 291, with each contains a tuple with the header and a dense vector of 10 elements. Multiplying these two numbers indicates a considerable amount of data to be loaded.", 
            "cell_type": "markdown", 
            "metadata": {}
        }, 
        {
            "source": "## Task 4) Create LabeledPoints \n\nDetermine whether the file is spam (i.e. the filename contains \u2019spmsg\u2019) and replace the filename by a 1 (spam) or 0 (non-spam) accordingly.", 
            "cell_type": "markdown", 
            "metadata": {}
        }, 
        {
            "execution_count": 9, 
            "cell_type": "code", 
            "metadata": {
                "pixiedust": {
                    "displayParams": {
                        "handlerId": "tableView"
                    }
                }
            }, 
            "outputs": [
                {
                    "output_type": "stream", 
                    "name": "stdout", 
                    "text": "[LabeledPoint(0.0, [0.0,0.0,0.0,0.0,0.0,0.0,1.0,0.0,0.0,0.0])]\n"
                }
            ], 
            "source": "from pyspark.mllib.regression import LabeledPoint\n\n# creatate labelled points of vector size N out of an RDD with normalised (filename [(word,count), ...]) items\ndef makeLabeledPoints(fn_vec_RDD): # RDD and N needed \n    cls_vec_RDD = fn_vec_RDD.map(lambda x: (1 if x[0].startswith('spmsg') else 0, x[1])) # use a conditional expression to get the class label (True or False)\n    \n    # now we can create the LabeledPoint objects with (class,vector) arguments\n    lp_RDD = cls_vec_RDD.map(lambda cls_vec: LabeledPoint(cls_vec[0],cls_vec[1]) ) \n    return lp_RDD \n\n# for testing\ntestLpRDD = makeLabeledPoints(rdd3) \nprint(testLpRDD.take(1)) "
        }, 
        {
            "source": "## Task 5) Complete the preprocessing \n\nCreate a single function to do the preprocessing", 
            "cell_type": "markdown", 
            "metadata": {}
        }, 
        {
            "execution_count": 10, 
            "cell_type": "code", 
            "metadata": {}, 
            "outputs": [
                {
                    "output_type": "stream", 
                    "name": "stdout", 
                    "text": "[LabeledPoint(0.0, [0.0,0.0,0.0,0.0,0.0,0.0,1.0,0.0,0.0,0.0])]\n[PosixPath('lemm/part5'), PosixPath('lemm/part8'), PosixPath('lemm/part2'), PosixPath('lemm/part9'), PosixPath('lemm/part3'), PosixPath('lemm/part6'), PosixPath('lemm/part1'), PosixPath('lemm/part4'), PosixPath('lemm/part7'), PosixPath('lemm/part10')]\nlen(rddList) 10\n[('file:/gpfs/global_fs01/sym_shared/YPProdSpark/user/s8bb-4c006897e4ec94-b51c55267e90/notebook/work/City-Data-Science/datasets/lingspam_public/lemm/part8/6-939msg2.txt', 'Subject: child language acquistion\\n\\ni be look for information on elicitation technique and grammaticality judgement for 2 - 4 year old child . i would be grateful for your help on thus s subject . cathy finlay . university of ulster .\\n')]\n[LabeledPoint(0.0, [0.0640664810574,0.156270149468,0.28272862321,0.158772994227,0.264182002173,0.575501960201,0.756375820848,0.360451806382,0.369625030955,0.575501960201,0.172630846172,0.405814869475,0.631754026472,0.0,0.379600845063,0.112351647251,0.102606114476,0.0829114902185,0.106840593517,0.24070669008,0.787256566569,0.149802196334,0.0,0.496232476108,0.0,0.301506600992,0.369799970474,0.0874274500912,0.261582779851,0.517892538515,0.333333333333,0.284135764403,0.311885889399,0.479051011293,0.633172878675,0.0,0.666666666667,0.455521014076,0.167962220107,0.25626592423,0.271946716128,0.380785440758,0.112633571765,0.330821650739,0.559168877584,0.566300385437,0.352242669564,0.13504381253,0.523165559702,0.128566185443,0.993161304897,0.379052415883,0.271193312311,0.0,0.158293219669,0.14665040527,0.140385015401,0.402008801323,0.226307614616,0.191557674608,0.133026506265,1.0,0.113260077087,0.113260077087,0.151746189069,0.115608144008,0.29330081054,0.0,0.111111111111,0.187252745418,0.344290519883,0.327814808358,0.0,0.292725449981,0.3141566617,0.261582779851,0.310589037924,0.12035334504,0.931185543855,0.279584438792,0.394462999725,0.279584438792,0.215813235075,0.467950051336,0.172033485639,0.35928825847,0.409768510447,0.687634152492,0.158293219669,0.104912940109,0.875303958925,0.418875548933,0.184347328191,0.59420476545,0.0,0.114261010077,0.444444444444,0.180225903191,0.320332405287,0.641043561103])]\n[LabeledPoint(0.0, [0.0,0.262098328856,0.161716304193,0.0965638401197,0.0,0.159632130456,0.0,0.350707225162,0.225014599217,0.182800482666,1.0,0.0,0.507971698186,0.452080730114,0.0,0.100180881367,0.0,0.173627147761,0.12636726198,0.0812144887354,0.183173947696,0.113591368182,0.110786940634,0.21980383993,0.0,0.297392551902,0.115543104418,0.0465609143526,0.0,0.0945406865319,0.0,0.923111908195,0.375374094802,0.185403170162,0.384552440624,0.303470500906,0.0641900972033,0.0643161851002,0.166040717435,0.0,0.0,0.154217701634,0.286295606643,0.0,0.109084797546,0.0793821935253,0.145818516571,0.0923685439525,0.0,0.0832919275075,0.386717326116,0.238548057933,0.114759424063,0.307941022214,0.0,0.0969658434672,0.471257448314,0.0735622887271,0.0,0.110300034387,0.0,0.0,0.233744172523,0.103058135128,0.0371825103396,0.154703626685,0.115851615767,0.0,0.0,0.0459494455867,0.086120516894,0.219119192182,0.0869866818387,0.0785477437194,0.147909821582,0.0784473733328,0.0,0.0624845286005,0.216497168687,0.0945406865319,0.144501827874,0.107500595375,0.0775159657901,0.0766364902716,0.0,0.267605049272,0.0,0.0961072710779,0.0,0.0736375796916,0.241993323204,0.116469759871,0.0860236329844,0.193327006019,0.0,0.230657841689,0.0,0.423953724234,0.108599512004,0.122365063358])]\n"
                }
            ], 
            "source": "# N is for controlling the vector size\ndef preprocess(rawRDD,N):\n    \"\"\" take a (filename,text) RDD and transform into LabelledPoint objects \n        with class labels and a TF.IDF vector with N dimensions. \"\"\"\n    tokenRDD = prepareTokenRDD(rawRDD) # task 2\n    tfIdfRDD = normTFIDF(tokenRDD,N) # task 3\n    lpRDD = makeLabeledPoints(tfIdfRDD) # task 4\n    return lpRDD # return RDD with LabeledPoints\n\n# and with this we can start the whole process from a directory, N is again the vector size\ndef loadAndPreprocess(directory,N):\n    \"\"\" load lingspam data from a directory and create a training and test set of preprocessed data \"\"\"\n    trainRDD_testRDD = makeTestTrainRDDs(directory) # read from the directory using the function created in task 1\n    (trainRDD,testRDD) = trainRDD_testRDD # unpack the returned tuple\n    \n    return (preprocess(trainRDD,N),preprocess(testRDD,N)) # apply the preprocessing funcion defined above\n\ntrainLpRDD = preprocess(trainRDD,testDim) # prepare the training data\nprint(testLpRDD.take(1)) #testing\n\ntrain_test_LpRDD = loadAndPreprocess('lemm',100) # Re-run with another vector size\n(trainLpRDD,testLpRDD) = train_test_LpRDD \n\nprint(testLpRDD.take(1))\nprint(trainLpRDD.take(1))"
        }, 
        {
            "source": "## Task 6) Train some classifiers \n\nUse the `LabeledPoint` objects to train a classifier, specifically the *LogisticRegression*, *Naive Bayes*, and *Support Vector Machine*. Calculate the accuracy of the model on the training set", 
            "cell_type": "markdown", 
            "metadata": {}
        }, 
        {
            "execution_count": 13, 
            "cell_type": "code", 
            "metadata": {
                "scrolled": false
            }, 
            "outputs": [
                {
                    "output_type": "stream", 
                    "name": "stdout", 
                    "text": "Starting to train the model\nTrained LR (model1)\nTrained NB (model2)\nTrained SVM (model3)\nAccuracy 84.9% (data items: 2602, correct: 2209)\n"
                }, 
                {
                    "execution_count": 13, 
                    "metadata": {}, 
                    "data": {
                        "text/plain": "0.8489623366641046"
                    }, 
                    "output_type": "execute_result"
                }
            ], 
            "source": "from pyspark.mllib.classification import (NaiveBayes, LogisticRegressionWithLBFGS, SVMWithSGD) \nimport numpy\n\ndef trainModel(lpRDD):\n    \"\"\" Train 3 classifier models on the given RDD with LabeledPoint objects. A list of trained model is returned. \"\"\"\n    lpRDD.persist(StorageLevel.MEMORY_ONLY)\n    \n    # Train a classifier model.\n    print('Starting to train the model') \n    model1 = LogisticRegressionWithLBFGS.train(lpRDD) # this is the best model\n    print('Trained LR (model1)')\n    #print('type(model1)')\n    model2 = NaiveBayes.train(lpRDD) # doesn't work well\n    print('Trained NB (model2)')\n    #print(type(model2))\n    model3 = SVMWithSGD.train(lpRDD) # or this ...\n    print('Trained SVM (model3)')\n    return [model1,model2,model3]\n\ndef testModel(model, lpRDD):\n    \"\"\" Tests the classification accuracy of the given model on the given RDD with LabeledPoint objects. \"\"\"\n    lpRDD.persist(StorageLevel.MEMORY_ONLY)\n    \n    # Make prediction and evaluate training set accuracy.\n    predictionAndLabel = lpRDD.map(lambda p: (model.predict(p.features), p.label)) # get the prediction and ground truth (label) for each item.\n    correct = predictionAndLabel.filter(lambda xv: xv[0] == xv[1]).count() # count the correct predictions \n    \n    accuracy = correct/(lpRDD.count()) # and calculate the accuracy \n    \n    print('Accuracy {:.1%} (data items: {}, correct: {})'.format(accuracy,lpRDD.count(), correct)) # report to console\n    return accuracy # and return the value  \n\nmodels = trainModel(trainLpRDD) # just for testing\ntestModel(models[2], trainLpRDD) # just for testing"
        }, 
        {
            "source": "## Task 7) Automate training and testing\n\nWe automate now the whole process from reading the files, through preprocessing, and training up to evaluating the models. In the end we have a single function that takes all the parameters we are interested in and produces trained models and an evaluation.", 
            "cell_type": "markdown", 
            "metadata": {}
        }, 
        {
            "execution_count": 14, 
            "cell_type": "code", 
            "metadata": {}, 
            "outputs": [
                {
                    "output_type": "stream", 
                    "name": "stdout", 
                    "text": "Start loading and preprocessing\n[PosixPath('lemm/part5'), PosixPath('lemm/part8'), PosixPath('lemm/part2'), PosixPath('lemm/part9'), PosixPath('lemm/part3'), PosixPath('lemm/part6'), PosixPath('lemm/part1'), PosixPath('lemm/part4'), PosixPath('lemm/part7'), PosixPath('lemm/part10')]\nlen(rddList) 10\n[('file:/gpfs/global_fs01/sym_shared/YPProdSpark/user/s8bb-4c006897e4ec94-b51c55267e90/notebook/work/City-Data-Science/datasets/lingspam_public/lemm/part8/6-939msg2.txt', 'Subject: child language acquistion\\n\\ni be look for information on elicitation technique and grammaticality judgement for 2 - 4 year old child . i would be grateful for your help on thus s subject . cathy finlay . university of ulster .\\n')]\nFinished loading and preprocessing\nStarting to train the model\nTrained LR (model1)\nTrained NB (model2)\nTrained SVM (model3)\nTraining\nAccuracy 100.0% (data items: 2602, correct: 2602)\nTesting\nAccuracy 97.9% (data items: 291, correct: 285)\nTraining\nAccuracy 96.5% (data items: 2602, correct: 2512)\nTesting\nAccuracy 94.8% (data items: 291, correct: 276)\nTraining\nAccuracy 88.9% (data items: 2602, correct: 2313)\nTesting\nAccuracy 89.0% (data items: 291, correct: 259)\n"
                }, 
                {
                    "execution_count": 14, 
                    "metadata": {}, 
                    "data": {
                        "text/plain": "[[1.0, 0.9654112221368178, 0.8889315910837817],\n [0.979381443298969, 0.9484536082474226, 0.8900343642611683]]"
                    }, 
                    "output_type": "execute_result"
                }
            ], 
            "source": "def trainTestModel(trainRDD,testRDD):\n    \"\"\" Trains 3 models and tests them on training and test data. Returns a matrix the training and testing (rows) accuracy values for all models (columns). \"\"\"\n    models = trainModel(trainRDD) # train models on the training set\n    results = [[],[]] # matrix for 2 modes (training/test) vs n models (currently 3)\n    \n    for mdl in models:\n        print('Training')\n        results[0].append(testModel(mdl, trainRDD)) # test the model on the training set\n        \n        print('Testing')\n        results[1].append(testModel(mdl, testRDD)) # test the model on the test set\n        \n    return results\n\ndef trainTestFolder(folder,N):\n    \"\"\" Reads data from a folder, preproceses the data, and trains and evaluates models on it. \"\"\"\n    print('Start loading and preprocessing')\n    train_test_LpRDD = loadAndPreprocess(folder,N) # create the RDDs\n    \n    print('Finished loading and preprocessing')\n    (trainLpRDD,testLpRDD) = train_test_LpRDD # unpack the RDDs \n    \n    return trainTestModel(trainLpRDD,testLpRDD) # train and test\n\ntrainTestFolder('lemm',1000) "
        }, 
        {
            "source": "## Task 8) Run experiments \n\nWe have now a single function that allows us to vary the vector size easily. Test vector sizes 3, 30, 300, 3000, 30000 and examine the effect on the classification accuracy in Experiment 1.\n\nUse the function from Task 7) to test different data types. The dataset has raw text in folder `bare`, lemmatised text in  `lemm` (similar to stemming, reduces to basic word forms), `stop` (with stopwords removed), and `lemm_stop` (lemmatised and stopwords removed). Test how the classification accuracy differs for these four data types in Experiment 2. Collect the results in a data structure that can be saved for later saving and analyis.\n\nComment on the results in a few sentences, considering the differences in performance between the different conditions as well as train an test values. 15%", 
            "cell_type": "markdown", 
            "metadata": {}
        }, 
        {
            "execution_count": 15, 
            "cell_type": "code", 
            "metadata": {
                "scrolled": false
            }, 
            "outputs": [
                {
                    "output_type": "stream", 
                    "name": "stdout", 
                    "text": "\nEXPERIMENT 1: Testing different vector sizes\nN = 3\nStart loading and preprocessing\n[PosixPath('bare/part5'), PosixPath('bare/part8'), PosixPath('bare/part2'), PosixPath('bare/part9'), PosixPath('bare/part3'), PosixPath('bare/part6'), PosixPath('bare/part1'), PosixPath('bare/part4'), PosixPath('bare/part7'), PosixPath('bare/part10')]\nlen(rddList) 10\n[('file:/gpfs/global_fs01/sym_shared/YPProdSpark/user/s8bb-4c006897e4ec94-b51c55267e90/notebook/work/City-Data-Science/datasets/lingspam_public/bare/part8/6-939msg2.txt', 'Subject: child language acquistion\\n\\ni am looking for information on elicitation techniques and grammaticality judgements for 2 - 4 year old children . i would be grateful for your help on thi s subject . cathy finlay . university of ulster .\\n')]\nFinished loading and preprocessing\nStarting to train the model\nTrained LR (model1)\nTrained NB (model2)\nTrained SVM (model3)\nTraining\nAccuracy 83.4% (data items: 2602, correct: 2170)\nTesting\nAccuracy 83.2% (data items: 291, correct: 242)\nTraining\nAccuracy 83.4% (data items: 2602, correct: 2170)\nTesting\nAccuracy 83.2% (data items: 291, correct: 242)\nTraining\nAccuracy 83.4% (data items: 2602, correct: 2170)\nTesting\nAccuracy 83.2% (data items: 291, correct: 242)\nN = 30\nStart loading and preprocessing\n[PosixPath('bare/part5'), PosixPath('bare/part8'), PosixPath('bare/part2'), PosixPath('bare/part9'), PosixPath('bare/part3'), PosixPath('bare/part6'), PosixPath('bare/part1'), PosixPath('bare/part4'), PosixPath('bare/part7'), PosixPath('bare/part10')]\nlen(rddList) 10\n[('file:/gpfs/global_fs01/sym_shared/YPProdSpark/user/s8bb-4c006897e4ec94-b51c55267e90/notebook/work/City-Data-Science/datasets/lingspam_public/bare/part8/6-939msg2.txt', 'Subject: child language acquistion\\n\\ni am looking for information on elicitation techniques and grammaticality judgements for 2 - 4 year old children . i would be grateful for your help on thi s subject . cathy finlay . university of ulster .\\n')]\nFinished loading and preprocessing\nStarting to train the model\nTrained LR (model1)\nTrained NB (model2)\nTrained SVM (model3)\nTraining\nAccuracy 88.8% (data items: 2602, correct: 2310)\nTesting\nAccuracy 86.3% (data items: 291, correct: 251)\nTraining\nAccuracy 83.4% (data items: 2602, correct: 2170)\nTesting\nAccuracy 83.2% (data items: 291, correct: 242)\nTraining\nAccuracy 83.4% (data items: 2602, correct: 2170)\nTesting\nAccuracy 83.2% (data items: 291, correct: 242)\nN = 300\nStart loading and preprocessing\n[PosixPath('bare/part5'), PosixPath('bare/part8'), PosixPath('bare/part2'), PosixPath('bare/part9'), PosixPath('bare/part3'), PosixPath('bare/part6'), PosixPath('bare/part1'), PosixPath('bare/part4'), PosixPath('bare/part7'), PosixPath('bare/part10')]\nlen(rddList) 10\n[('file:/gpfs/global_fs01/sym_shared/YPProdSpark/user/s8bb-4c006897e4ec94-b51c55267e90/notebook/work/City-Data-Science/datasets/lingspam_public/bare/part8/6-939msg2.txt', 'Subject: child language acquistion\\n\\ni am looking for information on elicitation techniques and grammaticality judgements for 2 - 4 year old children . i would be grateful for your help on thi s subject . cathy finlay . university of ulster .\\n')]\nFinished loading and preprocessing\nStarting to train the model\nTrained LR (model1)\nTrained NB (model2)\nTrained SVM (model3)\nTraining\nAccuracy 100.0% (data items: 2602, correct: 2602)\nTesting\nAccuracy 96.9% (data items: 291, correct: 282)\nTraining\nAccuracy 93.7% (data items: 2602, correct: 2437)\nTesting\nAccuracy 92.1% (data items: 291, correct: 268)\nTraining\nAccuracy 91.0% (data items: 2602, correct: 2367)\nTesting\nAccuracy 91.1% (data items: 291, correct: 265)\nN = 3000\nStart loading and preprocessing\n[PosixPath('bare/part5'), PosixPath('bare/part8'), PosixPath('bare/part2'), PosixPath('bare/part9'), PosixPath('bare/part3'), PosixPath('bare/part6'), PosixPath('bare/part1'), PosixPath('bare/part4'), PosixPath('bare/part7'), PosixPath('bare/part10')]\nlen(rddList) 10\n[('file:/gpfs/global_fs01/sym_shared/YPProdSpark/user/s8bb-4c006897e4ec94-b51c55267e90/notebook/work/City-Data-Science/datasets/lingspam_public/bare/part8/6-939msg2.txt', 'Subject: child language acquistion\\n\\ni am looking for information on elicitation techniques and grammaticality judgements for 2 - 4 year old children . i would be grateful for your help on thi s subject . cathy finlay . university of ulster .\\n')]\nFinished loading and preprocessing\nStarting to train the model\nTrained LR (model1)\nTrained NB (model2)\nTrained SVM (model3)\nTraining\nAccuracy 100.0% (data items: 2602, correct: 2602)\nTesting\nAccuracy 97.9% (data items: 291, correct: 285)\nTraining\nAccuracy 99.1% (data items: 2602, correct: 2579)\nTesting\nAccuracy 97.9% (data items: 291, correct: 285)\nTraining\nAccuracy 89.4% (data items: 2602, correct: 2325)\nTesting\nAccuracy 88.7% (data items: 291, correct: 258)\nN = 30000\nStart loading and preprocessing\n[PosixPath('bare/part5'), PosixPath('bare/part8'), PosixPath('bare/part2'), PosixPath('bare/part9'), PosixPath('bare/part3'), PosixPath('bare/part6'), PosixPath('bare/part1'), PosixPath('bare/part4'), PosixPath('bare/part7'), PosixPath('bare/part10')]\nlen(rddList) 10\n[('file:/gpfs/global_fs01/sym_shared/YPProdSpark/user/s8bb-4c006897e4ec94-b51c55267e90/notebook/work/City-Data-Science/datasets/lingspam_public/bare/part8/6-939msg2.txt', 'Subject: child language acquistion\\n\\ni am looking for information on elicitation techniques and grammaticality judgements for 2 - 4 year old children . i would be grateful for your help on thi s subject . cathy finlay . university of ulster .\\n')]\nFinished loading and preprocessing\nStarting to train the model\nTrained LR (model1)\nTrained NB (model2)\nTrained SVM (model3)\nTraining\nAccuracy 100.0% (data items: 2602, correct: 2602)\nTesting\nAccuracy 93.8% (data items: 291, correct: 273)\nTraining\nAccuracy 95.5% (data items: 2602, correct: 2486)\nTesting\nAccuracy 94.5% (data items: 291, correct: 275)\nTraining\nAccuracy 88.5% (data items: 2602, correct: 2303)\nTesting\nAccuracy 85.9% (data items: 291, correct: 250)\nEXPERIMENT 2: Testing different data types\nPath = bare\nStart loading and preprocessing\n[PosixPath('bare/part5'), PosixPath('bare/part8'), PosixPath('bare/part2'), PosixPath('bare/part9'), PosixPath('bare/part3'), PosixPath('bare/part6'), PosixPath('bare/part1'), PosixPath('bare/part4'), PosixPath('bare/part7'), PosixPath('bare/part10')]\nlen(rddList) 10\n[('file:/gpfs/global_fs01/sym_shared/YPProdSpark/user/s8bb-4c006897e4ec94-b51c55267e90/notebook/work/City-Data-Science/datasets/lingspam_public/bare/part8/6-939msg2.txt', 'Subject: child language acquistion\\n\\ni am looking for information on elicitation techniques and grammaticality judgements for 2 - 4 year old children . i would be grateful for your help on thi s subject . cathy finlay . university of ulster .\\n')]\nFinished loading and preprocessing\nStarting to train the model\nTrained LR (model1)\nTrained NB (model2)\nTrained SVM (model3)\nTraining\nAccuracy 100.0% (data items: 2602, correct: 2602)\nTesting\nAccuracy 97.9% (data items: 291, correct: 285)\nTraining\nAccuracy 99.1% (data items: 2602, correct: 2579)\nTesting\nAccuracy 97.9% (data items: 291, correct: 285)\nTraining\nAccuracy 89.4% (data items: 2602, correct: 2325)\nTesting\nAccuracy 88.7% (data items: 291, correct: 258)\nPath = stop\nStart loading and preprocessing\n[PosixPath('stop/part5'), PosixPath('stop/part8'), PosixPath('stop/part2'), PosixPath('stop/part9'), PosixPath('stop/part3'), PosixPath('stop/part6'), PosixPath('stop/part1'), PosixPath('stop/part4'), PosixPath('stop/part7'), PosixPath('stop/part10')]\nlen(rddList) 10\n[('file:/gpfs/global_fs01/sym_shared/YPProdSpark/user/s8bb-4c006897e4ec94-b51c55267e90/notebook/work/City-Data-Science/datasets/lingspam_public/stop/part8/6-939msg2.txt', 'Subject: child language acquistion\\n\\nam looking information elicitation techniques grammaticality judgements 2 - 4 old children . grateful help thi s subject . cathy finlay . university ulster .\\n')]\nFinished loading and preprocessing\nStarting to train the model\nTrained LR (model1)\nTrained NB (model2)\nTrained SVM (model3)\nTraining\nAccuracy 100.0% (data items: 2602, correct: 2602)\nTesting\nAccuracy 97.6% (data items: 291, correct: 284)\nTraining\nAccuracy 98.9% (data items: 2602, correct: 2573)\nTesting\nAccuracy 97.9% (data items: 291, correct: 285)\nTraining\nAccuracy 85.7% (data items: 2602, correct: 2229)\nTesting\nAccuracy 85.2% (data items: 291, correct: 248)\nPath = lemm\nStart loading and preprocessing\n[PosixPath('lemm/part5'), PosixPath('lemm/part8'), PosixPath('lemm/part2'), PosixPath('lemm/part9'), PosixPath('lemm/part3'), PosixPath('lemm/part6'), PosixPath('lemm/part1'), PosixPath('lemm/part4'), PosixPath('lemm/part7'), PosixPath('lemm/part10')]\nlen(rddList) 10\n[('file:/gpfs/global_fs01/sym_shared/YPProdSpark/user/s8bb-4c006897e4ec94-b51c55267e90/notebook/work/City-Data-Science/datasets/lingspam_public/lemm/part8/6-939msg2.txt', 'Subject: child language acquistion\\n\\ni be look for information on elicitation technique and grammaticality judgement for 2 - 4 year old child . i would be grateful for your help on thus s subject . cathy finlay . university of ulster .\\n')]\nFinished loading and preprocessing\nStarting to train the model\nTrained LR (model1)\nTrained NB (model2)\nTrained SVM (model3)\nTraining\nAccuracy 100.0% (data items: 2602, correct: 2602)\nTesting\nAccuracy 97.3% (data items: 291, correct: 283)\nTraining\nAccuracy 99.3% (data items: 2602, correct: 2583)\nTesting\nAccuracy 98.3% (data items: 291, correct: 286)\nTraining\nAccuracy 89.2% (data items: 2602, correct: 2321)\nTesting\nAccuracy 88.0% (data items: 291, correct: 256)\nPath = lemm_stop\nStart loading and preprocessing\n[PosixPath('lemm_stop/part5'), PosixPath('lemm_stop/part8'), PosixPath('lemm_stop/part2'), PosixPath('lemm_stop/part9'), PosixPath('lemm_stop/part3'), PosixPath('lemm_stop/part6'), PosixPath('lemm_stop/part1'), PosixPath('lemm_stop/part4'), PosixPath('lemm_stop/part7'), PosixPath('lemm_stop/part10')]\nlen(rddList) 10\n[('file:/gpfs/global_fs01/sym_shared/YPProdSpark/user/s8bb-4c006897e4ec94-b51c55267e90/notebook/work/City-Data-Science/datasets/lingspam_public/lemm_stop/part8/6-939msg2.txt', 'Subject: child language acquistion\\n\\ninformation elicitation technique grammaticality judgement 2 - 4 old child . grateful help thus s subject . cathy finlay . university ulster .\\n')]\nFinished loading and preprocessing\nStarting to train the model\nTrained LR (model1)\nTrained NB (model2)\nTrained SVM (model3)\nTraining\nAccuracy 100.0% (data items: 2602, correct: 2602)\nTesting\nAccuracy 97.6% (data items: 291, correct: 284)\nTraining\nAccuracy 99.0% (data items: 2602, correct: 2577)\nTesting\nAccuracy 97.9% (data items: 291, correct: 285)\nTraining\nAccuracy 85.8% (data items: 2602, correct: 2233)\nTesting\nAccuracy 84.5% (data items: 291, correct: 246)\n"
                }
            ], 
            "source": "from pyspark.sql import DataFrame\n\nfolder = 'bare'\nN = numpy.array([3,30,300,3000,30000]) \nprint('\\nEXPERIMENT 1: Testing different vector sizes')\nresults = []\nfor n in N:\n    print('N = {}'.format(n))\n    results.append(trainTestFolder(folder,n))\n    \nn = 3000\ntypeFolders = ['bare','stop','lemm','lemm_stop']\nprint('EXPERIMENT 2: Testing different data types')\nfor folder in typeFolders:\n    print('Path = {}'.format(folder))\n    results.append(trainTestFolder(folder,n))"
        }, 
        {
            "execution_count": 66, 
            "cell_type": "code", 
            "metadata": {}, 
            "outputs": [
                {
                    "output_type": "stream", 
                    "name": "stdout", 
                    "text": "============================ SUMMARY OF RESULTS ================================\n+-------------------+------------------------------+---------------------------+\n|                   |      Trainning accuracy      |     Testing accuracy      |\n|     Settings      |            Models            |          Models           |\n|                   |     LR      NB      SVM      |    LR      NB      SVM    |\n+===================+==============================+===========================+\n|       N = 3       |  [ 83.397  83.397  83.397]   | [ 83.162  83.162  83.162] |\n+-------------------+------------------------------+---------------------------+\n|      N = 30       |  [ 88.778  83.397  83.397]   | [ 86.254  83.162  83.162] |\n+-------------------+------------------------------+---------------------------+\n|      N = 300      | [ 100.      93.659   90.968] | [ 96.907  92.096  91.065] |\n+-------------------+------------------------------+---------------------------+\n|     N = 3000      | [ 100.      99.116   89.354] | [ 97.938  97.938  88.66 ] |\n+-------------------+------------------------------+---------------------------+\n|     N = 30000     | [ 100.      95.542   88.509] | [ 93.814  94.502  85.911] |\n+-------------------+------------------------------+---------------------------+\n|   Folder: bare    | [ 100.      99.116   89.354] | [ 97.938  97.938  88.66 ] |\n+-------------------+------------------------------+---------------------------+\n|   Folder: stop    | [ 100.      98.885   85.665] | [ 97.595  97.938  85.223] |\n+-------------------+------------------------------+---------------------------+\n|   Folder: lemm    | [ 100.      99.27    89.201] | [ 97.251  98.282  87.973] |\n+-------------------+------------------------------+---------------------------+\n| Folder: lemm_stop | [ 100.      99.039   85.819] | [ 97.595  97.938  84.536] |\n+-------------------+------------------------------+---------------------------+\n"
                }
            ], 
            "source": "from tabulate import tabulate\nvectorsize = np.array(results)*100\nheader = ['\\nSettings', 'Trainning accuracy\\nModels\\nLR      NB      SVM', 'Testing accuracy\\nModels\\nLR      NB      SVM']\nrow = ['N = 3', 'N = 30', 'N = 300', 'N = 3000', 'N = 30000', 'Folder: bare', 'Folder: stop', 'Folder: lemm', 'Folder: lemm_stop']\ntable1 = tabulate(vectorsize, header, tablefmt = 'grid', showindex = row, stralign = 'center')\nprint('============================ SUMMARY OF RESULTS ================================')\nprint(table1)"
        }, 
        {
            "source": "## <font color='green'>Comments</font>\n\n**Regarding hash vector size:** It can be observed that the bigger the hash vector, the better prediction, since differents words are less likely to be assigned to the same position. However, there appears to be no improvement to the accuracy as the vector size exceeds 3000, which can be justified as the vocabulary in each email rarely exceeds 3000. It is worth noting that all classifiers achieved ~84% accuracy even with the hash vector size of 3. This is due to the class imbalance: approximately 84% of the emails are non-spam, which means that the classifier are indicating all emails as non-spam. This emphasised the neccesity of using precision/recall as the main evaluation criteria.\n\n**Regarding lemmatisation and stop word removal:** In general, lemmatisation is expected to improve the models' accuracy as it groups words with _similar_ meanings together - this is consistent with the results obtained. Removing stop words does not significantly affect the result, as stop words are usually considered as containing very few information.\n\n**Regarding classifiers performance:** Logistic regression edged out the remaining two in all settings. Naives Bayes performed best with the hash vector size of 3000. Interestingly, normalizing samples to unit L1 or L2 norm by setting the normalizer parameter to 1 or 2 limits SVM's accuracy to ~84%, while setting this parameter to \"inf\" boosted SVM's accuracy to 90%. This can be regarded as an alternative to tunning SVM's kernel function. The result obtained appears to contradict empirical results, where SVM are proven to be the best classifier, followed by Naive Bayes then Logistic Regression. This might be attributed to the difference in datasets and model tuning.", 
            "cell_type": "markdown", 
            "metadata": {}
        }
    ], 
    "metadata": {
        "kernelspec": {
            "display_name": "Python 3.5 with Spark 2.1", 
            "name": "python3-spark21", 
            "language": "python"
        }, 
        "language_info": {
            "mimetype": "text/x-python", 
            "nbconvert_exporter": "python", 
            "version": "3.5.4", 
            "name": "python", 
            "file_extension": ".py", 
            "pygments_lexer": "ipython3", 
            "codemirror_mode": {
                "version": 3, 
                "name": "ipython"
            }
        }
    }, 
    "nbformat": 4
}