notebooks/data_science_tool1_midterms.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Data Science tools1 midterm winter 2019. \n",
    "- Total Time **1.45 Hours** 7.15 p.m - 9 p.m.\n",
    "- Please upload completed notebook to canvas before 9.05 p.m\n",
    "- You are allowed to use any thing to solve problem in this notebook\n",
    "- Talking online or to other person is not allowed\n",
    "- Cell with comment like  *write code, write command* will be graded only. Other cell are there to help you."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Q1(2 points) If you go to the site\n",
    "\n",
    "https://www.flydenver.com/security\n",
    "\n",
    "You will see **security wait time** on top left hand side.\n",
    "\n",
    "Using curl and grep find the portion of html source listing **table tag** for security wait line\n",
    "\n",
    "Hint: First line of output should start with **<table class ....\\>** and last line should be **</table\\>**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Write command pipeline here\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Q2(2 points) Apply grep with regular expression to the output of previous commands to find only wait time.\n",
    "\n",
    "Hint: output should be  similar to\n",
    "\n",
    "10 min<br>\n",
    "8 min<br>\n",
    ".<br>\n",
    "."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# write commands pipeline here\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Q3(2= 1+1 points) Redirect the output of Q1 to a file named security_line.html.  Using Pandas read api, read this file in pandas and display the DataFrame of this table."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Write the commands to redirect to file here"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# write pandas related code here"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Q4 (2.5 points)\n",
    "Let's say you for a e-commerce company. Company has 10 installment plans for some of its products. A customer is enrolled when he pays first installment. Installment is closed when he pays last installment. Due to a bug, some of the customer are getting charged a penny less in their last installment leaving their enrollment open.\n",
    "\n",
    "Assume that company has products with **integer** price value like \\\\$15 , \\\\$50.\n",
    "Your job is to figure out which customer are getting charged a penny less and manually close the enrollment.\n",
    "\n",
    "Using Commands, and payment_gateway.logxx file,  print **customer id, inst column and amount charged**.\n",
    "\n",
    "<font color='red'> Please run the following bash cell magic to create payment_gateway.logxx file </font>\n",
    "\n",
    "Hint:\n",
    "You output should look like\n",
    "\n",
    "2150:inst 10 charged:$15.99"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%bash\n",
    "echo \"customer:2150:credit_card:xxxx-xxxx-xxxx:inst 10 charged:\\$15.99\n",
    "customer:2350:credit_card:xxxx-xxxx-xxxx:inst 9 charged:\\$2\n",
    "Paymenet gateway: STATUS 200\n",
    "customer:3456 payment failed\n",
    "customer:3350:credit_card:xxxx-xxxx-xxxx:inst 1 charged:\\$7\n",
    "customer:4350:credit_card:xxxx-xxxx-xxxx:inst 10 charged:\\$3.99\n",
    "customer:4355:credit_card:xxxx-xxxx-xxxx:inst 10 charged:\\$5.49\n",
    "customer:2350:credit_card:xxxx-xxxx-xxxx:inst 2 charged:\\$2\n",
    "customer:2150:credit_card:xxxx-xxxx-xxxx:inst 3 charged:\\$16\n",
    "customer:2220:credit_card:xxxx-xxxx-xxxx:inst 10 charged:\\$5.5\n",
    "customer:3350:credit_card:xxxx-xxxx-xxxx:inst 4 charged:\\$7\n",
    "customer:2351:credit_card:xxxx-xxxx-xxxx:inst 10 charged:\\$9\n",
    "customer:2350:credit_card:xxxx-xxxx-xxxx:inst 6 charged:\\$2\n",
    "customer:6350:credit_card:xxxx-xxxx-xxxx:inst 10 charged:\\$2.49\n",
    "customer:23350:credit_card:xxxx-xxxx-xxxx:inst 10 charged:\\$16.99\" > payment_gateway.logxx"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!cat payment_gateway.logxx"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# write command pipeline here\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Q5(1 point) Look for github developer API documentation and see how to use *issue* endpoint. We want to get all the issues with data_science_tools1 repository (https://github.com/psnegi/data_science_tools1/issues ).\n",
    "\n",
    "build the api endpoint and get **all** the issues from this repository. Check the return status code. It has to be 200."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import requests\n",
    "# write your code here to build endpoint and make request"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# pretty print your resposne json to get an idea of returned json\n",
    "# inside pprint.pprint write your response.json()\n",
    "# It should be  list of dictionary\n",
    "import pprint\n",
    "pprint.pprint(???)# remove ??? and write your response json"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Q6 (2 point)  Using json from the  response build a new list of dictionary(*modifed_issues*). In the new dictionary only store *body, title, comments, created_at, closed_at and state* from the response json."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "\n",
    "modifed_issues = []\n",
    "#Write code to fill this list with modified dictionaries\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Check the list of dictionary. Each item in the list should be a dictionary with above mentioned keys.\n",
    "modifed_issues"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Q7 (2= .5 +.5+.5+.5)\n",
    "1. Build a panda DataFrame using  *modifed_issues* list.\n",
    "1. data type of closed_at and created_at are object type. We can't do any time arithmetic on these fields.Using pandas pandas.to_datetime  change datatype from object to datetime\n",
    "1. create a new field **time_to_close** which calculated the time it took to close an issue since opening.\n",
    "1. Calculate mean time to close issues in this repository."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Q8(2 = 1(.5+.5) + 1) Printing file name,  file size in byte, number of characters excluding white space in all the file with extension logxx. \n",
    "\n",
    "<font color='red'>Fill in the places marked with #??? to correct this script </font>\n",
    "\n",
    "Hint\n",
    "You output should look like\n",
    "\n",
    "payment_gateway2.logxx, 22, 18"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here is how we get the length of  string or word in bash"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%bash\n",
    "aa='dataset'\n",
    "echo \"Length of word $aa is ${#aa}\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#First ating one more file with extension logxx\n",
    "echo -e \"test line1\\ntest line2\" > payment_gateway2.logxx"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Finish the script in the next cell to print file name, file size in bytes and total word or character excluding spaces."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%bash\n",
    "\n",
    "file_info=`ls -l *.logxx |awk '  {print $5 \" \"  $NF}'`\n",
    "\n",
    "echo -e \"File size, # character, # words, # lines\\n\"\n",
    "\n",
    "echo \"$file_info\"|while read line\n",
    "do\n",
    " size=#???write code  to get size from line\n",
    " name=#???write code to  get the name from the variable line\n",
    " \n",
    " total_char=0\n",
    " for words in `cat $name`\n",
    "  do\n",
    "    total_char=#??? write correct expression to calculate total chracter\n",
    "  done\n",
    " echo $size, $name, $total_char\n",
    "done \n",
    "\n"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.5.2"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}