cwpk-35-module-iii-typologies-finish-packaging.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "CWPK \\#35: A Python Module, Part III: Custom Extractions and Finish Packaging\n",
    "=======================================\n",
    "\n",
    "Completing the Extraction Methods and Formal Packaging\n",
    "--------------------------\n",
    "\n",
    "<div style=\"float: left; width: 305px; margin-right: 10px;\">\n",
    "\n",
    "<img src=\"http://kbpedia.org/cwpk-files/cooking-with-kbpedia-305.png\" title=\"Cooking with KBpedia\" width=\"305\" />\n",
    "\n",
    "</div>\n",
    "\n",
    "This last installment in our mini-series of packaging our [*Cooking with Python and KBpedia*](https://www.mkbergman.com/cooking-with-python-and-kbpedia/) project will add one more flexible extraction routine, and complete packaging the *cowpoke* extraction module. This completion will set the template for how we will add additional clusters of functionality as we progress through the **CWPK** series. \n",
    "\n",
    "The new extraction routine is geared more to later analysis and use of [KBpedia](https://kbpedia.org/) than our current extraction routines. The routines we have developed so far are intended for nearly complete bulk extractions of KBpedia. We would like to add more fine-grained ways to extract various 'slices' from KBpedia, especially those that may arise from individual typology changes or [SPARQL](https://en.wikipedia.org/wiki/SPARQL) queries. We could benefit from having an intermediate specification for extractions that could be used for directing results from other functions or analyses."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Steps to Package a Project so Far\n",
    "\n",
    "To summarize, here are the major steps we have discovered so far to transition from prototype code in the notebook to a Python package:\n",
    "\n",
    "1. Generalize the prototype routines in the notebook\n",
    "1. Find the appropriate <code>.. Lib/site-packages</code> directory and define a new directory with your package name, short lowercase best\n",
    "1. Create an <code>\\_\\_init\\_\\_.py</code> file in that same directory (and any subsequent sub-package directories should you define the project deeper); add basic statements similar to above\n",
    "1. Create a <code>\\_\\_main\\_\\_.py</code> file for your shared functions; it is a toss-up whether shared variables should also go here or in <code>\\_\\_init\\_\\_.py</code>\n",
    "1. If you want a single point of editing for changing inputs for a given run via a 'record' metaphor, follow something akin to what I began with the <code>config.py</code>\n",
    "1. Create a <code>my_new_functions.py</code> module where the bulk of your new work resides (extraction in our current instance). The approach I found helpful was to ***NOT*** wrap my prototype functions in a function definition at first. Only in the next step, once the interpreter is able to step through the new functions without errors, do I then wrap the  routines in definitions\n",
    "1. In an interactive environment (such as Jupyter Notebook), start with a clean kernel and try to <code>import myproject</code> (where 'myproject' is the current bundle of functionality you are working on). Remember, an import will run the scripts as encountered, and if you have points of failure due to undefined variables or whatever, the traceback on the interpreter will tell you what the problem is and offer some brief diagnostics. Repeat until the new function code processes without error, then wrap in a definition, and move on to the next code block\n",
    "1. As defined functions are built, try to look for generalities of function and specification and desired inputs and outputs. These provide the grist for continued re-factoring of your code\n",
    "1. Document your code, which is only just beginning for this KBpedia project. I've been documenting much in the notebook pages, but not yet enough in the code itself."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Objectives for This Installment\n",
    "Here is what I want to accomplish and wrap up in this installment focusing on custom extractions. I want to:\n",
    "\n",
    "- Arbitrarily select one to many classes or properties for driving an extraction\n",
    "- Define one or multiple starting points for descendants() or one or multiple individual starting points. This entry point is provided by the variable <code>root</code> in our existing extraction routines\n",
    "- Allow the <code>extract_deck</code> specification to also define the rendering method (see next)\n",
    "- For iterations, specify input and output file forms, so need: iterator, base + extension logic, relate to existing annotations, etc., as necessary\n",
    "- This suggests to reuse and build from the existing extraction routines, and\n",
    "- Improve the file-based and -named orientation of the routines."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### The Render Function\n",
    "You may recall from the tip in [**CWPK #29**](https://www.mkbergman.com/2364/cwpk-29-extracting-object-and-data-properties/) that [owlready2](http://www.lesfleursdunormal.fr/static/informatique/owlready/index_en.html) comes with three rendering methods for its results: 1) a *default* method that has a short namespace prefix appended to all classes and properties; 2) a *label* method where no prefixes are provided; and 3) a full *iri* method where all three components of the *subject-predicate-object* (*s-p-o*) [semantic triple](https://en.wikipedia.org/wiki/Semantic_triple) are given their complete [IRI](https://en.wikipedia.org/wiki/Internationalized_Resource_Identifier). If you recall, here are those three function calls:\n",
    "\n",
    "<pre>\n",
    "set_render_func(default_render_func)\n",
    "\n",
    "set_render_func(render_using_label)\n",
    "\n",
    "set_render_func(render_using_iri)\n",
    "</pre>\n",
    "\n",
    "To provide these choices, we will add a <code>render</code> method specification in the <code>extract_deck</code> and provide the switch at the top of our extraction routines. We will use the same italicized names to specify which of the three rendering options has been chosen."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### A Custom Extractor\n",
    "The basic realization is that with just a few additions we are able to allow customization of our existing extraction routines. Initially, I thought I would need to write entirely new routines. But, fortunately, apparently our existing routines already are sufficiently general to enable this customization.\n",
    "\n",
    "Since it is a bit simpler, we will use the <code>struct_extractor</code> function to show where the customization enhancements need to go. We will also provide the code snippet where the insertion is noted. I provide comments on these additions below the code listing.\n",
    "\n",
    "Note these same changes are applied to the <code>annot_extractor</code> function as well (not shown). You can inspect the updated extraction module at the conclusion of this installment.\n",
    "\n",
    "OK, so let's explain these customizations:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def struct_extractor(**extract_deck):\n",
    "    print('Beginning structure extraction . . .')\n",
    "# 1 - render method goes here    \n",
    "    r_default = ''\n",
    "    r_label = ''\n",
    "    r_iri = ''\n",
    "    render = extract_deck.get('render')\n",
    "    if render == 'r_default':\n",
    "        set_render_func(default_render_func)\n",
    "    elif render == 'r_label':\n",
    "        set_render_func(render_using_label)\n",
    "    elif render == 'r_iri':\n",
    "        set_render_func(render_using_iri)\n",
    "    else:\n",
    "        print('You have assigned an incorrect render method--execution stopping.')\n",
    "        return\n",
    "# 2 - note about custom extractions\n",
    "    loop_list = extract_deck.get('loop_list')\n",
    "    loop = extract_deck.get('loop')\n",
    "    out_file = extract_deck.get('out_file')\n",
    "    class_loop = extract_deck.get('class_loop')\n",
    "    property_loop = extract_deck.get('property_loop')\n",
    "    descent_type = extract_deck.get('descent_type')\n",
    "    descent = extract_deck.get('descent') \n",
    "    single = extract_deck.get('single') \n",
    "    x = 1\n",
    "    cur_list = []\n",
    "    a_set = []\n",
    "    s_set = []\n",
    "    new_class = 'owl:Thing'\n",
    "# 5 - what gets passed to 'output'\n",
    "    with open(out_file, mode='w', encoding='utf8', newline='') as output:\n",
    "        csv_out = csv.writer(output)\n",
    "        if loop == 'class_loop':                                             \n",
    "            header = ['id', 'subClassOf', 'parent']\n",
    "            p_item = 'rdfs:subClassOf'\n",
    "        else:\n",
    "            header = ['id', 'subPropertyOf', 'parent']\n",
    "            p_item = 'rdfs:subPropertyOf'\n",
    "        csv_out.writerow(header)       \n",
    "# 3 - what gets passed to 'loop_list' \n",
    "        for value in loop_list:\n",
    "            print('   . . . processing', value)                                           \n",
    "            root = eval(value)\n",
    "# 4 - descendant or single here\n",
    "            if descent_type == 'descent':\n",
    "                a_set = root.descendants()\n",
    "                a_set = set(a_set)\n",
    "                s_set = a_set.union(s_set)\n",
    "            elif descent_type == 'single':\n",
    "                a_set = root\n",
    "                s_set.append(a_set)\n",
    "            else:\n",
    "                print('You have assigned an incorrect descent method--execution stopping.')\n",
    "                return                         \n",
    "        print('   . . . processing consolidated set.')\n",
    "        for s_item in s_set:\n",
    "            o_set = s_item.is_a\n",
    "            for o_item in o_set:\n",
    "                row_out = (s_item,p_item,o_item)\n",
    "                csv_out.writerow(row_out)\n",
    "                if loop == 'class_loop':\n",
    "                    if s_item not in cur_list:                \n",
    "                        row_out = (s_item,p_item,new_class)\n",
    "                        csv_out.writerow(row_out)\n",
    "                cur_list.append(s_item)\n",
    "                x = x + 1\n",
    "    print('Total unique IDs written to file:', x) "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The notes that follow pertain to the code listing above. \n",
    "\n",
    "The render method (**#1**) is just a simple switch set in the configuration file. Only three keyword options are allowed; if a wrong keyword is entered, the error is flagged and the routine ends. We also added a 'render' assignment at the top of the code block.\n",
    "\n",
    "What now makes this routine (**#2**) a custom one is the use of the configurable <code>custom_dict</code> and its configuration settings. The <code>custom_dict</code> dictionary is specified by assigning to the <code>loop_list</code> (**#3**). The <code>custom_dict</code> dictionary can take one or many *key:value* pairs. The first item, the *key*, should take the name that you wish to use as the internal variable name. The second item, the *value*, should correspond to the property or class with its namespace prefix. Here are the general rules and options available for a custom extraction:\n",
    "\n",
    "- You may enter properties **OR** classes into the <code>custom_dict</code> dictionary, but not both, in your pre-run configurations\n",
    "- The '<code>iri</code>' switch for the renderer is best suited for the <code>struct_extractor</code> function. It should probably not be used for annotations given the large number of output columns and loss of subsequent readability when using the full IRI. The choice of actual prefix is likely not that important since it is easy to do global search-and-replaces when in bulk mode\n",
    "- You may retrieve items in the <code>custom_dict</code> dictionary either singly or all of its descendants, depending on the use of the 'single' and 'descent' keyword options (see **#4** next). \n",
    "\n",
    "Item **#4** is another switch to either run the entries in the <code>custom_dict</code> dictionary as single 'roots' (thus no sub-classes or sub-properties) or with all descendants. The <code>descent_type</code> has been added to the <code>extract_deck</code> settings, plus we added the related assignments to the beginning of this code block.\n",
    "\n",
    "The last generalized capability we wanted to capture was the ability to print out all of the structural aspects of KBpedia's typologies, which suggested some code changes at roughly **#5** above. While I am sure I could have figured out a way to do this, because of interactions with the other customizations this addition proved to be more complicated than warranted. So, rather than spend undue time trying to cram everything into a single, generic function (<code>struct_extractor</code>), I decided the easier and quicker choice was to create its own function, picking up on many of the processing constructs developed for the other extractor routines.\n",
    "\n",
    "Basically, what we want in a typology extract is:\n",
    "\n",
    "- Separate extractions of individual typologies to their own named files\n",
    "- Removal of the need to find unique resources across multiple typologies. Rather, the intent is to capture the full scope of structural (<code>subClassOf</code> aspects in each typology\n",
    "- A design that enables us to load a typology as an individual ontology or knowledge graph into a tool such as [Protégé](https://en.wikipedia.org/wiki/Prot%C3%A9g%C3%A9_(software)).\n",
    "\n",
    "By focusing on a special extractor limited to classes, typologies, structure, and single output files per typology, we were able to make the function rather quickly and simply. Here is the result, the <code>typol_extractor</code>:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def typol_extractor(**extract_deck):\n",
    "    print('Beginning structure extraction . . .')\n",
    "    r_default = ''\n",
    "    r_label = ''\n",
    "    r_iri = ''\n",
    "    render = extract_deck.get('render')\n",
    "    if render == 'r_default':\n",
    "        set_render_func(default_render_func)\n",
    "    elif render == 'r_label':\n",
    "        set_render_func(render_using_label)\n",
    "    elif render == 'r_iri':\n",
    "        set_render_func(render_using_iri)\n",
    "    else:\n",
    "        print('You have assigned an incorrect render method--execution stopping.')\n",
    "        return\n",
    "    loop_list = extract_deck.get('loop_list')\n",
    "    loop = extract_deck.get('loop')\n",
    "    class_loop = extract_deck.get('class_loop')\n",
    "    base = extract_deck.get('base')\n",
    "    ext = extract_deck.get('ext')\n",
    "    new_class = 'owl:Thing'\n",
    "    if loop is not 'class_loop':\n",
    "        print(\"Needs to be a 'class_loop'; returning program.\")\n",
    "        return\n",
    "    header = ['id', 'subClassOf', 'parent']\n",
    "    p_item = 'rdfs:subClassOf'\n",
    "    for value in loop_list:\n",
    "        print('   . . . processing', value)\n",
    "        x = 1\n",
    "        s_set = []\n",
    "        cur_list = []\n",
    "        root = eval(value)\n",
    "        s_set = root.descendants()\n",
    "        frag = value.replace('kko.','')\n",
    "        out_file = (base + frag + ext)\n",
    "        with open(out_file, mode='w', encoding='utf8', newline='') as output:                                           \n",
    "            csv_out = csv.writer(output)\n",
    "            csv_out.writerow(header)       \n",
    "            for s_item in s_set:\n",
    "                o_set = s_item.is_a\n",
    "                for o_item in o_set:\n",
    "                    row_out = (s_item,p_item,o_item)\n",
    "                    csv_out.writerow(row_out)\n",
    "                    if s_item not in cur_list:                \n",
    "                        row_out = (s_item,p_item,new_class)\n",
    "                        csv_out.writerow(row_out)\n",
    "                cur_list.append(s_item)\n",
    "                x = x + 1\n",
    "        output.close()         \n",
    "        print('Total unique IDs written to file:', x)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Two absolute essentials for this routine are to set the <code>'loop'</code> key to <code>'class_loop'</code> and to set the <code>'loop_list'</code> key to <code>typol_dict.values()</code>.\n",
    "\n",
    "Note the code in the middle of the routine that creates the file name after replacing (removing) the 'kko.' prefix from the value name in the dictionary. We also needed to add two further entries to the <code>extract_deck</code> dictionary.\n",
    "\n",
    "With the caveat that your local file structure is likely different than what we set up for this project, should it be similar the following commands can be used to run these routines. Should you test different possibilities, make sure your input specifications in the <code>extract_deck</code> are modified appropriately. Remember, to always work from copies so that you may restore critical files in the case of an inadvertent overwrite.\n",
    "\n",
    "Here are the commands:\n",
    "\n",
    "<pre>\n",
    "from cowpoke.__main__ import *\n",
    "from cowpoke.config import *\n",
    "import cowpoke\n",
    "import owlready2\n",
    "\n",
    "cowpoke.typol_extractor(**cowpoke.extract_deck)\n",
    "</pre>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### The extract.py File\n",
    "Again, assuming you have set up your files and directories similar to what we have suggested, you can inspect the resulting extractor code in this new module (modify the path as necessary):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "with open(r'C:\\1-PythonProjects\\Python\\Lib\\site-packages\\cowpoke\\extract.py', 'r') as f:\n",
    "    print(f.read())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Summary of the Module\n",
    "\n",
    "OK, so we are now done with the development and packaging of the extractor module for *cowpoke*. Our efforts resulted in the addition of four files under the 'cowpoke' directory. These files are:\n",
    "\n",
    "- The <code>\\_\\_init\\_\\_.py</code> file that indicates the *cowpoke* package\n",
    "- The <code>\\_\\_main\\_\\_.py</code> file where shared start-up functions reside\n",
    "- The <code>config.py</code> file where we store our dictionaries and where we specify new run settings in the special <code>extract_deck</code> dictionary, and\n",
    "- The <code>extract.py</code> module where all of our extraction routines are housed.\n",
    "\n",
    "This module is supported by three dictionaries (and the fourth special one for the run configurations):\n",
    "\n",
    "- The <code>typol_dict</code> dictionary of typologies\n",
    "- The <code>prop_dict</code> dictionary of top-level property roots\n",
    "- The <code>custom_dict</code> dictionary for tailored starting point extractions, and\n",
    "- The <code>extract_deck</code> special dictionary for extraction run settings.\n",
    "\n",
    "In turn, most of these dictionaries can also be matched with three different extractor routines or functions:\n",
    "\n",
    "- The <code>annot_extractor</code> function for extracting annotations\n",
    "- The <code>struct_extractor</code> function for extracting the *is-a* relations in KBpedia, and\n",
    "- The <code>typol_extractor</code> dedicated function for extracting out the individual typologies into individual files.\n",
    "\n",
    "In our next **CWPK** installment we will discuss how we might manipulate this extracted information in a bulk manner using spreadsheets and other tools. These same extracted files, perhaps after bulk manipulations or other edits and changes, will then form the basis for the input files that we will use to build new versions of KBpedia (or your own extensions and changes to it) from scratch. We are now half-way around our [roundtrip](https://en.wikipedia.org/wiki/Round-trip_format_conversion).\n",
    "\n",
    "\n",
    " <div style=\"background-color:#efefff; border:1px dotted #ceceff; vertical-align:middle; margin:15px 60px; padding:8px;\"> \n",
    "  <span style=\"font-weight: bold;\">NOTE:</span> This article is part of the <a href=\"https://www.mkbergman.com/cooking-with-python-and-kbpedia/\" style=\"font-style: italic;\">Cooking with Python and KBpedia</a> series. See the <a href=\"https://www.mkbergman.com/cooking-with-python-and-kbpedia/\"><strong>CWPK</strong> listing</a> for other articles in the series. <a href=\"http://kbpedia.org/\">KBpedia</a> has its own Web site.\n",
    "  </div>\n",
    "\n",
    "<div style=\"background-color:#ebf8e2; border:1px dotted #71c837; vertical-align:middle; margin:15px 60px; padding:8px;\"> \n",
    "\n",
    "<span style=\"font-weight: bold;\">NOTE:</span> This <strong>CWPK \n",
    "installment</strong> is available both as an online interactive\n",
    "file <a href=\"https://mybinder.org/v2/gh/Cognonto/CWPK/master\" ><img src=\"https://mybinder.org/badge_logo.svg\" style=\"display:inline-block; vertical-align: middle;\" /></a> or as a <a href=\"https://github.com/Cognonto/CWPK\" title=\"CWPK notebook\" alt=\"CWPK notebook\">direct download</a> to use locally. Make sure and pick the correct installment number. For the online interactive option, pick the <code>*.ipynb</code> file. It may take a bit of time for the interactive option to load.</div>\n",
    "\n",
    "<div style=\"background-color:#feeedc; border:1px dotted #f7941d; vertical-align:middle; margin:15px 60px; padding:8px;\"> \n",
    "<div style=\"float: left; margin-right: 5px;\"><img src=\"http://kbpedia.org/cwpk-files/warning.png\" title=\"Caution!\" width=\"32\" /></div>I am at best an amateur with Python. There are likely more efficient methods for coding these steps than what I provide. I encourage you to experiment -- which is part of the fun of Python -- and to <a href=\"mailto:mike@mkbergman.com\">notify me</a> should you make improvements.    \n",
    "\n",
    "</div>"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}