Part 3 - Graph Analytics with Cypher.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "cf78ae6b-77b8-4844-acd9-033b82db1502",
   "metadata": {},
   "source": [
    "This is part 3 of a course called Full Stack Graph Machine Learning by <a href=\"https://linkedin.com/in/russelljurney\">Russell Jurney</a>. For team training inquiries contact <a href=\"mailto:rjurney@graphlet.ai\">rjurney@graphlet.ai</a>."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "5470c409-dd28-44f6-9259-cd7996f08bae",
   "metadata": {},
   "outputs": [],
   "source": [
    "import os\n",
    "from typing import Dict, List\n",
    "\n",
    "import pandas as pd\n",
    "\n",
    "from graphml_class.clean import clean_graph\n",
    "from graphml_class.palette import CATEGORICAL_PALETTE"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a8194767-3613-4c5b-ba2d-48894ecb093e",
   "metadata": {},
   "source": [
    "Part 3 - Graph Analytics with Cypher and Neo4j\n",
    "==============================================\n",
    "\n",
    "<center><img src=\"images/Neo4j-logo_color.png\" width=\"500px\" /></center>\n",
    "\n",
    "<br />\n",
    "\n",
    "In Part 3 of our course, we will use Neo4j and Bloom to learn Graph Analytics.\n",
    "\n",
    "The following resources will assist you in learning Cypher to query Neo4j and other graph databases via [openCypher](https://opencypher.org/) ([github](https://github.com/opencypher)):\n",
    "\n",
    "* [Neo4j Cypher Manual - Overview](https://neo4j.com/docs/cypher-manual/current/introduction/cypher_overview/)\n",
    "* [Neo4j Cypher Manual - Cypher and Neo4j](https://neo4j.com/docs/cypher-manual/current/introduction/cypher_neo4j/)\n",
    "* [Neo4j 5 Cypher Cheat Sheet](https://neo4j.com/docs/cypher-cheat-sheet/5/auradb-enterprise/)\n",
    "* [Neo4j Graph Academy Cypher Fundamentals - 1 Hour Video](https://graphacademy.neo4j.com/courses/cypher-fundamentals/)\n",
    "* [Neo4j Cypher Manual PDF](https://neo4j.com/docs/pdf/neo4j-cypher-manual-5.pdf)\n",
    "* [Neo4j Getting Started - Query a Neo4j database using Cypher](https://neo4j.com/docs/getting-started/cypher-intro/)\n",
    "* [Bite-Sized Neo4j for Data Scientists](https://neo4j.com/video/bite-sized-neo4j-for-data-scientists/) [[github code](https://github.com/cj2001/bite_sized_data_science)]\n",
    "* [Using Neo4j from Python](https://neo4j.com/developer/python/)\n",
    "\n",
    "We are going to call Neo4j from Python in this notebook. For simplicity's sake in setting up the course's software, I build a docker image and we run Neo4j Community on Docker via our [docker-compose.yml](docker-compose.yml) file. Docker only supports Neo4j Community unless you get a license (checkout [Neo4j for Startups](https://neo4j.com/startups/)) If you install [Neo4j Desktop](https://neo4j.com/download/) - which comes with a [developers' license]() to [Neo4j Enterprise]() - you can use [Neo4j Bloom](https://neo4j.com/product/bloom/). It is snazzy! :) It can visualize schemas and query results.\n",
    "\n",
    "# Read Learning Cypher\n",
    "\n",
    "I flash-learned Cypher based on experience with other query languages in a few hours by reading [Learning Cypher](https://learning.oreilly.com/library/view/learning-cypher/9781783287758/) by [Onofrio Panzarino](https://www.linkedin.com/in/onofriopanzarino/). While the resources above are useful, I recommend you buy this book to continue learning to do graph analytics with Neo4j."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ca87e767-b31e-44a2-9467-36d70f5f9fc8",
   "metadata": {},
   "source": [
    "# Connecting to Neo4j from Python\n",
    "\n",
    "We are going to be using two ways to query Cypher.\n",
    "\n",
    "1) The [neo4j](https://pypi.org/project/neo4j/) PyPi module [[github](https://github.com/neo4j/neo4j-python-driver)], [[Neo4j docs](https://neo4j.com/docs/api/python-driver/current/)]\n",
    "2) Neo4j's Graphistry Integration [[example notebook](https://github.com/graphistry/pygraphistry/blob/master/demos/demos_databases_apis/neo4j/official/graphistry_bolt_tutorial_public.ipynb)]\n",
    "\n",
    "We're going to use one or the other as the instructor prefers, depending on how important it is to visualize our results in a table, chart or as a network visualizaton using [Graphistry](https://graphistry.com).\n",
    "\n",
    "## Connecting Using `neo4j` PyPi Package\n",
    "\n",
    "The documentation for this library can be out of date, so I'd recommend consulting the source code on Github at [neo4j/neo4j-python-driver](https://github.com/neo4j/neo4j-python-driver) directly, if you run into trouble. I could not import a `neo4j.GraphDatabase` object... kind of an oversight :) It happens, enterprise software is difficult.\n",
    "\n",
    "This setup uses Docker, which publishes port 7687 to your local machine. Check out the `neo4j` service defined in [docker-compose.yml](docker-compose.yml) if you're curious."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "20503462-fd9e-4fc3-8352-c00c9abc3e48",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<neo4j._sync.driver.Neo4jDriver at 0x7fff95b8c890>"
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from neo4j import GraphDatabase, RoutingControl\n",
    "\n",
    "\n",
    "# Docker sends port 7687 to localhost\n",
    "URI = \"neo4j://localhost:7687\"\n",
    "# We have no authentication setup... in practice you may ;)\n",
    "# AUTH = (\"neo4j\", \"neo4j\")\n",
    "\n",
    "driver = GraphDatabase.driver(URI)\n",
    "driver"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7b51db24-b158-4ce4-8c3a-090eb9c21110",
   "metadata": {},
   "source": [
    "## A `cypher(query)` Utility\n",
    "\n",
    "The `neo4j` API is a little funny, so we write our own utility method `cypher(query)` to run queries and display a `pd.DataFrame`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "80501abe-db9d-49d7-93db-a4fa8c04b8b4",
   "metadata": {},
   "outputs": [],
   "source": [
    "def cypher(query: str) -> pd.DataFrame:\n",
    "    \"\"\"Run a cypher query\"\"\"\n",
    "    \n",
    "    def tx_query(tx):\n",
    "        result = tx.run(query)\n",
    "        return result.data()\n",
    "\n",
    "    with driver.session() as session:\n",
    "        data = session.execute_read(tx_query)\n",
    "\n",
    "    return pd.DataFrame(data)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "bfb31525-d6b6-4323-bcd2-452e8d09a9d0",
   "metadata": {},
   "source": [
    "### Hello World: What Node Labels? Edge Types?\n",
    "\n",
    "There are... a number. We can use a driver session to run a query in a transaction and get results back that we can convert to a pandas `pd.DataFrame`. Cool!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "53d872a4-f8a7-49e5-957a-a0ab118f4cdb",
   "metadata": {},
   "outputs": [],
   "source": [
    "cypher(\"CALL db.labels()\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "6d18db4b-c39e-449f-b5d0-8f112c69f81f",
   "metadata": {},
   "outputs": [],
   "source": [
    "cypher(\"CALL db.relationshipTypes()\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "1510279f-4e82-4654-9c23-44220cb50574",
   "metadata": {},
   "outputs": [],
   "source": [
    "pd.set_option('display.width', 1000)\n",
    "\n",
    "cypher(\"MATCH (n) RETURN DISTINCT labels(n) as Labels, COUNT(*) as Count ORDER BY Count DESC\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "9fdc5e1b-881e-477b-8b01-2e557562fb78",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "id": "6d5df2be-5ba9-462f-829e-3f7f3fa4823e",
   "metadata": {},
   "source": [
    "## Using Graphistry via `Plotter.cypher()`\n",
    "\n",
    "Graphistry has the ability to call Neo4j directlyl using its `graphistry.cypher()` method. First you need to initialize it using Graphistry bolt via `graphistry.register(bolt=NEO4J)`, which I'll show you below :)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "7b897485-7b84-4b65-a868-1711e527cd70",
   "metadata": {},
   "outputs": [],
   "source": [
    "import graphistry"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "id": "96782731-d375-41d1-b687-295b9cacc5ef",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Environment variable setup\n",
    "GRAPHISTRY_USERNAME = os.getenv(\"GRAPHISTRY_USERNAME\")\n",
    "GRAPHISTRY_PASSWORD = os.getenv(\"GRAPHISTRY_PASSWORD\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "id": "7c1ab414-7c71-47bc-ae14-8ac6340fa9e2",
   "metadata": {},
   "outputs": [],
   "source": [
    "graphistry.register(\n",
    "    api=3,\n",
    "    username=GRAPHISTRY_USERNAME,\n",
    "    password=GRAPHISTRY_PASSWORD,\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "id": "90bdf5d4-3bff-42b3-b73f-6971244ea927",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Configuration for Graphistry\n",
    "GRAPHISTRY_PARAMS = {\n",
    "    \"play\": 1000,\n",
    "    \"pointOpacity\": 0.7,\n",
    "    \"edgeOpacity\": 0.3,\n",
    "    \"edgeCurvature\": 0.3,\n",
    "    \"showArrows\": True,\n",
    "    \"gravity\": 0.15,\n",
    "    \"showPointsOfInterestLabel\": False,\n",
    "    \"labels\": {\n",
    "        \"shortenLabels\": False,\n",
    "    },\n",
    "    \"scalingRatio\": 0.5,\n",
    "}\n",
    "FAVICON_URL = \"https://graphlet.ai/assets/icons/favicon.ico\"\n",
    "LOGO = {\"url\": \"https://graphlet.ai/assets/Branding/Graphlet%20AI.svg\", \"dimensions\": {\"maxWidth\": 100, \"maxHeight\": 100}}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "id": "412dd689-5aed-45d9-bd08-ca1abb6c9743",
   "metadata": {},
   "outputs": [],
   "source": [
    "NEO4J = {\n",
    "    'uri': \"bolt://localhost:7687\", \n",
    "}\n",
    "\n",
    "graphistry.register(bolt=NEO4J)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "55ad5a45-df03-4a7d-8399-5b5422fe06a3",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Get the OFAC nodes and render them\n",
    "g = graphistry.cypher(\n",
    "\"\"\"\"\"\"\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "211c0caf-e0e6-4866-96f0-d249eec329fc",
   "metadata": {},
   "outputs": [],
   "source": [
    "g2 = g.bind(\n",
    "    point_title=\"caption\",\n",
    "    point_label=\"caption\",\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "194b9912-a3fa-4446-a7ad-e457f61fd97f",
   "metadata": {},
   "outputs": [],
   "source": [
    "g3 = (\n",
    "    g2\n",
    "    .addStyle(\n",
    "        page={\n",
    "            \"title\": \"Some Open Data\",\n",
    "            \"favicon\": FAVICON_URL\n",
    "        },\n",
    "        logo=LOGO,\n",
    "    )\n",
    "    .settings(\n",
    "        url_params=GRAPHISTRY_PARAMS,\n",
    "        height=800,\n",
    "    )\n",
    ")\n",
    "g3.plot()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a3ee2ef6-3d5a-49fa-bbc2-cecd94e54966",
   "metadata": {},
   "source": [
    "## Describe Our Network\n",
    "\n",
    "When you start working with a new graph database, you need to get oriented by describing the schema, then the network, then sampling some data and finally with some exploratory data analysis. From there you know enough to begin your own workflows.\n",
    "\n",
    "### Describing our Property Graph Model\n",
    "\n",
    "Let's start by describing our network. What kinds of nodes are there?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "2b84ff44-9cf6-4c90-bc49-d38f4557372c",
   "metadata": {},
   "outputs": [],
   "source": [
    "query = \"\"\"\"\"\"\n",
    "\n",
    "def distinct_labels_query(tx):\n",
    "    result = tx.run(query)\n",
    "    return pd.DataFrame.from_dict(result.data(), orient=\"columns\")\n",
    "\n",
    "with driver.session() as session:\n",
    "    df = session.execute_read(distinct_labels_query)\n",
    "\n",
    "df.p.apply(pd.Series)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "56552edd-be73-4cf2-917c-e9c493a34795",
   "metadata": {},
   "outputs": [],
   "source": [
    "df.p.apply(pd.Series)[0].apply(pd.Series)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "fd744190-4c39-4424-adca-ce992d6f4294",
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.11.10"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}