examples/08_recommending_for_categories.ipynb

{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-02-10T16:01:45.639135Z",
     "start_time": "2020-02-10T16:01:45.612577Z"
    },
    "jupyter": {
     "outputs_hidden": false
    },
    "pycharm": {
     "is_executing": false
    }
   },
   "outputs": [],
   "source": [
    "%load_ext autoreload\n",
    "%autoreload 2\n",
    "%config Completer.use_jedi = False"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "! pip install rs-datasets -q"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# The notebook shows an example of recommendation for product categories with CatPopRec model\n",
    "We will use movielens dataset and will consider genres as prodict categories. An item may belongs to several categories, but an interation is parametrized with one category. Assume the user select category at first and then select an item. CatPopRec calculates popularity of an item in categories and recommends the most popular items. \n",
    "\n",
    "Categories may be flat, but also may have structure of a tree. In this case we assume, that the category has only one parent and all items are mapped to only bottom-level (leaf) categories. CatPopRec allows to build recommendations not only for bottom level, but for any category, aggregating items' popularities in downstream categories.  \n",
    "1. Data loading and reindexing\n",
    "2. Categories preprocessing\n",
    "3. Assigning categories to interactions\n",
    "4. Recommending in case of flat categories structure\n",
    "6. Building categories tree\n",
    "4. Recommending in case of tree categories structure"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<style type='text/css'>\n",
       ".datatable table.frame { margin-bottom: 0; }\n",
       ".datatable table.frame thead { border-bottom: none; }\n",
       ".datatable table.frame tr.coltypes td {  color: #FFFFFF;  line-height: 6px;  padding: 0 0.5em;}\n",
       ".datatable .bool    { background: #DDDD99; }\n",
       ".datatable .object  { background: #565656; }\n",
       ".datatable .int     { background: #5D9E5D; }\n",
       ".datatable .float   { background: #4040CC; }\n",
       ".datatable .str     { background: #CC4040; }\n",
       ".datatable .time    { background: #40CC40; }\n",
       ".datatable .row_index {  background: var(--jp-border-color3);  border-right: 1px solid var(--jp-border-color0);  color: var(--jp-ui-font-color3);  font-size: 9px;}\n",
       ".datatable .frame tbody td { text-align: left; }\n",
       ".datatable .frame tr.coltypes .row_index {  background: var(--jp-border-color0);}\n",
       ".datatable th:nth-child(2) { padding-left: 12px; }\n",
       ".datatable .hellipsis {  color: var(--jp-cell-editor-border-color);}\n",
       ".datatable .vellipsis {  background: var(--jp-layout-color0);  color: var(--jp-cell-editor-border-color);}\n",
       ".datatable .na {  color: var(--jp-cell-editor-border-color);  font-size: 80%;}\n",
       ".datatable .sp {  opacity: 0.25;}\n",
       ".datatable .footer { font-size: 9px; }\n",
       ".datatable .frame_dimensions {  background: var(--jp-border-color3);  border-top: 1px solid var(--jp-border-color0);  color: var(--jp-ui-font-color3);  display: inline-block;  opacity: 0.6;  padding: 1px 10px 1px 5px;}\n",
       "</style>\n"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "from pyspark.sql import Window\n",
    "from pyspark.sql.functions import split\n",
    "import pyspark.sql.functions as sf\n",
    "\n",
    "from replay.data import Dataset, FeatureSchema, FeatureInfo, FeatureHint, FeatureType\n",
    "from replay.preprocessing.label_encoder import LabelEncoder, LabelEncodingRule\n",
    "from replay.models import CatPopRec\n",
    "from replay.utils.session_handler import State\n",
    "from replay.splitters import TimeSplitter\n",
    "from replay.utils.spark_utils import get_log_info, convert2spark\n",
    "from rs_datasets import MovieLens"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-02-10T15:59:09.227179Z",
     "start_time": "2020-02-10T15:59:06.427348Z"
    },
    "jupyter": {
     "outputs_hidden": false
    },
    "pycharm": {
     "is_executing": false
    }
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "23/11/08 22:30:12 WARN Utils: Your hostname, ecs-eemalov-large resolves to a loopback address: 127.0.1.1; using 10.11.10.44 instead (on interface eth0)\n",
      "23/11/08 22:30:12 WARN Utils: Set SPARK_LOCAL_IP if you need to bind to another address\n",
      "WARNING: An illegal reflective access operation has occurred\n",
      "WARNING: Illegal reflective access by org.apache.spark.unsafe.Platform (file:/root/anaconda3/envs/replay/lib/python3.9/site-packages/pyspark/jars/spark-unsafe_2.12-3.2.4.jar) to constructor java.nio.DirectByteBuffer(long,int)\n",
      "WARNING: Please consider reporting this to the maintainers of org.apache.spark.unsafe.Platform\n",
      "WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations\n",
      "WARNING: All illegal access operations will be denied in a future release\n",
      "Using Spark's default log4j profile: org/apache/spark/log4j-defaults.properties\n",
      "Setting default log level to \"WARN\".\n",
      "To adjust logging level use sc.setLogLevel(newLevel). For SparkR, use setLogLevel(newLevel).\n",
      "23/11/08 22:30:13 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable\n",
      "23/11/08 22:30:13 WARN SparkConf: Note that spark.local.dir will be overridden by the value set by the cluster manager (via SPARK_LOCAL_DIRS in mesos/standalone/kubernetes and LOCAL_DIRS in YARN).\n",
      "23/11/08 22:30:14 WARN Utils: Service 'SparkUI' could not bind on port 4040. Attempting port 4041.\n",
      "23/11/08 22:30:14 WARN Utils: Service 'SparkUI' could not bind on port 4041. Attempting port 4042.\n",
      "23/11/08 22:30:14 WARN Utils: Service 'SparkUI' could not bind on port 4042. Attempting port 4043.\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "\n",
       "            <div>\n",
       "                <p><b>SparkSession - hive</b></p>\n",
       "                \n",
       "        <div>\n",
       "            <p><b>SparkContext</b></p>\n",
       "\n",
       "            <p><a href=\"http://localhost:4043\">Spark UI</a></p>\n",
       "\n",
       "            <dl>\n",
       "              <dt>Version</dt>\n",
       "                <dd><code>v3.2.4</code></dd>\n",
       "              <dt>Master</dt>\n",
       "                <dd><code>local[*]</code></dd>\n",
       "              <dt>AppName</dt>\n",
       "                <dd><code>pyspark-shell</code></dd>\n",
       "            </dl>\n",
       "        </div>\n",
       "        \n",
       "            </div>\n",
       "        "
      ],
      "text/plain": [
       "<pyspark.sql.session.SparkSession at 0x7f0774ebcca0>"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "spark = State().session\n",
    "spark"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "spark.sparkContext.setLogLevel('ERROR')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 1. Data loading"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We will use MovieLens 10m dataset from rs_datasets package, which contains a list of recommendations datasets."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "ratings\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>user_id</th>\n",
       "      <th>item_id</th>\n",
       "      <th>rating</th>\n",
       "      <th>timestamp</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1</td>\n",
       "      <td>122</td>\n",
       "      <td>5.0</td>\n",
       "      <td>838985046</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>1</td>\n",
       "      <td>185</td>\n",
       "      <td>5.0</td>\n",
       "      <td>838983525</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>1</td>\n",
       "      <td>231</td>\n",
       "      <td>5.0</td>\n",
       "      <td>838983392</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   user_id  item_id  rating  timestamp\n",
       "0        1      122     5.0  838985046\n",
       "1        1      185     5.0  838983525\n",
       "2        1      231     5.0  838983392"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "items\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>item_id</th>\n",
       "      <th>title</th>\n",
       "      <th>genres</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1</td>\n",
       "      <td>Toy Story (1995)</td>\n",
       "      <td>Adventure|Animation|Children|Comedy|Fantasy</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>2</td>\n",
       "      <td>Jumanji (1995)</td>\n",
       "      <td>Adventure|Children|Fantasy</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>3</td>\n",
       "      <td>Grumpier Old Men (1995)</td>\n",
       "      <td>Comedy|Romance</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   item_id                    title  \\\n",
       "0        1         Toy Story (1995)   \n",
       "1        2           Jumanji (1995)   \n",
       "2        3  Grumpier Old Men (1995)   \n",
       "\n",
       "                                        genres  \n",
       "0  Adventure|Animation|Children|Comedy|Fantasy  \n",
       "1                   Adventure|Children|Fantasy  \n",
       "2                               Comedy|Romance  "
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "tags\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>user_id</th>\n",
       "      <th>item_id</th>\n",
       "      <th>tag</th>\n",
       "      <th>timestamp</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>15</td>\n",
       "      <td>4973</td>\n",
       "      <td>excellent!</td>\n",
       "      <td>1215184630</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>20</td>\n",
       "      <td>1747</td>\n",
       "      <td>politics</td>\n",
       "      <td>1188263867</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>20</td>\n",
       "      <td>1747</td>\n",
       "      <td>satire</td>\n",
       "      <td>1188263867</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   user_id  item_id         tag   timestamp\n",
       "0       15     4973  excellent!  1215184630\n",
       "1       20     1747    politics  1188263867\n",
       "2       20     1747      satire  1188263867"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n"
     ]
    }
   ],
   "source": [
    "data = MovieLens(\"10m\")\n",
    "data.info()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 1.1 Convert interaction log to RePlay format"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CPU times: user 78 ms, sys: 129 ms, total: 207 ms\n",
      "Wall time: 1.64 s\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "interactions = convert2spark(data.ratings)\n",
    "item_features = convert2spark(data.items)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "+-------+-------+------+---------+\n",
      "|user_id|item_id|rating|timestamp|\n",
      "+-------+-------+------+---------+\n",
      "|      1|    122|   5.0|838985046|\n",
      "|      1|    185|   5.0|838983525|\n",
      "|      1|    231|   5.0|838983392|\n",
      "|      1|    292|   5.0|838983421|\n",
      "|      1|    316|   5.0|838983392|\n",
      "+-------+-------+------+---------+\n",
      "only showing top 5 rows\n",
      "\n"
     ]
    }
   ],
   "source": [
    "interactions.show(5)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "+-------+----------------+--------------------+\n",
      "|item_id|           title|              genres|\n",
      "+-------+----------------+--------------------+\n",
      "|      1|Toy Story (1995)|Adventure|Animati...|\n",
      "|      2|  Jumanji (1995)|Adventure|Childre...|\n",
      "+-------+----------------+--------------------+\n",
      "only showing top 2 rows\n",
      "\n"
     ]
    }
   ],
   "source": [
    "item_features.show(2)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id='indexing'></a>\n",
    "### 1.2. Indexing"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Convert given users' and items' identifiers (\\_id) to integers starting at zero without gaps (\\_idx) with Indexer class."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "user_encoder = LabelEncoder([LabelEncodingRule(\"user_id\")])\n",
    "item_encoder = LabelEncoder([LabelEncodingRule(\"item_id\")])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "                                                                                \r"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CPU times: user 399 ms, sys: 25.8 ms, total: 425 ms\n",
      "Wall time: 5 s\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "<replay.preprocessing.label_encoder.LabelEncoder at 0x7f07764fbe20>"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "%%time\n",
    "user_encoder.fit(interactions)\n",
    "item_encoder.fit(interactions.select('item_id').union(item_features.select('item_id')))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "                                                                                \r"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "+------+---------+-------+-------+\n",
      "|rating|timestamp|user_id|item_id|\n",
      "+------+---------+-------+-------+\n",
      "|   5.0|838985046|  27764|   5178|\n",
      "|   5.0|838983525|  27764|   4962|\n",
      "+------+---------+-------+-------+\n",
      "only showing top 2 rows\n",
      "\n",
      "CPU times: user 888 ms, sys: 6.62 ms, total: 894 ms\n",
      "Wall time: 4.88 s\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "log_replay = user_encoder.transform(df=interactions)\n",
    "log_replay = item_encoder.transform(df=log_replay)\n",
    "log_replay.show(2)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "+--------------------+--------------------+-------+\n",
      "|               title|              genres|item_id|\n",
      "+--------------------+--------------------+-------+\n",
      "|Grumpier Old Men ...|      Comedy|Romance|    692|\n",
      "|      Jumanji (1995)|Adventure|Childre...|   1349|\n",
      "+--------------------+--------------------+-------+\n",
      "only showing top 2 rows\n",
      "\n",
      "CPU times: user 118 ms, sys: 3.13 ms, total: 121 ms\n",
      "Wall time: 548 ms\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "item_features_replay = item_encoder.transform(df=item_features)\n",
    "item_features_replay.show(2)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 1.3. Data split"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-02-10T15:59:50.986401Z",
     "start_time": "2020-02-10T15:59:42.042998Z"
    },
    "jupyter": {
     "outputs_hidden": false
    }
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "                                                                                \r"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "train info:\n",
      " total lines: 8000043, total users: 59522, total items: 8989\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "                                                                                \r"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "test info:\n",
      " total lines: 249418, total users: 3196, total items: 8180\n"
     ]
    }
   ],
   "source": [
    "# train/test split \n",
    "train_spl = TimeSplitter(\n",
    "    time_threshold=0.2,\n",
    "    drop_cold_items=True,\n",
    "    drop_cold_users=True,\n",
    "    query_column=\"user_id\",\n",
    "\n",
    ")\n",
    "train, test = train_spl.split(log_replay)\n",
    "print('train info:\\n', get_log_info(train, user_col='user_id', item_col='item_id'))\n",
    "print('test info:\\n', get_log_info(test, user_col='user_id', item_col='item_id'))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "False"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "train.is_cached"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 2. Categories preprocessing"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We will treat film genre as an item categories. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [],
   "source": [
    "genres = (\n",
    "    item_features_replay\n",
    "    .select(\n",
    "        \"item_id\",\n",
    "        split(\"genres\", \"\\|\").alias(\"genres\")\n",
    "    )\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "+-------+--------------------+\n",
      "|item_id|              genres|\n",
      "+-------+--------------------+\n",
      "|    692|   [Comedy, Romance]|\n",
      "|   1349|[Adventure, Child...|\n",
      "|   6814|[Comedy, Drama, R...|\n",
      "|   5659|[Action, Crime, T...|\n",
      "|   4274|[Adventure, Anima...|\n",
      "+-------+--------------------+\n",
      "only showing top 5 rows\n",
      "\n"
     ]
    }
   ],
   "source": [
    "genres.show(5)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<AxesSubplot:>"
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAjAAAAGdCAYAAAAMm0nCAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjUuMywgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/NK7nSAAAACXBIWXMAAA9hAAAPYQGoP6dpAAA0qUlEQVR4nO3dfXSU5Z3/8U8SkoEAEwRNJllCmkIFAkSeFKa2FCQkxiyrNdvVSoEK6oETqJAWabqIgYggLSKtCFopYY+kiq7aCkgy4AZKCQIpqTx0qVLa2ELCHi0ZAZkMmfn94cn8GHmcOA+5mPfrnDmH+76vueb7vTIJn9xzTybG6/V6BQAAYJDYSBcAAAAQKAIMAAAwDgEGAAAYhwADAACMQ4ABAADGIcAAAADjEGAAAIBxCDAAAMA4HSJdQKh4PB4dP35cXbt2VUxMTKTLAQAA18Dr9erTTz9VWlqaYmMvf57lug0wx48fV3p6eqTLAAAAbfDRRx+pZ8+elz1+3QaYrl27Svp8AaxWa9DmdbvdqqqqUm5uruLj44M2r0mifQ2ivX+JNaD/6O5fYg1C2b/T6VR6errv//HLuW4DTOvLRlarNegBJjExUVarNSqftBJrEO39S6wB/Ud3/xJrEI7+r3b5BxfxAgAA4xBgAACAcQgwAADAOAQYAABgHAIMAAAwDgEGAAAYhwADAACMQ4ABAADGIcAAAADjEGAAAIBxCDAAAMA4XyrALFmyRDExMZo1a5Zv37lz51RUVKQePXqoS5cuKiwsVGNjo9/96uvrVVBQoMTERCUnJ2vOnDk6f/6835jq6moNHTpUFotFffr0UXl5+ZcpFQAAXEfaHGD27t2rF154QdnZ2X77Z8+erbfffluvvfaatm/fruPHj+vee+/1HW9paVFBQYGam5u1a9curVu3TuXl5Zo/f75vzLFjx1RQUKAxY8aorq5Os2bN0kMPPaTKysq2lgsAAK4jbQowp0+f1oQJE/TLX/5SN9xwg29/U1OT1qxZo2eeeUZ33HGHhg0bprVr12rXrl3avXu3JKmqqkqHDx/Wyy+/rMGDBys/P19lZWVauXKlmpubJUmrV69WZmamli1bpv79+2vGjBn693//dy1fvjwILQMAANN1aMudioqKVFBQoJycHD355JO+/bW1tXK73crJyfHt69evn3r16qWamhqNHDlSNTU1GjRokFJSUnxj8vLyNH36dB06dEhDhgxRTU2N3xytYy58qeqLXC6XXC6Xb9vpdEr6/CO/3W53W9q8pNa5hi3cIpfnyh/13Z4cLM0L2lytaxDMdTVJtPcvsQb0H939S6xBKPu/1jkDDjCvvPKK/vCHP2jv3r0XHWtoaFBCQoK6devmtz8lJUUNDQ2+MReGl9bjrceuNMbpdOqzzz5Tp06dLnrsxYsXa8GCBRftr6qqUmJi4rU3eI3KhnuCPmcobd68OehzOhyOoM9pkmjvX2IN6D+6+5dYg1D0f/bs2WsaF1CA+eijj/Too4/K4XCoY8eObSosVEpKSlRcXOzbdjqdSk9PV25urqxWa9Aex+12y+Fw6PF9sVF9BsbhcGjcuHGKj48P2rymiPb+JdaA/qO7f4k1CGX/ra+gXE1AAaa2tlYnT57U0KFDfftaWlq0Y8cOPffcc6qsrFRzc7NOnTrldxamsbFRNptNkmSz2bRnzx6/eVvfpXThmC++c6mxsVFWq/WSZ18kyWKxyGKxXLQ/Pj4+JE8ulydGrhZzAkwo1iBUa2uKaO9fYg3oP7r7l1iDUPR/rfMFdBHv2LFjdeDAAdXV1fluw4cP14QJE3z/jo+P17Zt23z3OXLkiOrr62W32yVJdrtdBw4c0MmTJ31jHA6HrFarsrKyfGMunKN1TOscAAAgugV0BqZr164aOHCg377OnTurR48evv1Tp05VcXGxunfvLqvVqpkzZ8put2vkyJGSpNzcXGVlZWnixIlaunSpGhoaNG/ePBUVFfnOoEybNk3PPfecHnvsMU2ZMkXvvvuuNmzYoE2bNgWjZwAAYLg2vQvpSpYvX67Y2FgVFhbK5XIpLy9Pzz//vO94XFycNm7cqOnTp8tut6tz586aPHmyFi5c6BuTmZmpTZs2afbs2VqxYoV69uypl156SXl5wbuOAwAAmOtLB5jq6mq/7Y4dO2rlypVauXLlZe+TkZFx1XfFjB49Wvv37/+y5QEAgOsQn4UEAACMQ4ABAADGIcAAAADjEGAAAIBxCDAAAMA4BBgAAGAcAgwAADAOAQYAABiHAAMAAIxDgAEAAMYhwAAAAOMQYAAAgHEIMAAAwDgEGAAAYBwCDAAAMA4BBgAAGIcAAwAAjEOAAQAAxiHAAAAA43SIdAEIj6/8eFPQ5rLEebX0NmlgaaVcLTFBm/dS/rqkIKTzAwDMxBkYAABgHAIMAAAwDgEGAAAYhwADAACMQ4ABAADGIcAAAADjEGAAAIBxCDAAAMA4BBgAAGAcAgwAADAOAQYAABiHAAMAAIxDgAEAAMYJKMCsWrVK2dnZslqtslqtstvteuedd3zHR48erZiYGL/btGnT/Oaor69XQUGBEhMTlZycrDlz5uj8+fN+Y6qrqzV06FBZLBb16dNH5eXlbe8QAABcdzoEMrhnz55asmSJvva1r8nr9WrdunW6++67tX//fg0YMECS9PDDD2vhwoW++yQmJvr+3dLSooKCAtlsNu3atUsnTpzQpEmTFB8fr6eeekqSdOzYMRUUFGjatGlav369tm3bpoceekipqanKy8sLRs8AAMBwAQWY8ePH+20vWrRIq1at0u7du30BJjExUTab7ZL3r6qq0uHDh7V161alpKRo8ODBKisr09y5c1VaWqqEhAStXr1amZmZWrZsmSSpf//+2rlzp5YvX06AAQAAkgIMMBdqaWnRa6+9pjNnzshut/v2r1+/Xi+//LJsNpvGjx+vxx9/3HcWpqamRoMGDVJKSopvfF5enqZPn65Dhw5pyJAhqqmpUU5Ojt9j5eXladasWVesx+VyyeVy+badTqckye12y+12t7XNi7TOZYn1Bm1O07T2Ho41CObXLlhaa2qPtYVLtK8B/Ud3/xJrEMr+r3XOgAPMgQMHZLfbde7cOXXp0kVvvvmmsrKyJEkPPPCAMjIylJaWpvfff19z587VkSNH9MYbb0iSGhoa/MKLJN92Q0PDFcc4nU599tln6tSp0yXrWrx4sRYsWHDR/qqqKr+XsYKlbLgn6HOaJhxrsHnz5pA/Rls5HI5IlxBx0b4G9B/d/UusQSj6P3v27DWNCzjA9O3bV3V1dWpqatLrr7+uyZMna/v27crKytIjjzziGzdo0CClpqZq7NixOnr0qHr37h3oQwWkpKRExcXFvm2n06n09HTl5ubKarUG7XHcbrccDoce3xcrlycmaPOaxBLrVdlwT1jW4GBp+3vZsPU5MG7cOMXHx0e6nIiI9jWg/+juX2INQtl/6ysoVxNwgElISFCfPn0kScOGDdPevXu1YsUKvfDCCxeNHTFihCTpww8/VO/evWWz2bRnzx6/MY2NjZLku27GZrP59l04xmq1XvbsiyRZLBZZLJaL9sfHx4fkyeXyxMjVEp0BplU41qA9/2AI1XPLJNG+BvQf3f1LrEEo+r/W+b7034HxeDx+155cqK6uTpKUmpoqSbLb7Tpw4IBOnjzpG+NwOGS1Wn0vQ9ntdm3bts1vHofD4XedDQAAiG4BnYEpKSlRfn6+evXqpU8//VQVFRWqrq5WZWWljh49qoqKCt11113q0aOH3n//fc2ePVujRo1Sdna2JCk3N1dZWVmaOHGili5dqoaGBs2bN09FRUW+syfTpk3Tc889p8cee0xTpkzRu+++qw0bNmjTpk3B7x4AABgpoABz8uRJTZo0SSdOnFBSUpKys7NVWVmpcePG6aOPPtLWrVv17LPP6syZM0pPT1dhYaHmzZvnu39cXJw2btyo6dOny263q3Pnzpo8ebLf343JzMzUpk2bNHv2bK1YsUI9e/bUSy+9xFuoAQCAT0ABZs2aNZc9lp6eru3bt191joyMjKu+s2T06NHav39/IKUBAIAowmchAQAA4xBgAACAcQgwAADAOAQYAABgHAIMAAAwDgEGAAAYhwADAACMQ4ABAADGIcAAAADjEGAAAIBxCDAAAMA4BBgAAGAcAgwAADAOAQYAABiHAAMAAIxDgAEAAMYhwAAAAOMQYAAAgHEIMAAAwDgEGAAAYBwCDAAAMA4BBgAAGIcAAwAAjEOAAQAAxiHAAAAA4xBgAACAcQgwAADAOAQYAABgHAIMAAAwDgEGAAAYhwADAACMQ4ABAADGCSjArFq1StnZ2bJarbJarbLb7XrnnXd8x8+dO6eioiL16NFDXbp0UWFhoRobG/3mqK+vV0FBgRITE5WcnKw5c+bo/PnzfmOqq6s1dOhQWSwW9enTR+Xl5W3vEAAAXHcCCjA9e/bUkiVLVFtbq3379umOO+7Q3XffrUOHDkmSZs+erbfffluvvfaatm/fruPHj+vee+/13b+lpUUFBQVqbm7Wrl27tG7dOpWXl2v+/Pm+MceOHVNBQYHGjBmjuro6zZo1Sw899JAqKyuD1DIAADBdh0AGjx8/3m970aJFWrVqlXbv3q2ePXtqzZo1qqio0B133CFJWrt2rfr376/du3dr5MiRqqqq0uHDh7V161alpKRo8ODBKisr09y5c1VaWqqEhAStXr1amZmZWrZsmSSpf//+2rlzp5YvX668vLwgtQ0AAEzW5mtgWlpa9Morr+jMmTOy2+2qra2V2+1WTk6Ob0y/fv3Uq1cv1dTUSJJqamo0aNAgpaSk+Mbk5eXJ6XT6zuLU1NT4zdE6pnUOAACAgM7ASNKBAwdkt9t17tw5denSRW+++aaysrJUV1enhIQEdevWzW98SkqKGhoaJEkNDQ1+4aX1eOuxK41xOp367LPP1KlTp0vW5XK55HK5fNtOp1OS5Ha75Xa7A23zslrnssR6gzanaVp7D8caBPNrFyytNbXH2sIl2teA/qO7f4k1CGX/1zpnwAGmb9++qqurU1NTk15//XVNnjxZ27dvD7jAYFu8eLEWLFhw0f6qqiolJiYG/fHKhnuCPqdpwrEGmzdvDvljtJXD4Yh0CREX7WtA/9Hdv8QahKL/s2fPXtO4gANMQkKC+vTpI0kaNmyY9u7dqxUrVui+++5Tc3OzTp065XcWprGxUTabTZJks9m0Z88ev/la36V04ZgvvnOpsbFRVqv1smdfJKmkpETFxcW+bafTqfT0dOXm5spqtQba5mW53W45HA49vi9WLk9M0OY1iSXWq7LhnrCswcHS9nfdU+tzYNy4cYqPj490ORER7WtA/9Hdv8QahLL/1ldQribgAPNFHo9HLpdLw4YNU3x8vLZt26bCwkJJ0pEjR1RfXy+73S5JstvtWrRokU6ePKnk5GRJn6c3q9WqrKws35gv/tbtcDh8c1yOxWKRxWK5aH98fHxInlwuT4xcLdEZYFqFYw3a8w+GUD23TBLta0D/0d2/xBqEov9rnS+gAFNSUqL8/Hz16tVLn376qSoqKlRdXa3KykolJSVp6tSpKi4uVvfu3WW1WjVz5kzZ7XaNHDlSkpSbm6usrCxNnDhRS5cuVUNDg+bNm6eioiJf+Jg2bZqee+45PfbYY5oyZYreffddbdiwQZs2bQpwCQAAwPUqoABz8uRJTZo0SSdOnFBSUpKys7NVWVmpcePGSZKWL1+u2NhYFRYWyuVyKS8vT88//7zv/nFxcdq4caOmT58uu92uzp07a/LkyVq4cKFvTGZmpjZt2qTZs2drxYoV6tmzp1566SXeQg0AAHwCCjBr1qy54vGOHTtq5cqVWrly5WXHZGRkXPXCzNGjR2v//v2BlAYAAKIIn4UEAACMQ4ABAADGIcAAAADjEGAAAIBxCDAAAMA4BBgAAGAcAgwAADAOAQYAABiHAAMAAIxDgAEAAMYhwAAAAOMQYAAAgHEIMAAAwDgEGAAAYBwCDAAAMA4BBgAAGIcAAwAAjEOAAQAAxiHAAAAA4xBgAACAcQgwAADAOAQYAABgHAIMAAAwDgEGAAAYhwADAACMQ4ABAADGIcAAAADjEGAAAIBxCDAAAMA4BBgAAGAcAgwAADAOAQYAABiHAAMAAIwTUIBZvHixbr31VnXt2lXJycm65557dOTIEb8xo0ePVkxMjN9t2rRpfmPq6+tVUFCgxMREJScna86cOTp//rzfmOrqag0dOlQWi0V9+vRReXl52zoEAADXnYACzPbt21VUVKTdu3fL4XDI7XYrNzdXZ86c8Rv38MMP68SJE77b0qVLfcdaWlpUUFCg5uZm7dq1S+vWrVN5ebnmz5/vG3Ps2DEVFBRozJgxqqur06xZs/TQQw+psrLyS7YLAACuBx0CGbxlyxa/7fLyciUnJ6u2tlajRo3y7U9MTJTNZrvkHFVVVTp8+LC2bt2qlJQUDR48WGVlZZo7d65KS0uVkJCg1atXKzMzU8uWLZMk9e/fXzt37tTy5cuVl5cXaI8AAOA686WugWlqapIkde/e3W//+vXrdeONN2rgwIEqKSnR2bNnfcdqamo0aNAgpaSk+Pbl5eXJ6XTq0KFDvjE5OTl+c+bl5ammpubLlAsAAK4TAZ2BuZDH49GsWbN0++23a+DAgb79DzzwgDIyMpSWlqb3339fc+fO1ZEjR/TGG29IkhoaGvzCiyTfdkNDwxXHOJ1OffbZZ+rUqdNF9bhcLrlcLt+20+mUJLndbrnd7ra2eZHWuSyx3qDNaZrW3sOxBsH82gVLa03tsbZwifY1oP/o7l9iDULZ/7XO2eYAU1RUpIMHD2rnzp1++x955BHfvwcNGqTU1FSNHTtWR48eVe/evdv6cFe1ePFiLViw4KL9VVVVSkxMDPrjlQ33BH1O04RjDTZv3hzyx2grh8MR6RIiLtrXgP6ju3+JNQhF/xe+anMlbQowM2bM0MaNG7Vjxw717NnzimNHjBghSfrwww/Vu3dv2Ww27dmzx29MY2OjJPmum7HZbL59F46xWq2XPPsiSSUlJSouLvZtO51OpaenKzc3V1arNbAGr8DtdsvhcOjxfbFyeWKCNq9JLLFelQ33hGUNDpa2v2ueWp8D48aNU3x8fKTLiYhoXwP6j+7+JdYglP23voJyNQEFGK/Xq5kzZ+rNN99UdXW1MjMzr3qfuro6SVJqaqokyW63a9GiRTp58qSSk5MlfZ7grFarsrKyfGO++Ju3w+GQ3W6/7ONYLBZZLJaL9sfHx4fkyeXyxMjVEp0BplU41qA9/2AI1XPLJNG+BvQf3f1LrEEo+r/W+QK6iLeoqEgvv/yyKioq1LVrVzU0NKihoUGfffaZJOno0aMqKytTbW2t/vrXv+q3v/2tJk2apFGjRik7O1uSlJubq6ysLE2cOFF//OMfVVlZqXnz5qmoqMgXQKZNm6a//OUveuyxx/S///u/ev7557VhwwbNnj07kHIBAMB1KqAAs2rVKjU1NWn06NFKTU313V599VVJUkJCgrZu3arc3Fz169dPP/zhD1VYWKi3337bN0dcXJw2btyouLg42e12fe9739OkSZO0cOFC35jMzExt2rRJDodDt9xyi5YtW6aXXnqJt1ADAABJbXgJ6UrS09O1ffv2q86TkZFx1YszR48erf379wdSHgAAiBJ8FhIAADAOAQYAABiHAAMAAIxDgAEAAMYhwAAAAOMQYAAAgHEIMAAAwDgEGAAAYBwCDAAAMA4BBgAAGIcAAwAAjEOAAQAAxiHAAAAA4xBgAACAcQgwAADAOAQYAABgHAIMAAAwDgEGAAAYhwADAACMQ4ABAADGIcAAAADjEGAAAIBxCDAAAMA4BBgAAGAcAgwAADAOAQYAABiHAAMAAIxDgAEAAMYhwAAAAOMQYAAAgHEIMAAAwDgEGAAAYBwCDAAAME5AAWbx4sW69dZb1bVrVyUnJ+uee+7RkSNH/MacO3dORUVF6tGjh7p06aLCwkI1Njb6jamvr1dBQYESExOVnJysOXPm6Pz5835jqqurNXToUFksFvXp00fl5eVt6xAAAFx3Agow27dvV1FRkXbv3i2HwyG3263c3FydOXPGN2b27Nl6++239dprr2n79u06fvy47r33Xt/xlpYWFRQUqLm5Wbt27dK6detUXl6u+fPn+8YcO3ZMBQUFGjNmjOrq6jRr1iw99NBDqqysDELLAADAdB0CGbxlyxa/7fLyciUnJ6u2tlajRo1SU1OT1qxZo4qKCt1xxx2SpLVr16p///7avXu3Ro4cqaqqKh0+fFhbt25VSkqKBg8erLKyMs2dO1elpaVKSEjQ6tWrlZmZqWXLlkmS+vfvr507d2r58uXKy8sLUusAAMBUX+oamKamJklS9+7dJUm1tbVyu93KycnxjenXr5969eqlmpoaSVJNTY0GDRqklJQU35i8vDw5nU4dOnTIN+bCOVrHtM4BAACiW0BnYC7k8Xg0a9Ys3X777Ro4cKAkqaGhQQkJCerWrZvf2JSUFDU0NPjGXBheWo+3HrvSGKfTqc8++0ydOnW6qB6XyyWXy+XbdjqdkiS32y23293WNi/SOpcl1hu0OU3T2ns41qDvf24M+WMEyhLrVdlwadjCLXJ5Yi46frD0+j9L2Pp9EMzvLZPQf3T3L7EGoez/Wudsc4ApKirSwYMHtXPnzrZOEVSLFy/WggULLtpfVVWlxMTEoD9e2XBP0Oc0TbSvweX637x5c5griRyHwxHpEiKK/qO7f4k1CEX/Z8+evaZxbQowM2bM0MaNG7Vjxw717NnTt99ms6m5uVmnTp3yOwvT2Ngom83mG7Nnzx6/+VrfpXThmC++c6mxsVFWq/WSZ18kqaSkRMXFxb5tp9Op9PR05ebmymq1tqXNS3K73XI4HHp8X+wlf/uOBp+fgfBE7Rpcrf9oOQPjcDg0btw4xcfHR7qcsKP/6O5fYg1C2X/rKyhXE1CA8Xq9mjlzpt58801VV1crMzPT7/iwYcMUHx+vbdu2qbCwUJJ05MgR1dfXy263S5LsdrsWLVqkkydPKjk5WdLnCc5qtSorK8s35ou/xTocDt8cl2KxWGSxWC7aHx8fH5Inl8sTI1dL9P3nfaFoX4PL9R9NP8xC9f1lCvqP7v4l1iAU/V/rfAEFmKKiIlVUVOg3v/mNunbt6rtmJSkpSZ06dVJSUpKmTp2q4uJide/eXVarVTNnzpTdbtfIkSMlSbm5ucrKytLEiRO1dOlSNTQ0aN68eSoqKvIFkGnTpum5557TY489pilTpujdd9/Vhg0btGnTpkDKBQAA16mA3oW0atUqNTU1afTo0UpNTfXdXn31Vd+Y5cuX61//9V9VWFioUaNGyWaz6Y033vAdj4uL08aNGxUXFye73a7vfe97mjRpkhYuXOgbk5mZqU2bNsnhcOiWW27RsmXL9NJLL/EWagAAIKkNLyFdTceOHbVy5UqtXLnysmMyMjKueqHj6NGjtX///kDKAwAAUYLPQgIAAMYhwAAAAOMQYAAAgHEIMAAAwDgEGAAAYBwCDAAAMA4BBgAAGIcAAwAAjEOAAQAAxiHAAAAA4xBgAACAcQgwAADAOAQYAABgHAIMAAAwDgEGAAAYhwADAACMQ4ABAADGIcAAAADjEGAAAIBxCDAAAMA4BBgAAGAcAgwAADAOAQYAABiHAAMAAIxDgAEAAMYhwAAAAOMQYAAAgHEIMAAAwDgEGAAAYBwCDAAAMA4BBgAAGIcAAwAAjEOAAQAAxgk4wOzYsUPjx49XWlqaYmJi9NZbb/kd//73v6+YmBi/25133uk35pNPPtGECRNktVrVrVs3TZ06VadPn/Yb8/777+ub3/ymOnbsqPT0dC1dujTw7gAAwHUp4ABz5swZ3XLLLVq5cuVlx9x55506ceKE7/brX//a7/iECRN06NAhORwObdy4UTt27NAjjzziO+50OpWbm6uMjAzV1tbqpz/9qUpLS/Xiiy8GWi4AALgOdQj0Dvn5+crPz7/iGIvFIpvNdsljf/rTn7Rlyxbt3btXw4cPlyT94he/0F133aWf/exnSktL0/r169Xc3Kxf/epXSkhI0IABA1RXV6dnnnnGL+gAAIDoFHCAuRbV1dVKTk7WDTfcoDvuuENPPvmkevToIUmqqalRt27dfOFFknJychQbG6v33ntP3/72t1VTU6NRo0YpISHBNyYvL09PP/20/vnPf+qGG2646DFdLpdcLpdv2+l0SpLcbrfcbnfQemudyxLrDdqcpmntPVrX4Gr9B/P51l619hgNvV4K/Ud3/xJrEMr+r3XOoAeYO++8U/fee68yMzN19OhR/eQnP1F+fr5qamoUFxenhoYGJScn+xfRoYO6d++uhoYGSVJDQ4MyMzP9xqSkpPiOXSrALF68WAsWLLhof1VVlRITE4PVnk/ZcE/Q5zRNtK/B5frfvHlzmCuJHIfDEekSIor+o7t/iTUIRf9nz569pnFBDzD333+/79+DBg1Sdna2evfurerqao0dOzbYD+dTUlKi4uJi37bT6VR6erpyc3NltVqD9jhut1sOh0OP74uVyxMTtHlNYon1qmy4J2rX4Gr9HyzNi0BV4dX6fTBu3DjFx8dHupywo//o7l9iDULZf+srKFcTkpeQLvTVr35VN954oz788EONHTtWNptNJ0+e9Btz/vx5ffLJJ77rZmw2mxobG/3GtG5f7toai8Uii8Vy0f74+PiQPLlcnhi5WqLvP+8LRfsaXK7/aPphFqrvL1PQf3T3L7EGoej/WucL+d+B+fvf/66PP/5YqampkiS73a5Tp06ptrbWN+bdd9+Vx+PRiBEjfGN27Njh9zqYw+FQ3759L/nyEQAAiC4BB5jTp0+rrq5OdXV1kqRjx46prq5O9fX1On36tObMmaPdu3frr3/9q7Zt26a7775bffr0UV7e56fV+/fvrzvvvFMPP/yw9uzZo9///veaMWOG7r//fqWlpUmSHnjgASUkJGjq1Kk6dOiQXn31Va1YscLvJSIAABC9Ag4w+/bt05AhQzRkyBBJUnFxsYYMGaL58+crLi5O77//vv7t3/5NN998s6ZOnaphw4bpd7/7nd/LO+vXr1e/fv00duxY3XXXXfrGN77h9zdekpKSVFVVpWPHjmnYsGH64Q9/qPnz5/MWagAAIKkN18CMHj1aXu/l3z5bWVl51Tm6d++uioqKK47Jzs7W7373u0DLAwAAUYDPQgIAAMYhwAAAAOMQYAAAgHEIMAAAwDgEGAAAYBwCDAAAMA4BBgAAGIcAAwAAjEOAAQAAxiHAAAAA4xBgAACAcQgwAADAOAQYAABgHAIMAAAwDgEGAAAYhwADAACMQ4ABAADGIcAAAADjEGAAAIBxCDAAAMA4BBgAAGAcAgwAADAOAQYAABiHAAMAAIxDgAEAAMYhwAAAAOMQYAAAgHEIMAAAwDgEGAAAYBwCDAAAMA4BBgAAGIcAAwAAjEOAAQAAxgk4wOzYsUPjx49XWlqaYmJi9NZbb/kd93q9mj9/vlJTU9WpUyfl5OTogw8+8BvzySefaMKECbJarerWrZumTp2q06dP+415//339c1vflMdO3ZUenq6li5dGnh3AADguhRwgDlz5oxuueUWrVy58pLHly5dqp///OdavXq13nvvPXXu3Fl5eXk6d+6cb8yECRN06NAhORwObdy4UTt27NAjjzziO+50OpWbm6uMjAzV1tbqpz/9qUpLS/Xiiy+2oUUAAHC96RDoHfLz85Wfn3/JY16vV88++6zmzZunu+++W5L0X//1X0pJSdFbb72l+++/X3/605+0ZcsW7d27V8OHD5ck/eIXv9Bdd92ln/3sZ0pLS9P69evV3NysX/3qV0pISNCAAQNUV1enZ555xi/oAACA6BRwgLmSY8eOqaGhQTk5Ob59SUlJGjFihGpqanT//ferpqZG3bp184UXScrJyVFsbKzee+89ffvb31ZNTY1GjRqlhIQE35i8vDw9/fTT+uc//6kbbrjhosd2uVxyuVy+bafTKUlyu91yu91B67F1LkusN2hzmqa192hdg6v1H8znW3vV2mM09Hop9B/d/UusQSj7v9Y5gxpgGhoaJEkpKSl++1NSUnzHGhoalJyc7F9Ehw7q3r2735jMzMyL5mg9dqkAs3jxYi1YsOCi/VVVVUpMTGxjR5dXNtwT9DlNE+1rcLn+N2/eHOZKIsfhcES6hIii/+juX2INQtH/2bNnr2lcUANMJJWUlKi4uNi37XQ6lZ6ertzcXFmt1qA9jtvtlsPh0OP7YuXyxARtXpNYYr0qG+6J2jW4Wv8HS/MiUFV4tX4fjBs3TvHx8ZEuJ+zoP7r7l1iDUPbf+grK1QQ1wNhsNklSY2OjUlNTffsbGxs1ePBg35iTJ0/63e/8+fP65JNPfPe32WxqbGz0G9O63TrmiywWiywWy0X74+PjQ/Lkcnli5GqJvv+8LxTta3C5/qPph1movr9MQf/R3b/EGoSi/2udL6h/ByYzM1M2m03btm3z7XM6nXrvvfdkt9slSXa7XadOnVJtba1vzLvvviuPx6MRI0b4xuzYscPvdTCHw6G+ffte8uUjAAAQXQIOMKdPn1ZdXZ3q6uokfX7hbl1dnerr6xUTE6NZs2bpySef1G9/+1sdOHBAkyZNUlpamu655x5JUv/+/XXnnXfq4Ycf1p49e/T73/9eM2bM0P3336+0tDRJ0gMPPKCEhARNnTpVhw4d0quvvqoVK1b4vUQEAACiV8AvIe3bt09jxozxbbeGismTJ6u8vFyPPfaYzpw5o0ceeUSnTp3SN77xDW3ZskUdO3b03Wf9+vWaMWOGxo4dq9jYWBUWFurnP/+573hSUpKqqqpUVFSkYcOG6cYbb9T8+fN5CzUAAJDUhgAzevRoeb2Xf/tsTEyMFi5cqIULF152TPfu3VVRUXHFx8nOztbvfve7QMsDAABRgM9CAgAAxiHAAAAA4xBgAACAcQgwAADAOAQYAABgHAIMAAAwDgEGAAAYhwADAACMQ4ABAADGIcAAAADjEGAAAIBxCDAAAMA4BBgAAGAcAgwAADAOAQYAABiHAAMAAIzTIdIFAIi8r/x4U0DjLXFeLb1NGlhaKVdLTIiqurK/LimIyOMCaB84AwMAAIxDgAEAAMYhwAAAAOMQYAAAgHEIMAAAwDgEGAAAYBwCDAAAMA4BBgAAGIcAAwAAjEOAAQAAxiHAAAAA4xBgAACAcQgwAADAOAQYAABgHAIMAAAwDgEGAAAYJ+gBprS0VDExMX63fv36+Y6fO3dORUVF6tGjh7p06aLCwkI1Njb6zVFfX6+CggIlJiYqOTlZc+bM0fnz54NdKgAAMFSHUEw6YMAAbd269f8/SIf//zCzZ8/Wpk2b9NprrykpKUkzZszQvffeq9///veSpJaWFhUUFMhms2nXrl06ceKEJk2apPj4eD311FOhKBcAABgmJAGmQ4cOstlsF+1vamrSmjVrVFFRoTvuuEOStHbtWvXv31+7d+/WyJEjVVVVpcOHD2vr1q1KSUnR4MGDVVZWprlz56q0tFQJCQmhKBkAABgkJAHmgw8+UFpamjp27Ci73a7FixerV69eqq2tldvtVk5Ojm9sv3791KtXL9XU1GjkyJGqqanRoEGDlJKS4huTl5en6dOn69ChQxoyZMglH9Plcsnlcvm2nU6nJMntdsvtdgett9a5LLHeoM1pmtbeo3UNrtZ/MJ9v4WKJC+xr2R6eA5Fc59bHNvFrHQzR3r/EGoSy/2udM8br9Qb1J9A777yj06dPq2/fvjpx4oQWLFigf/zjHzp48KDefvttPfjgg35BQ5Juu+02jRkzRk8//bQeeeQR/e1vf1NlZaXv+NmzZ9W5c2dt3rxZ+fn5l3zc0tJSLViw4KL9FRUVSkxMDGaLAAAgRM6ePasHHnhATU1Nslqtlx0X9DMwFwaM7OxsjRgxQhkZGdqwYYM6deoU7IfzKSkpUXFxsW/b6XQqPT1dubm5V1yAQLndbjkcDj2+L1YuT0zQ5jWJJdarsuGeqF2Dq/V/sDQvAlV9OQNLK68+6ALt4TkQyXVu/Tkwbtw4xcfHR6yOSIn2/iXWIJT9t76CcjUheQnpQt26ddPNN9+sDz/8UOPGjVNzc7NOnTqlbt26+cY0Njb6rpmx2Wzas2eP3xyt71K61HU1rSwWiywWy0X74+PjQ/Lkcnli5GqJvv+8LxTta3C5/k38YdbWr2MknwPtYZ1D9fPFFNHev8QahKL/a50v5H8H5vTp0zp69KhSU1M1bNgwxcfHa9u2bb7jR44cUX19vex2uyTJbrfrwIEDOnnypG+Mw+GQ1WpVVlZWqMsFAAAGCPoZmB/96EcaP368MjIydPz4cT3xxBOKi4vTd7/7XSUlJWnq1KkqLi5W9+7dZbVaNXPmTNntdo0cOVKSlJubq6ysLE2cOFFLly5VQ0OD5s2bp6KiokueYQEAANEn6AHm73//u7773e/q448/1k033aRvfOMb2r17t2666SZJ0vLlyxUbG6vCwkK5XC7l5eXp+eef990/Li5OGzdu1PTp02W329W5c2dNnjxZCxcuDHapAADAUEEPMK+88soVj3fs2FErV67UypUrLzsmIyNDmzdvDnZpAADgOsFnIQEAAOMQYAAAgHEIMAAAwDgEGAAAYBwCDAAAMA4BBgAAGIcAAwAAjEOAAQAAxiHAAAAA4xBgAACAcQgwAADAOAQYAABgHAIMAAAwDgEGAAAYhwADAACMQ4ABAADGIcAAAADjEGAAAIBxCDAAAMA4BBgAAGAcAgwAADBOh0gXAABt8ZUfb4rYY1vivFp6mzSwtFKulphrvt9flxSEsCogunAGBgAAGIcAAwAAjEOAAQAAxiHAAAAA4xBgAACAcQgwAADAOAQYAABgHAIMAAAwDgEGAAAYhwADAACMQ4ABAADGadcBZuXKlfrKV76ijh07asSIEdqzZ0+kSwIAAO1Au/0wx1dffVXFxcVavXq1RowYoWeffVZ5eXk6cuSIkpOTI10eAESFS31oZls/zDJc+NDM6NBuz8A888wzevjhh/Xggw8qKytLq1evVmJion71q19FujQAABBh7fIMTHNzs2pra1VSUuLbFxsbq5ycHNXU1FzyPi6XSy6Xy7fd1NQkSfrkk0/kdruDVpvb7dbZs2fVwR2rFk/7+80jHDp4vDp71hO1a3C1/j/++OMIVPXldDh/JrDxPAfa1P/18txo71//cKxz6/8FH3/8seLj40P+eO1NKPv/9NNPJUler/fKA73t0D/+8Q+vJO+uXbv89s+ZM8d72223XfI+TzzxhFcSN27cuHHjxu06uH300UdXzArt8gxMW5SUlKi4uNi37fF49Mknn6hHjx6KiQnebwhOp1Pp6en66KOPZLVagzavSaJ9DaK9f4k1oP/o7l9iDULZv9fr1aeffqq0tLQrjmuXAebGG29UXFycGhsb/fY3NjbKZrNd8j4Wi0UWi8VvX7du3UJVoqxWa1Q+aS8U7WsQ7f1LrAH9R3f/EmsQqv6TkpKuOqZdXsSbkJCgYcOGadu2bb59Ho9H27Ztk91uj2BlAACgPWiXZ2Akqbi4WJMnT9bw4cN122236dlnn9WZM2f04IMPRro0AAAQYe02wNx33336v//7P82fP18NDQ0aPHiwtmzZopSUlIjWZbFY9MQTT1z0clU0ifY1iPb+JdaA/qO7f4k1aA/9x3i9V3ufEgAAQPvSLq+BAQAAuBICDAAAMA4BBgAAGIcAAwAAjEOACcCOHTs0fvx4paWlKSYmRm+99VakSwqbxYsX69Zbb1XXrl2VnJyse+65R0eOHIl0WWG1atUqZWdn+/5wk91u1zvvvBPpsiJmyZIliomJ0axZsyJdStiUlpYqJibG79avX79IlxVW//jHP/S9731PPXr0UKdOnTRo0CDt27cv0mWFzVe+8pWLngMxMTEqKiqKdGlh0dLSoscff1yZmZnq1KmTevfurbKysqt/blEItNu3UbdHZ86c0S233KIpU6bo3nvvjXQ5YbV9+3YVFRXp1ltv1fnz5/WTn/xEubm5Onz4sDp37hzp8sKiZ8+eWrJkib72ta/J6/Vq3bp1uvvuu7V//34NGDAg0uWF1d69e/XCCy8oOzs70qWE3YABA7R161bfdocO0fNj9J///Kduv/12jRkzRu+8845uuukmffDBB7rhhhsiXVrY7N27Vy0tLb7tgwcPaty4cfrOd74TwarC5+mnn9aqVau0bt06DRgwQPv27dODDz6opKQk/eAHPwhrLdHznRcE+fn5ys/Pj3QZEbFlyxa/7fLyciUnJ6u2tlajRo2KUFXhNX78eL/tRYsWadWqVdq9e3dUBZjTp09rwoQJ+uUvf6knn3wy0uWEXYcOHS77kSbXu6efflrp6elau3atb19mZmYEKwq/m266yW97yZIl6t27t771rW9FqKLw2rVrl+6++24VFBRI+vyM1K9//Wvt2bMn7LXwEhLapKmpSZLUvXv3CFcSGS0tLXrllVd05syZqPt4i6KiIhUUFCgnJyfSpUTEBx98oLS0NH31q1/VhAkTVF9fH+mSwua3v/2thg8fru985ztKTk7WkCFD9Mtf/jLSZUVMc3OzXn75ZU2ZMiWoHxrcnn3961/Xtm3b9Oc//1mS9Mc//lE7d+6MyC/3nIFBwDwej2bNmqXbb79dAwcOjHQ5YXXgwAHZ7XadO3dOXbp00ZtvvqmsrKxIlxU2r7zyiv7whz9o7969kS4lIkaMGKHy8nL17dtXJ06c0IIFC/TNb35TBw8eVNeuXSNdXsj95S9/0apVq1RcXKyf/OQn2rt3r37wgx8oISFBkydPjnR5YffWW2/p1KlT+v73vx/pUsLmxz/+sZxOp/r166e4uDi1tLRo0aJFmjBhQthrIcAgYEVFRTp48KB27twZ6VLCrm/fvqqrq1NTU5Nef/11TZ48Wdu3b4+KEPPRRx/p0UcflcPhUMeOHSNdTkRc+Ftmdna2RowYoYyMDG3YsEFTp06NYGXh4fF4NHz4cD311FOSpCFDhujgwYNavXp1VAaYNWvWKD8/X2lpaZEuJWw2bNig9evXq6KiQgMGDFBdXZ1mzZqltLS0sD8HCDAIyIwZM7Rx40bt2LFDPXv2jHQ5YZeQkKA+ffpIkoYNG6a9e/dqxYoVeuGFFyJcWejV1tbq5MmTGjp0qG9fS0uLduzYoeeee04ul0txcXERrDD8unXrpptvvlkffvhhpEsJi9TU1IvCev/+/fXf//3fEaoocv72t79p69ateuONNyJdSljNmTNHP/7xj3X//fdLkgYNGqS//e1vWrx4MQEG7ZPX69XMmTP15ptvqrq6Ouou3Lscj8cjl8sV6TLCYuzYsTpw4IDfvgcffFD9+vXT3Llzoy68SJ9f0Hz06FFNnDgx0qWExe23337Rn0/485//rIyMjAhVFDlr165VcnKy72LWaHH27FnFxvpfPhsXFyePxxP2WggwATh9+rTfb1rHjh1TXV2dunfvrl69ekWwstArKipSRUWFfvOb36hr165qaGiQJCUlJalTp04Rri48SkpKlJ+fr169eunTTz9VRUWFqqurVVlZGenSwqJr164XXfPUuXNn9ejRI2quhfrRj36k8ePHKyMjQ8ePH9cTTzyhuLg4ffe73410aWExe/Zsff3rX9dTTz2l//iP/9CePXv04osv6sUXX4x0aWHl8Xi0du1aTZ48OareRi99/m7MRYsWqVevXhowYID279+vZ555RlOmTAl/MV5cs//5n//xSrroNnny5EiXFnKX6luSd+3atZEuLWymTJnizcjI8CYkJHhvuukm79ixY71VVVWRLiuivvWtb3kfffTRSJcRNvfdd583NTXVm5CQ4P2Xf/kX73333ef98MMPI11WWL399tvegQMHei0Wi7dfv37eF198MdIlhV1lZaVXkvfIkSORLiXsnE6n99FHH/X26tXL27FjR+9Xv/pV73/+5396XS5X2GuJ8Xoj8OfzAAAAvgT+DgwAADAOAQYAABiHAAMAAIxDgAEAAMYhwAAAAOMQYAAAgHEIMAAAwDgEGAAAYBwCDAAAMA4BBgAAGIcAAwAAjEOAAQAAxvl/agbBYp96tNkAAAAASUVORK5CYII=",
      "text/plain": [
       "<Figure size 640x480 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "num_genres = genres.select(\"item_id\", sf.size(\"genres\").alias(\"num_genres\"))\n",
    "num_genres.toPandas()[\"num_genres\"].hist(bins=10)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Some films have more than one genre. To model the real life we will randomly sample one category (genre) the item is chosen from for each interaction."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Will asign a number of each genre for each item and sample random integer from interval [0, num_genres) to get a category of an interaction."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "+-------+-------+------------+\n",
      "|item_id|  genre|genre_number|\n",
      "+-------+-------+------------+\n",
      "|      0|  Drama|           0|\n",
      "|      0|Romance|           1|\n",
      "|      1|  Drama|           0|\n",
      "|      1|Mystery|           1|\n",
      "|      1|Romance|           2|\n",
      "+-------+-------+------------+\n",
      "only showing top 5 rows\n",
      "\n"
     ]
    }
   ],
   "source": [
    "genres = (genres\n",
    "          .select(\"item_id\", sf.explode(\"genres\").alias(\"genre\"))\n",
    "          .withColumn(\"genre_number\", sf.row_number().over(Window.orderBy(\"genre\").partitionBy(\"item_id\"))  - sf.lit(1)\n",
    "                     )\n",
    "         )\n",
    "genres.show(5)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Some films has several genres. Assume that each interaction with film is associated with a genre, e.g. the user select genre at first and then select film.\n",
    "Lets build the interactions dataset based on this assumption. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 3. Assigning categories to interactions"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "                                                                                \r"
     ]
    },
    {
     "data": {
      "text/plain": [
       "8000043"
      ]
     },
     "execution_count": 20,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "train_genre = train.join(num_genres, on=\"item_id\")\n",
    "train_genre = train_genre.withColumn(\"genre_number\", sf.floor(sf.rand() * sf.col(\"num_genres\")).alias(\"category\")).select(\"item_id\", \"user_id\", \"genre_number\")\n",
    "train_genre = train_genre.join(genres.withColumnRenamed(\"genre\", \"category\"), on=[\"item_id\", \"genre_number\"]).drop(\"genre_number\")\n",
    "train_genre.cache()\n",
    "train_genre.count()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "+-------+--------+---------------+\n",
      "|item_id|category|count(category)|\n",
      "+-------+--------+---------------+\n",
      "|      0|   Drama|           3424|\n",
      "|      0| Romance|           3456|\n",
      "|      1| Romance|           1981|\n",
      "|      1|   Drama|           2004|\n",
      "|      1| Mystery|           1880|\n",
      "+-------+--------+---------------+\n",
      "only showing top 5 rows\n",
      "\n"
     ]
    }
   ],
   "source": [
    "train_genre.groupBy(\"item_id\", \"category\").agg(sf.count(\"category\")).orderBy(\"item_id\").show(5)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "+-------+-------+---------+\n",
      "|item_id|user_id| category|\n",
      "+-------+-------+---------+\n",
      "|    243|  28939|   Action|\n",
      "|   2493|   2268| Thriller|\n",
      "|   2946|  37421|Adventure|\n",
      "|   3636|  51779| Thriller|\n",
      "|   4060|  57959|   Comedy|\n",
      "+-------+-------+---------+\n",
      "only showing top 5 rows\n",
      "\n"
     ]
    }
   ],
   "source": [
    "train_genre.sample(0.0001).show(5)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Prepare Dataset"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [],
   "source": [
    "feature_schema = FeatureSchema(\n",
    "    [\n",
    "        FeatureInfo(\n",
    "            column=\"user_id\",\n",
    "            feature_type=FeatureType.CATEGORICAL,\n",
    "            feature_hint=FeatureHint.QUERY_ID,\n",
    "        ),\n",
    "        FeatureInfo(\n",
    "            column=\"item_id\",\n",
    "            feature_type=FeatureType.CATEGORICAL,\n",
    "            feature_hint=FeatureHint.ITEM_ID,\n",
    "        ),\n",
    "        FeatureInfo(\n",
    "            column=\"category\",\n",
    "            feature_type=FeatureType.CATEGORICAL,\n",
    "        ),\n",
    "    ]\n",
    ")\n",
    "\n",
    "train_dataset = Dataset(\n",
    "    feature_schema=feature_schema,\n",
    "    interactions=train_genre,\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 4. Recommending in case of flat categories structure"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Building category structure"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "+------------------+----------+\n",
      "|          category|parent_cat|\n",
      "+------------------+----------+\n",
      "|       Documentary|      null|\n",
      "|           Fantasy|      null|\n",
      "|              IMAX|      null|\n",
      "|         Adventure|      null|\n",
      "|               War|      null|\n",
      "|         Animation|      null|\n",
      "|            Comedy|      null|\n",
      "|          Thriller|      null|\n",
      "|         Film-Noir|      null|\n",
      "|             Crime|      null|\n",
      "|            Sci-Fi|      null|\n",
      "|           Musical|      null|\n",
      "|           Mystery|      null|\n",
      "|             Drama|      null|\n",
      "|            Horror|      null|\n",
      "|           Western|      null|\n",
      "|           Romance|      null|\n",
      "|          Children|      null|\n",
      "|            Action|      null|\n",
      "|(no genres listed)|      null|\n",
      "+------------------+----------+\n",
      "\n"
     ]
    }
   ],
   "source": [
    "flat_cats = genres.select(\"genre\").distinct().withColumnRenamed(\"genre\", \"category\").withColumn(\"parent_cat\", sf.lit(None)).cache()\n",
    "flat_cats.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Model training"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [],
   "source": [
    "model = CatPopRec(flat_cats)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CPU times: user 9.36 ms, sys: 520 µs, total: 9.88 ms\n",
      "Wall time: 505 ms\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "model.fit(train_dataset)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The model stores number of interactions with each item in each leaf category."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "+------------------+-------+---------+\n",
      "|          category|item_id|relevance|\n",
      "+------------------+-------+---------+\n",
      "|(no genres listed)|   7621|        2|\n",
      "|            Action|   9399|    12314|\n",
      "|            Action|   9621|     8602|\n",
      "|            Action|   9815|     8005|\n",
      "|            Action|   1124|     7979|\n",
      "+------------------+-------+---------+\n",
      "only showing top 5 rows\n",
      "\n"
     ]
    }
   ],
   "source": [
    "model.cat_item_popularity.orderBy(sf.col(\"category\"), sf.col(\"relevance\").desc()).show(5)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Generating recommendations"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "+--------+-------+---------+----------------------------------------+-----------------------+\n",
      "|category|item_id|relevance|substring(title, 0, 40)                 |genres                 |\n",
      "+--------+-------+---------+----------------------------------------+-----------------------+\n",
      "|Action  |9399   |0.0168   |Terminator 2: Judgment Day (1991)       |Action|Sci-Fi          |\n",
      "|Action  |9621   |0.0117   |Raiders of the Lost Ark (Indiana Jones a|Action|Adventure       |\n",
      "|Action  |9815   |0.0109   |Braveheart (1995)                       |Action|Drama|War       |\n",
      "|Action  |1124   |0.0109   |Star Wars: Episode IV - A New Hope (a.k.|Action|Adventure|Sci-Fi|\n",
      "|Action  |4964   |0.0093   |Star Wars: Episode VI - Return of the Je|Action|Adventure|Sci-Fi|\n",
      "+--------+-------+---------+----------------------------------------+-----------------------+\n",
      "\n",
      "CPU times: user 36.7 ms, sys: 20.4 ms, total: 57 ms\n",
      "Wall time: 2.77 s\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "pred = model.predict([\"Action\"], k=5)\n",
    "(pred\n",
    " .join(item_features_replay, on=\"item_id\")\n",
    " .orderBy(sf.col(\"relevance\").desc())\n",
    " .select(\"category\", \"item_id\", sf.round(\"relevance\", 4).alias(\"relevance\"), sf.substring(\"title\", 0, 40), \"genres\")\n",
    ").show(5, truncate=False)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 5. Recommending in case of tree categories structure"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Building category structure"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's create three parent categories and assing a parent for each genre."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [],
   "source": [
    "parent_cats = spark.createDataFrame([(\"Dark\", None), (\"Bright\", None), (\"Neutral\", None)], schema=\"category string, parent_cat string\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "+--------+----------+\n",
      "|category|parent_cat|\n",
      "+--------+----------+\n",
      "|    Dark|      null|\n",
      "|  Bright|      null|\n",
      "| Neutral|      null|\n",
      "+--------+----------+\n",
      "\n"
     ]
    }
   ],
   "source": [
    "parent_cats.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "+------------------+----------+\n",
      "|          category|parent_cat|\n",
      "+------------------+----------+\n",
      "|       Documentary|   Neutral|\n",
      "|           Fantasy|   Neutral|\n",
      "|              IMAX|   Neutral|\n",
      "|         Adventure|   Neutral|\n",
      "|               War|      Dark|\n",
      "|         Animation|   Neutral|\n",
      "|            Comedy|    Bright|\n",
      "|          Thriller|      Dark|\n",
      "|         Film-Noir|      Dark|\n",
      "|             Crime|   Neutral|\n",
      "|            Sci-Fi|   Neutral|\n",
      "|           Musical|   Neutral|\n",
      "|           Mystery|   Neutral|\n",
      "|             Drama|      Dark|\n",
      "|            Horror|      Dark|\n",
      "|           Western|   Neutral|\n",
      "|           Romance|    Bright|\n",
      "|          Children|   Neutral|\n",
      "|            Action|   Neutral|\n",
      "|(no genres listed)|   Neutral|\n",
      "+------------------+----------+\n",
      "only showing top 20 rows\n",
      "\n"
     ]
    }
   ],
   "source": [
    "cat_tree = (flat_cats\n",
    "            .withColumn(\"parent_cat\", \n",
    "                        sf.when(sf.col(\"category\").isin([\"Romance\", \"Comedy\"]), \"Bright\")\n",
    "                        .otherwise(\n",
    "                            sf.when(sf.col(\"category\").isin([\"Drama\", \"Horror\", \"War\", \"Thriller\", \"Film-Noir\"]), \"Dark\")\n",
    "                            .otherwise(\"Neutral\"))\n",
    "                       )\n",
    "           .unionByName(parent_cats)\n",
    "           )\n",
    "cat_tree.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Model training"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {},
   "outputs": [],
   "source": [
    "model = CatPopRec(cat_tree)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CPU times: user 3.34 ms, sys: 1.74 ms, total: 5.08 ms\n",
      "Wall time: 85.6 ms\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "model.fit(train_dataset)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The model stores number of interactions with each item in each leaf category."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "+------------------+-------+---------+\n",
      "|          category|item_id|relevance|\n",
      "+------------------+-------+---------+\n",
      "|(no genres listed)|   7621|        2|\n",
      "|            Action|   9399|    12314|\n",
      "|            Action|   9621|     8602|\n",
      "|            Action|   9815|     8005|\n",
      "|            Action|   1124|     7979|\n",
      "+------------------+-------+---------+\n",
      "only showing top 5 rows\n",
      "\n"
     ]
    }
   ],
   "source": [
    "model.cat_item_popularity.orderBy(sf.col(\"category\"), sf.col(\"relevance\").desc()).show(5)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Generating recommendations"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "DataFrame[category: string, item_id: bigint, relevance: double]"
      ]
     },
     "execution_count": 35,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pred.unpersist()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "+--------+-------+--------------------+--------------------------------------+------------------------+\n",
      "|category|item_id|relevance           |title                                 |genres                  |\n",
      "+--------+-------+--------------------+--------------------------------------+------------------------+\n",
      "|Bright  |2021   |0.007996300880202234|Ace Ventura: Pet Detective (1994)     |Comedy                  |\n",
      "|Bright  |3825   |0.007493301805905117|Pretty Woman (1990)                   |Comedy|Romance          |\n",
      "|Bright  |241    |0.006798332442701945|Four Weddings and a Funeral (1994)    |Comedy|Romance          |\n",
      "|Bright  |3822   |0.006763260947613339|Dumb & Dumber (1994)                  |Comedy                  |\n",
      "|Bright  |661    |0.006537603564740596|Forrest Gump (1994)                   |Comedy|Drama|Romance|War|\n",
      "|Comedy  |2021   |0.01093788169438695 |Ace Ventura: Pet Detective (1994)     |Comedy                  |\n",
      "|Comedy  |3822   |0.009251246197653228|Dumb & Dumber (1994)                  |Comedy                  |\n",
      "|Comedy  |6574   |0.007837300272247712|Monty Python and the Holy Grail (1975)|Comedy                  |\n",
      "|Comedy  |232    |0.007703480390021834|Birdcage, The (1996)                  |Comedy                  |\n",
      "|Comedy  |5660   |0.006944746906835482|Clerks (1994)                         |Comedy                  |\n",
      "|Romance |3825   |0.01405497414989799 |Pretty Woman (1990)                   |Comedy|Romance          |\n",
      "|Romance |241    |0.01246433063421548 |Four Weddings and a Funeral (1994)    |Comedy|Romance          |\n",
      "|Romance |661    |0.012381967215927712|Forrest Gump (1994)                   |Comedy|Drama|Romance|War|\n",
      "|Romance |5652   |0.011688741778672336|Clueless (1995)                       |Comedy|Romance          |\n",
      "|Romance |3823   |0.01161667378767054 |Speed (1994)                          |Action|Romance|Thriller |\n",
      "+--------+-------+--------------------+--------------------------------------+------------------------+\n",
      "\n",
      "CPU times: user 31.6 ms, sys: 19.8 ms, total: 51.4 ms\n",
      "Wall time: 2.5 s\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "pred = model.predict([\"Bright\", \"Romance\", \"Comedy\"], k=5)\n",
    "(pred\n",
    " .join(item_features_replay, on=\"item_id\")\n",
    " .orderBy(sf.col(\"category\"), sf.col(\"relevance\").desc())\n",
    " .select(\"category\", \"item_id\", \"relevance\", \"title\", \"genres\")\n",
    ").show(truncate=False)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.12"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}