notebooks/01_get_started/introduction_to_sportslabkit.ipynb

{
 "cells": [
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Introduction to SportsLabKit\n",
    "\n",
    "This quick tutorial introduces the key concepts and basic features of `SportsLabKit` to help you get started with your projects."
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Multiple Object Tracking in `SportsLabKit`"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In a broad definition, Multiple Object Tracking (MOT) is the problem of automatically identifying multiple objects in a video and representing them as a set of trajectories. \n",
    "\n",
    "The typical approach to MOT algorithms follows the tracking-by-detection paradigm, which attempts to solve the problem by first detecting objects in each frame and then associating them with the objects in the previous frame.\n",
    "\n",
    "One large challenge in the tracking-by-detection paradigm is scalability. The detection model is typically a deep learning model, which is computationally expensive. The association model is also computationally expensive, as it requires reID features to be extracted for each bounding box. Recent approaches, such as [TrackFormer](https://arxiv.org/abs/2101.02702)/[TransTrack](https://arxiv.org/abs/2012.15460), have attempted to address this challenge by using a single deep learning model to perform both detection and association. However, there is no clear consensus on the best approach to MOT as tracking-by-detection models are still competitive ([ByteTrack](https://arxiv.org/abs/2110.06864)/[BoT-SORT](https://arxiv.org/abs/2206.14651)/[Strong-SORT](https://arxiv.org/abs/2202.13514)).\n"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "> IMHO, approaches that adhere to “The Bitter Lesson” are the most promising. [Unicorn: Towards Grand Unification of Object Tracking](https://arxiv.org/abs/2207.07078) demonstrates that a single network can solve four tracking problems (SOT, MOT, VOS, MOTS) simultaneously. I think this is a direction many will follow."
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "SportsLabKit implements the tracking-by-detection paradigm. \n",
    "\n",
    "In brief, the algorithm works as follows:\n",
    "\n",
    "1. The detection model (YOLOX, DETR, RCNN etc.) detects items of interest via bounding boxes in each frame, then \n",
    "2. Several feature extractors are used to obtain descripters of each detection (e.g. ReID features, optical flow features, etc.), then\n",
    "3. The association model (Minimum cost bipartite matching) associates detections in the current frame with detections in the previous frame / existing tracklets.\n",
    "\n",
    "For a more detailed explanation of the tracking-by-detection paradigm, please refer to the original [DeepSORT paper](https://arxiv.org/abs/1703.07402), [this blog](https://medium.com/augmented-startups/deepsort-deep-learning-applied-to-object-tracking-924f59f99104) explaining DeepSORT or our [SoccerTrack paper](https://openaccess.thecvf.com/content/CVPR2022W/CVSports/papers/Scott_SoccerTrack_A_Dataset_and_Tracking_Algorithm_for_Soccer_With_Fish-Eye_CVPRW_2022_paper.pdf).\n",
    "\n",
    "![](assets/tracking-by-detection.png)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We chose to start with this approach for two reasons: 1) it is a simple and modular approach and 2) we can explicitly control the use of appearance and motion features. "
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## DataFrames in `SportsLabKit`"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "`SportsLabKit` extends the popular data science library [pandas](https://pandas.pydata.org/) by adding an interface to handle tracking data. If you are not familiar with [pandas](https://pandas.pydata.org/), we recommend taking a quick look at its [Getting started](https://pandas.pydata.org/docs/getting_started/index.html#getting-started) documentation before proceeding.\n",
    "\n",
    "There are two core data structures in `SportsLabKit`, the BoundingBoxDataFrame and the CoordinateDataFrame. Both are,\n",
    "\n",
    "   1. Subclasses of pandas.DataFrame and inherit all of its functionality\n",
    "   2. Inherited from the `SLKMixin` and are designed to work with the `SportsLabKit` API"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![](./assets/dataframe_inheritance.png)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The main difference is that the CoordinateDataFrame has comes built-in with functionality that handles coordinates, while the BoundingBoxDataFrame is made to be compatibile with bounding box data. \n",
    "\n",
    "In a nutshell, both data structures are a multi-indexed `pandas.DataFrame` with a few extra methods and attributes/metadata. This means that we can use various dataframe method directly:\n",
    "\n",
    "* `df.head()` returns the first 5 rows of the dataframe\n",
    "* `df.columns` returns the column names\n",
    "* `df.iloc[0]` returns the first row of the dataframe\n",
    "\n",
    "While also extending functionality to additional convenience functions like:\n",
    "\n",
    "* `df.iter_frames()` iterates over the data frame by frame\n",
    "* `df.iter_teams()` iterates over the data frame by team\n",
    "* `df.iter_players()` iterates over the data frame by player\n",
    "* `df.iter_attributes()` iterates over the data frame by attribute\n",
    "\n",
    "See more about how to use dataframes in the [DataFrame Manipulation](../02_user_guide/dataframe_manipulation.ipynb) tutorial."
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Reading and writing files"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "First, we need to read some data."
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Reading files"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Assuming you have a file containing either bounding box or coordinates data, you can read it using `slk.read_data()`, which automatically detects the filetype and creates a `BBoxDataFrame` or a `CoordinateDataFrame`. This tutorial uses a sample from the SoccerTrack dataset, which is part of the SoccerTrack installation. Therefore, we use `slk.datasets.get_path()` to retrieve the path to the dataset."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "%load_ext autoreload\n",
    "%autoreload 2"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead tr th {\n",
       "        text-align: left;\n",
       "    }\n",
       "\n",
       "    .dataframe thead tr:last-of-type th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr>\n",
       "      <th>TeamID</th>\n",
       "      <th colspan=\"10\" halign=\"left\">0</th>\n",
       "      <th>...</th>\n",
       "      <th colspan=\"5\" halign=\"left\">1</th>\n",
       "      <th colspan=\"5\" halign=\"left\">3</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>PlayerID</th>\n",
       "      <th colspan=\"5\" halign=\"left\">1</th>\n",
       "      <th colspan=\"5\" halign=\"left\">10</th>\n",
       "      <th>...</th>\n",
       "      <th colspan=\"5\" halign=\"left\">9</th>\n",
       "      <th colspan=\"5\" halign=\"left\">0</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Attributes</th>\n",
       "      <th>bb_left</th>\n",
       "      <th>bb_top</th>\n",
       "      <th>bb_width</th>\n",
       "      <th>bb_height</th>\n",
       "      <th>conf</th>\n",
       "      <th>bb_left</th>\n",
       "      <th>bb_top</th>\n",
       "      <th>bb_width</th>\n",
       "      <th>bb_height</th>\n",
       "      <th>conf</th>\n",
       "      <th>...</th>\n",
       "      <th>bb_left</th>\n",
       "      <th>bb_top</th>\n",
       "      <th>bb_width</th>\n",
       "      <th>bb_height</th>\n",
       "      <th>conf</th>\n",
       "      <th>bb_left</th>\n",
       "      <th>bb_top</th>\n",
       "      <th>bb_width</th>\n",
       "      <th>bb_height</th>\n",
       "      <th>conf</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>frame</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>3543.0</td>\n",
       "      <td>607.0</td>\n",
       "      <td>30.0</td>\n",
       "      <td>52.5</td>\n",
       "      <td>1.0</td>\n",
       "      <td>3536.42</td>\n",
       "      <td>555.93</td>\n",
       "      <td>13.57</td>\n",
       "      <td>42.39</td>\n",
       "      <td>1.0</td>\n",
       "      <td>...</td>\n",
       "      <td>2919.31</td>\n",
       "      <td>538.44</td>\n",
       "      <td>23.59</td>\n",
       "      <td>47.18</td>\n",
       "      <td>1.0</td>\n",
       "      <td>3542.77</td>\n",
       "      <td>549.47</td>\n",
       "      <td>6.4</td>\n",
       "      <td>7.0</td>\n",
       "      <td>1.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>3542.0</td>\n",
       "      <td>609.0</td>\n",
       "      <td>32.0</td>\n",
       "      <td>51.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>3536.13</td>\n",
       "      <td>555.96</td>\n",
       "      <td>13.66</td>\n",
       "      <td>42.27</td>\n",
       "      <td>1.0</td>\n",
       "      <td>...</td>\n",
       "      <td>2919.44</td>\n",
       "      <td>538.55</td>\n",
       "      <td>23.59</td>\n",
       "      <td>47.18</td>\n",
       "      <td>1.0</td>\n",
       "      <td>3548.55</td>\n",
       "      <td>549.43</td>\n",
       "      <td>6.4</td>\n",
       "      <td>7.0</td>\n",
       "      <td>1.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>3542.0</td>\n",
       "      <td>611.0</td>\n",
       "      <td>32.0</td>\n",
       "      <td>50.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>3535.85</td>\n",
       "      <td>555.99</td>\n",
       "      <td>13.73</td>\n",
       "      <td>42.16</td>\n",
       "      <td>1.0</td>\n",
       "      <td>...</td>\n",
       "      <td>2919.57</td>\n",
       "      <td>538.66</td>\n",
       "      <td>23.59</td>\n",
       "      <td>47.18</td>\n",
       "      <td>1.0</td>\n",
       "      <td>3554.32</td>\n",
       "      <td>549.40</td>\n",
       "      <td>6.4</td>\n",
       "      <td>7.0</td>\n",
       "      <td>1.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>3542.0</td>\n",
       "      <td>613.0</td>\n",
       "      <td>32.0</td>\n",
       "      <td>49.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>3535.57</td>\n",
       "      <td>556.02</td>\n",
       "      <td>13.80</td>\n",
       "      <td>42.04</td>\n",
       "      <td>1.0</td>\n",
       "      <td>...</td>\n",
       "      <td>2919.70</td>\n",
       "      <td>538.77</td>\n",
       "      <td>23.59</td>\n",
       "      <td>47.18</td>\n",
       "      <td>1.0</td>\n",
       "      <td>3560.10</td>\n",
       "      <td>549.36</td>\n",
       "      <td>6.4</td>\n",
       "      <td>7.0</td>\n",
       "      <td>1.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>3539.0</td>\n",
       "      <td>615.0</td>\n",
       "      <td>36.0</td>\n",
       "      <td>46.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>3535.28</td>\n",
       "      <td>556.04</td>\n",
       "      <td>13.88</td>\n",
       "      <td>41.94</td>\n",
       "      <td>1.0</td>\n",
       "      <td>...</td>\n",
       "      <td>2919.84</td>\n",
       "      <td>538.88</td>\n",
       "      <td>23.59</td>\n",
       "      <td>47.18</td>\n",
       "      <td>1.0</td>\n",
       "      <td>3565.87</td>\n",
       "      <td>549.33</td>\n",
       "      <td>6.4</td>\n",
       "      <td>7.0</td>\n",
       "      <td>1.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>5 rows × 115 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "TeamID           0                                                           \\\n",
       "PlayerID         1                                      10                    \n",
       "Attributes bb_left bb_top bb_width bb_height conf  bb_left  bb_top bb_width   \n",
       "frame                                                                         \n",
       "0           3543.0  607.0     30.0      52.5  1.0  3536.42  555.93    13.57   \n",
       "1           3542.0  609.0     32.0      51.0  1.0  3536.13  555.96    13.66   \n",
       "2           3542.0  611.0     32.0      50.0  1.0  3535.85  555.99    13.73   \n",
       "3           3542.0  613.0     32.0      49.0  1.0  3535.57  556.02    13.80   \n",
       "4           3539.0  615.0     36.0      46.0  1.0  3535.28  556.04    13.88   \n",
       "\n",
       "TeamID                     ...        1                                  \\\n",
       "PlayerID                   ...        9                                   \n",
       "Attributes bb_height conf  ...  bb_left  bb_top bb_width bb_height conf   \n",
       "frame                      ...                                            \n",
       "0              42.39  1.0  ...  2919.31  538.44    23.59     47.18  1.0   \n",
       "1              42.27  1.0  ...  2919.44  538.55    23.59     47.18  1.0   \n",
       "2              42.16  1.0  ...  2919.57  538.66    23.59     47.18  1.0   \n",
       "3              42.04  1.0  ...  2919.70  538.77    23.59     47.18  1.0   \n",
       "4              41.94  1.0  ...  2919.84  538.88    23.59     47.18  1.0   \n",
       "\n",
       "TeamID            3                                  \n",
       "PlayerID          0                                  \n",
       "Attributes  bb_left  bb_top bb_width bb_height conf  \n",
       "frame                                                \n",
       "0           3542.77  549.47      6.4       7.0  1.0  \n",
       "1           3548.55  549.43      6.4       7.0  1.0  \n",
       "2           3554.32  549.40      6.4       7.0  1.0  \n",
       "3           3560.10  549.36      6.4       7.0  1.0  \n",
       "4           3565.87  549.33      6.4       7.0  1.0  \n",
       "\n",
       "[5 rows x 115 columns]"
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import sportslabkit as slk\n",
    "from sportslabkit.logger import show_df # This just makes the df viewable in the notebook.\n",
    "\n",
    "\n",
    "dataset_path = slk.datasets.get_path(\"wide_view\")\n",
    "path_to_csv = sorted(dataset_path.glob(\"annotations/*.csv\"))[0]\n",
    "\n",
    "bbdf = slk.load_df(path_to_csv)\n",
    "\n",
    "bbdf.head()"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To use the full soccertrack dataset, see [\"Dataset Preparation\"](../02_user_guide/dataset_preparation.ipynb)."
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Writing files\n",
    "\n",
    "To write back to file use `BBoxDataFrame.to_csv()`. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "bbdf.to_csv(\"assets/soccertrack_sample.csv\") "
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Visualization\n",
    "\n",
    "Now that we have a bounding box dataframe, we can visualize the results."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "path_to_mp4 = sorted(dataset_path.glob(\"videos/*.mp4\"))[0]"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "It is also possible to download full soccertrack dataset using `soccertrack.datasets.Downloader`. See [\"Dataset Preparation\"](../02_user_guide/dataset_preparation.ipynb). for more details."
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The `BBoxDataFrame` has a built-in `visualize_frame()` method that can be used to visualize the bounding boxes in a single frame."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sportslabkit.utils import cv2pil\n",
    "\n",
    "frame_idx = 50\n",
    "cam = slk.Camera(path_to_mp4)\n",
    "frame = cam.get_frame(frame_idx)\n",
    "resized_frame = cv2pil(bbdf.visualize_frame(frame_idx=frame_idx, frame=frame), False).resize((frame.shape[1]//8, frame.shape[0]//8))"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The `BBoxDataFrame` also has a `visualize_bbox()` method built in. It returns  generators containing a sequence of drawn bounding boxes, which can be passed to the `make_video` method of `soccertrack.utils` to output a video. \n",
    "\n",
    "The following notebook will output `bbox_FISH.mp4` in the current directory."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Writing video: 100it [00:08, 11.88it/s]\n"
     ]
    }
   ],
   "source": [
    "save_path = 'assets/visualize_frames.mp4'\n",
    "bbdf_short = bbdf.iloc[:100].copy()\n",
    "bbdf_short.visualize_frames(path_to_mp4, save_path)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Tracking"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Below we provide a snippiet of code that detects and tracks the players and ball in the downloaded video. First we define the components of our tracking pipeline.\n",
    "\n",
    "- `Camera` - A class that handles camera calibration and coordinate transformations\n",
    "- `detection_model`: A detection model, such as YOLOX, DETR, etc. \n",
    "- `motion_model`:　A motion model that predicts the next position of the players and ball, such as Kalman Filter, Constant Velocity, etc.\n",
    "- `SORTTracker`: A tracker based on the [SORT](https://arxiv.org/abs/1602.00763) algorithm.\n",
    "\n",
    "\n",
    "> Deciding how to architect the structure of the tracking part was very tricky and probably suboptimal. We are still working on improving this part of the library so there may be some breaking changes in the future."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Tracking Progress:   0%|          | 0/100 [00:00<?, ?it/s][W NNPACK.cpp:64] Could not initialize NNPACK! Reason: Unsupported hardware.\n",
      "Tracking Progress: 100%|██████████| 100/100 [03:47<00:00,  2.27s/it, Active: 1, Dead: 2]\n"
     ]
    }
   ],
   "source": [
    "from sportslabkit.utils import get_git_root\n",
    "from sportslabkit.mot import SORTTracker\n",
    "\n",
    "root = get_git_root()\n",
    "cam = slk.Camera(path_to_mp4)\n",
    "\n",
    "det_model = slk.detection_model.load('YOLOv8x', imgsz=640)\n",
    "motion_model = slk.motion_model.load('KalmanFilter', dt=1/30, process_noise=10000, measurement_noise=10)\n",
    "\n",
    "tracker = SORTTracker(detection_model=det_model, motion_model=motion_model)\n",
    "tracker.track(cam[:100])\n",
    "res = tracker.to_bbdf()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Writing video: 97it [00:21,  4.47it/s]\n"
     ]
    }
   ],
   "source": [
    "save_path = \"assets/tracking_results.mp4\"\n",
    "res.visualize_frames(cam.video_path, save_path)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## What next?"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "> This tutorial is a work in progress. If you have any questions or suggestions, please feel free to [reach out](https://twitter.com/AtomJamesScott)."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "soccertrack-KNSgM8DD-py3.9",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.12"
  },
  "orig_nbformat": 4,
  "vscode": {
   "interpreter": {
    "hash": "977a6a4298a6f74f911a0aa33bb1949bba79863f90698d4cbd5dc1167a3d3b34"
   }
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}