yolox-full-training-pipeline-for-cots-dataset.ipynb

{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"# Train YOLOX on COTS dataset (PART 1 - TRAINING)\n\nThis notebook shows how to train custom object detection model (COTS dataset) on Kaggle. It could be good starting point for build own custom model based on YOLOX detector. Full github repository you can find here - [YOLOX](https://github.com/Megvii-BaseDetection/YOLOX)\n\n<div align = 'center'><img src='https://github.com/Megvii-BaseDetection/YOLOX/raw/main/assets/logo.png'/></div>\n\n**Steps covered in this notebook:**\n* Install YOLOX \n* Prepare COTS dataset for YOLOX object detection training\n* Download Pre-Trained Weights for YOLOX\n* Prepare configuration files\n* YOLOX training\n* Run YOLOX inference on test images\n* Export YOLOX weights for Tensorflow inference (soon)\n\nNow I created notebook for learning and prototyping in YOLOX. Next step is too create better model (play with YOLOX experimentation parameters).","metadata":{"execution":{"iopub.status.busy":"2021-11-29T13:34:31.033138Z","iopub.execute_input":"2021-11-29T13:34:31.033449Z","iopub.status.idle":"2021-11-29T13:34:33.455468Z","shell.execute_reply.started":"2021-11-29T13:34:31.033368Z","shell.execute_reply":"2021-11-29T13:34:33.454141Z"}}},{"cell_type":"markdown","source":"<div class=\"alert alert-warning\">\n<strong>I found that there is no reference custom model training YOLOX notebook on Kaggle (or I am bad in searching ... ). Since we have such an opportunity this is my contribution to this competition. Feel free to use it and enjoy!\n    I really appreciate if you upvote this notebook. Thank you! </strong>\n</div>\n\n\n<div class=\"alert alert-success\" role=\"alert\">\nThis work consists of two parts:     \n    <ul>\n        <li> PART 1 - TRAIN CUSTOM MODEL (for COTS dataset) - > YoloX full training pipeline for COTS dataset -> this notebook</li>\n        <li> PART 2 - INFERENCE PART - YOLOX on Kaggle for COTS is available -> <a href=\"https://www.kaggle.com/remekkinas/yolox-inference-on-kaggle-for-cots\">YOLOX detections submission made on COTS dataset (PART 2 - DETECTION)</a></li>\n    </ul>\n    \n</div>","metadata":{}},{"cell_type":"code","source":"import warnings\nwarnings.filterwarnings(\"ignore\")\n\nimport ast\nimport os\nimport json\nimport pandas as pd\nimport torch\nimport importlib\nimport cv2 \n\nfrom shutil import copyfile\nfrom tqdm.notebook import tqdm\ntqdm.pandas()\nfrom sklearn.model_selection import GroupKFold\nfrom PIL import Image\nfrom string import Template\nfrom IPython.display import display\n\nTRAIN_PATH = '/kaggle/input/tensorflow-great-barrier-reef'","metadata":{"execution":{"iopub.status.busy":"2022-01-12T11:04:13.365322Z","iopub.execute_input":"2022-01-12T11:04:13.365813Z","iopub.status.idle":"2022-01-12T11:04:15.640332Z","shell.execute_reply.started":"2022-01-12T11:04:13.365773Z","shell.execute_reply":"2022-01-12T11:04:15.639609Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# check Torch and CUDA version\nprint(f\"Torch: {torch.__version__}\")\n!nvcc --version","metadata":{"execution":{"iopub.status.busy":"2022-01-12T11:04:15.641739Z","iopub.execute_input":"2022-01-12T11:04:15.642005Z","iopub.status.idle":"2022-01-12T11:04:16.320902Z","shell.execute_reply.started":"2022-01-12T11:04:15.641962Z","shell.execute_reply":"2022-01-12T11:04:16.320055Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# 1. INSTALL YOLOX","metadata":{}},{"cell_type":"code","source":"!git clone https://github.com/Megvii-BaseDetection/YOLOX -q\n\n%cd YOLOX\n!pip install -U pip && pip install -r requirements.txt\n!pip install -v -e . ","metadata":{"_kg_hide-output":true,"execution":{"iopub.status.busy":"2022-01-12T11:04:16.475874Z","iopub.execute_input":"2022-01-12T11:04:16.476130Z","iopub.status.idle":"2022-01-12T11:05:03.685932Z","shell.execute_reply.started":"2022-01-12T11:04:16.476100Z","shell.execute_reply":"2022-01-12T11:05:03.685026Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"!pip install 'git+https://github.com/cocodataset/cocoapi.git#subdirectory=PythonAPI'","metadata":{"_kg_hide-output":true,"execution":{"iopub.status.busy":"2022-01-12T11:05:03.689522Z","iopub.execute_input":"2022-01-12T11:05:03.689744Z","iopub.status.idle":"2022-01-12T11:05:20.264570Z","shell.execute_reply.started":"2022-01-12T11:05:03.689716Z","shell.execute_reply":"2022-01-12T11:05:20.263718Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# 2. PREPARE COTS DATASET FOR YOLOX\nThis section is taken from  notebook created by Awsaf [Great-Barrier-Reef: YOLOv5 train](https://www.kaggle.com/awsaf49/great-barrier-reef-yolov5-train)\n\n## A. PREPARE DATASET AND ANNOTATIONS","metadata":{}},{"cell_type":"code","source":"def get_bbox(annots):\n    bboxes = [list(annot.values()) for annot in annots]\n    return bboxes\n\ndef get_path(row):\n    row['image_path'] = f'{TRAIN_PATH}/train_images/video_{row.video_id}/{row.video_frame}.jpg'\n    return row","metadata":{"execution":{"iopub.status.busy":"2022-01-12T11:05:20.267022Z","iopub.execute_input":"2022-01-12T11:05:20.267780Z","iopub.status.idle":"2022-01-12T11:05:20.273125Z","shell.execute_reply.started":"2022-01-12T11:05:20.267741Z","shell.execute_reply":"2022-01-12T11:05:20.272382Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"df = pd.read_csv(\"/kaggle/input/tensorflow-great-barrier-reef/train.csv\")\ndf.head(5)","metadata":{"execution":{"iopub.status.busy":"2022-01-12T11:05:20.275605Z","iopub.execute_input":"2022-01-12T11:05:20.276143Z","iopub.status.idle":"2022-01-12T11:05:20.353392Z","shell.execute_reply.started":"2022-01-12T11:05:20.276107Z","shell.execute_reply":"2022-01-12T11:05:20.352619Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Taken only annotated photos\ndf[\"num_bbox\"] = df['annotations'].apply(lambda x: str.count(x, 'x'))\ndf_train = df[df[\"num_bbox\"]>0]\n\n#Annotations \ndf_train['annotations'] = df_train['annotations'].progress_apply(lambda x: ast.literal_eval(x))\ndf_train['bboxes'] = df_train.annotations.progress_apply(get_bbox)\n\n#Images resolution\ndf_train[\"width\"] = 1280\ndf_train[\"height\"] = 720\n\n#Path of images\ndf_train = df_train.progress_apply(get_path, axis=1)","metadata":{"execution":{"iopub.status.busy":"2022-01-12T11:05:20.354647Z","iopub.execute_input":"2022-01-12T11:05:20.355328Z","iopub.status.idle":"2022-01-12T11:05:24.299037Z","shell.execute_reply.started":"2022-01-12T11:05:20.355288Z","shell.execute_reply":"2022-01-12T11:05:24.298196Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"kf = GroupKFold(n_splits = 5) \ndf_train = df_train.reset_index(drop=True)\ndf_train['fold'] = -1\nfor fold, (train_idx, val_idx) in enumerate(kf.split(df_train, y = df_train.video_id.tolist(), groups=df_train.sequence)):\n    df_train.loc[val_idx, 'fold'] = fold\n\ndf_train.head(5)","metadata":{"execution":{"iopub.status.busy":"2022-01-12T11:05:24.300581Z","iopub.execute_input":"2022-01-12T11:05:24.300844Z","iopub.status.idle":"2022-01-12T11:05:24.336074Z","shell.execute_reply.started":"2022-01-12T11:05:24.300807Z","shell.execute_reply":"2022-01-12T11:05:24.334618Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"HOME_DIR = '/kaggle/working/' \nDATASET_PATH = 'dataset/images'\n\n!mkdir {HOME_DIR}dataset\n!mkdir {HOME_DIR}{DATASET_PATH}\n!mkdir {HOME_DIR}{DATASET_PATH}/train2017\n!mkdir {HOME_DIR}{DATASET_PATH}/val2017\n!mkdir {HOME_DIR}{DATASET_PATH}/annotations","metadata":{"execution":{"iopub.status.busy":"2022-01-12T11:05:24.337407Z","iopub.execute_input":"2022-01-12T11:05:24.337653Z","iopub.status.idle":"2022-01-12T11:05:27.847597Z","shell.execute_reply.started":"2022-01-12T11:05:24.337620Z","shell.execute_reply":"2022-01-12T11:05:27.846584Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"SELECTED_FOLD = 4\n\nfor i in tqdm(range(len(df_train))):\n    row = df_train.loc[i]\n    if row.fold != SELECTED_FOLD:\n        copyfile(f'{row.image_path}', f'{HOME_DIR}{DATASET_PATH}/train2017/{row.image_id}.jpg')\n    else:\n        copyfile(f'{row.image_path}', f'{HOME_DIR}{DATASET_PATH}/val2017/{row.image_id}.jpg') ","metadata":{"execution":{"iopub.status.busy":"2022-01-12T11:05:27.849181Z","iopub.execute_input":"2022-01-12T11:05:27.849445Z","iopub.status.idle":"2022-01-12T11:06:22.410527Z","shell.execute_reply.started":"2022-01-12T11:05:27.849408Z","shell.execute_reply":"2022-01-12T11:06:22.409796Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"print(f'Number of training files: {len(os.listdir(f\"{HOME_DIR}{DATASET_PATH}/train2017/\"))}')\nprint(f'Number of validation files: {len(os.listdir(f\"{HOME_DIR}{DATASET_PATH}/val2017/\"))}')","metadata":{"execution":{"iopub.status.busy":"2022-01-12T11:06:22.411958Z","iopub.execute_input":"2022-01-12T11:06:22.412367Z","iopub.status.idle":"2022-01-12T11:06:22.421325Z","shell.execute_reply.started":"2022-01-12T11:06:22.412330Z","shell.execute_reply":"2022-01-12T11:06:22.420496Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## B. CREATE COCO ANNOTATION FILES","metadata":{}},{"cell_type":"code","source":"def save_annot_json(json_annotation, filename):\n    with open(filename, 'w') as f:\n        output_json = json.dumps(json_annotation)\n        f.write(output_json)","metadata":{"execution":{"iopub.status.busy":"2022-01-12T11:06:22.425340Z","iopub.execute_input":"2022-01-12T11:06:22.425555Z","iopub.status.idle":"2022-01-12T11:06:22.429886Z","shell.execute_reply.started":"2022-01-12T11:06:22.425524Z","shell.execute_reply":"2022-01-12T11:06:22.429030Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"annotion_id = 0","metadata":{"execution":{"iopub.status.busy":"2022-01-12T11:06:22.431316Z","iopub.execute_input":"2022-01-12T11:06:22.431928Z","iopub.status.idle":"2022-01-12T11:06:22.437575Z","shell.execute_reply.started":"2022-01-12T11:06:22.431893Z","shell.execute_reply":"2022-01-12T11:06:22.436872Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def dataset2coco(df, dest_path):\n    \n    global annotion_id\n    \n    annotations_json = {\n        \"info\": [],\n        \"licenses\": [],\n        \"categories\": [],\n        \"images\": [],\n        \"annotations\": []\n    }\n    \n    info = {\n        \"year\": \"2021\",\n        \"version\": \"1\",\n        \"description\": \"COTS dataset - COCO format\",\n        \"contributor\": \"\",\n        \"url\": \"https://kaggle.com\",\n        \"date_created\": \"2021-11-30T15:01:26+00:00\"\n    }\n    annotations_json[\"info\"].append(info)\n    \n    lic = {\n            \"id\": 1,\n            \"url\": \"\",\n            \"name\": \"Unknown\"\n        }\n    annotations_json[\"licenses\"].append(lic)\n\n    classes = {\"id\": 0, \"name\": \"starfish\", \"supercategory\": \"none\"}\n\n    annotations_json[\"categories\"].append(classes)\n\n    \n    for ann_row in df.itertuples():\n            \n        images = {\n            \"id\": ann_row[0],\n            \"license\": 1,\n            \"file_name\": ann_row.image_id + '.jpg',\n            \"height\": ann_row.height,\n            \"width\": ann_row.width,\n            \"date_captured\": \"2021-11-30T15:01:26+00:00\"\n        }\n        \n        annotations_json[\"images\"].append(images)\n        \n        bbox_list = ann_row.bboxes\n        \n        for bbox in bbox_list:\n            b_width = bbox[2]\n            b_height = bbox[3]\n            \n            # some boxes in COTS are outside the image height and width\n            if (bbox[0] + bbox[2] > 1280):\n                b_width = bbox[0] - 1280 \n            if (bbox[1] + bbox[3] > 720):\n                b_height = bbox[1] - 720 \n                \n            image_annotations = {\n                \"id\": annotion_id,\n                \"image_id\": ann_row[0],\n                \"category_id\": 0,\n                \"bbox\": [bbox[0], bbox[1], b_width, b_height],\n                \"area\": bbox[2] * bbox[3],\n                \"segmentation\": [],\n                \"iscrowd\": 0\n            }\n            \n            annotion_id += 1\n            annotations_json[\"annotations\"].append(image_annotations)\n        \n        \n    print(f\"Dataset COTS annotation to COCO json format completed! Files: {len(df)}\")\n    return annotations_json","metadata":{"execution":{"iopub.status.busy":"2022-01-12T11:06:22.439205Z","iopub.execute_input":"2022-01-12T11:06:22.439456Z","iopub.status.idle":"2022-01-12T11:06:22.452693Z","shell.execute_reply.started":"2022-01-12T11:06:22.439421Z","shell.execute_reply":"2022-01-12T11:06:22.451891Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Convert COTS dataset to JSON COCO\ntrain_annot_json = dataset2coco(df_train[df_train.fold != SELECTED_FOLD], f\"{HOME_DIR}{DATASET_PATH}/train2017/\")\nval_annot_json = dataset2coco(df_train[df_train.fold == SELECTED_FOLD], f\"{HOME_DIR}{DATASET_PATH}/val2017/\")\n\n# Save converted annotations\nsave_annot_json(train_annot_json, f\"{HOME_DIR}{DATASET_PATH}/annotations/train.json\")\nsave_annot_json(val_annot_json, f\"{HOME_DIR}{DATASET_PATH}/annotations/valid.json\")","metadata":{"execution":{"iopub.status.busy":"2022-01-12T11:06:22.455013Z","iopub.execute_input":"2022-01-12T11:06:22.455194Z","iopub.status.idle":"2022-01-12T11:06:22.689025Z","shell.execute_reply.started":"2022-01-12T11:06:22.455173Z","shell.execute_reply":"2022-01-12T11:06:22.688233Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# 3. PREPARE CONFIGURATION FILE\n\nConfiguration files for Yolox:\n- [YOLOX-nano](https://github.com/Megvii-BaseDetection/YOLOX/blob/main/exps/default/nano.py)\n- [YOLOX-s](https://github.com/Megvii-BaseDetection/YOLOX/blob/main/exps/default/yolox_s.py)\n- [YOLOX-m](https://github.com/Megvii-BaseDetection/YOLOX/blob/main/exps/default/yolox_m.py)\n\nBelow you can find two (yolox-s and yolox-nano) configuration files for our COTS dataset training.\n\n<div align=\"center\"><img  width=\"800\" src=\"https://github.com/Megvii-BaseDetection/YOLOX/raw/main/assets/git_fig.png\"/></div>","metadata":{}},{"cell_type":"code","source":"# Choose model for your experiments NANO or YOLOX-S (you can adapt for other model type)\n\nNANO = False","metadata":{"execution":{"iopub.status.busy":"2022-01-12T11:06:22.690487Z","iopub.execute_input":"2022-01-12T11:06:22.690964Z","iopub.status.idle":"2022-01-12T11:06:22.696083Z","shell.execute_reply.started":"2022-01-12T11:06:22.690924Z","shell.execute_reply":"2022-01-12T11:06:22.695305Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## 3A. YOLOX-S EXPERIMENT CONFIGURATION FILE\nTraining parameters could be set up in experiment config files. I created custom files for YOLOX-s and nano. You can create your own using files from oryginal github repo.","metadata":{}},{"cell_type":"markdown","source":"<div class=\"alert alert-warning\">\n<strong> For YOLOX_s I use input size 960x960 but you can change it for your experiments.</strong> \n</div>","metadata":{}},{"cell_type":"code","source":"config_file_template = '''\n\n#!/usr/bin/env python3\n# -*- coding:utf-8 -*-\n# Copyright (c) Megvii, Inc. and its affiliates.\n\nimport os\n\nfrom yolox.exp import Exp as MyExp\n\n\nclass Exp(MyExp):\n    def __init__(self):\n        super(Exp, self).__init__()\n        self.depth = 0.33\n        self.width = 0.50\n        self.exp_name = os.path.split(os.path.realpath(__file__))[1].split(\".\")[0]\n        \n        # Define yourself dataset path\n        self.data_dir = \"/kaggle/working/dataset/images\"\n        self.train_ann = \"train.json\"\n        self.val_ann = \"valid.json\"\n\n        self.num_classes = 1\n\n        self.max_epoch = $max_epoch\n        self.data_num_workers = 2\n        self.eval_interval = 1\n        \n        self.mosaic_prob = 1.0\n        self.mixup_prob = 1.0\n        self.hsv_prob = 1.0\n        self.flip_prob = 0.5\n        self.no_aug_epochs = 2\n        \n        self.input_size = (960, 960)\n        self.mosaic_scale = (0.5, 1.5)\n        self.random_size = (10, 20)\n        self.test_size = (960, 960)\n'''","metadata":{"execution":{"iopub.status.busy":"2022-01-12T11:06:22.697801Z","iopub.execute_input":"2022-01-12T11:06:22.698190Z","iopub.status.idle":"2022-01-12T11:06:22.707119Z","shell.execute_reply.started":"2022-01-12T11:06:22.698151Z","shell.execute_reply":"2022-01-12T11:06:22.706299Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## 3B. YOLOX-NANO CONFIG FILE\n<div class=\"alert alert-warning\">\n<strong> For YOLOX_nano I use input size 460x460 but you can change it for your experiments.</strong> \n</div","metadata":{}},{"cell_type":"code","source":"if NANO:\n    config_file_template = '''\n\n#!/usr/bin/env python3\n# -*- coding:utf-8 -*-\n# Copyright (c) Megvii, Inc. and its affiliates.\n\nimport os\n\nimport torch.nn as nn\n\nfrom yolox.exp import Exp as MyExp\n\n\nclass Exp(MyExp):\n    def __init__(self):\n        super(Exp, self).__init__()\n        self.depth = 0.33\n        self.width = 0.25\n        self.input_size = (800, 1280)\n        self.mosaic_scale = (0.5, 1.5)\n        self.random_size = (10, 20)\n        self.test_size = (800, 1280)\n        self.exp_name = os.path.split(\n            os.path.realpath(__file__))[1].split(\".\")[0]\n        self.enable_mixup = False\n\n        # Define yourself dataset path\n        self.data_dir = \"/kaggle/working/dataset/images\"\n        self.train_ann = \"train.json\"\n        self.val_ann = \"valid.json\"\n\n        self.num_classes = 1\n\n        self.max_epoch = $max_epoch\n        self.data_num_workers = 2\n        self.eval_interval = 1\n\n    def get_model(self, sublinear=False):\n        def init_yolo(M):\n            for m in M.modules():\n                if isinstance(m, nn.BatchNorm2d):\n                    m.eps = 1e-3\n                    m.momentum = 0.03\n\n        if \"model\" not in self.__dict__:\n            from yolox.models import YOLOX, YOLOPAFPN, YOLOXHead\n            in_channels = [256, 512, 1024]\n            # NANO model use depthwise = True, which is main difference.\n            backbone = YOLOPAFPN(self.depth,\n                                 self.width,\n                                 in_channels=in_channels,\n                                 depthwise=True)\n            head = YOLOXHead(self.num_classes,\n                             self.width,\n                             in_channels=in_channels,\n                             depthwise=True)\n            self.model = YOLOX(backbone, head)\n\n        self.model.apply(init_yolo)\n        self.model.head.initialize_biases(1e-2)\n        return self.model\n\n'''","metadata":{"execution":{"iopub.status.busy":"2022-01-12T11:10:47.555898Z","iopub.execute_input":"2022-01-12T11:10:47.556236Z","iopub.status.idle":"2022-01-12T11:10:47.566777Z","shell.execute_reply.started":"2022-01-12T11:10:47.556199Z","shell.execute_reply":"2022-01-12T11:10:47.565865Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"<div class=\"alert alert-warning\">\n<strong> I trained model for 20 EPOCHS only .... This is for DEMO purposes only.</strong> \n</div>","metadata":{}},{"cell_type":"code","source":"PIPELINE_CONFIG_PATH='cots_config.py'\n\npipeline = Template(config_file_template).substitute(max_epoch = 20)\n\nwith open(PIPELINE_CONFIG_PATH, 'w') as f:\n    f.write(pipeline)","metadata":{"execution":{"iopub.status.busy":"2022-01-12T11:10:48.598457Z","iopub.execute_input":"2022-01-12T11:10:48.598719Z","iopub.status.idle":"2022-01-12T11:10:48.603985Z","shell.execute_reply.started":"2022-01-12T11:10:48.598691Z","shell.execute_reply":"2022-01-12T11:10:48.603202Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# ./yolox/data/datasets/voc_classes.py\n\nvoc_cls = '''\nVOC_CLASSES = (\n  \"starfish\",\n)\n'''\nwith open('./yolox/data/datasets/voc_classes.py', 'w') as f:\n    f.write(voc_cls)\n\n# ./yolox/data/datasets/coco_classes.py\n\ncoco_cls = '''\nCOCO_CLASSES = (\n  \"starfish\",\n)\n'''\nwith open('./yolox/data/datasets/coco_classes.py', 'w') as f:\n    f.write(coco_cls)\n\n# check if everything is ok    \n!more ./yolox/data/datasets/coco_classes.py","metadata":{"execution":{"iopub.status.busy":"2022-01-12T11:06:22.729468Z","iopub.execute_input":"2022-01-12T11:06:22.729744Z","iopub.status.idle":"2022-01-12T11:06:23.422155Z","shell.execute_reply.started":"2022-01-12T11:06:22.729709Z","shell.execute_reply":"2022-01-12T11:06:23.419499Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# 4. DOWNLOAD PRETRAINED WEIGHTS","metadata":{}},{"cell_type":"markdown","source":"List of pretrained models:\n* YOLOX-s\n* YOLOX-m\n* YOLOX-nano for inference speed (!)\n* etc.","metadata":{}},{"cell_type":"code","source":"sh = 'wget https://github.com/Megvii-BaseDetection/storage/releases/download/0.0.1/yolox_s.pth'\nMODEL_FILE = 'yolox_s.pth'\n\nif NANO:\n    sh = '''\n    wget https://github.com/Megvii-BaseDetection/storage/releases/download/0.0.1/yolox_nano.pth\n    '''\n    MODEL_FILE = 'yolox_nano.pth'\n\nwith open('script.sh', 'w') as file:\n  file.write(sh)\n\n!bash script.sh","metadata":{"execution":{"iopub.status.busy":"2021-11-30T16:24:16.590401Z","iopub.execute_input":"2021-11-30T16:24:16.591075Z","iopub.status.idle":"2021-11-30T16:24:17.977995Z","shell.execute_reply.started":"2021-11-30T16:24:16.591028Z","shell.execute_reply":"2021-11-30T16:24:17.977091Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# 5. TRAIN MODEL","metadata":{}},{"cell_type":"code","source":"!cp ./tools/train.py ./","metadata":{"execution":{"iopub.status.busy":"2022-01-12T11:06:23.425507Z","iopub.execute_input":"2022-01-12T11:06:23.425818Z","iopub.status.idle":"2022-01-12T11:06:24.238592Z","shell.execute_reply.started":"2022-01-12T11:06:23.425779Z","shell.execute_reply":"2022-01-12T11:06:24.237629Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"!python train.py \\\n    -f cots_config.py \\\n    -d 1 \\\n    -b 32 \\\n    --fp16 \\\n    -o \\\n    -c {MODEL_FILE}   # Remember to chenge this line if you take different model eg. yolo_nano.pth, yolox_s.pth or yolox_m.pth","metadata":{"execution":{"iopub.status.busy":"2022-01-12T11:06:24.240479Z","iopub.execute_input":"2022-01-12T11:06:24.240755Z","iopub.status.idle":"2022-01-12T11:06:31.093957Z","shell.execute_reply.started":"2022-01-12T11:06:24.240720Z","shell.execute_reply":"2022-01-12T11:06:31.093021Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# 6. RUN INFERENCE\n\n## 6A. INFERENCE USING YOLOX TOOL","metadata":{}},{"cell_type":"code","source":"!ls","metadata":{"execution":{"iopub.status.busy":"2022-01-12T11:06:56.691061Z","iopub.execute_input":"2022-01-12T11:06:56.691419Z","iopub.status.idle":"2022-01-12T11:06:57.390304Z","shell.execute_reply.started":"2022-01-12T11:06:56.691381Z","shell.execute_reply":"2022-01-12T11:06:57.389145Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# I have to fix demo.py file because it:\n# - raises error in Kaggle (cvWaitKey does not work) \n# - saves result files in time named directory eg. /2021_11_29_22_51_08/ which is difficult then to automatically show results\n\n%cp ../../input/yolox-kaggle-fix-for-demo-inference/demo.py tools/demo.py","metadata":{"execution":{"iopub.status.busy":"2022-01-12T11:06:57.393184Z","iopub.execute_input":"2022-01-12T11:06:57.393893Z","iopub.status.idle":"2022-01-12T11:06:58.071141Z","shell.execute_reply.started":"2022-01-12T11:06:57.393796Z","shell.execute_reply":"2022-01-12T11:06:58.070106Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"!ls ../../input/../input/yoloxbestepoch12subseqkfoldpreprocv3/\n","metadata":{"execution":{"iopub.status.busy":"2022-01-12T11:07:43.449229Z","iopub.execute_input":"2022-01-12T11:07:43.449509Z","iopub.status.idle":"2022-01-12T11:07:44.134706Z","shell.execute_reply.started":"2022-01-12T11:07:43.449478Z","shell.execute_reply":"2022-01-12T11:07:44.133876Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"TEST_IMAGE_PATH = \"/kaggle/working/dataset/images/val2017/0-4614.jpg\"\nMODEL_PATH = \"../../input/../input/yoloxbestepoch12subseqkfoldpreprocv3/best_ckpt.pth\"\n\n!python tools/demo.py image \\\n    -f cots_config.py \\\n    -c {MODEL_PATH} \\\n    --path {TEST_IMAGE_PATH} \\\n    --conf 0.1 \\\n    --nms 0.45 \\\n    --tsize 960 \\\n    --save_result \\\n    --device gpu","metadata":{"execution":{"iopub.status.busy":"2022-01-12T11:08:07.936774Z","iopub.execute_input":"2022-01-12T11:08:07.937145Z","iopub.status.idle":"2022-01-12T11:08:09.444715Z","shell.execute_reply.started":"2022-01-12T11:08:07.937100Z","shell.execute_reply":"2022-01-12T11:08:09.443874Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"OUTPUT_IMAGE_PATH = \"./YOLOX_outputs/cots_config/vis_res/0-4614.jpg\" \nImage.open(OUTPUT_IMAGE_PATH)","metadata":{"execution":{"iopub.status.busy":"2021-11-30T17:40:33.958824Z","iopub.execute_input":"2021-11-30T17:40:33.959161Z","iopub.status.idle":"2021-11-30T17:40:34.301375Z","shell.execute_reply.started":"2021-11-30T17:40:33.95912Z","shell.execute_reply":"2021-11-30T17:40:34.299544Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## 6B. INFERENCE USING CUSTOM SCRIPT (IT WOULD BE USED FOR COTS INFERENCE PART)\n\n### 6B.1 SETUP MODEL","metadata":{}},{"cell_type":"code","source":"!ls ../../input/","metadata":{"execution":{"iopub.status.busy":"2022-01-12T11:09:36.803774Z","iopub.execute_input":"2022-01-12T11:09:36.804531Z","iopub.status.idle":"2022-01-12T11:09:36.964773Z","shell.execute_reply.started":"2022-01-12T11:09:36.804488Z","shell.execute_reply":"2022-01-12T11:09:36.963980Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"from yolox.utils import postprocess\nfrom yolox.data.data_augment import ValTransform\n\nCOCO_CLASSES = (\n  \"starfish\",\n)\n\n# get YOLOX experiment\ncurrent_exp = importlib.import_module('cots_config')\nexp = current_exp.Exp()\n\n# set inference parameters\ntest_size = (800, 1280)\nnum_classes = 1\nconfthre = 0.1\nnmsthre = 0.45\n\n\n# get YOLOX model\nmodel = exp.get_model()\nmodel.cuda()\nmodel.eval()\n\n# get custom trained checkpoint\nckpt_file = \"../../input/yoloxbestepoch12subseqkfoldpreprocv3/best_ckpt.pth\"\nckpt = torch.load(ckpt_file, map_location=\"cpu\")\nmodel.load_state_dict(ckpt[\"model\"])","metadata":{"execution":{"iopub.status.busy":"2022-01-12T11:11:13.424760Z","iopub.execute_input":"2022-01-12T11:11:13.425315Z","iopub.status.idle":"2022-01-12T11:11:13.951897Z","shell.execute_reply.started":"2022-01-12T11:11:13.425278Z","shell.execute_reply":"2022-01-12T11:11:13.948293Z"},"collapsed":true,"jupyter":{"outputs_hidden":true},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### 6B.2 INFERENCE BBOXES","metadata":{}},{"cell_type":"code","source":"def yolox_inference(img, model, test_size): \n    bboxes = []\n    bbclasses = []\n    scores = []\n    \n    preproc = ValTransform(legacy = False)\n\n    tensor_img, _ = preproc(img, None, test_size)\n    tensor_img = torch.from_numpy(tensor_img).unsqueeze(0)\n    tensor_img = tensor_img.float()\n    tensor_img = tensor_img.cuda()\n\n    with torch.no_grad():\n        outputs = model(tensor_img)\n        outputs = postprocess(\n                    outputs, num_classes, confthre,\n                    nmsthre, class_agnostic=True\n                )\n\n    if outputs[0] is None:\n        return [], [], []\n    \n    outputs = outputs[0].cpu()\n    bboxes = outputs[:, 0:4]\n\n    bboxes /= min(test_size[0] / img.shape[0], test_size[1] / img.shape[1])\n    bbclasses = outputs[:, 6]\n    scores = outputs[:, 4] * outputs[:, 5]\n    \n    return bboxes, bbclasses, scores","metadata":{"execution":{"iopub.status.busy":"2021-11-30T17:44:42.016709Z","iopub.execute_input":"2021-11-30T17:44:42.017298Z","iopub.status.idle":"2021-11-30T17:44:42.024464Z","shell.execute_reply.started":"2021-11-30T17:44:42.017256Z","shell.execute_reply":"2021-11-30T17:44:42.023731Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### 6B.3 DRAW RESULT","metadata":{}},{"cell_type":"code","source":"def draw_yolox_predictions(img, bboxes, scores, bbclasses, confthre, classes_dict):\n    for i in range(len(bboxes)):\n            box = bboxes[i]\n            cls_id = int(bbclasses[i])\n            score = scores[i]\n            if score < confthre:\n                continue\n            x0 = int(box[0])\n            y0 = int(box[1])\n            x1 = int(box[2])\n            y1 = int(box[3])\n\n            cv2.rectangle(img, (x0, y0), (x1, y1), (0, 255, 0), 2)\n            cv2.putText(img, '{}:{:.1f}%'.format(classes_dict[cls_id], score * 100), (x0, y0 - 3), cv2.FONT_HERSHEY_PLAIN, 0.8, (0,255,0), thickness = 1)\n    return img","metadata":{"execution":{"iopub.status.busy":"2021-11-30T17:40:55.799936Z","iopub.execute_input":"2021-11-30T17:40:55.80021Z","iopub.status.idle":"2021-11-30T17:40:55.80991Z","shell.execute_reply.started":"2021-11-30T17:40:55.800179Z","shell.execute_reply":"2021-11-30T17:40:55.807375Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### 6B.4 ALL PUZZLES TOGETHER","metadata":{}},{"cell_type":"code","source":"TEST_IMAGE_PATH = \"/kaggle/working/dataset/images/val2017/0-4614.jpg\"\nimg = cv2.imread(TEST_IMAGE_PATH)\n\n# Get predictions\nbboxes, bbclasses, scores = yolox_inference(img, model, test_size)\n\n# Draw predictions\nout_image = draw_yolox_predictions(img, bboxes, scores, bbclasses, confthre, COCO_CLASSES)\n\n# Since we load image using OpenCV we have to convert it \nout_image = cv2.cvtColor(out_image, cv2.COLOR_BGR2RGB)\ndisplay(Image.fromarray(out_image))","metadata":{"execution":{"iopub.status.busy":"2021-11-30T17:44:46.826158Z","iopub.execute_input":"2021-11-30T17:44:46.826988Z","iopub.status.idle":"2021-11-30T17:44:47.169804Z","shell.execute_reply.started":"2021-11-30T17:44:46.826943Z","shell.execute_reply":"2021-11-30T17:44:47.169166Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"<div class=\"alert alert-success\" role=\"alert\">\n    Find this notebook helpful? :) Please give me a vote ;) Thank you\n </div>","metadata":{}},{"cell_type":"markdown","source":"# 7. SUBMIT TO COTS COMPETITION AND EVALUATE","metadata":{}},{"cell_type":"code","source":"import greatbarrierreef\n\nenv = greatbarrierreef.make_env()   # initialize the environment\niter_test = env.iter_test()  ","metadata":{"execution":{"iopub.status.busy":"2021-11-30T17:41:53.604564Z","iopub.execute_input":"2021-11-30T17:41:53.605106Z","iopub.status.idle":"2021-11-30T17:41:53.631616Z","shell.execute_reply.started":"2021-11-30T17:41:53.605068Z","shell.execute_reply":"2021-11-30T17:41:53.630689Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"submission_dict = {\n    'id': [],\n    'prediction_string': [],\n}\n\nfor (image_np, sample_prediction_df) in iter_test:\n \n    bboxes, bbclasses, scores = yolox_inference(image_np, model, test_size)\n    \n    predictions = []\n    for i in range(len(bboxes)):\n        box = bboxes[i]\n        cls_id = int(bbclasses[i])\n        score = scores[i]\n        if score < confthre:\n            continue\n        x_min = int(box[0])\n        y_min = int(box[1])\n        x_max = int(box[2])\n        y_max = int(box[3])\n        \n        bbox_width = x_max - x_min\n        bbox_height = y_max - y_min\n        \n        predictions.append('{:.2f} {} {} {} {}'.format(score, x_min, y_min, bbox_width, bbox_height))\n    \n    prediction_str = ' '.join(predictions)\n    sample_prediction_df['annotations'] = prediction_str\n    env.predict(sample_prediction_df)\n\n    print('Prediction:', prediction_str)","metadata":{"execution":{"iopub.status.busy":"2021-11-30T17:44:53.457988Z","iopub.execute_input":"2021-11-30T17:44:53.458918Z","iopub.status.idle":"2021-11-30T17:44:53.467057Z","shell.execute_reply.started":"2021-11-30T17:44:53.458871Z","shell.execute_reply":"2021-11-30T17:44:53.466236Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"sub_df = pd.read_csv('submission.csv')\nsub_df.head()","metadata":{"execution":{"iopub.status.busy":"2021-11-30T17:46:07.26512Z","iopub.execute_input":"2021-11-30T17:46:07.2654Z","iopub.status.idle":"2021-11-30T17:46:07.279331Z","shell.execute_reply.started":"2021-11-30T17:46:07.265368Z","shell.execute_reply":"2021-11-30T17:46:07.27862Z"},"trusted":true},"execution_count":null,"outputs":[]}]}