fashion_clip_api_demo.ipynb

{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%load_ext autoreload\n",
    "%autoreload 2"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "%reload_ext autoreload"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "Import fashion_clip abstractions and utils"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "import os\n",
    "from fashion_clip.fashion_clip import FashionCLIP, FCLIPDataset\n",
    "from fashion_clip.utils import get_cache_directory, display_images\n",
    "print(\"Cache is at {}\".format(get_cache_directory()))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "## Basic Setup\n",
    "Load dataset:\n",
    "- Download FF dataset\n",
    "- Construct your own dataset\n",
    "\n",
    "Dataset abstraction expects a list of dictionaries. Each element representing a single product/item.\n",
    "Each element __must__ contain at least: _id_, _image_ (filename), _caption_\n",
    "\n",
    "Currently, we assume images are all stored in some local folder"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "# download included dataset\n",
    "# dataset = FCLIPDataset('FF',\n",
    "#                           image_source_path=os.getenv('S3_IMAGES_PATH'),\n",
    "#                           image_source_type='s3')\n",
    "\n",
    "# OR\n",
    "\n",
    "# Loading a local dataset\n",
    "# Create some local dataset\n",
    "catalog = [\n",
    "    {'id': 1, 'image': '16867424.jpg', 'caption': 'light red polo shirt'},\n",
    "    {'id': 2, 'image': '16790484.jpg', 'caption': 'an adidas sneaker'},\n",
    "    {'id': 3, 'image': '16198646.jpg', 'caption': 'dark red polo shirt'},\n",
    "]\n",
    "dataset = FCLIPDataset('farfetch_local',\n",
    "                       image_source_path='./images',\n",
    "                       image_source_type='local',\n",
    "                       catalog=catalog)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "Create FCLIP Object by specifying\n",
    "1. Model (Pre-trained (storedd in S3) or Local File)\n",
    "2. Dataset (FCLIPDataset)\n",
    "\n",
    "Under the hood: if dataset and model combination is recognized via hashing, we download the\n",
    "pre-processed vectors, else upon instantiation we generate vectors for the dataset"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "# Instantiate FCLIP with pre-defined model and dataset\n",
    "fclip = FashionCLIP('fashion-clip', dataset)\n",
    "# fclip = FashionCLIP('openai/clip-vit-base-patch32', dataset)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "## Demo Functionality"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "1 . Zero-shot classification\n",
    "\n",
    "We do not use pre-processed vectors here, although there is a possiblity of adding the option to\n",
    "specify images as SKU/ID as pull pre-processed vectors accordingly."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "test_captions = [\n",
    "    \"nike sneakers\", \"adidas sneakers\", \"nike blue sneakers\", \"converse\", \"nike\",\n",
    "    \"library\", \"the flag of italy\", \"pizza\", \"a gucci dress\"\n",
    "]\n",
    "test_img_path = 'images/16790484.jpg'\n",
    "display_images([test_img_path])\n",
    "fclip.zero_shot_classification([test_img_path], test_captions)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "2 . Product Retrieval\n",
    "\n",
    "We perform retrieval over the images from the dataset specified at instantiation."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "candidates = fclip.retrieval(['shoes'])\n",
    "print(candidates)\n",
    "_ = dataset.display_products(dataset.ids[candidates[0]], fields=tuple(['id']))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "candidates = fclip.retrieval(['shirt'])\n",
    "print(candidates)\n",
    "_ = dataset.display_products(dataset.ids[candidates[0]], fields=tuple(['id']))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "3. Attention Masking\n",
    "\n",
    "Taken from [CLIP Italian](https://huggingface.co/spaces/clip-italian/clip-italian-demo)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "fclip.display_attention(image_path='images/nike_dress.jpg',\n",
    "                        query_text='nike',\n",
    "                        pixel_size=15,\n",
    "                        iterations=15\n",
    ")\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}