If you want to run a demo for single image inference, please refer to demo.
This is the official implementation of SeqTR: A Simple yet Universal Network for Visual Grounding, which simplifies and unifies the modelling for visual grounding tasks under a novel point prediction paradigm.
- 2022.07.04: Our work has been accepted as Oral by ECCV 2022.
pip install -r requirements.txt
wget https://github.com/explosion/spacy-models/releases/download/en_vectors_web_lg-2.1.0/en_vectors_web_lg-2.1.0.tar.gz -O en_vectors_web_lg-2.1.0.tar.gz
pip install en_vectors_web_lg-2.1.0.tar.gz
Then install SeqTR package in editable mode:
pip install -e .
- Download our preprocessed json files including the merged dataset for pre-training, and DarkNet-53 model weights trained on MS-COCO object detection task.
- Download the train2014 images from Joseph Redmon's mscoco mirror, original Flickr30K images, ReferItGame images, and Visual Genome images.
The project structure should look like the following:
| -- SeqTR
| -- data
| -- annotations
| -- flickr30k
| -- instances.json
| -- ix_to_token.pkl
| -- token_to_ix.pkl
| -- word_emb.npz
| -- referitgame-berkeley
| -- refcoco-unc
| -- refcocoplus-unc
| -- refcocog-umd
| -- refcocog-google
| -- mixed
| -- weights
| -- darknet.weights
| -- yolov3.weights
| -- images
| -- mscoco
| -- train2014
| -- COCO_train2014_000000000072.jpg
| -- ...
| -- saiaprtc12
| -- 25.jpg
| -- ...
| -- flickr30k
| -- 36979.jpg
| -- ...
| -- visual-genome
| -- 2412112.jpg
| -- ...
| -- configs
| -- seqtr
| -- tools
| -- teaser
Note that the darknet.weights excludes val/test images of RefCOCO/+/g datasets while yolov3.weights does not.
We train SeqTR to perform grouning at bounding box level on a single V100 GPU with 32 GB memory. The following script performs the training:
python tools/train.py configs/seqtr/detection/seqtr_det_[DATASET_NAME].py --cfg-options ema=True
[DATASET_NAME] is one of "flickr30k", "referitgame-berkeley", "refcoco-unc", "refcocoplus-unc", "refcocog-umd", and "refcocog-google".
To train SeqTR to generate the target sequence of ground-truth mask, which is then assembled into the predicted mask by connecting the points, run the following script:
python tools/train.py configs/seqtr/segmentation/seqtr_mask_[DATASET_NAME].py --cfg-options ema=True
Note that instead of sampling 18 points and does not shuffle the sequence for RefCOCO dataset, for RefCOCO+ and RefCOCOg, we uniformly sample 12 points on the mask contour and randomly shffle the sequence with 20% percentage. Therefore, to execute the training on RefCOCO+/g datasets, modify num_ray at line 1 to 12 and model.head.shuffle_fraction to 0.2 at line 35, in configs/seqtr/segmentation/seqtr_mask_darknet.py.
python tools/test.py [PATH_TO_CONFIG_FILE] --load-from [PATH_TO_CHECKPOINT_FILE] --cfg-options ema=True
The script will evaluate both the performance of the model trained with and without EMA.
We pre-train SeqTR on 8 V100 GPUs with 32 GB memory:
bash tools/dist_train.sh configs/seqtr/detection/seqtr_det_mixed.py 8 --cfg-options scheduler_config.max_epoch=15 scheduler_config.decay_steps=[12] scheduler_config.warmup_epochs=1
Then we fine-tune 5 epochs on down-stream datasets:
python tools/train.py configs/seqtr/detection/seqtr_det_[DATASET_NAME].py --finetune-from [PATH_TO_PRETRAINED_CHECKPOINT] --cfg-options scheduler_config.max_epoch=5 scheduler_config.decay_steps=[4] scheduler_config.warmup_epochs=0
Note that both during pre-training and fune-tuning stage, we disable Large Scale Jittering (LSJ) and Exponential Moving Average (EMA), so be cautious whether the LSJ is enabled or not during your fine-tuning stage. For example, one must comment out line 9-10 in configs/_base_/datasets/detection/refcoco-unc.py and do not comment out line 11, also, for evaluation, you should not add --cfg-options ema=True or set it to False.
| val | testA | testB | url | |
| SeqTR detection | 81.23 | 85.00 | 76.08 | model & log |
| SeqTR* detection | 83.72 | 86.51 | 81.24 | model & log |
| pre-trained + fine-tuned SeqTR detection | 87.00 | 90.15 | 83.59 | model & log |
| SeqTR segmentation | 67.26 | 69.79 | 64.12 | model & log |
| pre-trained + fine-tuned SeqTR segmentation | 71.70 | 73.31 | 69/82 | - |
| val | testA | testB | url | |
| SeqTR detection | 68.82 | 75.37 | 58.78 | model & log |
| SeqTR* detection | 71.45 | 76.26 | 64.88 | model & log |
| pre-trained + fine-tuned SeqTR detection | 78.69 | 84.51 | 71.87 | - |
| SeqTR segmentation | 54.14 | 58.93 | 48.19 | - |
| pre-trained + fine-tuned SeqTR segmentation | 63.04 | 66.73 | 58.97 | - |
| val-g | url | val-u | test-u | url | |
| SeqTR detection | - | - | 71.35 | 71.58 | model & log |
| SeqTR* detection | 71.50 | model & log | 74.86 | 74.21 | model & log |
| pre-trained + fine-tuned SeqTR detection | - | - | 82.69 | 83.37 | model & log |
| SeqTR segmentation | - | - | 55.67 | 55.64 | model & log |
| pre-trained + fine-tuned SeqTR segmentation | - | - | 64.69 | 65.74 | model & log |
SeqTR* denotes that its visual encoder is initialized with yolov3.weights, while the visual encoder of the rest are initialized with darknet.weights.
Our codes are highly modularized and flexible to be extended to new architectures,. For instance, one can register new components such as head, fusion to promote your research ideas, or register new data augmentation techniques just as in mmdetection library. Feel free to play :-).
@inproceedings{zhu2022seqtr,
title={Seqtr: A simple yet universal network for visual grounding},
author={Zhu, Chaoyang and Zhou, Yiyi and Shen, Yunhang and Luo, Gen and Pan, Xingjia and Lin, Mingbao and Chen, Chao and Cao, Liujuan and Sun, Xiaoshuai and Ji, Rongrong},
booktitle={Computer Vision--ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23--27, 2022, Proceedings, Part XXXV},
pages={598--615},
year={2022},
organization={Springer}
}
Our code is built upon the open-sourced mmcv, mmdetection and detectron2 libraries.