OphNet: A Large-Scale Video Benchmark for Ophthalmic Surgical Workflow Understanding

News

• [Nov, 2024] We have repaired several damaged videos. You can now download the dataset again.
• [Oct, 2024] We realeased OphNet2024 challenge dataset ! More information can be found in Data Preparation.
• [Jul, 2024] OphNet2024 is in preparation——larger scale, more accurate, and more experimental results!
• [Jul, 2024] OphNet was accepted by ECCV2024.
• [Jun, 2024] The manuscript can be found on arXiv.

Introduction

---

Dataset Preparation

Directory Structure

OphNet-benchmark
├── annotation
│   ├── OphNet2024_surgery.csv
│   ├── OphNet2024_loca_all.csv
│   ├── OphNet2024_loca_challenge.csv
│   ├── OphNet2024_loca_challenge_phase.csv
│   ├── OphNet2024_ori_operation_trimmed.csv
│   ├── OphNet2024_ori_phase_trimmed.csv
├── data_processing
│   ├── clipper.py

-annotation

• OphNet2024_surgery.csv: Annotated 1,969 untrimmed videos for surgical types, with the first label as the primary surgery. Selected 743 videos for time-boundary annotation.
• OphNet2024_loca_all.csv: The original version of the time boundary annotations.
• OphNet2024_loca_challenge.csv: Map phase and operation labels with fewer than 15 clips to numeric IDs 51 and 106, which can be interpreted as renaming labels with fewer than 15 instances as "Others."
• OphNet2024_loca_challenge_phase.csv: A complete phase clip in OphNet2024_challenge.csv may be split due to covering multiple operations. Therefore, in OphNet2024_challenge_phase.csv, we merge consecutive clips of the same phase.
• OphNet2024_ori_operation_trimmed.csv & OphNet2024_ori_phase_trimmed.csv: Follow the original labels without processing the tail data, similarly divided into two granularities: phase and operation.

-data_processing

• clipper.py: extract clips based on annotated time boundaries from untrimmed videos.

HuggingFace&Baidu Netdisk

OphNet2024
├── OphNet2024_all (≈305G, 1,969 untrimmed videos--original resolution and FPS)
│   ├── OphNet2024_all.tar.gz.00
│   ├── OphNet2024_all.tar.gz.01
│   ├── ...
├── OphNet2024_trimmed_operation (≈139G, 17,508 trimmed videos from 743 videos with time-boundary annotation--original resolution and FPS)
│   ├── OphNet2024_loca_challenge_trimmed.csv
│   ├── OphNet2024_trimmed_operation.tar.gz.00
│   ├── OphNet2024_trimmed_operation.tar.gz.01
│   ├── ...
├── OphNet2024_trimmed_phase (≈139G, 14,674 trimmed videos from 743 videos with time-boundary annotation--original resolution and FPS)
│   ├── OphNet2024_loca_challenge_phase_trimmed.csv
│   ├── OphNet2024_trimmed_phase.tar.gz.00
│   ├── OphNet2024_trimmed_phase.tar.gz.01
│   ├── ...

• OphNet2024_loca_challenge_trimmed.csv: The OphNet2024_loca_challenge.csv file with the version containing trimmed video names will be automatically created after running data_processing/cliper.py. (/OphNet2024_trimmed_operation)
• OphNet2024_loca_challenge_phase_trimmed.csv: The OphNet2024_loca_challenge_phase.csv file with the version containing trimmed video names will be automatically created after running data_processing/cliper.py. (/OphNet2024_trimmed_phase)

Download

• Label Description: The table with Chinese and English versions of surgery, phase, and operation names along with their ID mappings: OphNet2024_Label

• Untrimmed Videos Download Source: HuggingFace | Baidu Netdisk

  Use the following command to merge and extract the archive:

  cat OphNet2024_all.tar.gz.* | tar xzvf -

• Trimmed Videos Download Source: run the script we provided for trimming:

  python data_processing/cliper.py

  or use the link to download: HuggingFace | Baidu Netdisk. Use the following command to merge and extract the archive:

  operation level

  cat OphNet2024_trimmed_operation.tar.gz.* | tar xzvf -

  phase level

  cat OphNet2024_trimmed_phase.tar.gz.* | tar xzvf -

---

Baselines

Coming soon...

---

Challenge

Coming soon...

Discussion Group

If you have any questions about OphNet, please add this WeChat ID to the WeChat group discussion:

---

TO DO

• Release untrimmed videos
• Release trimmed videos--operation level
• Release trimmed videos--phase level
• Release annotation files
• Release baseline experimental results and checkpoints

---

Citation

@article{hu2024ophnet,
  title={OphNet: A Large-Scale Video Benchmark for Ophthalmic Surgical Workflow Understanding},
  author={Hu, Ming and Xia, Peng and Wang, Lin and Yan, Siyuan and Tang, Feilong and Xu, Zhongxing and Luo, Yimin and Song, Kaimin and Leitner, Jurgen and Cheng, Xuelian and others},
  journal={arXiv preprint arXiv:2406.07471},
  year={2024}
}