Event-based Frame Interpolation with Ad-hoc Deblurring

Lei Sun, Christos Sakaridis, Jingyun Liang, Peng Sun, Jiezhang Cao, Kai Zhang, Qi Jiang, Kaiwei Wang, Luc Van Gool

Paper

CVPR virtual poster

A Unified Framework for Event-based Frame Interpolation with Ad-hoc Deblurring in the Wild

Lei Sun, Daniel Gehrig, Christos Sakaridis, Mathias Gehrig,Jingyun Liang, Peng Sun, Zhijie Xu, Kaiwei Wang, Luc Van Gool, and Davide Scaramuzza

Work done in Robotics and Perception Group, UZH.

Paper

News

• Feb 2025: Our dataset for journal version is available.
• Feb 2025: The journal version: A Unified Framework for Event-based Frame Interpolation with Ad-hoc Deblurring in the Wild is accepted by T-PAMI!
• June 2023: The codes and dataset are publicly available.
• March 2023: The paper is accepted by CVPR 2023

Goal

Unified framework for both event-based sharp and blurry frame interpolation.

Sharp frame interpolation:

• Short exposure time
• Sharp reference frames

Blurry frame interpolation:

• Long exposure time
• Blurry reference frames

Model Architecture

Bi-directional event recurrent block

Event-guided adaptive channel attention

Results

Blurry frame interpolation (Click to expand)

Sharp frame interpolation (Click to expand)

Installation

This implementation based on BasicSR which is a open source toolbox for image/video restoration tasks.

python 3.8.5
pytorch 1.7.1
cuda 11.0

git clone https://github.com/AHupuJR/REFID
cd REFID
pip install -r requirements.txt
python setup.py develop --no_cuda_ext

HighREV dataset

HighREV dataset is a event camera dataset with high spatial resolution. It can be used for event-based image deblurring, event-based frame interpolation, event-based blurry frame interpolation and other event-based low-level image tasks.

HighREV dataset includes:

• Blurry images (png)
• Sharp image (png)
• Event stream (npy)

The blurry images are synthesized from 11 sharp images, and we use RIFE to upsample the framerate of the original frames by 4 times. Thus each blurry image is synthesized from 44 sharp images.

We skip every 1/3 sharp images between each blurry image for frame interpolation task evaluation.

Dataset Download

Because of the commercial reason, dataset download is allowed only with the authority of Alpsentek. Please contacting me or Alpsentek to get the authority from Alpsentek if needed.

Weights

---

GoPro

Blur VFI: 11+1; 11+3

Sharp VFI: 7 skip; 15 skip

Single image deblur

HighREV

Blur VFI: 11+1; 11+3

Sharp VFI: 7 skip; 15 skip

Single image deblur

Train

---

GoPro

• train

  • python -m torch.distributed.launch --nproc_per_node=4 --master_port=4321 basicsr/train.py -opt options/train/GoPro/REFID.yml --launcher pytorch

• eval

  • Download pretrained model to ./experiments/pretrained_models/
  • python basicsr/test.py -opt options/test/GoPro/REFID.yml

HighREV

• train

  • python -m torch.distributed.launch --nproc_per_node=4 --master_port=4321 basicsr/train.py -opt options/train/HighREV/REFID.yml --launcher pytorch

• eval

  • Download pretrained model to ./experiments/pretrained_models/REFID-REBlur.pth
  • python basicsr/test.py -opt options/test/HighREV/REFID.yml

Citations

@article{sun2023event,
  title={Event-Based Frame Interpolation with Ad-hoc Deblurring},
  author={Sun, Lei and Sakaridis, Christos and Liang, Jingyun and Sun, Peng and Cao, Jiezhang and Zhang, Kai and Jiang, Qi and Wang, Kaiwei and Van Gool, Luc},
  journal={arXiv preprint arXiv:2301.05191},
  year={2023}
}

Contact

Should you have any questions, please feel free to contact leosun0331@gmail.com or leo_sun@zju.edu.cn

License and Acknowledgement

This project is under the Apache 2.0 license, and it is based on BasicSR which is under the Apache 2.0 license.