NOVA: Rendering Virtual Worlds with Humans for Computer Vision Tasks

A sample panorama displaying procedurally generated humans by the NOVA framework in a controllable, configurable environment along with their annotations. The first half is photorealistic renderings transitioning between different times of day and the latter half is demonstrating some of the pixel-level annotations NOVA generates for use in various computer vision tasks: (from left to right) instance segmentation, semantic segmentation, optical flow, surface normals and the depth data.

Authors

Abdulrahman Kerim, Lancaster University, a.kerim@lancaster.ac.uk

Cem Aslan, Hacettepe University

Ufuk Celikcan, Hacettepe University

Erkut Erdem, Hacettepe University

Aykut Erdem, Koç University

Abstract

Today, the cutting edge of computer vision research greatly depends on the availability of large datasets, which are critical for effectively training and testing new methods. Manually annotating visual data, however, is not only a labor-intensive process but also prone to errors. In this study, we present NOVA, a versatile framework to create realistic-looking 3D rendered worlds containing procedurally generated humans with rich pixel-level ground truth annotations. NOVA can simulate various environmental factors such as weather conditions or different times of day, and bring an exceptionally diverse set of humans to life, each having a distinct body shape, gender and age. To demonstrate NOVA’s capabilities, we generate two synthetic datasets for person tracking. The first one includes 108 sequences, each with different levels of difficulty like tracking in crowded scenes or at nighttime and aims for testing the limits of current state-of-the-art trackers. A second dataset of 97 sequences with normal weather conditions is used to show how our synthetic sequences can be utilized to train and boost the performance of deep-learning based trackers. Our results indicate that the synthetic data generated by NOVA represents a good proxy of the real-world and can be exploited for computer vision tasks.

Acknowledgements

This work was supported in part by TUBITAK-1001 Program (Grant No. 217E029), GEBIP 2018 fellowship of Turkish Academy of Sciences awarded to E. Erdem, and BAGEP 2021 Award of the Science Academy awarded to A. Erdem.

Citation

• Paper published at Computer Graphics Forum Journal.

@article{0a8eb23d8bc348fcb2a2106e089a6fd9,
title = "NOVA: Rendering Virtual Worlds with Humans for Computer Vision Tasks",
year = "2021",
month = sep,
day = "30",
doi = "10.1111/cgf.14271",
language = "English",
volume = "40",
pages = "258--272",
journal = "Computer Graphics Forum",
issn = "0167-7055",
publisher = "Wiley-Blackwell",
number = "6",
}