+ 
+ 
Visual Effects
-- Using nerfies you can create fun visual effects. This Dolly zoom effect - would be impossible without nerfies since it would require going through a wall. -
- -
- Rethinking Interactive Image Segmentation
Nerfies: Deformable Neural Radiance Fields
+Rethinking Interactive Image Segmentation
with Low Latency, High Quality, and Diverse Prompts
Nerfies: Deformable Neural Radiance Fie + +
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
Abstract
-
- We present the first method capable of photorealistically reconstructing a non-rigidly
- deforming scene using photos/videos captured casually from mobile phones.
-
-
- Our approach augments neural radiance fields
- (NeRF) by optimizing an
- additional continuous volumetric deformation field that warps each observed point into a
- canonical 5D NeRF.
- We observe that these NeRF-like deformation fields are prone to local minima, and
- propose a coarse-to-fine optimization method for coordinate-based models that allows for
- more robust optimization.
- By adapting principles from geometry processing and physical simulation to NeRF-like
- models, we propose an elastic regularization of the deformation field that further
- improves robustness.
-
-
- We show that Nerfies can turn casually captured selfie
- photos/videos into deformable NeRF
- models that allow for photorealistic renderings of the subject from arbitrary
- viewpoints, which we dub "nerfies". We evaluate our method by collecting data
- using a
- rig with two mobile phones that take time-synchronized photos, yielding train/validation
- images of the same pose at different viewpoints. We show that our method faithfully
- reconstructs non-rigidly deforming scenes and reproduces unseen views with high
- fidelity.
-
-
+ The goal of interactive image segmentation is to delineate specific regions
+ within an image via visual or language prompts. Low-latency and high-quality
+ interactive segmentation with diverse prompts remain challenging for existing
+ specialist and generalist models. Specialist models, with their limited
+ prompts and task-specific designs, experience high latency because the image
+ must be recomputed every time the prompt is updated, due to the joint
+ encoding of image and visual prompts. Generalist models, exemplified by the
+ Segment Anything Model (SAM), have recently excelled in prompt diversity and
+ efficiency, lifting image segmentation to the foundation model era.
+ However, for high-quality segmentations, SAM still lags behind state-of-the-art
+ specialist models despite SAM being trained with ×100 more
+ segmentation masks. In this work, we delve deep into the
+ architectural differences between the two types of models.
+ We observe that dense representation and fusion of visual
+ prompts are the key design choices contributing to the high
+ segmentation quality of specialist models. In light of this,
+ we reintroduce this dense design into the generalist models, to facilitate the development of generalist models with
+ high segmentation quality. To densely represent diverse visual prompts, we propose to use a dense map to capture
+ five types: clicks, boxes, polygons, scribbles, and masks.
+ Thus, we propose SegNext, a next-generation interactive
+ segmentation approach offering low latency, high quality,
+ and diverse prompt support. Our method outperforms current state-of-the-art methods on HQSeg-44K and DAVIS,
+ both quantitatively and qualitatively.
+
+
-
-
-
- Video
-
-
-
-
-
Abstract
-
+ The goal of interactive image segmentation is to delineate specific regions
+ within an image via visual or language prompts. Low-latency and high-quality
+ interactive segmentation with diverse prompts remain challenging for existing
+ specialist and generalist models. Specialist models, with their limited
+ prompts and task-specific designs, experience high latency because the image
+ must be recomputed every time the prompt is updated, due to the joint
+ encoding of image and visual prompts. Generalist models, exemplified by the
+ Segment Anything Model (SAM), have recently excelled in prompt diversity and
+ efficiency, lifting image segmentation to the foundation model era.
+ However, for high-quality segmentations, SAM still lags behind state-of-the-art
+ specialist models despite SAM being trained with ×100 more
+ segmentation masks. In this work, we delve deep into the
+ architectural differences between the two types of models.
+ We observe that dense representation and fusion of visual
+ prompts are the key design choices contributing to the high
+ segmentation quality of specialist models. In light of this,
+ we reintroduce this dense design into the generalist models, to facilitate the development of generalist models with
+ high segmentation quality. To densely represent diverse visual prompts, we propose to use a dense map to capture
+ five types: clicks, boxes, polygons, scribbles, and masks.
+ Thus, we propose SegNext, a next-generation interactive
+ segmentation approach offering low latency, high quality,
+ and diverse prompt support. Our method outperforms current state-of-the-art methods on HQSeg-44K and DAVIS,
+ both quantitatively and qualitatively.
+
+ - We present the first method capable of photorealistically reconstructing a non-rigidly - deforming scene using photos/videos captured casually from mobile phones. -
-- Our approach augments neural radiance fields - (NeRF) by optimizing an - additional continuous volumetric deformation field that warps each observed point into a - canonical 5D NeRF. - We observe that these NeRF-like deformation fields are prone to local minima, and - propose a coarse-to-fine optimization method for coordinate-based models that allows for - more robust optimization. - By adapting principles from geometry processing and physical simulation to NeRF-like - models, we propose an elastic regularization of the deformation field that further - improves robustness. -
-- We show that Nerfies can turn casually captured selfie - photos/videos into deformable NeRF - models that allow for photorealistic renderings of the subject from arbitrary - viewpoints, which we dub "nerfies". We evaluate our method by collecting data - using a - rig with two mobile phones that take time-synchronized photos, yielding train/validation - images of the same pose at different viewpoints. We show that our method faithfully - reconstructs non-rigidly deforming scenes and reproduces unseen views with high - fidelity. -
-
-
- Video
-
-
-
- 
- 
-