Fail to reproduce performance in some scenes

[bbangsik13]

Congratulations on your acceptance of the paper!

I experimented with N3V dataset through the code you provided, but I can't reproduce the performance in certain scenes. Can I get some advice?

I tested with the environment you provided and COLMAP 3.9 & 3.10 version.

martini.mp4

salmon.mp4

[Alexander0Yang]

Have you made any modifications to the config? It looks like the bgmap didn’t work.

It seems that the view outside the window was completely occluded by the Gaussians distributed on the window, and there are some undesired accumulations opacity causing black shadows on the walls.

This might be because some Gaussians are less constrained in some training view. This could be related to initialization, but I have never encountered this situation. 😂
BTW, my colmap version is also 3.9.

Perhaps providing some intermediate results from the training would be more informative for identifying the problem.

If the issue is still not resolved, I would provide you with a checkpoint file in the next few days to help your reproduction.

[bbangsik13]

Thank you for your quick reply. 👍

I checked the point cloud input and the intermediate rendering results and found no problem.

render_ckpt7000.mp4

Additionally, the specific values of the config for the failed scenes (Martini & Salmon) and the well-reconstructed scenes (other scenes) are different, so I wonder if you meant it.
PipelineParams.env_map_res and PipelineParams.env_optimize_until have different values (see below)

[config/dynerf/coffee_martini.yaml] vs [config/dynerf/cook_spinach.yaml]

BTW, I didn't modify the config file you provided. 😂

[Alexander0Yang]

Hi @bbangsik13,
Here is my checkpoint, tensorboard record, and input point cloud generated during testing the open-sourced code.

I just verified the released code in a different environment and was still able to reproduce the similar result.

I checked the point cloud input and the intermediate rendering results and found no problem.

I mean the quantitative results recorded in tensorboard during training. Checking how these metrics changed might be helpful. As can be seen in our tensorboard, the psnr have not been so bad even in the 500 iterations.

It is very weird. Could you provide more detailed environment information for locating the problem?

[bbangsik13]

I think I've solved the problem.

I used the render code of #12 to measure the metric after rendering. When I debugged the render.py , there was a problem that the parameter env_map of the ckpt did not loaded. Changing the code to load it up, the result came out close to the result of the tensorboard you shared.

Only salmon and martini scenes use env_map, so I think the results came out right in other scenes.

Thank you for your assistance.

[Joaquin-Gajardo]

@bbangsik13 what was your specific change for loading up the env_map parameter correctly? btw I also tested the metrics for some scenes and get similar values: your exact same PSNR for cook_spinach, 33.33 for cut_roasted_beef and 33.79 for flame_steak.

Thank you!

I think I've solved the problem.

I used the render code of #12 to measure the metric after rendering. When I debugged the render.py , there was a problem that the parameter env_map of the ckpt did not loaded. Changing the code to load it up, the result came out close to the result of the tensorboard you shared.

Only salmon and martini scenes use env_map, so I think the results came out right in other scenes.

Thank you for your assistance.

[bbangsik13]

Hi @Joaquin-Gajardo, sorry for the late response.

As it's been a while since I made those modifications, I'm not sure about all details.

For the code from #12, It does not load envmap options from the yaml file and values from the checkpoint.
While I'm not entirely certain, maybe I hardcoded the envmap options and modified scene.__init__.py line 81 to use the restore function instead of create_from_pth. I could have organized the code better but preferred not to at the time. The only requirement is to load the trained env_map.

If you're still experiencing issues, please feel free to contact me via email and I'll be happy to help.