Custom materials

[csparker247]

My research group often works with objects that have been imaged under multiple modalities (e.g. RGB, spectral, XRF) in addition to 3D. For example, here's a 3D model of an object, and here's the same 3D surface but textured with an infrared image. We'd like the ability to switch between materials (i.e. textures) as part of the Voyager interface. As far as I can tell, this is not part of the current API.

Given that there is currently built-in material switching support in Voyager, it seems possible that it could be relatively simple to support user-defined materials, but I'm not that familiar with the code base. Has this been considered? Does this idea align with your plans for Voyager? We could possibly have a student contribute this feature if that would be helpful.

[gjcope]

Hi @csparker247 - This is definitely of interest to us. We have a number of folks here doing similar multi-modal imaging who would like to visualize in Voyager. We do have a somewhat hacky way to do it right now using overlay maps, but it is not a robust solution. It's certainly feasible though, and would not be much work. It's on our last to work on at some point, but we won't turn down a PR :)

[csparker247]

Awesome! I suppose extending the current Materials features makes the most sense? Where would you suggest I start looking in your code to get a sense of how to tackle this? In my experimentation with three.js, I've had to make an entirely new material to swap out the texture image.

Here's some initial implementation thoughts/notes I've had just so they're logged somewhere:

• Need to list alternate textures in the scene file along with some descriptive metadata (short name, description, etc.).
• Decide supported image formats. Is there a commonly used image transport for WebGL besides just referencing a jpg/png/webp directly?
• Most of my objects assume the UV map is valid for all alternative texture images, but that doesn't strictly have to be true. How to provide a secondary UV map + texture? Maybe this isn't important right now?
• Probably should start by making use of the built-in Materials panel, but the UI might need modification if many textures are supplied (more than 3-5?)

[gjcope]

I think a good start would be to integrate it into the model material settings in Story, similar to the overlay map. We are currently supporting swapping out this texture by setting the 'needsUpdate' flag of the material. Adding it to the Explorer Material UI is probably the way to as well, but it will need a little more thought.

Addressing your points:

• This can be done now in the 'assets' section of the scene.svx file using "Texture" asset type. The schema definition is here.
• We currently support jpg and png, though internally we almost always use jpg for web content.
• This is a great point. Supporting multiple UV channels is definitely possible, but would need to be implemented generically, not just for this feature. I think for now it would be fine to stick to this assumption and expand later.
• On the Story side, I think we can use a dropdown, similar to the overlay functionality. On the Explorer side something more flexible would definitely be needed.

[gjcope]

Hey @csparker247 I finally had some time to work on this. You can check out an initial implementation at https://github.com/Smithsonian/dpo-voyager/tree/dev-texture-swap. The branch name is a little misleading, I ended up extending our overlay functionality instead of swapping textures. But I think it's a little more efficient in the long run. Alternate maps are accessible via the material menu and can also be changed in tour step states. Happy to hear any feedback you may have for your use cases.

[csparker247]

This is awesome! I'll check it out soon! Thanks!

[csparker247]

Finally had a chance to look into this feature. Before anything else, thanks! I think this is going to be very useful for us. Here's a few things I noticed:

• I'm seeing a difference in overall brightness when using the material approach compared to loading a standalone model with the texture. Shading is also not rendered, which I think could be problematic for some use cases:

• When I was trying to figure out how this worked, I had the Zone asset under a different derivative than my primary model. This lead to a silent error (i.e. javascript console only) about Material [name] not found on current derivative. It just made me wonder about the failure behavior for current derivatives when the zone map for another derivative is enabled. Can that other derivative be force loaded when the material is enabled?

[gjcope]

• Right now it's an additive overlay

  dpo-voyager/source/client/shaders/uberPBRShader.frag

  Line 184 in 833cc1e

  gl_FragColor += vec4(zoneColor.rgb, 1.0);

  due to our initial overlay use-case rather than texture swapping. This explains brightness difference. The same shading should still be happening, but it may be getting blown out by the overlay. Definitely open to suggestions on how to better accommodate both use cases.
• Great point on the derivative support. My inclination is to not offer the texture if not available on the current derivative. We are looking at auto-loading bandwidth-appropriate derivatives, so we could potentially be forcing a user to load a model at resolution they aren't equipped to handle. Happy to discuss.

[gjcope]

Scratch that first bullet. Forgot this was in a branch. The appropriate shader line is:

dpo-voyager/source/client/shaders/uberPBRShader.frag

Line 185 in a263c73

gl_FragColor = vec4(mix(gl_FragColor.rgb, zoneColor.rgb, zoneColor.a), 1.0);

We can definitely look at tweaking it.

[csparker247]

Great point on the derivative support. My inclination is to not offer the texture if not available on the current derivative. We are looking at auto-loading bandwidth-appropriate derivatives, so we could potentially be forcing a user to load a model at resolution they aren't equipped to handle. Happy to discuss.

Yeah, I had been wondering how that might fit into any auto-load functionality. In the case of a material that isn't already enabled, I think a reasonable first step is to grey it out and add a tooltip to the button that indicates it requires a high-resolution derivative. In what you're planning for auto-load, is it possible that a high-res derivative will be in use and then switch to a low-res one (e.g. LOD for performance)? In that case, I think there should be a message in the lower right that indicates why the overlay has been disabled for the user.

Ideally, though, the user (or developer?) should have the manual option to force load any high-resolution derivative they choose, I would think. It might be worth it for some users to wait 1 min to download the ultra-high res model/texture/etc, but you want that to be optional and intention.

Scratch that first bullet. Forgot this was in a branch. The appropriate shader line is:

dpo-voyager/source/client/shaders/uberPBRShader.frag

Line 185 in a263c73

gl_FragColor = vec4(mix(gl_FragColor.rgb, zoneColor.rgb, zoneColor.a), 1.0);

We can definitely look at tweaking it.

I did try tweaking this a bit just to see if it got close:

gl_FragColor = vec4(zoneColor.rgb, 1.0);

I clearly don't understand much about the UberShader, though, because that didn't really seem to do anything. I'm sure you'll find a good solution.

I think I can go ahead and prep some of my documents to support this feature without killing my production env, and test out some of the bigger questions over the coming months (e.g. what happens when we have a lot of these textures in the list). Thanks!

[gjcope]

Agree, disabling/tooltip would be a smoother user experience. I will add this back to the list and play around with the shader.