Input on creating a client and server split for VTK9 based VisIt 3.4

[brugger1]

Problem

Currently we build all the VisIt components together and generate a distribution that contains both the client and server components. We build the viewer and GUI with OpenGL support and the engine with OSMesa support. Support for this was in early versions of VTK, but was deprecated several major releases ago. We have been able to patch VTK to continue to support this functionality up until the current VTK9.0 release. Since that release we have been unable to figure out a way to support that functionality in VTK.

Proposed Solution

Build two VisIt distributions. A client distribution and a server distribution. The client distribution contains the client components and links against a VTK built with OpenGL support. The server distribution contains the server components and links against a VTK build with OSMesa support. The two distributions could easily be combined into a single distribution by un-tarring the client distribution and server distributions into the same directory and then re-tarring up the combined installation.

A single client-server distribution could also be built with only OpenGL support. This would be the common way it is built for development. The client-server distribution with an OpenGL client and an OSMesa server would typically only be built for installations on clusters to support off-screen rendering.

Positives

1. We will be building VTK in a standard way without patches.
2. It will be easier for supercomputing centers that only want to support a server installation to build the distribution.

Negatives

1. Our binaries will be larger since we now have VTK built twice.
2. A few libraries that are currently built in _ser and _par modes would also need to be built
   in a _vwr mode. This is only 8 libraries.
3. Building a combined distribution will be a bit more complex.

Details of the solution

• build_visit will be enhanced to build two versions of VTK.

• They will be placed in vtk-mesagl or vtk-osmesa and vtk-qt within the third_party directory.

• To build both a client and server, I suggest create a build_client and build_server to replace build.

• The control of building the "client" and "server" will be with the CMake variables VISIT_CLIENT_COMPONENTS_ONLY and VISIT_SERVER_COMPONENTS_ONLY. Both will be set to ON by default. The meaning of VISIT_SERVER_COMPONENTS_ONLY will change. Can we remove the _ONLY?

• Either two config site files will be needed or users can specify the values of VISIT_CLIENT_COMPONENTS and VISIT_SERVER_COMPONENTS on the CMake line. Thoughts on which is better?

• The client will consist of the viewer, cli, gui, mdserver.

• The server will consist of the engine, vcl, mdserver.

• In a combined distribution will contain the server version of the mdserver.

• Within the installation the 2 versions will be put in vtk-qt and vtk-osmesa.

• The internal launcher will set the LD_LIBRARY_PATH appropriately for the component being launched.

• The following libraries will have a viewer version

libavtdatabase_vwr.so
libavtfilters_vwr.so
libavtmir_vwr.so
libavtpipeline_vwr.so
libavtplotter_vwr.so
libavtviswindow_vwr.so
libavtwriter_vwr.so
libviewercore_vwr.so

• For the plugins, some of the libraries will be for the client, some for the server and some shared.

Databases:

libE*_ser.so - server
libE*_par.so - server
libI*.so - shared
libM*.so - shared

Plots & Operators:

libE*_ser.so - server
libE*_par.so - server
libG*.so - client
libI*.so - shared
libS*.so - client
libV*.so - client

[markcmiller86]

It occurs to me that scalable-rendering is a key reason for OSMesa but not the only reason. What about rendering to 8kx8k window on my mac laptop. Is that possible with just an OpenGL install? I thought that doing those kinds of renderings required OSMesa. Likewise, what about visit -cli -nowin which is common on a laptop. Doesn't that too require OSMesa?

Next, I am not sure I understand your explanation of what we've currently done prior to VTK9. I mean, it sounds like we had one VTK installed and some of it was using OpenGL and other parts using OSMesa or maybe it could be instanced at run time to use one or the other and we just instanced it how we needed it at run time? And, now they've made this next to impossible?

Sounds like we need Morph-VTK (like there was once a Morph-MPI)...just kidding.

What is ParaView doing for these use cases? I know you were trying to explain it to my yesterday in your office but I didn't fully get it either.

Any ideas how would this effect our CI? Mainly just longer build times I guess??

Is there any way to namespace two VTK builds in the same executable and instance either one on demand in various places in VisIt...a sort of VTK instance factory I guess?

[markcmiller86]

By namespace, I mean a better way to mangle names using C++ features than what name-mangling has typically meant in the past.

[biagas]

Prior to VTK 9, VTK's build had an option VTK_OPENGL_HAS_OSMESA . Turning this option on had the effect of VTK building vtkOSOpenGLRenderWindow (the offscreen render window). The regular onscreen render window (vtkXOpenGLRenderWindow) was also built if you use X (like we always do on Linux). Within VisIt, we have a vtk factory to force the use of this offscreen window in place of vtkXOpenGLRenderWindow when we are running the engine or need offscreen. The osmesa libraries were installed in visit's lib/osmesa dir and internal launcher would load that instead of mesa/gl for engine and '-nowin' situations.

VTK 9 no longer allows you to specify both GL/Mesa with Xlib and OSMesa for the same build, so you can either get a regular RenderWindow or an Offscreen window but not both. (at least for Linux and vtkXOpenGLRenderWindow).

On Windows, the native render window can create an offscreen context, so OSMesa wasn't needed. Is the same true for Mac?

[cyrush]

Yes, before one VTK build would provide with both onscreen and offscreen rendering context.
Then VisIt would select at runtime which one was used, and also select which GL lib was needed (system GL or mesa GL)

Further in the past, we linked both GL libs (mesa and system) and we mangled symbols for the offscreen gl. Since both system GL and mesa were both providing OpenGL function, one had to be mangled. In theory this was great b/c both libs could be loaded in the same exe, however keeping the mangling in straight was hard & in practice we never really needed both at the same time.

[brugger1]

My understanding is the neither the MacOS nor Windows versions use OSMesa.

Good point on my explanation about what is currently being done prior to VTK9 regarding OpenGL versus OSMesa. Yes, we would dynamically at run time adjust the LD_LIBRARY_PATH to use OpenGL for the viewer and OSMesa for the engine.

ParaView long ago (maybe since the beginning) went with a client build and a server build. The client used OpenGL and the server used OSMesa. I'm not sure if they were ever able to use OpenGL for rendering in the server. Whenever Sandia spoke on the issue they always said that software rendering was plenty good enough, which matches what we observed and said.

For the CI, there would probably be a small increase in build times. I suspect the compiles wouldn't be impacted much, but running CMake and creating the tar file would increase since those can't be parallelized. Of course, both builds could be done in parallel with perfect scaling since the builds would be totally independent.

Interesting thought on a VTK instance factory, I'm not sure how to do it.

[cyrush]

Correct, macOS hasn't used osmesa for quite some time. (It did long ago when we still leveraged X11 on macOS)

macOS has some system limit for the offscreen frame buffer size (it might be 10k x 10k, or 16k x 16k)

[biagas]

Is there any way to namespace two VTK builds in the same executable and instance either one on demand in various places in VisIt...a sort of VTK instance factory I guess?

We can create 'ALIAS' libraries (named anything we want) for individual vtk libraries. But that is a lot of work given the number of vtk libraries. Also, after a 'Find' for a say the vtk-qt version, there would be problems with CMake (target collisions) when we did a 'Find' for the vtk-osmesa version.

If there was a way to tell VTK to namespace itself differently when we build and install it that might work.
Guess we could create a VTK patch to change the 'VTK::' namespace to something else. Would probably have to change all uses of NAMESPACE in the VTK source tree so there are no collisions between the two ( they have different namespaces for vtk libs, and wrapped modules, etc).

[markcmiller86]

Not sure how related this is but will mention. First, do we need to do anything special to do hardware rendering from the viewer on multiple GPUs that might be available on the client? I think that situation is more likely now than it was several years ago. I dunno if it influences thinking.

Also, we've had a long standing issue with duplicated rendering-related functionality in both the engine and viewer. That was supposed to all be handled in the VisWindow classes I think but over time, things related to driving rendering and re-execution of plots, etc. has evolved and diffused out of just VisWindow classes and now is duplicated in both engine and viewer.

If we think of rendering as always a parallel thing (MPI and/or threads...also thinking multiple GPUs in the client), can we encapsulate all the rendering functionality in a new thingy in VisIt that is shared by both engine and viewer but maybe compiled differently or instanced differently?