Add initial documentation about graphics in GTK and WPE ports #126
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
TingPing
reviewed
Jun 25, 2025
carlosgcampos
force-pushed
the
coordinated-graphics
branch
from
91d7154 to
0ad9698
Compare
spenap
reviewed
Jun 30, 2025
|
|
||
| 1. Non-accelerated single process model: At the beginning the graphics model was quite simple, all web contents were rendered into a graphics context in a single process. | ||
| 2. Multi-process model: Then, the multi-process model was introduced ([WebKit2](../../Deep Dive/Architecture/WebKit2.md)) and the rendering model was more or less the same but in a web process. So, we had to find a way to send the rendered contents from the web process to the UI process. The web process rendered into a Cairo image surface and the resulting pixels were copied to a shared memory buffer that the UI process read to create a Cairo surface to pass to the GTK widget. | ||
| 3. Accelerated compositing: The web became more and more complex with animations and other features that required an accelerated compositing model to work properly. The contents were split in layers that were rendered with Cairo image surfaces and then upload as textures with OpenGL. The TextureMapper was introduced to compose all those textures and produce a composited frame. On every paint request, all the layers were flushed, rendered and then composited in the main thread. The shared memory approach to share the rendered contents with the UI process was no longer possible, since the output now is not on main memory but on GPU. Copying the pixels from GPU to main memory for every frame would be very slow. We added a redirected X composite window, used as the target surface of the GL context where the frames were composited and the XDamage extension to notify the UI process when the window was updated. On the UI process a Cairo X11 surface was used to render the generated XPixmap into the GTK widget. |
There was a problem hiding this comment.
Suggested change
| 3. Accelerated compositing: The web became more and more complex with animations and other features that required an accelerated compositing model to work properly. The contents were split in layers that were rendered with Cairo image surfaces and then upload as textures with OpenGL. The TextureMapper was introduced to compose all those textures and produce a composited frame. On every paint request, all the layers were flushed, rendered and then composited in the main thread. The shared memory approach to share the rendered contents with the UI process was no longer possible, since the output now is not on main memory but on GPU. Copying the pixels from GPU to main memory for every frame would be very slow. We added a redirected X composite window, used as the target surface of the GL context where the frames were composited and the XDamage extension to notify the UI process when the window was updated. On the UI process a Cairo X11 surface was used to render the generated XPixmap into the GTK widget. | |
| 3. Accelerated compositing: The web became more and more complex with animations and other features that required an accelerated compositing model to work properly. The contents were split into layers that were rendered with Cairo image surfaces and then uploaded as textures with OpenGL. The TextureMapper was introduced to compose all those textures and produce a composited frame. On every paint request, all the layers were flushed, rendered and then composited in the main thread. The shared memory approach to share the rendered contents with the UI process was no longer possible, since the output is now not on main memory but on GPU, and copying the pixels from GPU to main memory for every frame would be very slow. We added a redirected X composite window, used as the target surface of the GL context where the frames were composited and the XDamage extension to notify the UI process when the window was updated. On the UI process a Cairo X11 surface was used to render the generated XPixmap into the GTK widget. |
| 2. Multi-process model: Then, the multi-process model was introduced ([WebKit2](../../Deep Dive/Architecture/WebKit2.md)) and the rendering model was more or less the same but in a web process. So, we had to find a way to send the rendered contents from the web process to the UI process. The web process rendered into a Cairo image surface and the resulting pixels were copied to a shared memory buffer that the UI process read to create a Cairo surface to pass to the GTK widget. | ||
| 3. Accelerated compositing: The web became more and more complex with animations and other features that required an accelerated compositing model to work properly. The contents were split in layers that were rendered with Cairo image surfaces and then upload as textures with OpenGL. The TextureMapper was introduced to compose all those textures and produce a composited frame. On every paint request, all the layers were flushed, rendered and then composited in the main thread. The shared memory approach to share the rendered contents with the UI process was no longer possible, since the output now is not on main memory but on GPU. Copying the pixels from GPU to main memory for every frame would be very slow. We added a redirected X composite window, used as the target surface of the GL context where the frames were composited and the XDamage extension to notify the UI process when the window was updated. On the UI process a Cairo X11 surface was used to render the generated XPixmap into the GTK widget. | ||
| 4. Wayland: The way to share composited frames with the UI process was X11 specific, so when Wayland support was introduced we had to find a different way. For Wayland we added a nested compositor running in the UI process. The web process connected to the nested compositor to create a surface that was used as the target surface of the GL context where the frames were composited. In the UI process the wayland surface was rendered into the GTK widget by using EGL images when GDK supported it, or downloading the texture to a Cairo image surface otherwise. | ||
| 5. Threaded compositor: Composition was moved to a dedicated thread to release the main thread while compositing layers. The challenge here was to coordinate the layer changes between the main and composited threads. We reused the CoordinatedGraphics implementation that Qt port was using at the time to do the composition in the UI process, adapting it to our threaded model. Performance improved a lot, but the code was more and more complex. |
There was a problem hiding this comment.
Suggested change
| 5. Threaded compositor: Composition was moved to a dedicated thread to release the main thread while compositing layers. The challenge here was to coordinate the layer changes between the main and composited threads. We reused the CoordinatedGraphics implementation that Qt port was using at the time to do the composition in the UI process, adapting it to our threaded model. Performance improved a lot, but the code was more and more complex. | |
| 5. Threaded compositor: Composition was moved to a dedicated thread to release the main thread while compositing layers. The challenge here was to coordinate the layer changes between the main and composited threads. We reused the CoordinatedGraphics implementation that Qt port was using at the time to do the composition in the UI process, adapting it to our threaded model. Performance improved a lot, but the code became increasingly complex. |
| - If the buffer is accelerated and dirty region is smaller than tile area: the dirty region texture needs to be copied into the tile texture. `BitmapTexture::copyFromExternalTexture()` is called to perform a GPU to GPU copy. | ||
| - If the buffer is accelerated and dirty region is the same as the tile area: in this case we can just use the dirty region texture as the tile texture without any copies. `BitmapTexture::swapTexture()` is called in this case. | ||
|
|
||
| Later, when the texture mapper paints the layer `CoordinatedBackingStore::paintToTextureMapper()` is called which iterates the tiles calling `TextureMapper::drawTexture()` with each tile texture. |
There was a problem hiding this comment.
Suggested change
| Later, when the texture mapper paints the layer `CoordinatedBackingStore::paintToTextureMapper()` is called which iterates the tiles calling `TextureMapper::drawTexture()` with each tile texture. | |
| Later, when the texture mapper paints the layer, `CoordinatedBackingStore::paintToTextureMapper()` is called. This iterates the tiles calling `TextureMapper::drawTexture()` with each tile texture. |
|
|
||
| #### Media | ||
|
|
||
| In the case of media, the video frames are provided by the media player at its own pace and updates are not handled by layer flush, but passed directly to the compositor. The class used to handle video frames is `CoordinatedPlatformLayerBufferProxy`. The media player creates a `CoordinatedPlatformLayerBufferProxy` on construction that is returned as its platform layer. `GraphicsLayer::setContentsToPlatformLayer()` is called to set the `CoordinatedPlatformLayerBufferProxy` on the graphics layer. `GraphicsLayer::setContentsToPlatformLayer()` calls `CoordinatedPlatformLayerBufferProxy::setTargetLayer()` passing its `CoordinatedPlatformLayer` to associate the proxy with the platform layer. The first time the proxy is set the layer is marked as changed as usual and if at the time of layer flush there was a pending buffer, it's set to the platrform layer calling `CoordinatedPlatformLayer::setContentsBuffer()` the same way it's done for delegated content. The next frames are handled directly by the proxy and sent to the compositor without a layer flush. Whenever a new frame is available (usually from a GStreamer thread) the media player creates a `CoordinatedPlatformLayerBufferVideo` for the current frame and calls `CoordinatedPlatformLayerBufferProxy::setDisplayBuffer()`. `CoordinatedPlatformLayerBufferProxy::setDisplayBuffer()` saves the buffer as pending if the proxy hasn't been attached to a platform layer or calls `CoordinatedPlatformLayer::setContentsBuffer()` on the attached platform layer and asks the layer to request a composition to its client (`LayerTreeHost`). In `CoordinatedPlatformLayer` the media buffers are handled exactly the same way as the delegated content buffers. `CoordinatedPlatformLayerBufferVideo` is a class that contains another `CoordinatedPlatformLayerBuffer` that depends on the type of video frame and whether the frame contains a DMABuf buffer, GL texture or main memory. |
There was a problem hiding this comment.
Suggested change
| In the case of media, the video frames are provided by the media player at its own pace and updates are not handled by layer flush, but passed directly to the compositor. The class used to handle video frames is `CoordinatedPlatformLayerBufferProxy`. The media player creates a `CoordinatedPlatformLayerBufferProxy` on construction that is returned as its platform layer. `GraphicsLayer::setContentsToPlatformLayer()` is called to set the `CoordinatedPlatformLayerBufferProxy` on the graphics layer. `GraphicsLayer::setContentsToPlatformLayer()` calls `CoordinatedPlatformLayerBufferProxy::setTargetLayer()` passing its `CoordinatedPlatformLayer` to associate the proxy with the platform layer. The first time the proxy is set the layer is marked as changed as usual and if at the time of layer flush there was a pending buffer, it's set to the platrform layer calling `CoordinatedPlatformLayer::setContentsBuffer()` the same way it's done for delegated content. The next frames are handled directly by the proxy and sent to the compositor without a layer flush. Whenever a new frame is available (usually from a GStreamer thread) the media player creates a `CoordinatedPlatformLayerBufferVideo` for the current frame and calls `CoordinatedPlatformLayerBufferProxy::setDisplayBuffer()`. `CoordinatedPlatformLayerBufferProxy::setDisplayBuffer()` saves the buffer as pending if the proxy hasn't been attached to a platform layer or calls `CoordinatedPlatformLayer::setContentsBuffer()` on the attached platform layer and asks the layer to request a composition to its client (`LayerTreeHost`). In `CoordinatedPlatformLayer` the media buffers are handled exactly the same way as the delegated content buffers. `CoordinatedPlatformLayerBufferVideo` is a class that contains another `CoordinatedPlatformLayerBuffer` that depends on the type of video frame and whether the frame contains a DMABuf buffer, GL texture or main memory. | |
| In the case of media, the video frames are provided by the media player at its own pace and updates are not handled by layer flush, but passed directly to the compositor. The class used to handle video frames is `CoordinatedPlatformLayerBufferProxy`. The media player creates a `CoordinatedPlatformLayerBufferProxy` on construction that is returned as its platform layer. `GraphicsLayer::setContentsToPlatformLayer()` is called to set the `CoordinatedPlatformLayerBufferProxy` on the graphics layer. `GraphicsLayer::setContentsToPlatformLayer()` calls `CoordinatedPlatformLayerBufferProxy::setTargetLayer()` passing its `CoordinatedPlatformLayer` to associate the proxy with the platform layer. The first time the proxy is set the layer is marked as changed as usual and if at the time of layer flush there was a pending buffer, it's set to the platform layer calling `CoordinatedPlatformLayer::setContentsBuffer()` the same way it's done for delegated content. The next frames are handled directly by the proxy and sent to the compositor without a layer flush. Whenever a new frame is available (usually from a GStreamer thread) the media player creates a `CoordinatedPlatformLayerBufferVideo` for the current frame and calls `CoordinatedPlatformLayerBufferProxy::setDisplayBuffer()`. `CoordinatedPlatformLayerBufferProxy::setDisplayBuffer()` saves the buffer as pending if the proxy hasn't been attached to a platform layer or calls `CoordinatedPlatformLayer::setContentsBuffer()` on the attached platform layer and asks the layer to request a composition to its client (`LayerTreeHost`). In `CoordinatedPlatformLayer` the media buffers are handled exactly the same way as the delegated content buffers. `CoordinatedPlatformLayerBufferVideo` is a class that contains another `CoordinatedPlatformLayerBuffer` that depends on the type of video frame and whether the frame contains a DMABuf buffer, GL texture or main memory. |
| 4. If the surface was resized, `glViewport()` is called with the new size. | ||
| 5. Surface is asked to clear if needed to call `glClear()` with transparent color if the surface is not opaque. | ||
| 6. Scene is updated: `flushCompositingState()` is called for all the currently active layers in `CoordinatedSceneState`. | ||
| 7. Layer are now painted into the texture mapper. |
There was a problem hiding this comment.
Suggested change
| 7. Layer are now painted into the texture mapper. | |
| 7. Layers are now painted into the texture mapper. |
| 7. Layer are now painted into the texture mapper. | ||
| 8. If there are running animations another composition is requested. | ||
| 9. The composition response identifier is set to the current request identifier and `LayerTreeHost` is notified calling `didComposite()` from a main thread timer passing the composition response identifier. | ||
| 10. The surface is notified that frame has been rendered. In the case of `AcceleratedSurfaceDMABuf` the `Frame` message is sent to the UI process. |
There was a problem hiding this comment.
Suggested change
| 10. The surface is notified that frame has been rendered. In the case of `AcceleratedSurfaceDMABuf` the `Frame` message is sent to the UI process. | |
| 10. The surface is notified that a frame has been rendered. In the case of `AcceleratedSurfaceDMABuf` the `Frame` message is sent to the UI process. |
# for free
to join this conversation on GitHub.
Already have an account?
# to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
0ad9698