RAM usage seems excessive during ".rrd" file export

[adujardin]

Describe the bug
This is probably an edge case, when testing the C++ API I used it to log a pretty big amount of ptclouds. The clouds itself are fairly small but logged aggressively in multiple threads. The logging part is handled very well (nice job! 👍) and as expected it consumes a bit of RAM, in the current config of the sample, 15GB.

When trying to save the rrd file in the viewer however, the RAM usage spikes to ~50GB (maybe more but it gets hard to keep going).

Is there options or other format to save the scene that could improve this?

To Reproduce
Steps to reproduce the behavior:

1. Run the snippet that reproduces the isolated behavior: https://github.com/adujardin/zed-rerun-io/blob/main/rerun_dummy_ptclouds/src/main.cpp
   The size and number of logged data can be reduced, the point is it seems to have a factor of ~3-4 of RAM usage between logging and exporting.
2. In the viewer, Click on save to 'rrd' file

Expected behavior
Limited additional RAM allocation for rrd file export.

Desktop (please complete the following information):

• OS: Ubuntu 20

Rerun version
0.10

[teh-cmc]

That unfortunately doesn't surprise me too much 😞 the "export from viewer" feature really needs some a lot of love ASAP.

A workaround in the mean time would be to use RecordingStream::save to save your data to an .rrd file on disk first, and then load that in the Viewer (you can even split the recording into multiple files if needed).

The drawback of that method is that you will need to wait for the logging process to be done before you can load the full file in the viewer:

• Viewer should be able to stream in .rrd files whilst they're still being written to #4056

Another alternative would be to log simultaneously to the viewer and to a file, but we don't support that yet:

• Support multiple sinks at once (tee logging) #1998

[adujardin]

Got it! I understand yes. Simultaneously logging in to the viewer and a file would be great 👍

[teh-cmc]

That should be fixed -- Menu > Save barely allocates nor copies these days (0.19).