pyo3 large memory footprint when making lists

[Renmusxd]

🐛 Bug Reports

Not entirely sure if bug or working as intended, but lists created from returned pyo3 calls appear to have nearly twice the memory footprint as native python versions, even though the underlying rust representation of the lists can be trivial by comparison. I ran into this issue when running into out of memory issues which I had not expected.

I have been testing the memory usage using memory-profiler from PyPi. I will try others, but the high memory usage seems to agree with some out of memory issues I was experiencing in other places, so I'm confident in the output. See here for the repo with the testing code.

I tested two situations:
py_function returning a list of 100,000,000 bools, this is ~ 20MB in rust, and nearly a GB in python, but pyo3 uses about 1750MB to create the list, after creation memory returns to normal.

py_function returns nested list of tuples with lists inside, also a total of 100,000,000 bools (+16 floats). This again uses only a few MB in rust, nearly a GB in python, but the rust produced version never moves past its peak memory usage and stays at 1750MB until the object is deleted. This doesn't appear to be a memory leak since it is removed with the object, so I'm not sure it fits with #311 / #793. Although this may be fixed with #679 if there isn't another issue.

🌍 Environment

• Your operating system and version: OSX 10.15.3 and Ubuntu 18.04
• Your python version: 3.6.9
• How did you install python (e.g. apt or pyenv)? Did you use a virtualenv?: virtualenv
• Your rust version (rustc --version): 1.44.0-nightly
• Are you using the latest pyo3 version? Have you tried using latest master (replace version = "0.x.y" with git = "https://github.com/PyO3/pyo3")? I've been using 0.9.2, using git has no effect on results.

💥 Reproducing

Here is a repo with the code I used to test these claims.

Final Note

This is a great project, I use it every day for physics research + simulation!

[davidhewitt]

Thanks for reporting, that's a nice comparison. You're right that it doesn't look like #311, and I'm not sure whether #679 is related. Will have to explore...

[kngwyu]

@Renmusxd
Thank you for reporting!

Although this may be fixed with #679 if there isn't another issue.

IntoPy<PyObject> for Vec<T> uses PyObject::into_ptr so I don't think this is related to #679. Maybe it's helpful to compare PyO3's performance with native C extension or Cython.

[kngwyu]

And if you need a 'quick remedy' for this, I recommend using rust-numpy. Numpy array is much more efficient than list.

[Renmusxd]

Thanks, rust-numpy reduced the memory footprint substantially.

[davidhewitt]

I just revisited, bumping the sample pyo3_memory package in the OP from PyO3 0.9 to 0.20, using Python 3.12.

Amazingly everything compiled without changes. Also it looks like whatever the original issue was, it's been resolved. Here's the profile I got this morning:

The Rust peak is now actually lower than the pure-Python peak! 👀