Skip to content
New issue

Have a question about this project? # for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “#”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? # to your account

Unexpected instance dict size increase #210

Closed
Jongy opened this issue Jul 11, 2020 · 3 comments · Fixed by #216
Closed

Unexpected instance dict size increase #210

Jongy opened this issue Jul 11, 2020 · 3 comments · Fixed by #216
Assignees

Comments

@Jongy
Copy link
Contributor

Jongy commented Jul 11, 2020

Since PEP 412 (landed in Python 3.3/3.4), Python has support for split dictionaries: values and keys can be kept separately to allow for multiple dictionaries (with different values) to share the same keys, reducing memory usage. This was implemented solely for dictionaries holding instance attributes (as of today Python doesn't use this trick anywhere else) and it greatly reduces the size of each instance.

This optimization is seamless, but some actions will disable it for a given type object:

  • Adding additional attributes after multiple instances have been created: for example, an__init__ method that adds 4 attributes, then another method adds 3 more. This unshares the particular dictionary, plus disables the optimization for newly created instances.
  • Deleting an attribute of an instance: del instance.a. This also disables the optimization for new instances.
  • Using non-str key in the instance __dict__ - this makes no sense though. For example instance.__dict__[1] = 1. This unshares the particular dictionary (doesn't disable for future instances)

The last case is not so interesting, but the first and second are not unimaginable, and they have a huge effect on memory consumption (can be demonstrated easily with sys.getsizeof or tracemalloc).

@satwikkansal
Copy link
Owner

Yes, the third case might be too contrived. But we can illustrate the add/remove of instance attributes and the change in the memory footprint.

I'll try to think of a succinct example to demonstrate this and add it in the next update. If you have any ideas / snippets handy, let me know :)

@Jongy
Copy link
Contributor Author

Jongy commented Jul 29, 2020

I'd be happy to write example snippets for the first 2 cases and post a PR (also IIRC I have some snippets demonstrating this somewhere)

@satwikkansal
Copy link
Owner

Awesome, thanks, looking forward to your PR.

Jongy added a commit to Jongy/wtfpython that referenced this issue Jul 31, 2020
Jongy added a commit to Jongy/wtfpython that referenced this issue Jul 31, 2020
# for free to join this conversation on GitHub. Already have an account? # to comment
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants