Potential memory leak when accessing the Diagnostics attribute of an IntegrityError

[davidcoffin]

We recently had a memory leak on one of our servers and it stopped as soon as we commented out the line of code assigning the Diagnostics attribute of an IntegrityError to a local variable within the exception handler.

The code below replicates the issue for Python 3.5.2, Psycopg 2.6.1. When I run this the amount of resident memory used by the process steadily increases, although not every time func() is called.

#!/usr/bin/env python3

import psycopg2
import itertools
import gc
import os

conn = psycopg2.connect(os.getenv('DSN'))

def main():
    with conn:
        with conn.cursor() as cur:
            cur.execute('CREATE TABLE IF NOT EXISTS leaky (id integer UNIQUE)')

    for n in itertools.count(1):
        print(n, resident_memory_usage())
        func()

def func():
    with conn:
        with conn.cursor() as cur:
            try:
                cur.execute("INSERT INTO leaky VALUES (1)")
            except psycopg2.IntegrityError as e:
                diag = e.diag # this assignment results in a memory leak unless it's explicitly deleted
                #e.diag # just accessing the attribute does not result in a memory leak
    #del diag # explicitly deleting stops the leak
    #gc.collect() # <-- garbage collecting doesn't help even though the only reference should be circular

def resident_memory_usage():
    with open('/proc/self/status') as status:
        for line in status:
            parts = line.split()
            if parts[0][2:-1].lower() == 'rss':
                return int(parts[1])

if __name__ == '__main__':
    main()

[dvarrazzo]

Thank you for the report and the script to reproduce, @davidcoffin !

I've played a bit with it... only to make it worse: implementing object traversal in the Diagnostics object makes only sure that the leak stays there even with the del diag :\

I'll look further into it...

[GuardianRain]

What is the latest progress?