BUG: memory leak when slice series var assigning to itself

[hefvcjm]

Pandas version checks

• I have checked that this issue has not already been reported.

• I have confirmed this bug exists on the latest version of pandas.

• I have confirmed this bug exists on the main branch of pandas.

Reproducible Example

import gc
import pandas as pd

class Test(object):

    def __init__(self, ab):
        self.__a = ab


a = pd.Series([Test(i) for i in range(100000)])
a = a[-1:]
gc.collect()

Issue Description

The memory allocating in the line a = pd.Series([Test(i) for i in range(100000)]) does not free when slicing var a and assigning to itself.

Expected Behavior

Free the memory after slicing like build-in list behavior.

Installed Versions

INSTALLED VERSIONS

commit : 0691c5c
python : 3.12.8
python-bits : 64
OS : Linux
OS-release : 5.4.0-204-generic
Version : #224-Ubuntu SMP Thu Dec 5 13:38:28 UTC 2024
machine : x86_64
processor : x86_64
byteorder : little
LC_ALL : None
LANG : en_US.UTF-8
LOCALE : en_US.UTF-8

pandas : 2.2.3
numpy : 2.2.1
pytz : 2024.2
dateutil : 2.9.0.post0
pip : 24.3.1
Cython : None
sphinx : None
IPython : None
adbc-driver-postgresql: None
adbc-driver-sqlite : None
bs4 : None
blosc : None
bottleneck : None
dataframe-api-compat : None
fastparquet : None
fsspec : None
html5lib : None
hypothesis : None
gcsfs : None
jinja2 : None
lxml.etree : None
matplotlib : None
numba : None
numexpr : None
odfpy : None
openpyxl : None
pandas_gbq : None
psycopg2 : None
pymysql : None
pyarrow : None
pyreadstat : None
pytest : None
python-calamine : None
pyxlsb : None
s3fs : None
scipy : None
sqlalchemy : None
tables : None
tabulate : None
xarray : None
xlrd : None
xlsxwriter : None
zstandard : None
tzdata : 2024.2
qtpy : None
pyqt5 : None

[daxh733]

Thank you for reporting this issue. I appreciate the detailed description and steps you've provided.

1. Explicitly delete the original Series:
   a = pd.Series([Test(i) for i in range(100000)])
   a = a[-1:]
   del a
   gc.collect()

[hefvcjm]

Thank you for reporting this issue. I appreciate the detailed description and steps you've provided.

1. Explicitly delete the original Series:
   a = pd.Series([Test(i) for i in range(100000)])
   a = a[-1:]
   del a
   gc.collect()

Thanks for your replying. But then, after del a, I could not use a. What I expected is a is a series that just holds last item after slicing while the other items memory should free.

[daxh733]

ok!

import pandas as pd
import gc

a = pd.Series([f"Test({i})" for i in range(100000)])

a = a[-1:].copy() ///Using .copy() creates an independent object.
gc.collect()

print(a)

i hope this method using copy will fix the issue

[rhshadrach]

Thanks for the report! Confirmed on main and with copy_on_write=True, further investigations and PRs to fix are welcome.

i hope this method using copy will fix the issue

This is not sufficient. Even without copy, users lose all access to the values and they should be garbage collected. In other words, users should not need to invoke copy here.

[niranjanorkat]

@rhshadrach , I think we have a problem here.

If we internally modify _slice to solve this issue (by breaking the reference to the underlying data block) and free up memory, it seems like we lose functionality that was explicitly designed as a feature.

In particular, tests in copy_view/test_indexing and copy_view/test_methods start failing, which suggests this shared-memory behavior is intentional and tied to pandas' copy-on-write system.

I'm not sure how best to proceed here — would really appreciate your thoughts here.