Skip to content

BUG: DataFrame.explode doesn't explode when using pyarrow large list type #61091

New issue

Have a question about this project? # for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “#”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? # to your account

Closed
3 tasks done
MarcoGorelli opened this issue Mar 9, 2025 · 1 comment · Fixed by #61105
Closed
3 tasks done

BUG: DataFrame.explode doesn't explode when using pyarrow large list type #61091

MarcoGorelli opened this issue Mar 9, 2025 · 1 comment · Fixed by #61105
Assignees
Labels
Arrow pyarrow functionality Bug Reshaping Concat, Merge/Join, Stack/Unstack, Explode

Comments

@MarcoGorelli
Copy link
Member

Pandas version checks

  • I have checked that this issue has not already been reported.

  • I have confirmed this bug exists on the latest version of pandas.

  • I have confirmed this bug exists on the main branch of pandas.

Reproducible Example

In [4]: pd.DataFrame({'a': [[1, 2], [3, 4]]}, dtype=pd.ArrowDtype(pa.list_(pa.int64()))).explode('a')
Out[4]:
   a
0  1
0  2
1  3
1  4

In [5]: pd.DataFrame({'a': [[1, 2], [3, 4]]}, dtype=pd.ArrowDtype(pa.large_list(pa.int64()))).explode('a')
Out[5]:
       a
0  [1 2]
1  [3 4]

Issue Description

For list_, it explodes correctly, but for large_list, it's a no-op

Expected Behavior

   a
0  1
0  2
1  3
1  4

Installed Versions

INSTALLED VERSIONS

commit : 57fd502
python : 3.10.12
python-bits : 64
OS : Linux
OS-release : 5.15.167.4-microsoft-standard-WSL2
Version : #1 SMP Tue Nov 5 00:21:55 UTC 2024
machine : x86_64
processor : x86_64
byteorder : little
LC_ALL : None
LANG : C.UTF-8
LOCALE : en_US.UTF-8

pandas : 3.0.0.dev0+1979.g57fd50221e
numpy : 1.26.4
dateutil : 2.9.0.post0
pip : 25.0.1
Cython : 3.0.12
sphinx : 8.1.3
IPython : 8.33.0
adbc-driver-postgresql: None
adbc-driver-sqlite : None
bs4 : 4.13.3
blosc : None
bottleneck : 1.4.2
fastparquet : 2024.11.0
fsspec : 2025.2.0
html5lib : 1.1
hypothesis : 6.127.5
gcsfs : 2025.2.0
jinja2 : 3.1.5
lxml.etree : 5.3.1
matplotlib : 3.10.1
numba : 0.61.0
numexpr : 2.10.2
odfpy : None
openpyxl : 3.1.5
psycopg2 : 2.9.10
pymysql : 1.4.6
pyarrow : 19.0.1
pyreadstat : 1.2.8
pytest : 8.3.5
python-calamine : None
pytz : 2025.1
pyxlsb : 1.0.10
s3fs : 2025.2.0
scipy : 1.15.2
sqlalchemy : 2.0.38
tables : 3.10.1
tabulate : 0.9.0
xarray : 2024.9.0
xlrd : 2.0.1
xlsxwriter : 3.2.2
zstandard : 0.23.0
tzdata : 2025.1
qtpy : None
pyqt5 : None

@MarcoGorelli MarcoGorelli added Bug Needs Triage Issue that has not been reviewed by a pandas team member labels Mar 9, 2025
@rhshadrach rhshadrach added Arrow pyarrow functionality Reshaping Concat, Merge/Join, Stack/Unstack, Explode and removed Needs Triage Issue that has not been reviewed by a pandas team member labels Mar 11, 2025
@snitish
Copy link
Member

snitish commented Mar 11, 2025

take

# for free to join this conversation on GitHub. Already have an account? # to comment
Labels
Arrow pyarrow functionality Bug Reshaping Concat, Merge/Join, Stack/Unstack, Explode
Projects
None yet
Development

Successfully merging a pull request may close this issue.

3 participants