Skip to content

Fix pyarrow FileInfo import #9078

New issue

Have a question about this project? # for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “#”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? # to your account

Open
wants to merge 2 commits into
base: main
Choose a base branch
from

Conversation

jrbourbeau
Copy link
Member

Custom serialization for pyarrow.fs.FileInfo was recently added to help with Parquet in Dask DataFrame (xref #9025) but this import isn't quite right

In [1]: import pyarrow

In [2]: pyarrow.fs.FileInfo
---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)
Cell In[2], line 1
----> 1 pyarrow.fs.FileInfo

AttributeError: module 'pyarrow' has no attribute 'fs'

I think this went unnoticed because in the Dask DataFrame case we trigger this correct import first

https://github.com/dask/dask/blob/c8ef947bd38e2b739ad3eccdbcce42e08d8c3f4a/dask/dataframe/dask_expr/io/parquet.py#L21

first which allows the import here to work.

However in the non-Dask DataFrame parquet case, that correct import doesn't happen and folks hit the AttributeError

@jrbourbeau jrbourbeau requested a review from fjetter as a code owner May 15, 2025 17:34
Copy link
Contributor

github-actions bot commented May 15, 2025

Unit Test Results

See test report for an extended history of previous test failures. This is useful for diagnosing flaky tests.

    27 files  ±    0      27 suites  ±0   10h 44m 4s ⏱️ - 32m 54s
 4 113 tests ±    0   3 992 ✅ +    1    111 💤 ± 0   10 ❌  -  1 
50 325 runs   - 1 244  47 995 ✅  - 1 242  2 222 💤  - 62  108 ❌ +60 

For more details on these failures, see this check.

Results for commit 76596c0. ± Comparison against base commit a890b85.

♻️ This comment has been updated with latest results.

# for free to join this conversation on GitHub. Already have an account? # to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant