BUG: Impossible creation of array with dtype=string

[Manju080]

closes #61155

• Tests added and passed if fixing a bug or adding a new feature
• All code checks passed.

Hello @rhshadrach ,

I’ve created a fix that raises a ValueError when trying to create a StringArray from a list of lists with inconsistent lengths or non-character elements. This aligns the behavior for both consistent and inconsistent input formats and also tested.

I've would like to hear opinion to raise an error when a list of lists is passed for dtype=StringDtype, to avoid ambiguous behavior. If preferred, we could instead join the inner lists into strings automatically — happy to adjust based on guidance.
Example case : pd.array([["t", "e", "s", "t"], ["w", "o", "r", "d"]], dtype="string")
output : <StringArray> ['test', 'word'] Length: 2, dtype: string

Thanks

[rhshadrach]

Also, please add a test for this.

[Manju080]

pre-commit.ci autofix

[rhshadrach]

We use pytest for testing, you'll need to add a test using that format. See here:

https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#using-pytest

The general pytest introduction may also be useful:

https://docs.pytest.org/en/7.1.x/getting-started.html

[Manju080]

We use pytest for testing, you'll need to add a test using that format. See here:

https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#using-pytest

The general pytest introduction may also be useful:

https://docs.pytest.org/en/7.1.x/getting-started.html

Thank you for the details, will work on it

[Manju080]

@rhshadrach I’ve been testing the following case in test_lib.py

def test_ensure_string_array_list_of_lists():
# GH#61155: ensure list of lists doesn't get converted to string
arr = [['t', 'e', 's', 't'], ['w', 'o', 'r', 'd']]
result = lib.ensure_string_array(arr)

# Each item in result should still be a list, not a stringified version
assert isinstance(result[0], list)
assert isinstance(result[1], list)
assert result[0] == ['t', 'e', 's', 't']
assert result[1] == ['w', 'o', 'r', 'd']

However, the test fails with
FAILED pandas/tests/libs/test_lib.py::test_ensure_string_array_list_of_lists - AssertionError
DEBUG RESULT: ["['t', 'e', 's', 't']" "['w', 'o', 'r', 'd']"] <class 'numpy.ndarray'> <class 'str'>

So currently, the list of lists gets converted into a 1D NumPy array of strings.
With the current implementation, arr becomes a 1D object array of lists (as intended), but it seems that downstream processing stringifies each list.
Do you want me to guard against this case inside ensure_string_array to preserve the list structure? Or is the stringification expected behavior in this context?

Thanks!

[rhshadrach]

I believe converting to a 1-dimesional ndarray of strings is the expected behavior of enusure_string_array. Perhaps I'm misunderstanding; what is the alternative?

[Manju080]

Thanks for the clarification!

You're right — the behavior of ensure_string_array producing a 1D ndarray of stringified inner lists (when given a list of lists like [list("test"), list("word")]) is consistent with the current expectations of the function.

def test_ensure_string_array_list_of_lists():
arr = [list("test"), list("word")]
result = lib.ensure_string_array(arr)
assert isinstance(result, np.ndarray)
assert result.dtype == object
assert result[0] == "['t', 'e', 's', 't']"
assert result[1] == "['w', 'o', 'r', 'd']"
print("DEBUG RESULT:", result)

My initial assumption was that it should preserve the list structure instead of converting to strings, but after re-evaluating and running the test, I see that the 1D array of strings is indeed the intended behavior. The test has now been updated and passes successfully and got the below output
[1/1] Generating write_version_file with a custom command
================================================= test session starts
==================================================
platform linux -- Python 3.12.3, pytest-8.3.5, pluggy-1.5.0
rootdir: /mnt/c/Users/HP/Documents/Python_pandas_op/pandas
configfile: pyproject.toml
plugins: hypothesis-6.131.6
collected 84 items
pandas/tests/libs/test_lib.py ...................................................................................DEBUG RESULT: ["['t', 'e', 's', 't']" "['w', 'o', 'r', 'd']"]`

----------------- generated xml file: /mnt/c/Users/HP/Documents/Python_pandas_op/pandas/test-data.xml ------------------
================================================= slowest 30 durations
=================================================
0.09s setup pandas/tests/libs/test_lib.py::TestMisc::test_max_len_string_array

(29 durations < 0.005s hidden. Use -vv to show these durations.)

Please let me know if I need to change anything

[rhshadrach]

@Manju080 - the last change I'm seeing is from 3 weeks ago. Perhaps you need to push some commits?

[Manju080]

That's right, I just wanna make sure before committing the changes.

[rhshadrach]

Looking good!

[Manju080]

Apologies for the causing confusion, I will work this to fix.

[Manju080]

@rhshadrach Thank you very much, required changes are done.
Let me know if there is anything

[Manju080]

pre-commit.ci autofix

[rhshadrach]

lgtm