BUG: \0 null bytes in str not preserved in pandas.CategoricalIndex or pandas.MultiIndex

[dutc]

Pandas version checks

• I have checked that this issue has not already been reported.

• I have confirmed this bug exists on the latest version of pandas.

• I have confirmed this bug exists on the main branch of pandas.

Reproducible Example

from sys import version_info as py_version_info
from pandas import __version__ as pd_version

assert py_version_info[:3] == (3, 13, 2)
assert pd_version == '2.2.3' or pd_version == '3.0.0.dev0+2028.gb64f438cc8'

from pandas import CategoricalIndex, MultiIndex

entities = [b'abc', b'abc\0']

# CORRECT
cat = CategoricalIndex(entities)
assert cat.tolist() == entities
assert len({*cat.tolist()}) == len({*entities})

# CORRECT
idx = MultiIndex.from_product([entities])
assert idx.get_level_values(0).tolist() == entities
assert len({*idx.get_level_values(0).tolist()}) == len({*entities})

entities = ['abc', 'abc\0']

# INCORRECT
cat = CategoricalIndex(entities)
assert cat.tolist() != entities
assert len({*cat.tolist()}) < len({*entities})

# INCORRECT
idx = MultiIndex.from_product([entities])
assert idx.get_level_values(0).tolist() != entities
assert len({*idx.get_level_values(0).tolist()}) < len({*entities})

entities = ['abc', 'abc\0def']

# INCORRECT
cat = CategoricalIndex(entities)
assert cat.tolist() != entities
assert len({*cat.tolist()}) < len({*entities})

# INCORRECT
idx = MultiIndex.from_product([entities])
assert idx.get_level_values(0).tolist() != entities
assert len({*idx.get_level_values(0).tolist()}) < len({*entities})

Issue Description

When constructing a pandas.CategoricalIndex or pandas.MultiIndex from Python str values, any code points following a '\0' are discarded. This does not occur with bytes inputs.

Expected Behavior

The null bytes should be preserved exactly.

Installed Versions

>>> from pandas import show_versions >>> show_versions() # trimmed

INSTALLED VERSIONS

commit : b64f438
python : 3.13.2
python-bits : 64
OS : Linux
OS-release : 6.12.20-1-lts
Version : #1 SMP PREEMPT_DYNAMIC Sun, 23 Mar 2025 08:02:10 +0000
machine : x86_64
processor :
byteorder : little
LC_ALL : None
LANG : en_US.UTF-8
LOCALE : en_US.UTF-8

pandas : 3.0.0.dev0+2028.gb64f438cc8
numpy : 2.3.0.dev0+git20250325.2a6f4f0
dateutil : 2.9.0.post0
pip : 24.3.1
tzdata : 2025.2

[camriddell]

To help further pinpoint, it seems that the deeper issue may lie within pandas.factorize

import pandas as pd
print(f'{pd.__version__ = }') # 2.2.3

entities = ['abc', 'abc\0']
s = pd.Series(entities)

assert s.tolist() == entities

print(
    # incorrect unique value detection
    s.factorize(),   # (array([0, 0]), Index(['abc'], dtype='object'))
    pd.factorize(s), # (array([0, 0]), Index(['abc'], dtype='object'))
    sep='\n'
)

Building more confidence, if we explicitly create a CategoricalDtype we can round trip safely

import pandas as pd
print(f'{pd.__version__ = }') # 2.2.3

entities = ['abc', 'abc\0']
dtype = pd.CategoricalDtype(categories=entities)

assert pd.Categorical(entities, dtype=dtype).tolist() == entities
assert pd.Series(entities, dtype=dtype).tolist() == entities

[asishm]

Yes, there's a few related open issues on this #34551 #53720

[rhshadrach]

Thanks for the report, and @asishm for tracking down those issues. Closing as a duplicate of #34551.