Skip to content

BUG: Incorrect Parsing of Timestamps in pd.to_datetime with Series with format="ISO8601" and UTC=True #61389

New issue

Have a question about this project? # for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “#”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? # to your account

Closed
2 of 3 tasks
PaulCalot opened this issue May 2, 2025 · 3 comments · Fixed by #61400
Closed
2 of 3 tasks
Assignees
Labels
Bug Datetime Datetime data dtype Timezones Timezone data dtype

Comments

@PaulCalot
Copy link

PaulCalot commented May 2, 2025

Pandas version checks

  • I have checked that this issue has not already been reported.

  • I have confirmed this bug exists on the latest version of pandas.

  • I have confirmed this bug exists on the main branch of pandas.

Reproducible Example

import pandas as pd

# Single timestamp
raw = "2023-10-15T14:30:00"
single = pd.to_datetime(raw, utc=True, format="ISO8601")
print(single)
# Output: 2023-10-15 14:30:00+00:00 (correct)

# Series of timestamps
series = pd.Series([0, 0], index=["2023-10-15T10:30:00-12:00", raw])
converted = pd.to_datetime(series.index, utc=True, format="ISO8601")
print(converted)
# Output: 2023-10-16 02:30:00+00:00 for the second one (incorrect)
# error depends on the previous one timezone

Issue Description

When using pd.to_datetime to parse a Series of timestamps with format="ISO8601" and utc=True, the parsing of a timestamp without an explicit timezone offset is incorrect and appears to depend on the timezone offset of the previous timestamp in the Series. This behavior does not occur when parsing a single timestamp.

Expected Behavior

In this configuration, behavior should not depend on the previous timestamp timezone. Result should be the same as when individually passed.

Installed Versions

INSTALLED VERSIONS

commit : 0691c5c
python : 3.10.0
python-bits : 64
OS : Linux
OS-release : 5.10.0-34-amd64
Version : #1 SMP Debian 5.10.234-1 (2025-02-24)
machine : x86_64
processor :
byteorder : little
LC_ALL : None
LANG : en_US.UTF-8
LOCALE : en_US.UTF-8

pandas : 2.2.3
numpy : 1.26.4
pytz : 2024.2
dateutil : 2.9.0.post0
pip : 24.2
Cython : None
sphinx : None
IPython : None
adbc-driver-postgresql: None
adbc-driver-sqlite : None
bs4 : None
blosc : None
bottleneck : None
dataframe-api-compat : None
fastparquet : None
fsspec : None
html5lib : None
hypothesis : None
gcsfs : None
jinja2 : 3.1.6
lxml.etree : None
matplotlib : 3.10.1
numba : None
numexpr : None
odfpy : None
openpyxl : None
pandas_gbq : None
psycopg2 : None
pymysql : None
pyarrow : 15.0.2
pyreadstat : None
pytest : 8.3.5
python-calamine : None
pyxlsb : None
s3fs : None
scipy : 1.15.2
sqlalchemy : 2.0.40
tables : None
tabulate : None
xarray : None
xlrd : None
xlsxwriter : None
zstandard : None
tzdata : 2025.2
qtpy : None
pyqt5 : None
Traceback (most recent call last):
File "", line 1, in
NameError: name 'version' is not defined

@PaulCalot PaulCalot added Bug Needs Triage Issue that has not been reviewed by a pandas team member labels May 2, 2025
@arthurlw
Copy link
Member

arthurlw commented May 2, 2025

Thanks for the clear issue! Confirmed on main.

When you batch‐parse ["2023-10-15T10:30:00-12:00", "2023-10-15T14:30:00"] with format="ISO8601", utc=True, the second (naive) timestamp wrongly reuses the “–12:00” offset and becomes 2023-10-16T02:30:00+00:00 instead of 2023-10-15T14:30:00+00:00. The ISO8601 parser is retaining its last‐seen offset between parses.

I believe when utc=True, naive timestamps should be treated as UTC.

@rhshadrach
Copy link
Member

An interesting case indeed. If this inconsistency can be fixed without hurting performance, I'm certainly positive on it. However I think the performance of what I believe is the common case (consistent timezone data) should weigh in here as well.

Further investigations are welcome.

@rhshadrach rhshadrach added Datetime Datetime data dtype Timezones Timezone data dtype and removed Needs Triage Issue that has not been reviewed by a pandas team member labels May 3, 2025
@myenugula
Copy link
Contributor

take

# for free to join this conversation on GitHub. Already have an account? # to comment
Labels
Bug Datetime Datetime data dtype Timezones Timezone data dtype
Projects
None yet
4 participants