Fiona no longer recognizes pd.Timestamp (as datetime.datetime) since 1.10 a2

[jorisvandenbossche]

Example that works with released fiona, using a pd.Timestamp scalar in the data passed to writerecords for a "datetime" field:

In [1]: import fiona

In [2]: fiona.__version__
Out[2]: '1.9.6'

In [3]: schema = {'geometry': 'Point', 'properties': {'a': 'float', 'b': 'datetime'}}

In [4]: with fiona.open("test_fiona110.geojson", mode="w", schema=schema) as col:
   ...:     col.writerecords([{'id': '0', 'type': 'Feature', 'properties': {'a': 1.0, 'b': pd.Timestamp('2021-11-21 01:07:43.018000')}, 'geometry': {'type': 'Point', 'coordinates': (0.0, 0.0)}}])
   ...: 

In [5]: !cat test_fiona110.geojson
{
"type": "FeatureCollection",
"features": [
{ "type": "Feature", "properties": { "a": 1.0, "b": "2021-11-21T01:07:43.018" }, "geometry": { "type": "Point", "coordinates": [ 0.0, 0.0 ] } }
]
}

Starting with 1.10 a2, this seems to silently drop that column:

In [1]: import fiona

In [2]: fiona.__version__
Out[2]: '1.10a2'

In [3]: schema = {'geometry': 'Point', 'properties': {'a': 'float', 'b': 'datetime'}}

In [4]: with fiona.open("test_fiona110.geojson", mode="w", schema=schema) as col:
   ...:     col.writerecords([{'id': '0', 'type': 'Feature', 'properties': {'a': 1.0, 'b': pd.Timestamp('2021-11-21 01:07:43.018000')}, 'geometry': {'type': 'Point', 'coordinates': (0.0, 0.0)}}])
   ...: 

In [5]: !cat test_fiona110.geojson
{
"type": "FeatureCollection",
"name": "test_fiona110",
"features": [
{ "type": "Feature", "properties": { "a": 1.0 }, "geometry": { "type": "Point", "coordinates": [ 0.0, 0.0 ] } }
]
}

If I use datetime.datetime.fromisoformat(..) instead of pd.Timestamp(..) in the code above (i.e. passing a datetime.datetime object), then it works again.
But pd.Timestamp is subclass of datetime.datetime (and so things like isinstance(value, datetime.datetime) should work), and this worked in the past.

We are running into this issue in GeoPandas' CI because we pass the result of GeoDataFrame.iterfeatures() to writerecords. And it seems that this is creating a json-like structure that uses pd.Timestamp instead of datetime.datetime objects.

[jorisvandenbossche]

Enabling logging gives:

WARNING:fiona.ogrext:Skipping field b: invalid type (11, 0, 'Timestamp')

This is because of using type(value).__name__ to get the name of the type in the fieldkey lookup for the setter function (from https://github.com/Toblerity/Fiona/pull/1366/files#diff-10438bbcfe51863df4f21eb3b3ef0782ab5289899b0d408e220bf7ae43c67648R763)

Fiona/fiona/ogrext.pyx

Lines 762 to 765 in 6581da6

	schema_type = session._schema_normalized_field_types[key]
	fieldkey = (*FIELD_TYPES_MAP2[NAMED_FIELD_TYPES[schema_type]], type(value).__name__)
	if fieldkey in self.property_setter_cache:
	setter = self.property_setter_cache[fieldkey]

[sgillies]

@jorisvandenbossche thanks for the report. Sorry about this, I'll make sure it's fixed for 1.10b1.

[jorisvandenbossche]

GeoPandas CI is green again!