BUG/API: Clarify the behaviour of fillna downcasting

[sinhrks]

From #11497. Found inconsistencyr regarding fillna downcasting.

I understand fillna(downcast=None) should downcast values appropriately. It doesn't work on float dtype.

# NG
pd.Series([1, 2, np.nan, 4]).fillna(3)
#0    1
#1    2
#2    3
#3    4
# dtype: float64

# OK
pd.Series([1, 2, np.nan, 4], dtype=object).fillna(3)
#0    1
#1    2
#2    3
#3    4
# dtype: int64

Based on the internal logic, downcast can accept dtype. It works on float, but not on object.

• https://github.com/pydata/pandas/blob/master/pandas/core/internals.py#L366

# OK
pd.Series([1, 2, np.nan, 4]).fillna(3, downcast='int')
#0    1
#1    2
#2    3
#3    4
# dtype: int64

# NG
pd.Series([1, 2, np.nan, 4], dtype=object).fillna(3, downcast='int')
#0    1
#1    2
#2    3
#3    4
# dtype: object

I understood the expected behavior as below:

• all dtypes should be downcast by default (downcast=None)
• if any dtype is passed via downcast kw, downcast to the specified dtype if possible.
• downcast=False will not perform downcast.
• Add Index.fillna to API (follow-up of ENH: Add Index.fillna #11343)

[jreback]

floats are never downcast by default
it's not cheap to do this

[sinhrks]

Thanks. Is the following correct? If so, I'll include it to the document in #11497.

• by default (downcast=None), only object dtype will be downcasted
• Downcast can be controlled explicitly by:
  • Specifying downcast=False prohibits the downcast
  • Specifying dtype via downcast forces the downcast to the specified type. If unable to downcast, dtype is kept as it is without showing error.

Based on the last condition, this should result in int64?

pd.Series([1, 2, np.nan, 4], dtype=object).fillna(3, downcast='int')
# 0    1
# 1    2
# 2    3
# 3    4
# dtype: object

[jreback]

@sinhrks IIRC that looks right. yeah it is meant for explicit use.

though to be honest maybe we should just take this out and always downcast object dtypes. We downcast for all other operations; the problem is I didn't have a performant way to downcast floats->ints.