DEPR: observed=False default in groupby

[rhshadrach]

• closes DEPR: Change default to observed=True in DataFrame.groupby #43999 (Replace xxxx with the GitHub issue number)
• closes PERF: groupby with many empty groups memory blowup #30552
• Tests added and passed if fixing a bug or adding a new feature
• All code checks passed.
• Added type annotations to new arguments/methods/functions.
• Added an entry in the latest doc/source/whatsnew/vX.X.X.rst file if fixing a bug or adding a new feature.

[topper-123]

for users who work interaactively (ipython, jupyter,..), Is there way to avoid the FutureWarning coming repeatedly? It will probably get quite annoying after a while...

Otherwise LGTM.

[rhshadrach]

Thanks @topper-123

for users who work interaactively (ipython, jupyter,..), Is there way to avoid the FutureWarning coming repeatedly? It will probably get quite annoying after a while...

Users can specify observed=True|False. Also, just to be sure, this will only warn if one of the groupings is categorical.

I think you're thinking of some global option? I would be -1 on that, and hope this will be rare enough that it won't pester users. They'll need to specify observed=True if they want to adopt future behavior or observed=False if they want to maintain current behavior eventually.

[mroeschke]

Could you mention this will be True in the future?

[rhshadrach]

Good call - done.

[mroeschke]

Thanks @rhshadrach