More accurate memory accounting in external sort

[2010YOUY01]

Is your feature request related to a problem or challenge?

#14644 fixed an external sorting bug. Each batch's memory overhead is estimated as 2 * batch memory size for the extra row conversion overhead. It works for common cases but still can fail. See #14644 (comment) and #14644 (comment).

To implement a more accurate memory size estimation for those edge cases, we can first do col->row conversion and measure the memory consumption. (depends on #7053)

Describe the solution you'd like

No response

Describe alternatives you've considered

No response

Additional context

No response

[2010YOUY01]

There is a small optimization can be done after we have accurate memory accounting #15017 (comment)

[alamb]

I wonder if this is ready to do now?

[2010YOUY01]

I wonder if this is ready to do now?

Yes, I think so. However, this might cause a temporary slowdown due to an extra row <-> col conversion. Since the performance can be brought back by reusing converted rows between operators, and also I believe correctness is more important, this is not a hard blocker.
I plan to work on this ticket in the future, but if anyone is available to take it now, please feel free.