Optimize the record-level RLE algorithm for a denser compression of shorter-than-declared strings and sets of subsequent NULLs [CORE4401] #4723
Assignees
Comments
Modified by: @dyemanovsummary: Optimizer the record-level RLE algorithm for a denser compression of shorter-than-declared strings and sets of subsequent NULLs => Optimize the record-level RLE algorithm for a denser compression of shorter-than-declared strings and sets of subsequent NULLs |
Modified by: @dyemanovassignee: Dmitry Yemanov [ dimitr ] |
Modified by: @dyemanovFix Version: 4.0 Beta 1 [ 10750 ] => |
# for free
to join this conversation on GitHub.
Already have an account?
# to comment
Submitted by: @dyemanov
Votes: 3
The RLE algorithm used by the engine compresses sets of up to 128 equal bytes into two bytes. This means that every empty (or NULL) VARCHAR(32000) is compressed into 500 bytes. This storage overhead is too big. The same issue applies to many shorter fields that are all NULLs -- they along with their padding also represent the long chain of zero bytes that cannot be compressed more efficiently than with a 64x ratio.
The text was updated successfully, but these errors were encountered: