[Epic] Native StringView support for string functions

[alamb]

Is your feature request related to a problem or challenge?

We are working to add complete StringView support in DataFusion, which permits potentially much faster processing of string data. See #10918 for more background.

Today, most DataFusion string functions support DataType::Utf8 and DataType::LargeUtf8 and when called with a StringView argument DataFusion will cast the argument back to DataType::Utf8 which is expensive.

To realize the full speed of StringView, we need to ensure that all string functions support the DataType::Utf8View directly.

Describe the solution you'd like

Port all string functions

• Implement native support StringView for character length #11676
• Initial support for regex_replace on StringViewArray #11556
• Support starts_with for Utf8View #11786
• Update the ASCII scalar function to support Utf8View #11834
• Update the BTRIM scalar function to support Utf8View #11835
• Update the CONCAT scalar function to support Utf8View #11836
• Update concat_ws scalar function to support Utf8View #11837
• Update CONTAINS scalar function to support Utf8View #11838
• Update ENDS_WITH scalar function to support Utf8View #11852
• Update INITCAP scalar function to support Utf8View #11853
• Update levenshtein scalar function to support Utf8View #11854
• Update LOWER scalar function to support Utf8View #11855
• Update LTRIM scalar function to support Utf8View #11856
• Update LPAD scalar function to support Utf8View #11857
• Update OCTET_LENGTH scalar function to support Utf8View #11858
• Update SPLIT_PART scalar function to support Utf8View #11950
• Update STRPOS scalar function to support Utf8View #11951
• Update SUBSTR scalar function to support Utf8View #11952
• Update TRANSLATE scalar function to support Utf8View #11953
• Update FIND_IN_SET scalar function to support Utf8View #11954
• Implement native support StringView for REPEAT #11962
• Support Utf8View for string function bit_length #13195

Describe alternatives you've considered

No response

Additional context

See coordination plan with @tshauck and myself here: #11787 (comment)

[alamb]

One thing I have noticed during implementations is that some functions such as ltrim/rtrim/btrim could be more efficient if they produced Utf8View as output in addition to accepting them as input

For example, in #11920 (comment) from @Kev1n8 it is actually probably a good idea to always generate StringView as output (rather than StringArray) as it could avoid a copy.

I am thinking once we get the string functions so they can support StringView as input then we can do a second pass and optimize some functions so they produce StringView as output

[2010YOUY01]

Inspired by @Omega359 's great PR #11941, I have some suggestion on testing Utf8View support for functions:

Although most implementation is adapted from existing implementation, but the execution takes another path, so I think comprehensive end-to-end tests are still needed.
The good news is there already exists sqllogictests for original string functions, the only thing to do is to duplicate existing testings with Utf8View

Here are the examples on how to adapt existing test cases for Utf8View input

1. For functions takes scalar value, use arrow_cast() like https://github.com/apache/datafusion/pull/11941/files#diff-51757b2b1d0a07b88551d88eabeba7f74e11b5217e44203ac7c6f613c0221196
2. For functions read from a table, string column can be converted to Utf8View column like

   datafusion/datafusion/sqllogictest/test_files/string_view.slt

   Lines 30 to 42 in 2cf0956

   	# Table with the different combination of column types
   	statement ok
   	create table test as
   	SELECT
   	arrow_cast(column1, 'Utf8') as column1_utf8,
   	arrow_cast(column2, 'Utf8') as column2_utf8,
   	arrow_cast(column1, 'LargeUtf8') as column1_large_utf8,
   	arrow_cast(column2, 'LargeUtf8') as column2_large_utf8,
   	arrow_cast(column1, 'Utf8View') as column1_utf8view,
   	arrow_cast(column2, 'Utf8View') as column2_utf8view,
   	arrow_cast(column1, 'Dictionary(Int32, Utf8)') as column1_dict,
   	arrow_cast(column2, 'Dictionary(Int32, Utf8)') as column2_dict
   	FROM test_source;

[alamb]

We are making pretty good progress here -- just a few more functions left 🚀

[alamb]

I think we can claim this is completed. Follow on work is tracked in

• [Epic] A Collection of Additional UTF8View support tickets #13504

[alamb]

🎉