[BUG] cudf::io::json::detail::normalize_single_quotes outputs incorrect result when the input has \n character

[ttnghia]

Reproducible with this input:

{\"a\": \"1\n2\"}
{\'a\': 12}

The output tokens, generated by cudf::io::json::detail::get_token_stream after preprocessing with cudf::io::json::detail::normalize_single_quotes are:

Input:
{"a": "1
2"}{'a': 12}
Tokens:
0, 4, 6, 7, 8, 9, 5, 1, 0, 1
Token indices:
0, 1, 1, 3, 6, 10, 11, 11, 0, 0

If remove the \n character then the output is correct:

Input:
{"a": "12"}{"a": 12}
Tokens:
0, 4, 6, 7, 8, 9, 5, 1, 0, 4, 6, 7, 10, 11, 5, 1
Token indices:
0, 1, 1, 3, 6, 9, 10, 10, 12, 13, 13, 15, 18, 20, 20, 20

Note:

• Line delimiter between JSON objects is \0, not \n.
• allow_unquoted_control is set to true.
• Token indices are the positions of the tokens in the input string.
• Token numbers are static_cast from enum token_t at

  cudf/cpp/include/cudf/io/detail/tokenize_json.hpp

  Line 88 in 06b3f83

  enum token_t : PdaTokenT {

I suspect that it is due to the leftover character \n in

cudf/cpp/src/io/json/json_normalization.cu

Lines 76 to 77 in 06b3f83

	std::array<std::vector<SymbolT>, NUM_SYMBOL_GROUPS - 1> const qna_sgs{
	{{'\"'}, {'\''}, {'\\'}, {'\n'}}};

, but I'm not 100% sure.