jflex warnings in multiple lex files about too long Unicode escape sequences

[vladak]

Warning in file "/home/vkotal/opengrok-vladak-scratch/opengrok-indexer/src/main/jflex/analysis/php/PhpXref.lex" (line 139): 
Unicode escape sequence is too long. Use \u{...} to disambiguate.
Identifier = [a-zA-Z_\u007F-\u10FFFF] [a-zA-Z0-9_\u007F-\u10FFFF]*
                                  ^

Warning in file "/home/vkotal/opengrok-vladak-scratch/opengrok-indexer/src/main/jflex/analysis/php/PhpXref.lex" (line 139): 
Unicode escape sequence is too long. Use \u{...} to disambiguate.
Identifier = [a-zA-Z_\u007F-\u10FFFF] [a-zA-Z0-9_\u007F-\u10FFFF]*
                                                              ^

Warning in file "/home/vkotal/opengrok-vladak-scratch/opengrok-indexer/src/main/jflex/analysis/php/PhpXref.lex" (line 175): 
Unicode escape sequence is too long. Use \u{...} to disambiguate.
HtmlNameStart = [a-zA-Z_\u00C0-\u10FFFFFF]
                                     ^
[INFO]   generated /home/vkotal/opengrok-vladak-scratch/opengrok-indexer/target/generated-sources/jflex/org/opengrok/indexer/analysis/php/PhpXref.java

Warning in file "/home/vkotal/opengrok-vladak-scratch/opengrok-indexer/src/main/jflex/analysis/php/PhpSymbolTokenizer.lex" (line 75): 
Unicode escape sequence is too long. Use \u{...} to disambiguate.
Identifier = [a-zA-Z_\u007F-\u10FFFF] [a-zA-Z0-9_\u007F-\u10FFFF]*
                                  ^

Warning in file "/home/vkotal/opengrok-vladak-scratch/opengrok-indexer/src/main/jflex/analysis/php/PhpSymbolTokenizer.lex" (line 75): 
Unicode escape sequence is too long. Use \u{...} to disambiguate.
Identifier = [a-zA-Z_\u007F-\u10FFFF] [a-zA-Z0-9_\u007F-\u10FFFF]*
                                                              ^

[vladak]

The limit in JFlex is either 4 or 6 characters in the sequence (LexScan.flex):

  {Unicode4}  { maybeWarnUnicodeMatch(4);
                string.append( (char) Integer.parseInt(yytext().substring(2,6), 16));
              }
  {Unicode6}  { maybeWarnUnicodeMatch(6);
                int codePoint = Integer.parseInt(yytext().substring(2,8), 16);
                if (codePoint <= getMaximumCodePoint()) {
                  string.append(Character.toChars(codePoint));
                } else {
                  throw new ScannerException(file,ErrorMessages.CODEPOINT_OUT_OF_RANGE, yyline, yycolumn+2);
                }
              }

  "\\u{"      { yybegin(STRING_CODEPOINT_SEQUENCE); }

The maybeWarnUnicodeMatch() is the call which emits the warning:

  /**
   * Warn if the matched length of a Unicode escape sequence is longer than expected. Push back the
   * extra characters to be matched again.
   *
   * @param len expected Unicode escape sequence length
   */
  public void maybeWarnUnicodeMatch(int len) {
    // 2 for "\"" followed by "u" or "U" at start of match
    len += 2;
    if (lexLength() > len) {
      Out.warning(file, ErrorMessages.UNICODE_TOO_LONG, lexLine(), lexColumn() + len);
      lexPushback(lexLength() - len);
    }
  }

so it basically allows up to 4 characters after the \u, not 6.

[vladak]

The Java SE 17 documentation says in section 3.3 Unicode Escapes:

One Unicode escape can represent characters in the range U+0000 to U+FFFF; representing supplementary characters in the range U+010000 to U+10FFFF requires two consecutive Unicode escapes.

[vladak]

Looks like JFlex does not support using 2 consecutive Unicode escapes to specify character range in a regexp. Anyhow, given that it most likely used \u007F-\u10FF (sans the 3rd FF byte) up till now and given that The set of characters from U+0000 to U+FFFF is sometimes referred to as the Basic Multilingual Plane (BMP) is most likely sufficient for the identifiers and such, it could be fine to use that upper bound.