Unicode character not parsed correctly

[goretkin]

julia> CSTParser.parse("∪⃑")
  1:6   ErrorToken
  1:3    OP: UNION
  4:6    ErrorToken( CSTParser.UnexpectedToken)
  4:6     ErrorToken( CSTParser.Unknown)


julia> Meta.parse("∪⃑")
:∪⃑

Manifests as false positive in julia-vscode linter:

Even though running it gives:

julia> using LintTest: ∪⃑


julia> (a=1, b=2) ∪⃑ (a=10, c=30)
(a = 10, b = 2, c = 30)

For clarity:

help?> ∪⃑
"∪⃑" can be typed by \cup<tab>\rightharpoonaccent<tab>

search:

  No documentation found.

  Binding ∪⃑ does not exist.

[goretkin]

I suppose this is the problem:

julia> CSTParser.Tokenize.Lexers.isopsuffix(collect("∪⃑")[2])
false

[goretkin]

I see that this actually is a problem with less esoteric symbols:

julia> dump(Meta.parse("1 +̄ 2"))
Expr
  head: Symbol call
  args: Array{Any}((3,))
    1: Symbol +̄
    2: Int64 1
    3: Int64 2

julia> CSTParser.parse("1 +̄ 2")
  1:7   call
  1:1    OP: +
  2:3    INTEGER: 1
  4:7    errortoken
  4:6     errortoken( CSTParser.UnexpectedToken)
  4:6      errortoken( CSTParser.Unknown)
  7:7     INTEGER: 2

[pfitzseb]

Closing in favor of JuliaLang/Tokenize.jl#180.