Support paths with dynamic interpolation from Nix 2.4+

[Ma27]

⚠️ Warning: not ready to be merged. This is my first deep-dive and I'm a little afraid of regressions, so I'd first like to use this with rnix-lsp locally to test it a bit :)

Summary & Motivation

Since Nix 2.4 it's possible to perform string interpolations within
paths:

nix-repl> let a = "rnix"; in /home/ma27/Projects/${a}-parser
/home/ma27/Projects/rnix-parser

Please note that the following expressions are not supported (by both
Nix and this patch):

• NIX_PATH imports, i.e. <nixpkgs/${foo}>
• Interpolations before the first slash, i.e. foo${bar}/baz.

The newly introduced token NODE_PATH_WITH_INTERPOL is only used for
these interpolated paths. Literal paths are still TOKEN_PATH inside
NODE_LITERAL for backwards compatibility.

Backwards-incompatible changes

none

Further context

Closes #44

---

cc @nerdypepper

[Ma27]

The changes can be tested via https://github.com/Ma27/rnix-lsp/tree/experimental.

[Ma27]

Seems to be working for me.

cc @nerdypepper @aaronjanse for a review.

[oppiliappan]

Amazing, thanks! I'll take a look over the weekend.

[aaronjanse]

Thank you @Ma27!! I'll do some review & local testing over the next few days. I'm excited to have support for this syntax

[aaronjanse]

It would be nice if the error were more isolated (rather than consuming the rest of the expression in code like a${b}/c + /tmp), but I think we can figure that out once we merge some of the very helpful error handling code that @oberblastmeister has been working on

[Ma27]

I'll need to think of this later today, but actually we should split 0.10 in 0.10 and 0.11. The reason for this is that I'd like to have a rnix-parser which doesn't provide invalid syntax-errors for new language-features that are part of the current stable Nix (2.4) which will be stable Nix in the upcoming 21.11.

That said, yes, I'd love to have @oberblastmeister's changes at some point, definitely, but I think we should cut 0.10 a bit earlier than planned IMHO :)

[aaronjanse]

This all LGTM. I want to later re-visit error handling in this code, but in the meantime I think that parsing correct Nix syntax is a higher priority.

I'm in favor of merging this PR. I think we should consider releasing 0.10 quickly with just this PR, moving the already-merged breaking-change commits like 56131e0 to 0.11 (then maybe assigning 0.12 to things currently assigned to 0.11).

[ncfavier]

This is not completely in sync with the syntax supported by Nix, for example ./${"a"}.b gives

error: unexpected TOKEN_PATH at 9..10, wanted any of [TOKEN_IDENT]
error: unexpected end of file, wanted any of [TOKEN_IDENT]
NODE_ROOT 0..10 {
  NODE_SELECT 0..10 {
    NODE_PATH_WITH_INTERPOL 0..8 {
      TOKEN_PATH("./") 0..2
      NODE_STRING_INTERPOL 2..8 {
        TOKEN_INTERPOL_START("${") 2..4
        NODE_STRING 4..7 {
          TOKEN_STRING_START("\"") 4..5
          TOKEN_STRING_CONTENT("a") 5..6
          TOKEN_STRING_END("\"") 6..7
        }
        TOKEN_INTERPOL_END("}") 7..8
      }
    }
    TOKEN_DOT(".") 8..9
    NODE_ERROR 9..10 {
      TOKEN_PATH("b") 9..10
    }
  }
}

whereas this syntax is supported by Nix since 2.4 (NixOS/nix#5066).

[Ma27]

Yes, I realized that this is a problem ~a week ago, I started working on a fix, but my time is slightly constrained unfortunately.

[Ma27]

Opened a ticket to keep track of it: #85 :)

[ncfavier]

Fixed in #86