Add local_for_callback option to Macro.Env.expand_import

[lukaszsamson]

This PR adds a new option :local_for_callback to Macro.Env.expand_import in order to allow hooking into local macro resolution and providing a custom resolver.

Rationale:
In ElixirSense minicompiler I need to resolve local macros using the metadata extracted from the AST. The current mechanism relying on :elixir_def.local_for is not going to work correctly when there is no real compilation in process. The current hacky code using extra filled with __info__(:macros) only returns public macros and with that only those already compiled and loadable. The result is improper tracking of references in traversed code.

Example code using the new option:

case Macro.Env.expand_import(env, meta, fun, arity,
  trace: true,
  allow_locals: allow_locals,
  check_deprecations: false,
  local_for_callback: fn meta, name, arity, kinds, e ->
    # a custom local resolver that uses mods_funs_to_positions def dictionary
    case state.mods_funs_to_positions[{e.module, name, arity}] do
      nil ->
        # no info found - treat as reference to a not existing local
        false

      %ModFunInfo{} = info ->
        category = ModFunInfo.get_category(info)
        definition_line = info.positions |> List.first() |> elem(0)
        usage_line = meta |> Keyword.get(:line)

        if ModFunInfo.get_def_kind(info) in kinds and
            (category != :macro or usage_line >= definition_line) do
          # found a local function or macro that is visible in the context
          if macro_exported?(e.module, name, arity) do
            # public macro found - return the expander
            proper_name = :"MACRO-#{name}"
            proper_arity = arity + 1
            Function.capture(e.module, proper_name, proper_arity)
          else
            # return a fake macro expander
            true
          end
        else
          # local not found - return false
          false
        end
    end
  end
) do
  {:macro, module, callback} ->
    expand_macro(meta, module, fun, args, callback, state, env)
  _ ->
    # omitted
end

The PR is based on the current code used in ElixirSense. I'm opening it as a draft. If there is will to merge this I'll work on tests.

Related PR hooking into define_import:
#13628

[josevalim]

Instead of adding a new option, we can augment allow_locals. If it is false, it is disabled, if it is true, it is the default implementation (elixir_locals), otherwise it is a custom function.

[lukaszsamson]

Done on the erlang side. Do you also want that on the elixir API side?

[josevalim]

We can definitely get this in. I have dropped some comments, some of them are about refactoring the current code, but I can also do that after merging.

[lukaszsamson]

@josevalim I removed the Extra and added a resolver but with that I started to run into bootstrap issues and strange test failures like:

     error: unknown bitstring specifier: signed_16()
     │
 303 │       byte_size(sec_data)::size(1)-signed_16(),
     │                          ^
     │
     └─ test/elixir/kernel/binary_test.exs:303:26: Kernel.BinaryTest."test bitsyntax macro"/1

     error: unknown bitstring specifier: refb_spec()
     │
 301 │       byte_size(refb)::refb_spec(),
     │                      ^
     │
     └─ test/elixir/kernel/binary_test.exs:301:22: Kernel.BinaryTest."test bitsyntax macro"/1

[josevalim]

@lukaszsamson so please keep the extra for now. we can merge it and i will investigate it later :)

[josevalim]

We don't really need the kinds, do we? I think it can be hidden from the API so we keep this as a 4 arity function?

[lukaszsamson]

We could hide kinds and assume it's a macro resolver.

[josevalim]

Let’s do that, since it is for expansion (so it has to be macros)

[lukaszsamson]

done

[josevalim]

Do we return any or does it have to be AST?

[lukaszsamson]

Actually the return type is false | function()

[josevalim]

Ah, so we should update it!

[lukaszsamson]

fixed