Stabilize count, ignore, index, and len (macro_metavar_expr)

[c410-f3r]

Stabilization proposal

This PR proposes the stabilization of a subset of #![feature(macro_metavar_expr)] or more specifically, the stabilization of count, ignore, index and len.

Tracking issue: #83527
Version: 1.80 (June 13 2024 => beta, July 25 2024 => stable).

What is stabilized

Count

The number of times a meta variable repeats in total.

The output of count depends on where it is placed as well the provided index. If no index is provided, then it will always start at the innermost level.

macro_rules! count_idents {
    ( $( $i:ident ),* ) => {
        ${count($i)}
    };
}

fn main() {
    assert_eq!(count_idents!(a, b, c), 3);
}

Ignore

Binds a meta variable for repetition, but expands to nothing.

macro_rules! count {
    ( $( $i:stmt ),* ) => {{
        0 $( + 1 ${ignore($i)} )*
    }};
}

fn main() {
    assert_eq!(count!(if true {} else {}, let _: () = (), || false), 3);
}

Index

The current index of the inner-most repetition.

index is a counter that starts from 0 until the end of the repetition.

trait Foo {
    fn bar(&self) -> usize;
}

macro_rules! impl_tuple {
    ( $( $name:ident ),* ) => {
        impl<$( $name, )*> Foo for ($( $name, )*)
        where
            $( $name: AsRef<[u8]>, )*
        {
            fn bar(&self) -> usize {
                let mut sum: usize = 0;
                $({
                    const $name: () = ();
                    sum = sum.wrapping_add(self.${index()}.as_ref().len());
                })*
                sum
            }
        }
    };
}

impl_tuple!(A, B, C, D);

fn main() {
}

len

The current index starting from the inner-most repetition.

len indicates the sum of meta variable elements, aka length, of the current repetition.

macro_rules! array_3d {
    ( $( $( $number:literal ),* );* ) => {
        [
            $(
                [
                    $( $number + ${len()}, )*
                ],
            )*
        ]
    };
}

fn main() {
    assert_eq!(array_3d!(0, 1; 2, 3; 4, 5), [[2, 3], [4, 5], [6, 7]]);
}

Motivation

Meta variable expressions not only facilitate the use of macros but also allow things that can't be done today like in the $index example.

An initial effort to stabilize this feature was made in #111908 but ultimately reverted because of possible obstacles related to syntax and expansion.

Nevertheless, #83527 (comment) tried to address some questions and fortunately the lang team accept #117050 the unblocking suggestions.

Here we are today after ~4 months so everything should be mature enough for wider use.

After RFC changes

In order to unblock progress, some changes were applied in #118958. These changes did not trigger any opposition debates and were quickly integrated.

1. The depth direction of count changed from outermost-to-innermost to innermost-to-outermost.
2. Metavariables were originally referenced by their names like in count(some_metavariable) but now they must be preceded by $. For example, count($some_metavariable).

What isn't stabilized

$$ is not being stabilized due to unresolved concerns.

History

• 2021-02-22, RFC: Declarative macro metavariable expressions
• 2021-03-26, Tracking Issue for RFC 3086: macro metavariable expressions
• 2022-02-01, Implement macro meta-variable expressions
• 2022-02-25, [1/2] Implement macro meta-variable expressions
• 2022-03-11, [2/2] Implement macro meta-variable expressions
• 2022-03-12, Fix remaining meta-variable expression TODOs
• 2019-03-21, [macro-metavar-expr] Fix generated tokens hygiene
• 2022-04-07, Kickstart the inner usage of macro_metavar_expr
• 2022-04-07, [macro_metavar_expr] Add tests to ensure the feature requirement
• 2023-05-24, [RFC-3086] Restrict the parsing of count
• 2023-10-22, [RFC 3086] Attempt to try to resolve blocking concerns

Tests

This list is a subset of https://github.com/rust-lang/rust/tree/master/tests/ui/macros/rfc-3086-metavar-expr.

• Ensures that nested macros have correct behavior

• Compares produced tokens to assert expected outputs

• Checks the declarations of the features

• Verifies all possible errors that can occur due to incorrect user input

Possible future work

With enough consensus, other nightly expressions can be added for experimentation and possibly stabilized in the future. For example, #118958.

Thanks @markbt for creating the RFC and thanks to @petrochenkov and @mark-i-m for reviewing the implementations.

[rustbot]

r? @fmease

rustbot has assigned @fmease.
They will have a look at your PR within the next two weeks and either review your PR or reassign to another reviewer.

Use r? to explicitly pick a reviewer

[fmease]

@c410-f3r Has this been documented in the Reference already? The box is not ticked over at the tracking issue. If not, stabilization is blocked on adding sufficient documentation.

[joshtriplett]

@rfcbot merge

@rfcbot concern document-in-reference

[rfcbot]

Team member @joshtriplett has proposed to merge this. The next step is review by the rest of the tagged team members:

• @joshtriplett
• @nikomatsakis
• @pnkfelix
• @scottmcm
• @tmandry

Concerns:

• depth levels are confusing (Stabilize count, ignore, index, and len (macro_metavar_expr) #122808 (comment))
• document-in-reference resolved by Stabilize count, ignore, index, and len (macro_metavar_expr) #122808 (comment)
• hygiene-handled? resolved by Stabilize count, ignore, index, and len (macro_metavar_expr) #122808 (comment)

Once a majority of reviewers approve (and at most 2 approvals are outstanding), this will enter its final comment period. If you spot a major issue that hasn't been raised at any point in this process, please speak up!

cc @rust-lang/lang-advisors: FCP proposed for lang, please feel free to register concerns.
See this document for info about what commands tagged team members can give me.

[RalfJung]

AFAIK it is currently equivalent to count($x, 0), which does align with my intuition once I re-familiarized myself with how nested capture groups work. I don't know what infinite depth would even mean.

[tmandry]

I'm not sure what count(x, 0) would mean, based on this from the RFC:

If repetitions are nested, then an optional depth parameter can be used to limit the number of nested repetitions that are counted. For example, a macro expansion like:

${count(x, 1)} ${count(x, 2)} ${count(x, 3)} $( a $( b $( $x )* )* )*

The three values this expands to are the number of outer-most repetitions (the number of times a would be generated), the sum of the number of middle repetitions (the number of times b would be generated), and the total number of repetitions of $x.

Reading this over again though, I no longer think it makes sense to limit to 1 by default. ${count(x)} should of course count the number of expansions for $x, not the number of expansions for the next-expandable metavariable that $x happens to be nested in. We could be conservative in only allowing it in the same contexts where $x can already be expanded, but I can see it being useful outside of those contexts.

I still think specifying a max depth level is non-obvious and of dubious value, since you could always rewrite the excerpt above as something like the following; are there cases where you can't do this?

 ${count(a)} ${count(b)} ${count(x)} $( a $( b $( $x )* )* )*

[RalfJung]

I'm not sure what count(x, 0) would mean, based on this from the RFC:

I have linked this already above, see here. That's what I reverse engineered based on my experiments, anyway. The default is definitely 0, according to that PR.

Maybe the semantics changed since the RFC was accepted?

Would be good to get an up=to-date description of the proposed semantics confirmed by the people involved in the implementation -- Cc @c410-f3r , not sure who else?

I still think specifying a max depth level is non-obvious and of dubious value, since you could always rewrite the excerpt above as something like the following; are there cases where you can't do this?

I think the issue is around matchers like $( [[ $( {{ $( $x, )* }} )* ]] )* where there is no variable in the outer repetition groups that one can refer to.

[c410-f3r]

I will try to address all questions this weekend in my free time.

cc rust-lang/reference#1485 (comment)

[c410-f3r]

I see it as a major downside, quite possibly a failure, of the lang team process that we got this far without such syntax concerns being surfaced.

Thank you. macro_metavar_expr has been around for +3 years and its implementation has been available for +2 years. Although part of the game, it is surprising and demotivating to see syntax concerns in stabilizing PRs (#95860).

The only point that gives me pause around count is -- if/when we have named repetition groups, how will they work with count, and is that compatible with today's behavior?

It is more complex but not impossible IMO. One way is to attach a corresponding matcher depth to each named repetition and then evaluate which ones can be associated based on the current transcriber position.

Syntax-wise, the distinction between numbers and strings can create appropriated implementation branches.

What I meant is, why not make count($x) equivalent to count($x, 1) instead of count($x, ∞)? We would then have to invent syntax for the latter if we wanted to preserve expressiveness, but this default aligns more with my expectations.

AFAIK it is currently equivalent to count($x, 0),

Before #117050 count($x) was equal to count($x, N) with N being the number of nested repetitions minus one. Now count($x) is equal to count($x, 0) which can also be seen as the innermost repetition.

rust-lang/reference#1485 (comment) will probably provide a better overview.

I still think specifying a max depth level is non-obvious and of dubious value, since you could always rewrite the excerpt above as something like the following; are there cases where you can't do this?

${count(a)} ${count(b)} ${count(x)} $( a $( b $( $x )* )* )*

Unfortunately it is not possible to count without a metavariable. AFAICT, a and b are just references used in the RFC to illustrative how the algorithm works.

But yeah, the specification of a depth is non-obvious as it depends on the number of nested matcher repetitions as well as the positioning.

the people involved in the implementation

@markbt, @petrochenkov and @mark-i-m

[tmandry]

I see, I was confused by reading the RFC for more detail, but that is now outdated. @c410-f3r Can you please include a list of differences from the RFC in the stabilization report (with rationale; feel free to summarize and link to more context)?

[c410-f3r]

I see, I was confused by reading the RFC for more detail, but that is now outdated. @c410-f3r Can you please include a list of differences from the RFC in the stabilization report (with rationale; feel free to summarize and link to more context)?

Sure. The report has been updated.

[alex-semenyuk]

@c410-f3r
From triage team. Does it ready for review? if so could you please fix merge conflicts before

[c410-f3r]

Well, I can rebase but I don't think it will make much of a difference. The actual issue lies within the decision around the use of indices in count, index and len.

This PR has been waiting for more than 2 months for a response from the responsible team. AFAICT, all questions posted here as well as in rust-lang/reference#1485 have already been answered to the best of my ability.

[fmease]

What's the context? Why was this closed?

[Lokathor]

Since the person who closed it is the author, who in September had already been waiting for "more than 2 months", and it's been many months since, and it's even been a month since it was Lang nominated with no update from T-lang, I have to assume the author lost interest in going any further with this.

[ZaneErebos]

I really hope this does not get abandoned. I was tracking this for a while and it would be a great addition. In any case, thank you to the author for all their hard work and patience

[jhpratt]

I will take this over (whether as this PR or a new one) if the author is no longer interested.