How to correctly use padded() after repeated() in v1.0.0.alpha.7

[Divy1211]

Hi, I have a lexer with the following code:

pub type Span = SimpleSpan<usize>;
pub type Spanned<T> = (T, Span);

pub fn lexer<'src>() -> impl Parser<
    'src, &'src str, Vec<Spanned<Token>>, extra::Err<Rich<'src, char, Span>>
> {
    choice((
        comment(),
        literal(),
        keyword(),
        operator(),
        punctuation(),
    ))        
        .map_with(|tok, info| (tok, info.span()))
        .padded()
        // .recover_with(skip_then_retry_until(any().ignored(), end()))
        .repeated()
        .collect()
}

When I have an empty code file: "", the output is fine, but if the empty code file has stray whitespaces: " ", then I get a lexing error about unexpected end of input.

I suspect this happens because the empty string "" is seen as being repeated 0 times, so choice(...).padded() is never invoked, but the string with spaces is seen as repeated once, and after the spaces are consumed by the padded() parser, the choice(...) parser returns an error because it sees EoF instead of a valid token.

• I initially tried src.trim() but that makes the code spans incorrect for the original string, messing up the Ariadne output.
• I'm leaning towards just checking if src.trim() == "" and simply skipping invoking the lexer, but I'm not sure if this is the best way or if I missed something I can do with combinators themselves.

Any help is appreciated!

[Zij-IT]

I believe this would work fine if you add a .padded() after repeated. This should allow whitespace surrounding the item repeated, even if it’s repeated 0 times.

[Divy1211]

Ahhh, I was also trying to add a padded() call after repeated(), but that was giving me a trait-bound type error. But, I just realised that I can move the padded() call to after collect(), which works:

pub type Span = SimpleSpan<usize>;
pub type Spanned<T> = (T, Span);

pub fn lexer<'src>() -> impl Parser<
    'src, &'src str, Vec<Spanned<Token>>, extra::Err<Rich<'src, char, Span>>
> {
    choice((
        comment(),
        literal(),
        keyword(),
        operator(),
        punctuation(),
    ))        
        .map_with(|tok, info| (tok, info.span()))
        .padded()
        // .recover_with(skip_then_retry_until(any().ignored(), end()))
        .repeated()
        .collect()
        .padded() // this call needs to go after collect()
}

Thanks for the help!!