macros: improve expansion performance #37569

jseyfried · 2016-11-03T22:46:36Z

#34908 regressed expansion performance on recursive, tt-heavy workloads.
This PR fixes that regression, further improves performance on recursive, tt-heavy workloads, and makes a variety of other improvements to parsing and expansion performance.

Expansion performance improvements:

Test case	Run-time	Memory usage
libsyntax	8%	10%
librustc	15%	6%
librustc_trans	30%	6%
#37074	20%	15%
#34630	40%	8%

r? @eddyb

jseyfried · 2016-11-03T22:47:22Z

The ast::ExprKind commit is the AST version of #37445 (h/t @nnethercote).
cc @durka @nrc

eddyb · 2016-11-03T23:00:36Z

@Manishearth r=me

Manishearth · 2016-11-03T23:07:40Z

@bors r=eddyb

cc @dtolnay

bors · 2016-11-03T23:07:41Z

📌 Commit e652060 has been approved by eddyb

`Token::Interpolated(Nonterminal)` -> `Token::Interpolated(Rc<Nonterminal>)`.

This reverts commit 41745f3.

bluss · 2016-11-04T03:34:36Z

Since not needing ast coercion is a feature, it should have tests so that it is not broken by this improvement. Are there any tests? I didn't find any in the tree.

jseyfried · 2016-11-04T03:40:30Z

By "not needing ast coercion", do you mean #22819? That is tested here.

bluss · 2016-11-04T08:44:42Z

great! I overlooked compile-fail.

jseyfried · 2016-11-04T22:50:11Z

@bors r=eddyb

bors · 2016-11-04T22:50:13Z

📌 Commit 51104e5 has been approved by eddyb

@eddyb

…=eddyb macros: improve expansion performance This PR fixes that regression, further improves performance on recursive, `tt`-heavy workloads, and makes a variety of other improvements to parsing and expansion performance. Expansion performance improvements: | Test case | Run-time | Memory usage | | -------------- | -------- | ------------ | | libsyntax | 8% | 10% | | librustc | 15% | 6% | | librustc_trans | 30% | 6% | | rust-lang#37074 | 20% | 15% | | rust-lang#34630 | 40% | 8% | r? @eddyb

Rollup of 24 pull requests - Successful merges: #37255, #37317, #37408, #37410, #37422, #37427, #37470, #37501, #37537, #37556, #37557, #37564, #37565, #37566, #37569, #37574, #37577, #37579, #37583, #37585, #37586, #37587, #37589, #37596 - Failed merges: #37521, #37547

@eddyb

…=eddyb macros: improve expansion performance This PR fixes that regression, further improves performance on recursive, `tt`-heavy workloads, and makes a variety of other improvements to parsing and expansion performance. Expansion performance improvements: | Test case | Run-time | Memory usage | | -------------- | -------- | ------------ | | libsyntax | 8% | 10% | | librustc | 15% | 6% | | librustc_trans | 30% | 6% | | rust-lang#37074 | 20% | 15% | | rust-lang#34630 | 40% | 8% | r? @eddyb

Rollup of 24 pull requests - Successful merges: #37255, #37317, #37408, #37410, #37422, #37427, #37470, #37501, #37537, #37556, #37557, #37564, #37565, #37566, #37569, #37574, #37577, #37579, #37583, #37585, #37586, #37587, #37589, #37596 - Failed merges: #37521, #37547

nnethercote · 2016-11-07T01:28:30Z

AFAICT this PR regressed the speed of html5ever compilation in rustc-benchmarks. perf.rust-lang.org is showing a 6% regression for html5ever this week while other benchmarks are basically unchanged. This PR is one of the PRs that show up in the regressing measurement, and I have profiling data that corroborates.

One cause of this is the span_to_filename call in Parser::new_with_doc_flag, which causes 800k extra allocations. Another cause is an extra 800k allocations in macro_parser::parse. I think this is because of the Box::new(rdr) at the top, but I'm not certain. (I found these by comparing DHAT output for before and after.)

I think these are not the only parts of the change that lead to regressions, but it's less obvious from my profiling what else might be a factor, other than vec and slice operations are showing up higher in the profile. (I got that from the Cachegrind output.)

jseyfried · 2016-11-07T01:33:45Z

@nnethercote Thanks, I'll fix the regression ASAP.

@jseyfried

…fried Macro parser performance improvements and refactoring This PR locally increased performance of #37074 by ~6.6 minutes. Follow up to #37569, but doesn't focus explicitly on expansion performance. History is relatively clean, but I can/will do some more polishing if this is deemed mergeable. Partially posting this now so I can get Travis to run tests for me. r? @jseyfried

rust-highfive assigned eddyb Nov 3, 2016

jseyfried added 7 commits November 3, 2016 23:48

Clean up parser.parse_token_tree().

5f280a5

Reduce the size of Token and make it cheaper to clone by refactoring

eb3ac29

`Token::Interpolated(Nonterminal)` -> `Token::Interpolated(Rc<Nonterminal>)`.

Revert "macros: Improve tt fragments"

3b71646

This reverts commit 41745f3.

Reimplement "macros: Improve tt fragments" with better performance.

6e9bf12

Make ast::ExprKind smaller.

b60bcba

Avoid recontructing the Parser in macro_parser.rs.

e2b3fec

Move doc comment desugaring into the parser.

7ae0833

jseyfried force-pushed the improve_expansion_perf branch from e652060 to 91c778a Compare November 3, 2016 23:48

jseyfried added 2 commits November 4, 2016 02:38

Improve tt-heavy expansion performance.

23ad6fd

Remove field TtReader::next_tok.

b7eed53

jseyfried force-pushed the improve_expansion_perf branch from 91c778a to b7eed53 Compare November 4, 2016 02:40

Fix fallout in tests.

51104e5

alexcrichton mentioned this pull request Nov 4, 2016

Rollup of 24 pull requests #37597

Merged

jseyfried deleted the improve_expansion_perf branch November 6, 2016 09:27

bors merged commit 51104e5 into rust-lang:master Nov 6, 2016

Mark-Simulacrum mentioned this pull request Nov 11, 2016

Macro parser performance improvements and refactoring #37701

Merged

jseyfried mentioned this pull request Dec 6, 2016

possible regression around paths and macro-generated modules #38190

Closed

Mark-Simulacrum mentioned this pull request May 16, 2017

Compiling rust-uinput takes almost 20 minutes #37074

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

macros: improve expansion performance #37569

macros: improve expansion performance #37569

jseyfried commented Nov 3, 2016 •

edited

Loading

jseyfried commented Nov 3, 2016 •

edited

Loading

eddyb commented Nov 3, 2016

Manishearth commented Nov 3, 2016

bors commented Nov 3, 2016

bluss commented Nov 4, 2016

jseyfried commented Nov 4, 2016 •

edited

Loading

bluss commented Nov 4, 2016

jseyfried commented Nov 4, 2016

bors commented Nov 4, 2016

nnethercote commented Nov 7, 2016

jseyfried commented Nov 7, 2016

macros: improve expansion performance #37569

macros: improve expansion performance #37569

Conversation

jseyfried commented Nov 3, 2016 • edited Loading

jseyfried commented Nov 3, 2016 • edited Loading

eddyb commented Nov 3, 2016

Manishearth commented Nov 3, 2016

bors commented Nov 3, 2016

bluss commented Nov 4, 2016

jseyfried commented Nov 4, 2016 • edited Loading

bluss commented Nov 4, 2016

jseyfried commented Nov 4, 2016

bors commented Nov 4, 2016

nnethercote commented Nov 7, 2016

jseyfried commented Nov 7, 2016

jseyfried commented Nov 3, 2016 •

edited

Loading

jseyfried commented Nov 3, 2016 •

edited

Loading

jseyfried commented Nov 4, 2016 •

edited

Loading