Allow Check targets needed for optional doc-scraping to fail without killing the build #11450

willcrichton · 2022-12-03T07:53:48Z

What does this PR try to resolve?

In doing a Crater run of -Zrustdoc-scrape-examples, I found that the only remaining regressions are cases where:

A library does not type-check
The library has examples
Cargo tries to scrape the examples, which requires checking the library
The Check unit for the library fails, crashing the build

The core issue is that the Check unit should be able to fail without killing the build. This PR fixes this issue by checking for this condition, and then allowing the unit to fail.

Specifically, I added a new method BuildContext::unit_can_fail_for_docscraping that determines the conditions for whether a unit is allowed to fail. This method is used both in JobQueue to interpret process failure, and in the rustc/rustdoc functions to emit a warning upon failure. I modified rustc to handle the case of failure similar to rustdoc, but with a slightly different diagnostic.

How should we test and review this PR?

The unit test no_fail_bad_lib has been extended with example files to test this case.

r? @weihanglo

…killing the build

…provements, r=notriddle Improve Rustdoc scrape-examples UI This PR combines a few different improvements to the scrape-examples UI. See a live demo here: https://willcrichton.net/misc/scrape-examples/small-first-example/clap/struct.Arg.html ### 1. The first scraped example now takes up significantly less screen height. Inserting the first scraped example takes up a lot of vertical screen space. I don't want this addition to overwhelm users, so I decided to reduce the height of the initial example in two ways: (A) the default un-expanded height is reduced from 240px (10 LOC) to 120px (5 LOC), and (B) the link to the example is now positioned *over* the example instead of *atop* the example (only on desktop though, not mobile). The changes to `scrape-examples.js` and `rustdoc.css` implement this fix. Here is what an example docblock now looks like: ![Screen Shot 2022-12-06 at 10 02 21 AM](https://user-images.githubusercontent.com/663326/205987450-3940063c-5973-4a34-8579-baff6a43aa9b.png) ### 2. Expanding all docblocks will not expand "More examples". The "More examples blocks" are huge, so fully expanding everything on the page would take up too much vertical space. The changes to `main.js` implement this fix. This is tested in `scrape-examples-toggle.goml`. ### 3. Examples from binary crates are sorted higher than examples from library crates. Code that is written as an example of an API is probably better for learning than code that happens to use an API, but isn't intended for pedagogic purposes. Unfortunately Rustc doesn't know whether a particular crate comes from an example target (only Cargo knows this). But we can at least create a proxy that prefers examples from binary crates over library crates, which we know from `--crate-type`. This change is implemented by adding a new field `bin_crate` in `Options` (see `config.rs`). An `is_bin` field has been added to the scraped examples metadata (see `scrape_examples.rs`). Then the example sorting metric uses `is_bin` as the first entry of a lexicographic sort on `(is_bin, example_size, display_name)` (see `render/mod.rs`). Note that in the future we can consider adding another flag like `--scrape-examples-cargo-target` that would pass target information from Cargo into the example metadata. But I'm proposing a less intrusive change for now. ### 4. The scrape-examples help page has been updated to reflect the latest Cargo interface. See `scrape-examples-help.md`. r? `@notriddle` P.S. once this PR and rust-lang/cargo#11450 are merged, then I think the scrape-examples feature is officially ready for deployment on docs.rs!

…provements, r=notriddle Improve Rustdoc scrape-examples UI This PR combines a few different improvements to the scrape-examples UI. See a live demo here: https://willcrichton.net/misc/scrape-examples/small-first-example/clap/struct.Arg.html ### 1. The first scraped example now takes up significantly less screen height. Inserting the first scraped example takes up a lot of vertical screen space. I don't want this addition to overwhelm users, so I decided to reduce the height of the initial example in two ways: (A) the default un-expanded height is reduced from 240px (10 LOC) to 120px (5 LOC), and (B) the link to the example is now positioned *over* the example instead of *atop* the example (only on desktop though, not mobile). The changes to `scrape-examples.js` and `rustdoc.css` implement this fix. Here is what an example docblock now looks like: ![Screen Shot 2022-12-06 at 10 02 21 AM](https://user-images.githubusercontent.com/663326/205987450-3940063c-5973-4a34-8579-baff6a43aa9b.png) ### 2. Expanding all docblocks will not expand "More examples". The "More examples blocks" are huge, so fully expanding everything on the page would take up too much vertical space. The changes to `main.js` implement this fix. This is tested in `scrape-examples-toggle.goml`. ### 3. Examples from binary crates are sorted higher than examples from library crates. Code that is written as an example of an API is probably better for learning than code that happens to use an API, but isn't intended for pedagogic purposes. Unfortunately Rustc doesn't know whether a particular crate comes from an example target (only Cargo knows this). But we can at least create a proxy that prefers examples from binary crates over library crates, which we know from `--crate-type`. This change is implemented by adding a new field `bin_crate` in `Options` (see `config.rs`). An `is_bin` field has been added to the scraped examples metadata (see `scrape_examples.rs`). Then the example sorting metric uses `is_bin` as the first entry of a lexicographic sort on `(is_bin, example_size, display_name)` (see `render/mod.rs`). Note that in the future we can consider adding another flag like `--scrape-examples-cargo-target` that would pass target information from Cargo into the example metadata. But I'm proposing a less intrusive change for now. ### 4. The scrape-examples help page has been updated to reflect the latest Cargo interface. See `scrape-examples-help.md`. r? ``@notriddle`` P.S. once this PR and rust-lang/cargo#11450 are merged, then I think the scrape-examples feature is officially ready for deployment on docs.rs!

…provements, r=notriddle Improve Rustdoc scrape-examples UI This PR combines a few different improvements to the scrape-examples UI. See a live demo here: https://willcrichton.net/misc/scrape-examples/small-first-example/clap/struct.Arg.html ### 1. The first scraped example now takes up significantly less screen height. Inserting the first scraped example takes up a lot of vertical screen space. I don't want this addition to overwhelm users, so I decided to reduce the height of the initial example in two ways: (A) the default un-expanded height is reduced from 240px (10 LOC) to 120px (5 LOC), and (B) the link to the example is now positioned *over* the example instead of *atop* the example (only on desktop though, not mobile). The changes to `scrape-examples.js` and `rustdoc.css` implement this fix. Here is what an example docblock now looks like: ![Screen Shot 2022-12-06 at 10 02 21 AM](https://user-images.githubusercontent.com/663326/205987450-3940063c-5973-4a34-8579-baff6a43aa9b.png) ### 2. Expanding all docblocks will not expand "More examples". The "More examples blocks" are huge, so fully expanding everything on the page would take up too much vertical space. The changes to `main.js` implement this fix. This is tested in `scrape-examples-toggle.goml`. ### 3. Examples from binary crates are sorted higher than examples from library crates. Code that is written as an example of an API is probably better for learning than code that happens to use an API, but isn't intended for pedagogic purposes. Unfortunately Rustc doesn't know whether a particular crate comes from an example target (only Cargo knows this). But we can at least create a proxy that prefers examples from binary crates over library crates, which we know from `--crate-type`. This change is implemented by adding a new field `bin_crate` in `Options` (see `config.rs`). An `is_bin` field has been added to the scraped examples metadata (see `scrape_examples.rs`). Then the example sorting metric uses `is_bin` as the first entry of a lexicographic sort on `(is_bin, example_size, display_name)` (see `render/mod.rs`). Note that in the future we can consider adding another flag like `--scrape-examples-cargo-target` that would pass target information from Cargo into the example metadata. But I'm proposing a less intrusive change for now. ### 4. The scrape-examples help page has been updated to reflect the latest Cargo interface. See `scrape-examples-help.md`. r? `@notriddle` P.S. once this PR and rust-lang/cargo#11450 are merged, then I think the scrape-examples feature is officially ready for deployment on docs.rs!

weihanglo

Looks reasonable. Thank you!

src/cargo/core/compiler/rustdoc.rs

src/cargo/core/compiler/mod.rs

…s, r=notriddle Improve Rustdoc scrape-examples UI This PR combines a few different improvements to the scrape-examples UI. See a live demo here: https://willcrichton.net/misc/scrape-examples/small-first-example/clap/struct.Arg.html ### 1. The first scraped example now takes up significantly less screen height. Inserting the first scraped example takes up a lot of vertical screen space. I don't want this addition to overwhelm users, so I decided to reduce the height of the initial example in two ways: (A) the default un-expanded height is reduced from 240px (10 LOC) to 120px (5 LOC), and (B) the link to the example is now positioned *over* the example instead of *atop* the example (only on desktop though, not mobile). The changes to `scrape-examples.js` and `rustdoc.css` implement this fix. Here is what an example docblock now looks like: ![Screen Shot 2022-12-06 at 10 02 21 AM](https://user-images.githubusercontent.com/663326/205987450-3940063c-5973-4a34-8579-baff6a43aa9b.png) ### 2. Expanding all docblocks will not expand "More examples". The "More examples blocks" are huge, so fully expanding everything on the page would take up too much vertical space. The changes to `main.js` implement this fix. This is tested in `scrape-examples-toggle.goml`. ### 3. Examples from binary crates are sorted higher than examples from library crates. Code that is written as an example of an API is probably better for learning than code that happens to use an API, but isn't intended for pedagogic purposes. Unfortunately Rustc doesn't know whether a particular crate comes from an example target (only Cargo knows this). But we can at least create a proxy that prefers examples from binary crates over library crates, which we know from `--crate-type`. This change is implemented by adding a new field `bin_crate` in `Options` (see `config.rs`). An `is_bin` field has been added to the scraped examples metadata (see `scrape_examples.rs`). Then the example sorting metric uses `is_bin` as the first entry of a lexicographic sort on `(is_bin, example_size, display_name)` (see `render/mod.rs`). Note that in the future we can consider adding another flag like `--scrape-examples-cargo-target` that would pass target information from Cargo into the example metadata. But I'm proposing a less intrusive change for now. ### 4. The scrape-examples help page has been updated to reflect the latest Cargo interface. See `scrape-examples-help.md`. r? `@notriddle` P.S. once this PR and rust-lang/cargo#11450 are merged, then I think the scrape-examples feature is officially ready for deployment on docs.rs!

weihanglo

Thank you! It's good to merge :)

weihanglo · 2022-12-11T22:27:13Z

@bors r+

bors · 2022-12-11T22:27:15Z

📌 Commit 29407d0 has been approved by weihanglo

It is now in the queue for this repository.

bors · 2022-12-11T22:27:23Z

⌛ Testing commit 29407d0 with merge 4a8d17e...

bors · 2022-12-11T23:14:51Z

☀️ Test successful - checks-actions
Approved by: weihanglo
Pushing 4a8d17e to master...

8 commits in 70898e522116f6c23971e2a554b2dc85fd4c84cd..cc0a320879c17207bbfb96b5d778e28a2c62030d 2022-12-05 19:43:44 +0000 to 2022-12-14 14:46:57 +0000 - artifact deps should works when target field specified coexists with `optional = true` (rust-lang/cargo#11434) - Add `home` crate to cargo crates (rust-lang/cargo#11359) - Use proper git URL for GitHub repos (rust-lang/cargo#11475) - Fixes flock(fd, LOCK_UN) emulation on Solaris. (rust-lang/cargo#11474) - Allow Check targets needed for optional doc-scraping to fail without killing the build (rust-lang/cargo#11450) - fix: gleaning rustdocflags from `target.cfg(…)` is not supported (rust-lang/cargo#11323) - Fix typo (rust-lang/cargo#11470) - resolver.md: Fix naming in example (rust-lang/cargo#11469)

Update cargo 8 commits in 70898e522116f6c23971e2a554b2dc85fd4c84cd..cc0a320879c17207bbfb96b5d778e28a2c62030d 2022-12-05 19:43:44 +0000 to 2022-12-14 14:46:57 +0000 - artifact deps should works when target field specified coexists with `optional = true` (rust-lang/cargo#11434) - Add `home` crate to cargo crates (rust-lang/cargo#11359) - Use proper git URL for GitHub repos (rust-lang/cargo#11475) - Fixes flock(fd, LOCK_UN) emulation on Solaris. (rust-lang/cargo#11474) - Allow Check targets needed for optional doc-scraping to fail without killing the build (rust-lang/cargo#11450) - fix: gleaning rustdocflags from `target.cfg(…)` is not supported (rust-lang/cargo#11323) - Fix typo (rust-lang/cargo#11470) - resolver.md: Fix naming in example (rust-lang/cargo#11469) r? `@ghost`

rustbot assigned weihanglo Dec 3, 2022

rustbot added the S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. label Dec 3, 2022

Allow Check targets needed for optional doc-scraping to fail without …

de34d60

…killing the build

willcrichton force-pushed the example-analyzer branch from 8e647ba to de34d60 Compare December 3, 2022 18:13

willcrichton mentioned this pull request Dec 6, 2022

Improve Rustdoc scrape-examples UI rust-lang/rust#105387

Merged

weihanglo reviewed Dec 9, 2022

View reviewed changes

Only compute failed_scrape_diagnostic when needed

29407d0

weihanglo approved these changes Dec 11, 2022

View reviewed changes

bors added S-waiting-on-bors Status: Waiting on bors to run and complete tests. Bors will change the label on completion. and removed S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. labels Dec 11, 2022

bors merged commit 4a8d17e into rust-lang:master Dec 11, 2022

willcrichton mentioned this pull request Dec 13, 2022

Add -Zrustdoc-scrape-examples to default Cargo args rust-lang/docs.rs#1954

Merged

weihanglo mentioned this pull request Dec 14, 2022

Update cargo rust-lang/rust#105704

Merged

weihanglo mentioned this pull request Dec 27, 2022

Tracking Issue for doc --scrape-examples #9910

Open

ehuss added this to the 1.68.0 milestone Dec 30, 2022

weihanglo added the Z-scrape-examples Nightly: rustdoc scrape examples label Sep 20, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Allow Check targets needed for optional doc-scraping to fail without killing the build #11450

Allow Check targets needed for optional doc-scraping to fail without killing the build #11450

willcrichton commented Dec 3, 2022 •

edited

Loading

weihanglo left a comment

weihanglo left a comment

weihanglo commented Dec 11, 2022

bors commented Dec 11, 2022

bors commented Dec 11, 2022

bors commented Dec 11, 2022

Allow Check targets needed for optional doc-scraping to fail without killing the build #11450

Allow Check targets needed for optional doc-scraping to fail without killing the build #11450

Conversation

willcrichton commented Dec 3, 2022 • edited Loading

What does this PR try to resolve?

How should we test and review this PR?

weihanglo left a comment

Choose a reason for hiding this comment

weihanglo left a comment

Choose a reason for hiding this comment

weihanglo commented Dec 11, 2022

bors commented Dec 11, 2022

bors commented Dec 11, 2022

bors commented Dec 11, 2022

willcrichton commented Dec 3, 2022 •

edited

Loading