Generating a documentation for tests #130463

ifxfrancois · 2024-09-17T11:00:54Z

This PR add the new option --document-tests to rustdoc.
When this option is set:

the test modules are not filtered out by rustdoc when generating the documentation,
the functions marked as #[test] are collected into a new category called Tests instead of Functions

Note: tests are often not marked as public so the option --document-private-items may be required to see the tests in the documentation.

A short example of the generated docs can be found here.

Motivation

During the development of embedded software, Quality or certifications teams need a quick overview of the tests of a project. Typical relevant information are what, how and why a test tests.

What would be a short description of the test.
How would be a short list of the important tests steps.
Why would be references to the relevant detailed architecture or requirements documents.

This information can easily be written as markdown in the documentation of the test functions and the documentation be generated with the additional option. The generated documentation can then be provided to relevant teams.

Limitations

There is no integration with cargo and integrations test can not be added to the same documentation as the rest of the crate documentation.

rustbot · 2024-09-17T11:01:03Z

Thanks for the pull request, and welcome! The Rust team is excited to review your changes, and you should hear from @GuillaumeGomez (or someone else) some time within the next two weeks.

Please see the contribution instructions for more information. Namely, in order to ensure the minimum review times lag, PR authors and assigned reviewers should ensure that the review label (S-waiting-on-review and S-waiting-on-author) stays updated, invoking these commands when appropriate:

@rustbot author: the review is finished, PR author should check the comments and take action accordingly
@rustbot review: the author is ready for a review, this PR will be queued again in the reviewer's queue

rustbot · 2024-09-17T11:01:05Z

Some changes occurred in src/librustdoc/clean/types.rs

cc @camelid

Some changes occurred in HTML/CSS/JS.

cc @GuillaumeGomez, @jsha

jieyouxu · 2024-09-17T11:11:27Z

src/librustdoc/clean/types.rs

+            ItemKind::FunctionItem(_)
+            | ItemKind::MethodItem(_, _)
+            | ItemKind::TyMethodItem(_)
+            | ItemKind::TestItem(_) => {


Modelling a #[test] fn as a distinct ItemKind seems a bit strange: #[test] fns are still function items, I wonder if you can distinguish non-#[test] fns versus #[test] fns by looking at the existence/absence of the #[test] attribute.

But yeah, rendering docs for tests do seem useful. Though a question that immediately comes to my mind is that do test-affiliated module docs get rendered too? As in e.g. docs on #[cfg(test)] mod tests;. I would think that they do, but I haven't looked closely if they do in this implementation.

Yes, they are rendered in this implementation though they are most likely not pub therefore the additional flag --document-private-items may be needed.

I agree with @jieyouxu, but also I like this difference as it makes it simpler to know what's being manipulated... I suppose it's fine as is for now. :)

Another approach could be to add a bool value saying whether or not it's a test function.

As for cfged out items, unfortunately you will need to pass --cfg test to have them (alongside --document-private-items).

src/librustdoc/formats/item_type.rs

src/librustdoc/html/render/mod.rs

GuillaumeGomez · 2024-09-17T12:47:24Z

src/librustdoc/html/render/mod.rs

@@ -2238,6 +2248,7 @@ impl ItemSection {
            Self::Unions => "Unions",
            Self::Enums => "Enums",
            Self::Functions => "Functions",
+            Self::Tests => "Tests",


Maybe "Unit tests" instead? Same above.

This bikeshed can continue as part of the FCP, I think.

But, for my own 2c, I think it should be "Test cases"

GuillaumeGomez · 2024-09-17T12:47:40Z

src/librustdoc/html/render/print_item.rs

@@ -186,6 +186,7 @@ pub(super) fn print_item(cx: &mut Context<'_>, item: &clean::Item, buf: &mut Buf
            }
        }
        clean::FunctionItem(..) | clean::ForeignFunctionItem(..) => "Function ",
+        clean::TestItem(..) => "Test ",


"Unit test "

@GuillaumeGomez We also want to call this documentation build on integration tests. If we change this to "Unit Test", the integration tests will also appear as Unit tests.
With a command like cargo rustdoc --test lib_test -- -Z unstable-options --document-tests --document-private-items

"Test case "

GuillaumeGomez · 2024-09-17T12:48:35Z

I think it's a good idea. Just one thing missing: an entry in the rustdoc book about the command line flag and that it needs to be used with --document-private-items and likely with --cfg test as well.

ifxfrancois · 2024-09-17T16:00:01Z

I think it's a good idea. Just one thing missing: an entry in the rustdoc book about the command line flag and that it needs to be used with --document-private-items and likely with --cfg test as well.

@GuillaumeGomez I don't know about the correct page where to add it, should I add it to the page "Unstable features" or to the page "Command-line arguments".

GuillaumeGomez · 2024-09-17T16:11:53Z

Sorry should have precised. Since the option is unstable, please put it into "Unstable features".

rustbot · 2024-09-18T11:53:12Z

There are merge commits (commits with multiple parents) in your changes. We have a no merge policy so these commits will need to be removed for this pull request to be merged.

You can start a rebase with the following commands:

$ # rebase
$ git pull --rebase https://github.com/rust-lang/rust.git master
$ git push --force-with-lease

The following commits are merge commits:

Urgau · 2024-09-18T12:23:23Z

I think it's a good idea. Just one thing missing: an entry in the rustdoc book about the command line flag and that it needs to be used with --document-private-items and likely with --cfg test as well.

It would be quite easy for rustdoc to automaticly set the test cfg when --document-tests is passed, in the same vain as done with the doc cfg.

rust/src/librustdoc/core.rs

Lines 206 to 207 in 82d17a4

    
           // Add the doc cfg into the doc build. 
        
           cfgs.push("doc".to_string());

GuillaumeGomez · 2024-09-18T14:03:45Z

I think it's a good idea. Just one thing missing: an entry in the rustdoc book about the command line flag and that it needs to be used with --document-private-items and likely with --cfg test as well.

It would be quite easy for rustdoc to automaticly set the test cfg when --document-tests is passed, in the same vain as done with the doc cfg.

rust/src/librustdoc/core.rs

Lines 206 to 207 in 82d17a4

// Add the doc cfg into the doc build.

cfgs.push("doc".to_string());

Yes it's easy to do. The question is more: "should we do it?".

ifxfrancois · 2024-09-18T16:18:30Z

src/librustdoc/core.rs

@@ -229,7 +229,8 @@ pub(crate) fn create_config(
        if proc_macro_crate { vec![CrateType::ProcMacro] } else { vec![CrateType::Rlib] };
    let resolve_doc_links =
        if *document_private { ResolveDocLinks::All } else { ResolveDocLinks::Exported };
-    let test = scrape_examples_options.map(|opts| opts.scrape_tests).unwrap_or(false);
+    let test =
+        scrape_examples_options.map(|opts| opts.scrape_tests).unwrap_or(false) || *document_tests;


I think it's a good idea. Just one thing missing: an entry in the rustdoc book about the command line flag and that it needs to be used with --document-private-items and likely with --cfg test as well.

It would be quite easy for rustdoc to automaticly set the test cfg when --document-tests is passed, in the same vain as done with the doc cfg.

rust/src/librustdoc/core.rs

Lines 206 to 207 in 82d17a4

// Add the doc cfg into the doc build.

cfgs.push("doc".to_string());

Yes it's easy to do. The question is more: "should we do it?".

In this implementation, ' --cfg test` is already passed automatically here.
It may not be the cleanest way to pass it.

Having it passed automatically is convenient, I think it would be a bit counter-intuitive if the option "document-tests" did not actually document tests except if an additional flag is passed.

I think it's an important thing to discuss as it might not be obvious for users. I would personally pass --cfg test as I wouldn't expect an option to modify the cfgs.

I think it might be good to not set --cfg test automatically in rustdoc.
I can see that in the future we might want to create a cargo wrapper for generating documentation of tests. In this case cargo could provide a user friendly way using only a single command or flag which would set all the required rustdoc flags. Still there would be full control over all flags on the rustdoc level.

In this case cargo could provide a user friendly way using only a single command or flag which would set all the required rustdoc flags. Still there would be full control over all flags on the rustdoc level.

If this feature were being added in a vacuum, I might agree. But we already have a ton of cases where it's Rustc, not Cargo, that takes care of deciding all the cfg options (except feature=, of course).

For example, rustc --test sets cfg(test), rustc --target sets the target cfgs, rustdoc sets cfg(doc), and rustdoc --test sets cfg(doctest). The tool itself makes these decisions, not Cargo.

possibly relevant MCP rust-lang/compiler-team#785

FYI, rust-lang/compiler-team#785 has now entered FCP.

Starting with 1.85.0, the test cfg is consider to be a "userspace" config despite being also set by rustc and should be managed by the build system itself.

https://doc.rust-lang.org/nightly/rustc/check-cfg.html

bors · 2024-09-23T05:23:23Z

☔ The latest upstream changes (presumably #130724) made this pull request unmergeable. Please resolve the merge conflicts.

rustbot · 2024-10-07T14:30:57Z

There are merge commits (commits with multiple parents) in your changes. We have a no merge policy so these commits will need to be removed for this pull request to be merged.

You can start a rebase with the following commands:

$ # rebase
$ git pull --rebase https://github.com/rust-lang/rust.git master
$ git push --force-with-lease

The following commits are merge commits (since this message was last posted):

bors · 2024-10-12T03:07:08Z

☔ The latest upstream changes (presumably #131573) made this pull request unmergeable. Please resolve the merge conflicts.

Dylan-DPC · 2024-11-20T14:04:37Z

Can we move on and merge this? Or should we wait for more comments?

This will need another review by the reviewer.

bors · 2024-12-20T10:35:13Z

☔ The latest upstream changes (presumably #134550) made this pull request unmergeable. Please resolve the merge conflicts.

bors · 2024-12-21T04:27:44Z

☔ The latest upstream changes (presumably #134590) made this pull request unmergeable. Please resolve the merge conflicts.

bors · 2025-01-04T16:09:14Z

☔ The latest upstream changes (presumably #135031) made this pull request unmergeable. Please resolve the merge conflicts.

bors · 2025-01-24T05:10:17Z

☔ The latest upstream changes (presumably #135947) made this pull request unmergeable. Please resolve the merge conflicts.

bors · 2025-03-02T16:14:01Z

☔ The latest upstream changes (presumably #137425) made this pull request unmergeable. Please resolve the merge conflicts.

bors · 2025-03-03T23:13:57Z

☔ The latest upstream changes (presumably #137914) made this pull request unmergeable. Please resolve the merge conflicts.

bors · 2025-03-05T06:26:37Z

☔ The latest upstream changes (presumably #138021) made this pull request unmergeable. Please resolve the merge conflicts.

bors · 2025-03-18T09:30:41Z

☔ The latest upstream changes (presumably #138630) made this pull request unmergeable. Please resolve the merge conflicts.

notriddle · 2025-04-12T17:02:45Z

src/librustdoc/visit_ast.rs

+                        let path: String = path
+                            .segments
+                            .iter()
+                            .map(|s| {
+                                if s.ident.name == kw::PathRoot {
+                                    ""
+                                } else {
+                                    s.ident.name.as_str()
+                                }
+                            })
+                            .intersperse("::")
+                            .collect();
+                        if path == "test::TestDescAndFn" {


Suggested change

let path: String = path

.segments

.iter()

.map(|s| {

if s.ident.name == kw::PathRoot {

""

} else {

s.ident.name.as_str()

}

})

.intersperse("::")

.collect();

if path == "test::TestDescAndFn" {

let path_names = path

.segments

.iter()

.map(|s| {

s.ident.name

});

if path_names.eq([sym::test, sym::TestDescAndFn]) {

I think, to follow this suggestion, you'd need to add TestDescAndFn to https://doc.rust-lang.org/nightly/nightly-rustc/src/rustc_span/symbol.rs.html#169

notriddle · 2025-04-12T17:14:33Z

src/librustdoc/html/render/mod.rs

@@ -2238,6 +2248,7 @@ impl ItemSection {
            Self::Unions => "Unions",
            Self::Enums => "Enums",
            Self::Functions => "Functions",
+            Self::Tests => "Tests",


This bikeshed can continue as part of the FCP, I think.

But, for my own 2c, I think it should be "Test cases"

The new option --document-tests is unstable and documented as such. In order to use it is needed to add `--cfg test` and in case the tests are not marked public to add `--document-private-items`. The implementation hide the auto generate main test function and constants.

notriddle · 2025-04-14T16:03:00Z

@rfcbot fcp merge

rfcbot · 2025-04-14T16:03:02Z

Team member @notriddle has proposed to merge this. The next step is review by the rest of the tagged team members:

Concerns:

doctests (Generating a documentation for tests #130463 (comment))
integration-tests (Generating a documentation for tests #130463 (comment))
motivation (Generating a documentation for tests #130463 (comment))

Once a majority of reviewers approve (and at most 2 approvals are outstanding), this will enter its final comment period. If you spot a major issue that hasn't been raised at any point in this process, please speak up!

See this document for info about what commands tagged team members can give me.

Manishearth · 2025-04-14T17:50:11Z

@rfcbot concern motivation

During the development of embedded software, Quality or certifications teams need a quick overview of the tests of a project.

I'm not really convinced of this motivation. This seems perfect for a domain specific tool, perhaps one consuming rustdoc JSON output.

If we are going down this path I would expect a lot more work on what test documentation might look like, how it should be presented, etc, and probably an RFC.

My subsequent two concerns somewhat reflect my issue with this motivation; as something written with a rather domain specific motivation, it overindexes on a particular way of testing Rust code.

@rfcbot concern doctests

What about doctests? Different projects have different conventions here: some projects have doctests as their primary method of testing, some have unit/integration tests that cover everything important and just run doctests as a way of keeping the examples working.

@rfcbot concern integration-tests

What about integration tests? These are not handled here at all.

bors · 2025-04-15T22:53:37Z

☔ The latest upstream changes (presumably #139881) made this pull request unmergeable. Please resolve the merge conflicts.

rustbot assigned GuillaumeGomez Sep 17, 2024

rustbot added A-rustdoc-json Area: Rustdoc JSON backend S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. T-rustdoc Relevant to the rustdoc team, which will review and decide on the PR/issue. labels Sep 17, 2024

rustbot added has-merge-commits PR has merge commits, merge with caution. S-waiting-on-author Status: This is awaiting some action (such as code changes or more information) from the author. labels Sep 17, 2024

jieyouxu reviewed Sep 17, 2024

View reviewed changes

GuillaumeGomez reviewed Sep 17, 2024

View reviewed changes

src/librustdoc/formats/item_type.rs Show resolved Hide resolved

GuillaumeGomez reviewed Sep 17, 2024

View reviewed changes

src/librustdoc/html/render/mod.rs Show resolved Hide resolved

GuillaumeGomez reviewed Sep 17, 2024

View reviewed changes

This comment has been minimized.

# to view

ifxfrancois commented Sep 18, 2024

View reviewed changes

ifxfrancois force-pushed the test_documentation branch from 0bef9fe to f8de95c Compare October 7, 2024 14:49

This comment has been minimized.

# to view

ifxfrancois force-pushed the test_documentation branch from f8de95c to fa3e84f Compare October 8, 2024 16:47

rustbot added the A-run-make Area: port run-make Makefiles to rmake.rs label Oct 8, 2024

This comment was marked as off-topic.

# to view

This comment has been minimized.

# to view

lucagladiator force-pushed the test_documentation branch from b8fb053 to c94e982 Compare December 20, 2024 10:59

lucagladiator force-pushed the test_documentation branch 2 times, most recently from 9183070 to cd61af9 Compare January 2, 2025 10:58

lucagladiator force-pushed the test_documentation branch from cd61af9 to 965e60c Compare January 7, 2025 10:46

This comment has been minimized.

# to view

lucagladiator force-pushed the test_documentation branch from 965e60c to aea4a22 Compare January 7, 2025 11:06

lucagladiator force-pushed the test_documentation branch from aea4a22 to d4cf95c Compare January 27, 2025 10:44

lucagladiator force-pushed the test_documentation branch from d4cf95c to d12b7b5 Compare March 3, 2025 09:56

rustbot added the T-rustdoc-frontend Relevant to the rustdoc-frontend team, which will review and decide on the web UI/UX output. label Mar 3, 2025

lucagladiator force-pushed the test_documentation branch from d12b7b5 to 964cf00 Compare March 4, 2025 08:18

lucagladiator force-pushed the test_documentation branch from 964cf00 to da5e555 Compare March 5, 2025 09:04

lucagladiator force-pushed the test_documentation branch from da5e555 to 14f00fc Compare March 18, 2025 13:25

notriddle approved these changes Apr 12, 2025

View reviewed changes

lucagladiator force-pushed the test_documentation branch from 14f00fc to 4f4be34 Compare April 14, 2025 15:35

rfcbot added proposed-final-comment-period Proposed to merge/close by relevant subteam, see T-<team> label. Will enter FCP once signed off. disposition-merge This issue / PR is in PFCP or FCP with a disposition to merge it. labels Apr 14, 2025

	// Add the doc cfg into the doc build.
	cfgs.push("doc".to_string());

Generating a documentation for tests #130463

Are you sure you want to change the base?

Generating a documentation for tests #130463

Conversation

ifxfrancois commented Sep 17, 2024

Motivation

Limitations

rustbot commented Sep 17, 2024

rustbot commented Sep 17, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

GuillaumeGomez Sep 17, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

notriddle Apr 12, 2025 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ifxfrancois Sep 17, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

GuillaumeGomez commented Sep 17, 2024

ifxfrancois commented Sep 17, 2024

GuillaumeGomez commented Sep 17, 2024

rustbot commented Sep 18, 2024

Urgau commented Sep 18, 2024

This comment has been minimized.

GuillaumeGomez commented Sep 18, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jieyouxu Oct 8, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

bors commented Sep 23, 2024

rustbot commented Oct 7, 2024

This comment has been minimized.

This comment was marked as off-topic.

This comment has been minimized.

bors commented Oct 12, 2024

Dylan-DPC commented Nov 20, 2024 • edited Loading

bors commented Dec 20, 2024

bors commented Dec 21, 2024

bors commented Jan 4, 2025

This comment has been minimized.

bors commented Jan 24, 2025

bors commented Mar 2, 2025

bors commented Mar 3, 2025

bors commented Mar 5, 2025

bors commented Mar 18, 2025

Choose a reason for hiding this comment

notriddle Apr 12, 2025 • edited Loading

Choose a reason for hiding this comment

notriddle commented Apr 14, 2025

rfcbot commented Apr 14, 2025 • edited Loading

Manishearth commented Apr 14, 2025

bors commented Apr 15, 2025

GuillaumeGomez Sep 17, 2024 •

edited

Loading

notriddle Apr 12, 2025 •

edited

Loading

ifxfrancois Sep 17, 2024 •

edited

Loading

jieyouxu Oct 8, 2024 •

edited

Loading

Dylan-DPC commented Nov 20, 2024 •

edited

Loading

notriddle Apr 12, 2025 •

edited

Loading

rfcbot commented Apr 14, 2025 •

edited

Loading