custom compile progress via env var template #6070

matthiaskrgr · 2018-09-21T16:31:45Z

Allows to customize the compile progress via env var.
This was inspired by NINJA_STATUS environmental variable:
https://ninja-build.org/manual.html#_environment_variables

I had to add some workarounds to make sure the CARGO_STATUS var does not mess with the git fetch/crate download progress indicators.
Right now this should only change the output of "cargo build/check"

Docs from the readme:

CARGO_STATUS — If this is set, cargo will read it as compile progress template.
The default format is [%b] %f/%t: %n which will look like this:
Building [=====> ] 115/145: libssh2-sys(build), thread_l...
The following parameters are supported:
* %b progress bar =======>
* %s number of started jobs
* %f number of finished jobs
* %t number of total jobs of the build
* %u number or remaining jobs to start
* %r number of active (currently running) jobs
* %p progress percentage with decimals
* %P progress percentage without decimals
* %% plain % character (can be used for %P%% => 14%)
* %n (truncated) list of names of running jobs
* %e elapsed time in seconds
* %E elapsed time in seconds (human readable)
* %o overall time per job
* %O overall time per job (human readable)
* %c jobs per second

rust-highfive · 2018-09-21T16:31:56Z

r? @alexcrichton

(rust_highfive has picked a reviewer for you, use r? to override)

matthiaskrgr · 2018-09-21T16:32:08Z

src/doc/src/reference/environment-variables.md

@@ -34,6 +34,20 @@ system:
  will otherwise be used.
 * `CARGO_CACHE_RUSTC_INFO` — If this is set to 0 then Cargo will not try to cache
  compiler version information.
+* `CARGO_STATUS` — If this is set, cargo will read it as compile progress template.
+  The default format is `[%b] %f/%t: %n` which will looke like this:


typo, will fix later

matthiaskrgr · 2018-09-21T16:33:29Z

src/cargo/util/progress.rs

+        }
+        let length_of_formatters = template.len() - template_skelleton.len();
+
+        // Render the percentage at the far right and then figure how long the progress bar iscustom.compile_progress


will also fix later

alexcrichton · 2018-09-24T20:15:55Z

Thanks for the PR! In general though Cargo's output isn't customizable to this degree and we try to avoid mini-DSLs wherever possible in Cargo. I would personally expect a feature like this to likely require an RFC rather than being accepted through a PR. Others on @rust-lang/cargo may feel different though.

joshtriplett · 2018-09-24T21:43:30Z

@alexcrichton I agree. If there's information not being provided via the build log, we should provide it. And if there's a different format we need when stdout isn't a TTY, we should provide that too. But let's not make it customizable.

matthiaskrgr · 2018-09-26T08:56:46Z

mini-DSLs

what does DSL stand for?

When using ninja, I find the number of currently running jobs and the current build time very useful to know, but I didn't want to change the default progress of cargo which is why I made it customisable.

dwijnand · 2018-09-26T09:14:27Z

what does DSL stand for?

Domain-specific language. (https://en.wikipedia.org/wiki/Domain-specific_language)

alexcrichton · 2018-09-26T15:31:04Z

I think it's fine to tweak and update Cargo's default output, and it's likely beneficial to everyone to have a better default output rather than allowing one-off customization.

joshtriplett · 2018-09-26T22:43:02Z

We discussed this in the @rust-lang/cargo meeting, and just as discussed here in this issue, this degree of output customization is not something we're looking to support. We'd prefer to make the output as usable as possible for various use cases, and if people have extremely specialized needs, they may need the full flexibility of build plans and custom build processes.

Given that:

@rfcbot fcp close

joshtriplett · 2018-09-26T22:43:16Z

(oops, accidentally hit "close and comment")

joshtriplett · 2018-09-26T22:43:42Z

@rfcbot fcp close

rfcbot · 2018-09-26T22:58:44Z

Team member @joshtriplett has proposed to close this. The next step is review by the rest of the tagged teams:

No concerns currently listed.

Once a majority of reviewers approve (and none object), this will enter its final comment period. If you spot a major issue that hasn't been raised at any point in this process, please speak up!

See this document for info about what commands tagged team members can give me.

matthiaskrgr · 2018-09-26T23:30:30Z

Well ok.

In the latest commit I added a bunch of time-related formatters.

So far the NINJA_STATUS template that has served me well is CARGO_STATUS="[%P%%/%f/%r/%t/%e]".
This would look like this: Building [ 12%/50/4/402/22] ([percentage/done/active/total/seconds].
The current cargo default is [%b] %f/%t: %n ([progress bar] done/total: job_names)

If we are going to display additional information (without customizale formatting), I would vote for something like
"%P%% [%b] %f/%r/%t (%E): %n" as new default:
Building 29% [==========> ] 17/4/58 (14.19s): termcolor, ansi_term, natord, glob
(github is truncating away some whitespaces unfortunately..)
It shows the progress percentage, bar, and afterwards the number of finished jobs, number of currently running/active jobs (this is easier to see than counting the job names (which might get truncated anyway)) and how long the build has been running so far in human readable format, completed by the names of current jobs.

I would like to know what others would want to see or if they are happy with the current compile progress layout. :)

alexcrichton · 2018-09-27T05:28:39Z

That looks reasonable to me, but the 17/4/58 I feel is a bit confusing in that it's sort of just three numbers where it's not clear what to do with it. The 29% I feel may also be redundant with the number of crates to build and total crates?

alexcrichton · 2018-10-12T17:30:01Z

@matthiaskrgr have you had a chance to game this out and see what it would look like to change Cargo's default progress bar?

matthiaskrgr · 2018-10-14T11:57:09Z

I'll hopefully get to this early in the upcomming week.

matthiaskrgr · 2018-10-17T11:34:50Z

That looks reasonable to me, but the 17/4/58 I feel is a bit confusing in that it's sort of just three numbers where it's not clear what to do with it. The 29% I feel may also be redundant with the number of crates to build and total crates?

So you would opt for just the current progress bar plus the build time?
( CARGO_STATUS="[%b] %f/%t (%E): %n")

have you had a chance to game this out and see what it would look like to change Cargo's default progress bar?

Hm, I'm not sure if I understand "what it would look like"...
On the branch we can try everything out and once we decided for a new(?) format, I can remove all the unnecessary (:() code that made it customizable and hardcode/update to the new format.

matthiaskrgr · 2018-10-31T14:38:50Z

bump?

alexcrichton · 2018-10-31T19:10:39Z

Er sorry for the delay, but I can't really visualize what this would look like given the syntax above, can a gif be provided?

I don't think we're going to land this as-is. We can land, however, a change to the defaults

matthiaskrgr · 2018-11-01T14:05:52Z

Ok:
default format [%b] %f/%t: %n
https://asciinema.org/a/eU1MJqRZTWVbPG3ZhAN4p7qFX

what I suggested earlier but is probably overkill %P%% [%b] %f/%r/%t (%E): %n
https://asciinema.org/a/SSbDV7HVfVDYk5GZ7S1lh7PPu

default format + build time "[%b] %f/%t (%E): %n"
https://asciinema.org/a/6HkyFGkJigy8ViAML3bIJVyAo

… progress_status()

…ss formatting You can pass CARGO_STATUS env var to cargo to change the formatting. %b is the progress bar %s is the number of done jobs %t is the number of total jobs %p is progress percentage %n is a list of names of running jobs The default formatting is "[%b] %s/%t%n" You can pass alternatives like this: CARGO_STATUS="%s of %t, running: %n, [%b]" cargo build

…ng with download progress. When downloading crates, use preset, non-costumizable compile-progress formats.

…well.

…esses (%r)

…inting '%' char itself

…or job names

change %s (number of started jobs) to %f (number of finished jobs) since that's what it actually is add new %s which represents the actual number of started jobs add %u representing the number of remaining jobs to start

In a pattern like %%s, we would previously evaluate %% first and then have a bare s char %%s => %s Evaluate %% last so that %%s => %2

%e elapsed time in seconds %E elapsed time (human readable) %o overall time per job %O overall time per job (human readable) %c jobs done per second

alexcrichton · 2018-11-01T18:17:13Z

Ok thanks for the gifs! I agree that the second one is probably too much information, but for the last one I'm not personally a huge fan of the timer only being updated when the output is otherwise updated, it looks sort of bad during long compilations where no activity is happening and the timer just sits there

bors · 2018-11-06T02:34:43Z

☔ The latest upstream changes (presumably #6130) made this pull request unmergeable. Please resolve the merge conflicts.

rfcbot · 2018-11-09T02:55:20Z

🔔 This is now entering its final comment period, as per the review above. 🔔

psst @joshtriplett, I wasn't able to add the final-comment-period label, please do so.

rfcbot · 2018-11-19T02:59:28Z

The final comment period, with a disposition to close, as per the review above, is now complete.

psst @joshtriplett, I wasn't able to add the finished-final-comment-period label, please do so.

rust-highfive assigned alexcrichton Sep 21, 2018

matthiaskrgr commented Sep 21, 2018

View reviewed changes

matthiaskrgr changed the title ~~[WIP] custom compile progress via env var template~~ custom compile progress via env var template Sep 21, 2018

matthiaskrgr force-pushed the custom_compile_progress_pr branch from 54b953e to 938f92d Compare September 22, 2018 15:25

joshtriplett closed this Sep 26, 2018

joshtriplett reopened this Sep 26, 2018

joshtriplett added the T-cargo Team: Cargo label Sep 26, 2018

matthiaskrgr added 5 commits November 1, 2018 15:22

compile-progress: use string.repeat() instead of loop

6e4dbcd

compile-progress: pass down number of active jobs all the way down to…

7e11daa

… progress_status()

compile-progress: work around customizable compile-progress interferi…

a515649

…ng with download progress. When downloading crates, use preset, non-costumizable compile-progress formats.

compile-progress: make the ": " before the job names costumizable as …

c99a1ab

…well.

matthiaskrgr added 10 commits November 1, 2018 15:22

compile-progress: add support to print number of current running proc…

529c954

…esses (%r)

compile progress: support printing percentage without decimals and pr…

4d6b905

…inting '%' char itself

compile-progress: fater code paths for when we can skip progress bar …

a13381c

…or job names

compile-progress: change %s to %f, add %s and %u

54201cd

change %s (number of started jobs) to %f (number of finished jobs) since that's what it actually is add new %s which represents the actual number of started jobs add %u representing the number of remaining jobs to start

compile-progress: refine todo comment

1db2bcb

docs: mention CARGO_STATUS in enviromental-variables.md documentation.

d9db820

NFC: fix typos and rustfmt src/cargo/util/progress.rs

70618cf

compile-progress: add tests

a526f96

compile-progress: make sure that %% is evaluated last

495112b

In a pattern like %%s, we would previously evaluate %% first and then have a bare s char %%s => %s Evaluate %% last so that %%s => %2

compile progress: add time-realted options: %e %E %o %O %c

b973718

%e elapsed time in seconds %E elapsed time (human readable) %o overall time per job %O overall time per job (human readable) %c jobs done per second

matthiaskrgr force-pushed the custom_compile_progress_pr branch from 2b7a7d3 to b973718 Compare November 1, 2018 14:23

dwijnand added the final-comment-period FCP — a period for last comments before action is taken label Nov 9, 2018

joshtriplett removed the final-comment-period FCP — a period for last comments before action is taken label Nov 19, 2018

joshtriplett closed this Nov 19, 2018

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

custom compile progress via env var template #6070

custom compile progress via env var template #6070

matthiaskrgr commented Sep 21, 2018 •

edited

Loading

rust-highfive commented Sep 21, 2018

matthiaskrgr Sep 21, 2018

matthiaskrgr Sep 21, 2018

alexcrichton commented Sep 24, 2018

joshtriplett commented Sep 24, 2018

matthiaskrgr commented Sep 26, 2018 •

edited

Loading

dwijnand commented Sep 26, 2018

alexcrichton commented Sep 26, 2018

joshtriplett commented Sep 26, 2018

joshtriplett commented Sep 26, 2018

joshtriplett commented Sep 26, 2018

rfcbot commented Sep 26, 2018 •

edited by aturon

Loading

matthiaskrgr commented Sep 26, 2018 •

edited

Loading

alexcrichton commented Sep 27, 2018

alexcrichton commented Oct 12, 2018

matthiaskrgr commented Oct 14, 2018

matthiaskrgr commented Oct 17, 2018

matthiaskrgr commented Oct 31, 2018

alexcrichton commented Oct 31, 2018

matthiaskrgr commented Nov 1, 2018

alexcrichton commented Nov 1, 2018

bors commented Nov 6, 2018

rfcbot commented Nov 9, 2018

rfcbot commented Nov 19, 2018

custom compile progress via env var template #6070

custom compile progress via env var template #6070

Conversation

matthiaskrgr commented Sep 21, 2018 • edited Loading

rust-highfive commented Sep 21, 2018

matthiaskrgr Sep 21, 2018

Choose a reason for hiding this comment

matthiaskrgr Sep 21, 2018

Choose a reason for hiding this comment

alexcrichton commented Sep 24, 2018

joshtriplett commented Sep 24, 2018

matthiaskrgr commented Sep 26, 2018 • edited Loading

dwijnand commented Sep 26, 2018

alexcrichton commented Sep 26, 2018

joshtriplett commented Sep 26, 2018

joshtriplett commented Sep 26, 2018

joshtriplett commented Sep 26, 2018

rfcbot commented Sep 26, 2018 • edited by aturon Loading

matthiaskrgr commented Sep 26, 2018 • edited Loading

alexcrichton commented Sep 27, 2018

alexcrichton commented Oct 12, 2018

matthiaskrgr commented Oct 14, 2018

matthiaskrgr commented Oct 17, 2018

matthiaskrgr commented Oct 31, 2018

alexcrichton commented Oct 31, 2018

matthiaskrgr commented Nov 1, 2018

alexcrichton commented Nov 1, 2018

bors commented Nov 6, 2018

rfcbot commented Nov 9, 2018

rfcbot commented Nov 19, 2018

matthiaskrgr commented Sep 21, 2018 •

edited

Loading

matthiaskrgr commented Sep 26, 2018 •

edited

Loading

rfcbot commented Sep 26, 2018 •

edited by aturon

Loading

matthiaskrgr commented Sep 26, 2018 •

edited

Loading