Skip to content
New issue

Have a question about this project? # for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “#”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? # to your account

Provide either progress or ETA info when running a pipeline #300

Closed
pombredanne opened this issue Aug 25, 2021 · 2 comments
Closed

Provide either progress or ETA info when running a pipeline #300

pombredanne opened this issue Aug 25, 2021 · 2 comments
Assignees

Comments

@pombredanne
Copy link
Member

Some steps doing heavy lifting (such as file-level license or copyright detection) can take a good amount of time and it can be frustrating not to have much more feedback than a given step is started.
For instance the https://github.com/nexB/scancode.io/blob/e574fa9e815c763ac5025f540dfc05d61122d23e/scanpipe/pipelines/scan_codebase.py#L44 step will get us only a Step [run_scancode] starting...
It would be great to have a way to get some progress information. I reckon that the main project web page can be refreshed by hand (F5) and we can see some progress based on the counters and graphics being updated, but this is indirect.

Instead it would be great to get either or both of:

  1. some progress bar of sorts displayed on a project page or on the projects lists
  2. some indication of ETA left for the current step (and that part is super hard to get right but some indication would go a long way)
@tdruez tdruez self-assigned this Aug 26, 2021
@tdruez tdruez added this to the 2021-08 milestone Aug 26, 2021
@tdruez
Copy link
Contributor

tdruez commented Aug 26, 2021

We do have logging for the scancode direct API calls pipes: https://github.com/nexB/scancode.io/blob/main/scanpipe/pipes/scancode.py#L204

We are missing those when calling the toolkit cli through a subprocess.
Let's see if we can stream the progress output from the toolkit for those calls.

@tdruez
Copy link
Contributor

tdruez commented Aug 26, 2021

some indication of ETA left for the current step (and that part is super hard to get right but some indication would go a long way)

@pombredanne does the scancode cli provide any of those ETA indication?

tdruez added a commit that referenced this issue Aug 26, 2021
Signed-off-by: Thomas Druez <tdruez@nexb.com>
tdruez added a commit that referenced this issue Aug 26, 2021
Signed-off-by: Thomas Druez <tdruez@nexb.com>
tdruez added a commit that referenced this issue Aug 30, 2021
* Log the outputs of `run_scancode` as progress indication #300

Signed-off-by: Thomas Druez <tdruez@nexb.com>

* Add changelog entry #300

Signed-off-by: Thomas Druez <tdruez@nexb.com>
tdruez added a commit that referenced this issue Aug 30, 2021
* Log the outputs of `run_scancode` as progress indication #300

Signed-off-by: Thomas Druez <tdruez@nexb.com>

* Add changelog entry #300

Signed-off-by: Thomas Druez <tdruez@nexb.com>
@tdruez tdruez removed this from the Task queue improvements milestone Oct 8, 2021
tdruez added a commit that referenced this issue Jul 25, 2022
Signed-off-by: Thomas Druez <tdruez@nexb.com>
tdruez added a commit that referenced this issue Jul 25, 2022
Signed-off-by: Thomas Druez <tdruez@nexb.com>
tdruez added a commit that referenced this issue Nov 2, 2022
Signed-off-by: Thomas Druez <tdruez@nexb.com>
tdruez added a commit that referenced this issue Nov 2, 2022
Signed-off-by: Thomas Druez <tdruez@nexb.com>
tdruez added a commit that referenced this issue Nov 2, 2022
Signed-off-by: Thomas Druez <tdruez@nexb.com>
tdruez added a commit that referenced this issue Nov 2, 2022
Signed-off-by: Thomas Druez <tdruez@nexb.com>
tdruez added a commit that referenced this issue Nov 7, 2022
Signed-off-by: Thomas Druez <tdruez@nexb.com>
tdruez added a commit that referenced this issue Nov 7, 2022
Signed-off-by: Thomas Druez <tdruez@nexb.com>
tdruez added a commit that referenced this issue Nov 7, 2022
Signed-off-by: Thomas Druez <tdruez@nexb.com>
tdruez added a commit that referenced this issue Nov 7, 2022
@tdruez tdruez closed this as completed Jul 10, 2023
# for free to join this conversation on GitHub. Already have an account? # to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants