Skip to content
New issue

Have a question about this project? # for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “#”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? # to your account

Support test execution order sorted by descending duration #6075

Open
oojacoboo opened this issue Dec 9, 2024 · 7 comments
Open

Support test execution order sorted by descending duration #6075

oojacoboo opened this issue Dec 9, 2024 · 7 comments

Comments

@oojacoboo
Copy link

oojacoboo commented Dec 9, 2024

We use PHPUnit with Paratest, allowing for parallel testrunners. This speeds up our tests on multi-core machines by a lot. One of the issues though, in terms of total test run time, is that some of our longer running tests don't get picked up until near the end. If these tests were executed first, it'd actually speed up the overall run time, since there would be other runners executing in parallel. As it is now, we might have 2 parallel runners executing long running tests at the very end, extending our total runtime by at least 30 seconds or more.

This is a request to add duration,reverse to the list of available order-by options.

@oojacoboo oojacoboo added the type/enhancement A new idea that should be implemented label Dec 9, 2024
@sebastianbergmann sebastianbergmann changed the title --order-by (executionOrder) duration,reverse Support test execution order sorted by descending duration Dec 9, 2024
@sebastianbergmann
Copy link
Owner

My initial thought, and I cannot really explain why, is to not make reverse combinable with duration but rather introduce duration-ascending as an alias for duration and duration-descending for your use case.

I am curious whether @Slamdunk thinks this should be tackled in PHPUnit or in Paratest. After all, the case could be made that not only partitioning the tests but also sorting them for parallel execution should be the responsibility of the parallel test runner.

@sebastianbergmann sebastianbergmann added the feature/test-runner CLI test runner label Dec 9, 2024
@oojacoboo
Copy link
Author

oojacoboo commented Dec 9, 2024

@sebastianbergmann that's a good point. Overall, paratest would be a more accurate place to handle this, since it can accurately distribute amongst the runners. Here is a comment on the subject: paratestphp/paratest#846 (comment). It may be significantly more challenging though, since it's not currently making use of the result cache.

@Slamdunk
Copy link
Contributor

Slamdunk commented Dec 9, 2024

I am curious whether @Slamdunk thinks this should be tackled in PHPUnit or in Paratest. After all, the case could be made that not only partitioning the tests but also sorting them for parallel execution should be the responsibility of the parallel test runner.

@sebastianbergmann I think the opposite: sorting the tests has nothing to do with partitioning, as it has never been.
Please implement the feature in PHPUnit and I'll ensure it works the same with ParaTest.

We are currently hindered to use the cache, but I'll open a different issue for that.

@sebastianbergmann
Copy link
Owner

Please implement the feature in PHPUnit and I'll ensure it works the same with ParaTest.

Okay. PR welcome (for main), no promise when I might get to this myself.

@Slamdunk
Copy link
Contributor

@oojacoboo just to be clear, I'll work (someday) on #6078 but not on this very issue, which I consider up to you or anyone willing to step forward.

@Slamdunk
Copy link
Contributor

Full support of result cache operations, and thus all the current and future --order-by features, has been released in ParaTest v7.7.0

@staabm
Copy link
Contributor

staabm commented Dec 15, 2024

I think we see the OPs problem also in phpstan-src, where slow tests run pretty late and therefore we loose some time unnecessarily.

And I think we see it although we are using random ordering

We are using brianium/paratest ^6.5

# for free to join this conversation on GitHub. Already have an account? # to comment
Projects
None yet
Development

No branches or pull requests

4 participants