IDE Launch Configurations for individual Dynamic Tests will often run incorrect tests

[k1w1m8]

When using Intellij IDEA, if you re-run a JUnit Test from the Test results window, it creates a launch configuration for you, so that you can run that test again later without having to run the entire set.

This works well for normal tests, you can go back to a run configuration that you created few days back and run that test.

However, for Dynamic Tests, this doesn't work well. It will often run an incorrect test. This breaks the principle of least astonishment.

Why is this? Well, turns out that IDEA stores the identifier of the test to run into the Run/Debug Configuration as a JUnit5 UniqueId.

For example for a Dynamic Test with the following path "a container/another container/a test", it might create (broken down one segment per line)

[engine:junit-jupiter]/
[class:pkg.JUnit5DynamicTest]/
[test-factory:createDynamicTests()]/
[dynamic-container:#7]/
[dynamic-container:#14]/
[dynamic-test:#3]

When reading the UniqueId inside the launch configuration, its not clear which dynamic test instance that it is trying to target, as the "display name" of the containers and tests are not visible in the UniqueId.

According to @sbrannen in #3682 (comment),

• JUnit generates the IDs based on an incrementing counter which denotes the position within the stream
• the unique IDs are not meant to be human readable

Problems here:

• IDEA is exposing a non human readable value to its users via the UI
• The numeric IDs are not stable over time, and therefore launch configurations relying on them are easily broken.

Ways to break a launch configuration:

• select a different subset of tests to run
• add or remove a test before the test
• add or remove a container before any of the parent containers

This instability is not good, and makes Dynamic Tests much less convenient to use than normal tests, especially if you often run different subsets.

We would like the ability to re-run DynamicTests using run configurations which are consistently stable over time, just as they are for normal tests.

It is not clear to me if the issue here lies with IDEA or JUnit or both.

So, I've started with this ticket, but we can move to IDEA once it is clear what would need to be fixed there.

See also

• Use names in UniqueIds of dynamic tests so that IDEs and humans can find dynamic tests easily #3682

[marcphilipp]

Thanks for your detailed problem description!

Team decision: Unfortunately, it's in the nature of the problem space that dynamic tests don't have a unique identifier that can be referenced safely across code changes or subsequent invocations of the same dynamic test (in case the Stream the dynamic tests are created with generates different invocations). Rerunning dynamic tests by unique ID is only intended to be used within the same development session.

[k1w1m8]

Thanks for the reply @marcphilipp.

Disappointed this was closed without any discussion.

A few points here.

• Changes to the test set (add/delete/rename) can easily happen within the same development session. Therefore, the existing approach is not stable within the same development session.
• Unfortunately, IDEs don't store launch configurations only for use within the same development session. They store them permanently. Therefore, working well with IDEs requires unique ids in launch configurations that are stable over time.
• Dynamic tests do contain a display name which likely to be unique in a lot of cases, if only because the IDE experience of using duplicate display names for tests is very poor.

Now that we've established the IDE ergonomic issues with dynamic tests, I will raise another ticket to see if we can improve things given the existing design, #3775.