Improving the IDE ergonomics for individual dynamic test re-run

[k1w1m8]

In #3686, we saw that the ergonomics of re-running individual dynamic tests from IDE launch configurations are less than ideal.

In summary, IDEs represent the individual dynamic tests using their display names in the test results pane and re-run launch configuration title but must use unique id internally to select the test to re-run. Simple changes to the test set can break the mapping of unique id to display name, causing unexpected mass invalidation of existing re-run launch configurations. Attempts to use these invalidated launch configurations will result in a test running which does not match the title of the launch configuration / the display name of the test it was created from, resulting in confused and frustrated users.

To fix this, we need JUnit to provide a unique id for re-run which is stable over time, to match the IDE which stores the re-run launch configuration permanently.

Now let's consider two different classes of users of dynamic tests to see if anything can be done to improve their situation, given the limitations above.

The first class of user works with the individual dynamic tests anonymously. These users might pass in a fixed string for the display name for every test. When they run the tests, the IDE test results pane will show duplicate display names for each individual test that is run. Given the obvious lack of unique names, the current unique id creation seems that best that can be done here.

The second class of user works with the individual dynamic tests using proper names, which are unique and long lived. An example of this might be creating tests from the filesystem, where containers map to folders and tests map to files. Since folder and files have unique names according to the filesystem, the display names of tests created are all unique. The users know the folder and file of the test they want to run individually. However the re-run IDE launch configurations for individual tests do not use their carefully curated, stable and unique naming.

Given the IDE usability issues associated with the test results pane of the first class of users, I suspect most users will try and create unique names for their tests. This should lead to most usages of dynamic tests being of the second class.

Now, can we easily improve things, e.g. without API changes?

The first class of users will still require synthetic unique ids.

For the second class of users, we could improve the way that the unique id is created to a "best effort" combination of both the display name and synthetic numeric ids. For example:

• if the display name is unique, use it as it is.
• if the display name is not unique, add synthetic numeric ids to make it unique.

This would mean the first group would get the same experience as now, whereas the second group would have an improved experience where launch configurations reliably pick the correct test to run.

For example, consider dynamic tests created with this structure and display naming

green/
  bad.test
  good.test

The IDE test results pane will display the test according to the above structure and naming, and any launch configuration created via re-run will be titled consistently.

However, the IDE launch configurations will have synthetic unique ids, which are inconsistent with the Run Test pane and launch configuration title, and unstable over time.

.../[dynamic-container:#1]/[dynamic-test:#1]
.../[dynamic-container:#1]/[dynamic-test:#2]

With the suggestion above, the IDE launch configurations will have UniqueIds which (verbosely) match the Run Test pane and launch configuration title, and are stable over time.

.../[dynamic-container:green]/[dynamic-test:bad.test]
.../[dynamic-container:green]/[dynamic-test:good.test]

Duplicate handling neccessarily introduces some synthetic ids, for example

green/
  clash.test
  clash.test

gives the following, which is an improvement on the purely synthetic ids

.../[dynamic-container:green]/[dynamic-test:clash.test#1]
.../[dynamic-container:green]/[dynamic-test:clash.test#2]

This approach would also automatically handle any conflicts introduced by name santisation, if required.

[k1w1m8]

I'd consider doing a PR for this.

[sormuras]

Would also consider a programmatic and stable solution? Something like:

@Test
void palindrome12321() {
    palindromes("12321");
}

@ParameterizedTest @ValueSource(strings = { "racecar", "radar", "able was I ere I saw elba" })
void palindromes(String candidate) {
    assertTrue(StringUtils.isPalindrome(candidate));
}

[k1w1m8]

@sormuras, I don't understand your question. Please can you explain further?