Command line format is confusing: it implies all tests work on all OSes

[stuartlangridge]

Expected behavior

Since a quicktest command looks like ./quicktest test_boot_to_login alpine v3.19, this seems to imply that tests and what they're run on are separate from one another. That is: it looks like you specify a test, and an OS, and a version, and quicktest runs that test on that OS. So if I like the test_boot_to_login test, I might decide to run it on Fedora 18, or Ubuntu 12.10, or Elementary, with a command line like ./quicktest test_boot_to_login ubuntu v12.04.

Actual behavior

This is not the case. In fact, any given test only exists for certain OS and version combinations. ./quicktest test_boot_to_login ubuntu v12.04 doesn't work because there is no test_boot_to_login supplied for Ubuntu 12.04. Quicktest knows this; it already knows which tests work on which OSes. But the command line syntax hides this, and implies that all tests run on all OSes even though they do not. ./quicktest test_boot_to_login ubuntu v12.04 will fail, not because that test doesn't work on that OS but because that test doesn't exist on that OS.

The test name is being treated almost like a subcommand, like git add or systemctl start. But the way subcommands work generally is that they're defined on everything: git add somefile.txt would be seen as strange if it only worked on text files and threw an error on others because it wasn't "defined" for PNGs.

Obviously, this doesn't actually affect the program itself. But I think the command line syntax could be more helpful here.

The main issue that the CLI syntax has is that it implies that a test has some separate independent existence from the OSes it runs on, and this isn't the case. So it might be clearer in people's minds if tests were presented as subordinate to the OSes they run on. For example, an syntax might look like this: ./quicktest ubuntu/24.04/test_boot_to_login. This makes it clear that there are "ubuntu" tests, within those there are "ubuntu 24.04" tests, and within that is the "test_boot_to_login" test. This mental model looks like a filesystem -- even though there isn't actually a filesystem visible with the tests in, people understand the idea of this hierarchy. The existence of the ./quicktest ubuntu/24.04/test_boot_to_login command does not necessarily imply the existence of a ./quicktest fedora/daily-live/test_boot_to_login command.

One of the downsides of this "filesystem" presentation is that it suggests that the "test_boot_to_login" test on ubuntu/24.04 and the "test_boot_to_login" test on ubuntu-mate/24.04 are unrelated. This might also be seen as confusing. But this is an inevitable consequence of having nearly-but-not-identical tests across some-but-not-all OSes.

[popey]

Thanks for the thoughtful issue @stuartlangridge - I appreciate the time you took to articulate this.

I'm inclined to agree. So the directory structure will likely stay the same, but just the invocation would change. I think also that the output of quicktest -ls would need to change, to just show os/release/testcase which could easily be grepped or copy/pasted.

I'd like (for example) to be able to for t in $(quicktest -ls); do quicktest "$t"; done - kind of thing, to just buzz through all tests.

[ali1234]

I think most tests should be able to run on any OS. In the most extreme example, the "gather screenshots" test should be available anywhere, without having to duplicate it. Further down the line I can imagine tests being built from standardized pieces that follow a specific interface. Then you can write a test in a generic way like:

• Boot to desktop
• Install package
• Run command
• Find and click button labelled

Then quicktest will see "Boot to desktop" and know that it needs to run "boot_to_desktop_$os_$version" and just do that automatically depending on what you asked for on the CLI.

I think this is not only desirable but also required for maintainability of the tests. Without it there will be a huge amount of duplicated code. For example across all the Ubuntu flavours that use the same installer.