Upgrading tests

Upgrading to the new test interface

Why upgrade?

If you wrote a package with test suites before Cabal made it easy, you're probably now seeing a message like this:

Warning: In the use of `runTests' (imported from Distribution.Simple, but defined in Distribution.Simple.UserHooks):
    Deprecated: "Please use the new testing interface instead!"

This guide will help you upgrade your test suites to use the new interface. The improved test interface eliminates the need for common boilerplate code and assures users of software quality by allowing Cabal to automatically run test suites and correctly interpret their results. Eventually, the test interface will be integrated with Hackage to make it easier for users to recognize high-quality libraries.

This guide will show the most common patterns from the old interface and demonstrate their equivalents in the new interface. Inevitably, some patterns will be missed; for that reason, contributions to this wiki are encouraged.

What Cabal expects

Cabal expects that each of your test suites compiles to a single executable that:

1. Runs one or more test cases
2. (Optionally) prints human-readable information about their results to the terminal
3. Returns a non-zero exit code if the tests failed or zero otherwise.

Your test suites likely already fit this model. However, if you are using an old version of QuickCheck or HUnit, your executable may not be returning the correct error code. Other test libraries, such as test-framework, handle this correctly; check the documentation if you're unsure.

What Cabal does for you

So far, it sounds like business as usual. Why upgrade the test interface? Here's what Cabal will do for you:

1. Conditional compilation: no more gymnastics with flags to enable and disable your test suites.
2. Automatic testing: no need for user hooks or custom setup files.
3. Automatic logging: Cabal can summarize test results, so there's no need to clutter up your terminal with output from successful tests.

As you can see, Cabal can manage the overhead of maintaining test suites for you.

How to upgrade

Remove user hooks

The old interface required you to implement user hooks telling Cabal how to run your test suites. In turn, this required you to use a custom setup script. All the information Cabal needs about your test suites is now specified in the package description (.cabal) file, so you should remove the runTests hook from your custom Setup.hs file.

If this was the only user hook you implemented, you can actually go back to using the default Setup.hs:

import Distribution.Simple
main = defaultMain

Then, you can use the Simple build type instead of Custom. In your .cabal file, the field

build-type: Custom

should become

build-type: Simple

Don't do this if you are still using a custom Setup.hs because you have other user hooks!

Convert executable sections

Your test suite is likely included in your .cabal file as an executable section such as:

executable test-foo
    main-is: test-foo.hs
    build-depends: ...
    ...

Your executable may vary; in particular, it might have some conditional blocks and a flag called something like tests. That situation will be addressed shortly, but first, you must tell Cabal that this executable is actually a test. The section above would become:

test-suite test-foo
    type: exitcode-stdio-1.0
    main-is: test-foo.hs
    build-depends: ...
    ...

The only thing that changes is the name of the section and the addition of the field type. Any build information field that is permitted in an executable section is permitted here.

Remove conditionals

If you were using a flag, such as tests, to control compilation of your test suites, you should remove the flag and the conditional blocks it controlled. The test-suite sections should be written as if tests are always enabled; Cabal will enable or disable them as appropriate.

Enjoy!

That's it! Building and testing your package requires only a minor change to your workflow:

$ cabal configure --enable-tests
$ cabal build
$ cabal test

By default, Cabal configures packages with tests disabled; obviously, cabal configure --disable-tests does the same.