Sequence of Execution

Tests in JarSpec are created dynamically (via a call to it) rather than statically (by declaring a test method). This results in its sequence of execution being a little different to most other unit test frameworks for statically-typed languages like Java.

For each Specification, JarSpec's runner carries out the following steps:

• Instantiate the spec class
• Call the root method on the spec class
  • This will recursively call all of the describe blocks, in order to build the whole spec
• Execute each test from the spec in turn

Comparison with JUnit

JarSpec runs under JUnit, so follows the same model insofar as each class is executed independently. However, within a single class (a Specification in the case of JarSpec), the execution order is quite different.

For the sake of comparison, consider the following JUnit class:

public class JUnitTest {
    public JUnitTest() { System.out.println("  constructor"); }

    @BeforeClass public static void beforeClass() { System.out.println("beforeClass"); }
    @Before public void before() { System.out.println("  before"); }
    @Test public void test1() { System.out.println("    test1 execution"); }
    @Test public void test2() { System.out.println("    test2 execution"); }
    @After public void after() { System.out.println("  after"); }
    @AfterClass public static void afterClass() { System.out.println("afterClass"); }
}

The above class produces the following output:

beforeClass
  constructor
  before
    test1 execution
  after
  constructor
  before
    test2 execution
  after
afterClass

Now consider the following JarSpec Specification:

@RunWith(JarSpecJUnitRunner.class)
public class JarSpecSpec implements Specification {
    public JarSpecSpec() { System.out.println("constructor"); }

    @Override
    public SpecificationNode root() {
        System.out.println("root initialisation");
        return describe("major unit", () -> {
            System.out.println("Major unit initialisation");
            return byAllOf(
                it("has a top-level statement", () ->
                    System.out.println("  Top-level statement execution")),
                describe("nested minor unit", () -> {
                    System.out.println("  Minor unit initialisation");
                    return byAllOf(
                        it("has a nested statement", () ->
                            System.out.println("    First nested statement execution")),
                        it("has another nested statement", () ->
                            System.out.println("    Second nested statement execution"))
                    );
                })
            );
        });
    }
}

The above class produces the following output:

constructor
root initialisation
Major unit initialisation
  Minor unit initialisation
  Top-level statement execution
    First nested statement execution
    Second nested statement execution

The constructor (and root method) are only invoked once, then all the describe blocks are invoked, and finally all of the statement tests.

Implications

Setup and teardown (before/after methods)

JarSpec does not have a direct equivalent of JUnit's Before/After annotations (or RSpec's before/after hooks or Jasmine and Mocha's before/after functions). However, it does have full support for JUnit Rules, which are the recommended approach within JUnit itself, as per the JUnit Java documentation:

TestRules can do everything that could be done previously with methods annotated with Before, After, BeforeClass, or AfterClass, but they are more powerful, and more easily shared between projects and classes.

For more information on JUnit Rules, see the Rules page in the JUnit wiki. For examples of using JUnit Rules with JarSpec, see the SharedStateSpec and ExpensiveDependencySpec fixtures, and the ExceptionBehaviour mixin.

The withRule method applies a rule to all tests within a unit. The withBlockRule method applies a rule to the unit itself (and not individually to its descendants). The execution order of rules is as follows: Rules on outer blocks are always applied around rules on inner blocks. For a given block, rules are specified outermost-first, and block-level rules are always applied around other rules. Consider the following specification and output:

@RunWith(JarSpecJUnitRunner.class)
public class RulesSpec implements Specification {
    @Override
    public SpecificationNode root() {
        return describe("major unit",
            it("has a top-level statement", () ->
                System.out.println("  Top-level statement execution")),
            describe("nested minor unit",
                it("has a nested statement", () ->
                    System.out.println("    First nested statement execution")
                ).withRule(log("    Individual test rule")),
                it("has another nested statement",
                    () -> System.out.println("    Second nested statement execution"))
            ).withRule(log("  First nested rule"))
            .withRule(log("  Second nested rule"))
            .withBlockRule(log("  Nested block rule"))
        ).withRule(log("First top-level rule"))
        .withRule(log("Second top-level rule"))
        .withBlockRule(log("Top-level block rule"));
    }

    private static TestRule log(String name) {
        return new LoggingRule(name);
    }
}

Top-level block rule before()
First top-level rule before()
Second top-level rule before()
  Top-level statement execution
Second top-level rule after()
First top-level rule after()
  Nested block rule before()
First top-level rule before()
Second top-level rule before()
  First nested rule before()
  Second nested rule before()
    Individual test rule before()
    First nested statement execution
    Individual test rule after()
  Second nested rule after()
  First nested rule after()
Second top-level rule after()
First top-level rule after()
First top-level rule before()
Second top-level rule before()
  First nested rule before()
  Second nested rule before()
    Second nested statement execution
  Second nested rule after()
  First nested rule after()
Second top-level rule after()
First top-level rule after()
  Nested block rule after()
Top-level block rule after()

Error behaviour

Errors generally only affect their immediate scope, including its children: If the test for an individual statement throws an exception, then that statement is reported as a failing test. If an exception is thrown within a describe block but outside of a statement test, then a single failure is reported for the whole unit, and the tests within that unit will not run, but other units in the same specification will run as normal. If the class constructor or root method throws an exception, then the entire class is marked as a failure (as with JUnit).

The only point to be aware of is that, since JarSpec tests are created dynamically, there will be no failure recorded for tests that were never reached due to errors in their containing unit.

Test isolation

Since the class is not recreated for each test, then member fields are shared between all tests in the class. Furthermore, any local variables declared in the root() implementation or in any describe blocks are available to all tests that they contain. The compiler will ensure that such variables cannot be redefined by tests, since variables captured by lambda scopes must be treated as final. However, if these variables contain objects that are themselves mutable, then it's possible for tests to interfere with one another.

To avoid any unexpected side-effects between tests, it is best to design objects to be immutable, or at least severely limit the scope of mutable objects. Fortunately, these are good rules for software development in general, not just for working with JarSpec!

If you really need to use shared state, perhaps because you have some stub object that's expensive to recreate for each test (e.g. an in-process webserver or database), you can still isolate your tests using JUnit TestRules (see above).