Sequence of Execution

Tests in JarSpec are created dynamically (via a call to it) rather than statically (by declaring a test method). This results in its sequence of execution being a little different to most other unit test frameworks for statically-typed languages like Java.

For each Specification, JarSpec's runner carries out the following steps:

• Instantiate the spec class
• Call the root method on the spec class
  • This will recursively call all of the describe blocks, in order to build the whole spec
• Execute each test from the spec in turn

Comparison with JUnit

JarSpec runs under JUnit, so follows the same model insofar as each class is executed independently. However, within a single class (a Specification in the case of JarSpec), the execution order is quite different.

For the sake of comparison, consider the following JUnit class:

public class JUnitTest {
    public JUnitTest() { System.out.println("  constructor"); }

    @BeforeClass public static void beforeClass() { System.out.println("beforeClass"); }
    @Before public void before() { System.out.println("  before"); }
    @Test public void test1() { System.out.println("    test1 execution"); }
    @Test public void test2() { System.out.println("    test2 execution"); }
    @After public void after() { System.out.println("  after"); }
    @AfterClass public static void afterClass() { System.out.println("afterClass"); }
}

The above class produces the following output:

beforeClass
  constructor
  before
    test1 execution
  after
  constructor
  before
    test2 execution
  after
afterClass

Now consider the following JarSpec Specification:

@RunWith(JarSpecJUnitRunner.class)
public class JarSpecSpec implements Specification {
    public JarSpecSpec() { System.out.println("constructor"); }

    @Override
    public SpecificationNode root() {
        System.out.println("root initialisation");
        return describe("major unit", () -> {
            System.out.println("Major unit initialisation");
            return byAllOf(
                    it("has a top-level statement", () ->
                            System.out.println("  Top-level statement execution")),
                    describe("nested minor unit", () -> {
                        System.out.println("  Minor unit initialisation");
                        return byAllOf(
                                it("has a nested statement", () ->
                                        System.out.println("    First nested statement execution")),
                                it("has another nested statement", () ->
                                        System.out.println("    Second nested statement execution"))
                        );
                    }).withReset(() -> System.out.println("  nested reset"))
            );
        }).withReset(() -> System.out.println("top-level reset"));
    }
}

The above class produces the following output:

constructor
root initialisation
Major unit initialisation
  Minor unit initialisation
top-level reset
  Top-level statement execution
top-level reset
  nested reset
    First nested statement execution
top-level reset
  nested reset
    Second nested statement execution

The constructor (and root method) are only invoked once, then all the describe blocks are invoked, and finally all of the statement tests.

Implications

Error behaviour

Errors generally only affect their immediate scope, including its children: If the test for an individual statement throws an exception, then that statement is reported as a failing test. If an exception is thrown within a describe block but outside of a statement test, then a single failure is reported for the whole unit, and the tests within that unit will not run, but other units in the same specification will run as normal. If the class constructor or root method throws an exception, then the entire class is marked as a failure (as with JUnit).

The only issue is that, since JarSpec tests are created dynamically, there will be no failure recorded for tests that were never reached due to errors in their containing unit.

Test Isolation

Since the class is not recreated for each test, then member fields are shared between all tests in the class. Furthermore, any local variables declared in the root() implementation or in any describe blocks are available to all tests that they contain. The compiler will ensure that such variables cannot be redefined by tests, since variables captured by lambda scopes must be treated as final. However, if these variables contain objects that are themselves mutable, then it's possible for tests to interfere with one another.

To avoid any unexpected side-effects between tests, it is best to design objects to be immutable, or at least severely limit the scope of mutable objects. Fortunately, these are good rules for software development in general, not just for working with JarSpec!