junitlogger report xml file content not correctly generated

[lauterry]

Hello

Please assume that I have the following test :

test( "AccountsView", function() {
    "use strict";
    ok( 1 === 1, "Passed!" );
    ok( 2 === 2, "Passed!" );
    ok( 3 === 3, "Passed!" );
});

Running junitlogger generate the following xml report :

<?xml version="1.0" encoding="UTF-8"?>
<testsuites hostname="localhost" tests="3" failures="0" errors="0" time="0.515" timestamp="2013-01-16T16:21:17Z">
    <testsuite id="0" name="undefined" hostname="localhost" tests="0" failures="0" errors="0" time="0"
               timestamp="2013-01-16T16:21:17Z">
        <testcase name="AccountsView" tests="3" failures="0" errors="0" time="0" timestamp="2013-01-16T16:21:17Z">
        </testcase>
    </testsuite>
</testsuites>

I expect the testsuite tag to have a name "AccountsView" instead of undefined and I have only one testcase whereas I expect 3. The report content do not seem to be correctly built.

Please correct me if I'm wrong

Best regards

[JamesMGreene]

testsuite == module in the QUnit world, though I would've expected a better default testsuite name, e.g. "default" or perhaps the page's URL.

testcase == test/asyncTest, so it is accurate that your testcase element has a name of "AccountsView".

Finally, the tests count on the testcase is equivalent to the number of assertions executed in that test, e.g. in this case, your 3 ok calls.

I am pondering if the testsuite element's tests/failures/errors attributes should be based on tests (i.e. testcase elements) rather than just being a sum of the assertion-based counts of the tests. In other words, your example would look more like the following (noting the counts of the tests attributes as they roll-up):

<?xml version="1.0" encoding="UTF-8"?>
<testsuites hostname="localhost"
            tests="1" failures="0" errors="0" time="0.515" timestamp="2013-01-16T16:21:17Z">
    <testsuite name="http://localhost/yourTestPageUrl" hostname="localhost"
               tests="1" failures="0" errors="0" time="0" timestamp="2013-01-16T16:21:17Z">
        <testcase name="AccountsView"
                  tests="3" failures="0" errors="0" time="0" timestamp="2013-01-16T16:21:17Z" />
    </testsuite>
</testsuites>

Thoughts?

[JamesMGreene]

@lauterry: I'm in the process of rewriting the JUnitLogger already as it does not mesh well with the Composite addon, so feel free to give any other suggestions/feedback by filing additional new issues.

@jzaefferer @Krinkle Please make me the owner of this issue, thanks!

[Krinkle]

@JamesMGreene Issues can only be assigned to repo collabs. Since assignees should also take care of review and eventual revert/backport/release, I'll oversee this one allowing you to provide a PR, which I'll then review.

[JamesMGreene]

Fair enough, thanks @Krinkle.

[lauterry]

I noticed this issue by giving the test xml report to Jenkins. The displayed report is not correct.

Having the following report processed by jenkins display a more correct report 👍

<?xml version="1.0" encoding="UTF-8" ?>
<testsuites>
<testsuite name="AccountsView" errors="0" tests="3" failures="0" time="0.078" timestamp="2013-01-15T15:42:33">
  <testcase classname="AccountsView" name="Passed!" time="0.031"></testcase>
  <testcase classname="AccountsView" name="Passed!" time="0.016"></testcase>
  <testcase classname="AccountsView" name="Passed!" time="0.031"></testcase>
</testsuite>
</testsuites>

So for Jenkins, testsuite should be Qunit test and <testcase> should be Qunit assertions.

I don't know what would be the equivalent to the Qunit module for Jenkins. Maybe the package attribute of testsuite tag.

[lauterry]

@JamesMGreene : How do you display your generated XML report? What do you use?

[JamesMGreene]

@lauterry Jenkin's XSD for JUnit shows that their testcase element can have an optional assertions attribute, which clearly means that testcase === QUnit.test.

However, the testcase element also does not expect the attributes tests (akin to assertions, I guess), failures, or errors, all of which we are currently providing. Those appear to only be expected on the testsuite element.

[JamesMGreene]

@lauterry We don't display our XML report, just have a CI build step (not on Jenkins) to parse the resulting XML file to verify there were no failures or errors.

[JamesMGreene]

@lauterry Also: if we did it your way, we would be showing every assertion even if it passed, which is not a behavior that I've ever seen in my [limited] experience with JUnit XML reports.

[JamesMGreene]

Here's an explanation of the end-behavior I'm intending for the JUnitLogger addon — hopefully with actual tests to verify these example results.

Report Logic

So, taking a deeper look at the JUnit XSD for Jenkins again, here's the report logic that I've extracted. I've used a bit of JavaScript pseudo-code plus XPath notations to shorten the descriptions of some of the conceptual values, hopefully that it makes sense to everyone:

<?xml version="1.0" encoding="UTF-8" ?>
<testsuites
  name="{window.location.href || undefined}"
  tests="{count(./testsuite)}"
  failures="{count(./testsuite[number(@failures) > 0])}"
  errors="{count(./testsuite[number(@errors) > 0])}"
  time="{sum(./testsuite/@time)}"
>
    <testsuite
      id="{./position()}"
      name="{qunitModuleName || 'default'}"
      tests="{count(./testcase)}"
      failures="{count(./testcase[@status = 'fail'])}"
      errors="{count(./testcase[@status = 'error'])}"
      time="{sum(./testcase/@time)}"
      timestamp="{startingDateTime as ISO8601}"
      package="{qunitCompositeSuiteName || ../@name || undefined}"
      hostname="{window.location.host || require('os').hostname() || 'localhost'}"
    >
        <properties>
            <property name="userAgent" value="{window.navigator.userAgent || 'node'}" />
            <property name="testFramework" value="QUnit" />
        </properties>

        <!-- Passed test -->
        <testcase
          name="{qunitTestName}"
          assertions="{count(`QUnit.push` calls)}"
          time="{durationInSeconds}"
          status="pass"
          classname="{../@name}"
        />

        <!-- Failed test -->
        <testcase
          name="{qunitTestName}"
          assertions="{count(`QUnit.push` calls)}"
          time="{durationInSeconds}"
          status="fail"
          classname="{../@name}"
        >
            <failure type="{assertionType}" message="{assertionMessage}" />
        </testcase>

        <!-- Erred test -->
        <testcase
          name="{qunitTestName}"
          assertions="{count(`QUnit.push` calls)}"
          time="{durationInSeconds}"
          status="error"
          classname="{../@name}"
        >
            <error type="{errorType}" message="{errorMessage}" />
        </testcase>

        <system-out>
            /* Redirect all `console.*` output (minus `console.error`), perhaps? */
        </system-out>
        <system-err>
            /* Redirect all `console.error` output, perhaps? */
        </system-err>
    </testsuite>
</testsuites>

There are a number of things about this output that differ from my current output (using a proprietary version of the JUnitLogger addon that pre-dates it) which I care enough about to call out:

1. testcase elements do not have a count of failures or errors. This makes sense for most languages/frameworks as a test typically halts immediately upon its first failure/error but QUnit does not. As such, it's great to know the actual failure/error counts per test. 😕
2. failure elements do not allow child elements, e.g. actual and expected elements indicating the value differences.
3. error elements (and upstream errors counts) are not currently pertinent to QUnit as all errors are captured and turned into failures instead. Is it worthwhile to distinguish them? Most testing frameworks do.

[JamesMGreene]

An interesting post trying to dissect Jenkins' display of a JUnit XML report from a black box perspective: http://nelsonwells.net/2012/09/how-jenkins-ci-parses-and-displays-junit-output/

This would suggest that I'll need to revise the attribute values for testsuite/@package, testsuite/@name, and testcase/@classname in order to get anything decent to show in Jenkins.

[gboissinot]

For now, Jenkins junit integration doesn't use a strict mechanism to parse JUnit output but a custom Jenkins implementation. The Jenkins XSD mentioned refers to the xUnit plugin (https://wiki.jenkins-ci.org/display/JENKINS/xUnit+Plugin).
xUnit plugin is the de-facto plugin for the ntegration of reprorts in a format different of Junit. From your analysis, if you are able to make an extensible style sheet (XSL), it could be integrated to Jenkins xUnit plugin (and therefore Jenkins) as soon as possible.
Do you think is it possible for you to make an XSL?

[JamesMGreene]

@gboissinot: I'm well acquainted with XSL but I don't understand your question. You want an XSL stylesheet (yes, I know that's redundant but that's what I call them) to do what?

[gboissinot]

The objective is to apply an XSL to transform the current QUnit output to JUnit compatible to JUnit's Jenkins.

[JamesMGreene]

@gboissinot: Ah, I see. I think that's putting the cart before the horse since our data isn't structured appropriately yet and we have instances of missing assertion details during nested suite runs. :)

Will Jenkins choke if there are unrecognized attribute nodes (e.g. failures and errors on a testcase element) or elements (e.g. actual and expected child elements, or just a text node child with the stack trace, under the failure/error elements) in the XML?

[JamesMGreene]

@gboissinot: Also, are you aware of an official JUnit XML spec/XSD/DTD somewhere that I can look at? We just extended the XML we knew of to best suit our needs but we can certainly adhere to a standard if we know what it is. :)

[gboissinot]

There is no strict standard in Jenkins for now. In general, in my knowledge, JUnit does not have an XSD. As I said, Jenkins uses a custom implementation. However if you use the previous XSD (mentioned above), Jenkins is able to read it correctly.

[Krinkle]

@lauterry : @ mentions are auto-linked :-) and <code><foo></code> doesn't work as <foo> will be filtered out . Use backticks instead:

Foo `<bar>` baz.

(fixed it for you)

[JamesMGreene]

Personally, I am thinking that it would make more sense to have a separate JUnitLogger (for whatever current clients) vs. XUnitLogger (for Jenkins). Thoughts?

This also fits into the idea I've mentioned before (e.g. in #378, etc.) about making QUnit extensible for custom reporters (like Mocha).

[jzaefferer]

I'd like to get a JUnit module in place that takes care of generating JUnit-style XML output in JavaScript, with a testsuite, usable both in node and the browser. See also jzaefferer/node-testswarm#8

[JamesMGreene]

@jzaefferer How would you expect said module to be used? Various test output formats (i.e. JUnit vs. XUnit) generally require some different data that may not be easy to reconcile.

As such, this idea makes me think of a generic json2xml converter, which I believe there are already Node.js modules for (though they may not be browser-compliant, i.e. expecting fs access, etc.).

[jzaefferer]

Regarding json2xml, I've used https://github.com/estheban/node-json2xml recently. That works, though there is no npm release for the version in master. The published 0.1.1 code is useless. Might make sense to fork that and publish on npm under a new name.

Anyway, I hope such a module, as long as it outputs something that Jenkins can use, would abstract enough details that QUnit or node-testswarm or other grunt plugins can use it without worrying about producing valid output.

[Krinkle]

Moved to JamesMGreene/qunit-reporter-junit#1.