Export JUnit XML Report format

[byarbrough]

Feature Request

Feature description :
I would like to ingest reports into CI/CD pipelines to avoid having to scroll through logs.
Pytest currently does this via JUnit reports. Gitlab is able to ingest those reports.

Suggested Solution description ( if you have any ) :
Add a --junitxml=path option that outputs results to a file, ready to be ingested by CI/CD.

Describe alternatives you've considered :
The format does not have to be JUnit, but it should be a format widely understood by various CI/CD tools.

Additional context :
Report could display the Scenario text as the header, similarly to:

[eerkunt]

Hello @byarbrough,

Thanks to the underlying library we use, we already have this functionality. You can pass --junit-xml=<junitxml> while you are running terraform-compliance :)

[byarbrough]

Oh, I see now! It isn't mentioned as an Additional Parameters, but it is available via Radish junit-xml. If I get a chance I will look at making that more clear in the documentation and send a PR.

Unfortunately, when I add --junit-xml I get an error:

$ terraform-compliance --junit-xml /tmp/result.xml -f ../compliance/* -p tfplan
...

5 features (1 passed, 3 failed, 1 skipped)
6 scenarios (1 passed, 3 failed, 2 skipped)
21 steps (12 passed, 3 failed, 4 skipped)
Run 1589986063 finished within a moment

Run 1589986063 finished within a moment
❗ ERROR: Hook 'generate_junit_xml' from /Users/x/code/env-lint/lib/python3.7/site-packages/radish/extensions/junit_xml_writer.py:74 raised: 'TypeError: expected string or bytes-like object'

Traceback (most recent call last):
  File "/Users/x/code/env-lint/lib/python3.7/site-packages/radish/hookregistry.py", line 132, in call
    func(model, *args, **kwargs)
  File "/Users/x/code/env-lint/lib/python3.7/site-packages/radish/extensions/junit_xml_writer.py", line 163, in generate_junit_xml
    self._strip_ansi(step.failure.traceback),
  File "/Users/x/code/env-lint/lib/python3.7/site-packages/radish/extensions/junit_xml_writer.py", line 72, in _strip_ansi
    return pattern.sub("", text)
TypeError: expected string or bytes-like object

Positioning of the argument and = vs space doesn't seem to matter

terraform v0.12.24
terraform-compliance v1.2.1
On a Mac, reproduced on Ubuntu 20

[eerkunt]

Now this is interesting. Having a look on this!

[byarbrough]

@eerkunt I am only able to reproduce the issue if I have features that fail. If everything passes, the XML generates just fine, shown here.

<?xml version='1.0' encoding='utf-8'?>
<testsuites time="0.003">
  <testsuite name="Ensure all resources have tags" failures="0" errors="0" skipped="0" tests="1" time="0.003">
    <testcase classname="Ensure all resources have tags" name="Ensure all resources have tags" time="0.001"/>
  </testsuite>
</testsuites>

If you provide some guidance I will be more than happy to help fix the bug!

[Kudbettin]

Hi @byarbrough, could you have try with release 1.2.6?

[byarbrough]

Yup, that took care of it! XML file looks good as well. Thank you!

[ghost]

This issue's conversation is now locked. If you want to continue this discussion please open a new issue.