Unit tests

[jacobwilliams]

Need some rigorous unit tests.

[zbeekman]

First let me start by saying I'm a big fan of any and all software testing. Unit tests are great.

I just want to point out that there is a decent testing capability and automated framework using the test/sample code @jacobwilliams wrote along with output validation using jsonlint (see #26) that is automated in the CMake build system. The tests, while perhaps somewhat ad hoc, have good coverage and test that the code behaves as expected, on a coarse level, and tests for regressions.

I am posting this because I want people to know that it is at least moderately well tested code as is, even without more fine grained unit testing.

[zbeekman]

Close? Or to ask it differently, what needs to happen for this issue to be resolved?

[jacobwilliams]

Let's keep open. Now that there's a good system in place, I'm going to add some more unit test.

[zbeekman]

I put this comment regarding a better way to deal with stuff in the files directory in PR #73 but this might be a better place to discuss it, so I’m copying the relevant portion here:

I’m wondering if we should have a convention to distinguish test inputs from test outputs. That way we can use file globbing to vacuum up the appropriate input files and make sure they’re copied to the build tree. At the same time if we had a convention for the test outputs, we could also automatically send them through jsonlint and compare the actual output with the anticipated output. What do you think of this idea?

[jacobwilliams]

Agreed. Ideally, it should be completely automated.

[zbeekman]

I think we should either create subdirectories of the files directory named inputs where inputs go and outputs where the files that should match the outputs go. Then each test can read from files/inputs/file.json if it requires input, but write to files/file.json so that files/file.json can be compared with files/ouputs/file.json (Here “file.json” is just a stand in for whatever the actual file name is.) Thoughts?

[zbeekman]

I just added a page to the wiki to track the procedures that aren’t called in any of the unit tests. Ideally, tests will be added in the future to address all issues there, and the wiki updated, to reflect the progress.

[jacobwilliams]

Agree. One thing that shouldn't happen (that currently does), is files committed in the repo shouldn't be overwritten in place by the unit tests. That seems like bad practice. The "truth" files should be in one dir, and the unit test outputs generated in another, then compared.

Great idea for the wiki. Definitely will work on getting all these routines tested.

[zbeekman]

I updated the page on the wiki with the output of the FoBiS.py gcov analyzer markdown. I manually edited some of it and moved things around, but I think we could get this to auto-deploy in the future if that interests you, the same way we auto-deploy the ROBODoc documentation. Then, we could have a slightly nicer coverage display page than the one coveralls uses, with he breakdown of covered and uncovered procedures.

Also, PR #86 address the issue wherein tests overwrite files committed to the repo.

[zbeekman]

Here is a link to the updated wiki page with the test coverage: https://github.com/jacobwilliams/json-fortran/wiki/Coverage-Analysis

[szaghi]

@jacobwilliams and @zbeekman

The gcov analyzer in FoBiS.py is very young. If you want to use it, I can introduce any feature you want, e.g. for customizing the report theme and/or reports comments.

Let me know of any features you like.

See you soon.

[zbeekman]

One thing that would be nice is to make the uncovered procedures report come before the covered procedure report, since they are more important to test. Also I customized the colors for the pie charts (green = covered = good, red = uncovered = bad!) but in general I think the output is great as is.

[szaghi]

Tomorrow, I will try to push a new version with customizable colors and with inverted order of covered/uncovered.

See you soon.

[zbeekman]

@jacobwilliams I’m wondering if we should close this given the decent test coverage we’re getting… we could always open a new issue regarding improving test coverage to some target, but having an issue that’s just “unit tests” makes it seem like there aren’t any and that the project doesn’t have good QA measures.

[jacobwilliams]

Fair enough. Yes, we can continue to add to them.