What statistics are appropriate for unit tests?

[AnthonyMastrean]

There comes a time, it seems, in any team, when you have to port a codebase, or update a major framework, or dramatically change some tooling and, inevitably, a ton of unit tests fail, for some period.

Someone usually decides to work up some stats on the data from, say, the last run of the tests. They might present a table like this...

     Project | Tests | Pass | Fail |  %
    ---------|-------|------|------|-----
     A       |   102 |   91 |   11 | 89%
     B       |    27 |   26 |    1 | 96%
     C       |    39 |   25 |   14 | 64%
     D       |     2 |    0 |    2 |  0%
     E       |     8 |    1 |    7 | 13%

And then they would conclude...

The average success rate is 52.4%

And maybe they'd present this information day-over-day or week-over-week until the number are "good" or the effort is over.

I'm having a really hard time with these numbers...

• What does it mean to be 89% successful?
• What does it mean if all your failures are caused by a single problem, that can be fixed once?
• How are you using continuous terms to describe discrete data (pass/fail)?
• How does one project relate to another? How does one test relate to another (in the same project, even)?

Problems...

• Continuous vs. discrete data and methods
• Data may not be related over time
• "Average" is only appropriate for normal distributions

Considerations...

• A unit tests should have a single assertion, so that the result (pass/fail) is 1:1 with the assertion (if you have multiple assertions, a test might only be 75% failing -- WAT)
• A test is either pass/fail, a test suite (project?) is either pass/fail, a test run is either pass/fail. The concept of partially passing or failing needs to be nonsense.

Todo...

• Explore discrete statistics and probability distributions: http://blog.minitab.com/blog/adventures-in-statistics-2/understanding-and-using-discrete-distributions