Improve coverage for umbrella apps

[xadhoom]

I want to report here others work made to support a better coverage reporting for umbrella apps, in order to not lose it, since it may be helpful. I think the orginal author did not had time to further push it here.

Sometimes within umbrella apps some integration testing is done, to check that everything works fine and to test also those bits that may be too expensive (speaking of test setup) to do as unit test.

Unfortunately the overall coverage cannot be computed with the current way of running tests in umbrella apps.

Links to the real work (not made by me):

• original issue: Umbrella projects do not report numbers overall parroty/excoveralls#23
• discussion: https://groups.google.com/forum/#!topic/elixir-lang-core/8wX8i5sEtFg
• passing paths to cover tool from mix test task: parroty@16dbd2e
• how the cover tool may handle it: parroty/excoveralls@501d67e

Reporting it here because to allow cover tools to do proper overall coverage requires some improvements into Elixir itself. Hope that can be helpful to further improve it :)

[josevalim]

Elixir today only provides a wrap around cover and a pluggable wrapper. So there isn't much we can do on the Elixir front. In fact, if the discussion eventually goes towards evaluating a solution from scratch, then I would prefer if the community developed a new cover-like tool.

Regarding umbrellas, I can think of two solutions:

1. Someone writes a tool from scratch
2. Someone writes a tool that can merge the reports in different apps into a consolidated one

I am really not aware how easy or hard those efforts will be because, as said, this is really not a topic Elixir tackles. We just provide a convenient wrapper around cover and a pluggable API. What you do then is completely up to you.

[josevalim]

And btw, if we need to do improvements in Elixir itself, then please go ahead and open up specific proposals about those improvements. :) My main point is that you shouldn't expect that Elixir will actually be the one measuring or grouping coverage - that's the job of the pluggable tool.

[xadhoom]

well, one improvement was already linked into the first message.

But I can see your point, even if doing a such tool may need to replicate a lot of stuff that current mix test tasks already do.
After all (on the provided example), on elixir side is just passing all the paths down to the cover tool, it does not have to do any cover, so basically is just enriching the informations in order to semplify what the cover tool has to do.

[josevalim]

well, one improvement was already linked into the first message.

Apologies but it is not clear enough what it is expected from Elixir in the first message. If there is a more concrete proposal, we will be glad to take a look at.

[xadhoom]

Ok,

got it. I'll check if is possible to hack around from the coverage tool itself, otherwise report in a different channel, forum or ml. basically I just wanted to start a discussion and this may be not the best location :)

[josevalim]

@xadhoom yes, especially because we ask to not use the issues tracker for discussions. ;) Please post a link here though if the discussion starts elsewhere but I am afraid I do not have enough knowledge to be of much help.

The only advice I can give is, if by any chance cover generates an intermediate format which is used by coveralls (which is a great tool btw) then you could have a command like "mix test.umbrella" that runs "mix test" outputting to this intermediate format. Since "mix test" is recursive, then all you need to do is to aggregate the results of those intermediate files. Although I believe you will also need a mechanism to tell mix test which should instrument the whole project and not individual applications. That may require changing Elixir.