Nurminen table detection

[mcharters]

This branch implements a more sophisticated table detection algorithm based off Anssi Nurminen's masters thesis (more or less) which can be found here: http://dspace.cc.tut.fi/dpub/bitstream/handle/123456789/21520/Nurminen.pdf?sequence=3

With this algorithm, 49 of the 67 ground truth table detection tests pass. The remaining failures are mostly either tricky tables or false positives (which I'm guessing are more useful to tabula than not finding anything).

Note that this branch is based off mcharters:add-table-detection-tests, so it's got some extra changes in there.

[mcharters]

OK, I was able to reuse Ruling.collapseOrientedRulings and SpreadsheetExtractionAlgorithm.findCells (which also let me get rid of my duplicate crossing point algorithm).

I'm still using my own getTableAreasFromCells method because I allow cells that are a small distance apart to be part of the same table (this cuts down on a few failures in the ground truth data). I'll keep it that way unless we should be doing that for the SpreadsheetExtractionAlgorithm as well?

[jazzido]

OK, I was able to reuse Ruling.collapseOrientedRulings and SpreadsheetExtractionAlgorithm.findCells (which also let me get rid of my duplicate crossing point algorithm).

Awesome. Deleting huge chunks of code always makes me happy :)

I'm still using my own getTableAreasFromCells method because I allow cells that are a small distance apart to be part of the same table (this cuts down on a few failures in the ground truth data). I'll keep it that way unless we should be doing that for the SpreadsheetExtractionAlgorithm as well?

Can you point me to the difference between your method and ours? If incorporating that change to SpreadsheetExtractionAlgorithm.findSpreadsheetsFromCells doesn't break our tests, we can get rid of one of them.

[mcharters]

OK, from looking closer at the findSpreadhseetsFromCells method it's much more sophisticated than mine. I'm just taking each cell (from top-left-most to bottom-right-most) and checking cell by cell to see if the next cell shares any corners with any cells in my current "table" and that's where I add the distance threshold - cells can be a little bit away and be considered part of the table.

I think to modify the findSpreadhseetsFromCells algorithm to be similar it would involve writing more code to snap all the points in the pointSet to some sort of grid before finding the horizontal and vertical lines - unless I'm missing something. This seems like it would be tricky to me?

[jazzido]

Small refactor suggestion:

We already have a top-leftmost Point2D comparator here — We could move it to Utils and reuse it from NurminenDetectionAlgorithm.

[jazzido]

OK, from looking closer at the findSpreadhseetsFromCells method it's much more sophisticated than mine […]

Right. Let's keep yours for now, then.

[jazzido]

I think it now looks good for merging into master. Last request: can you set up the tests so the ones that pass at this point are ran by default in the test suite? That way we make sure not to introduce regressions in the detector.

Thanks!

[mcharters]

@jazzido it looks like the tests are failing because PDFBox is logging too much. Do you know if there's an easy way to temporarily turn off logging or will I have to add log4j as a dependency and do it programatically for the tests?

[mcharters]

Well that's fun. I guess there must be enough of a difference between my flavour of java and travis' flavour that there are failures in the integration tests that don't appear locally.

I'll try to think of some clever way around this but suggestions are welcome!

[mcharters]

OK I can reproduce the failures in an old vagrant box. I'll dig into the failures.

[mcharters]

Phew! OK, by adding a little more "fuzziness" to a couple parts of the algorithm I was able to align the tests over a bunch of platforms (windows, os x & ubuntu). Might be ready to go now?

[jazzido]

Merged! Thanks a lot @mcharters. Your PR is the biggest contribution we received in the history of Tabula.

[jazzido]

I've just integrated the new detector in Tabula (web). This is awesome

(@jeremybmerrill , @mtigas, take a look :))