Unit test crash on Solaris

[koheiw]

There has been a mysterious but persistent errors on tests on Solaris ODS on CRAN and RHub. This does not happen on any other major OS but "Segmentation Fault" is worrying.

* checking tests ...
  Running ‘spelling.R’
  Running ‘testthat.R’Segmentation Fault

 ERROR
Running the tests in ‘tests/testthat.R’ failed.
Last 13 lines of output:
  34: withOneRestart(expr, restarts[[1L]])
  35: withRestarts(testthat_abort_reporter = function() NULL, force(code))
  36: with_reporter(reporter = reporter, start_end_reporter = start_end_reporter,     {        reporter$start_file(basename(path))        lister$start_file(basename(path))        source_file(path, new.env(parent = env), chdir = TRUE,             wrap = wrap)        reporter$.end_context()        reporter$end_file()    })
  37: FUN(X[[i]], ...)
  38: lapply(paths, test_file, env = env, reporter = current_reporter,     start_end_reporter = FALSE, load_helpers = FALSE, wrap = wrap)
  39: force(code)
  40: doWithOneRestart(return(expr), restart)
  41: withOneRestart(expr, restarts[[1L]])
  42: withRestarts(testthat_abort_reporter = function() NULL, force(code))
  43: with_reporter(reporter = current_reporter, results <- lapply(paths,     test_file, env = env, reporter = current_reporter, start_end_reporter = FALSE,     load_helpers = FALSE, wrap = wrap))
  44: test_files(paths, reporter = reporter, env = env, stop_on_failure = stop_on_failure,     stop_on_warning = stop_on_warning, wrap = wrap)
  45: test_dir(path = test_path, reporter = reporter, env = env, filter = filter,     ..., stop_on_failure = stop_on_failure, stop_on_warning = stop_on_warning,     wrap = wrap)
  46: test_package_dir(package = package, test_path = test_path, filter = filter,     reporter = reporter, ..., stop_on_failure = stop_on_failure,     stop_on_warning = stop_on_warning, wrap = wrap)
  47: test_check("quanteda.core")
  An irrecoverable exception occurred. R is aborting now ...

[koheiw]

We suspected that the caching mechanism of recently added textstat_summary() is causing the problem, but it turned out not after a week of investigation on a test branch and the quanteda.core package.

The real cause is a test for tokens_compile()

https://github.com/quanteda/quanteda/blob/e16b4babe75dd70fa292c66855e4420b089f9342/tests/testthat/test-tokens_recompile.R#L137-L144

The it underlying C++ function throws an error when tokens ID do not match token types, but it does not work on Solaris for some reason.

When it is tested without testthat, it produces

  > attr(toks, 'types') <- c('a', 'b', 'c') # Shorter than 10
  > attr(toks, 'class') <- 'tokens'
  > tryCatch(
  +     quanteda.core:::tokens_recompile(toks, 'C++')
  + )
  
   *** caught segfault ***
  address 0, cause 'memory not mapped

I could confirm that this is the cause in a branch that activates all the tests on Solaris but the above. It still errors but only minor problems:

* checking tests ...
  Running ‘spelling.R’
  Running ‘test.R’
  Running ‘testthat.R’
 ERROR
Running the tests in ‘tests/testthat.R’ failed.
Last 13 lines of output:
  
  x[3]: "good ice"
  y[3]: "ice cream"
  
  x[4]: "ice cream"
  y[4]: "like good"
  
  ══ testthat results  ═══════════════════════════════════════════════════════════
  [ OK: 2805 | SKIPPED: 15 | WARNINGS: 13 | FAILED: 3 ]
  1. Failure: test phrase for collocations (@test-phrases.R#37) 
  2. Failure: textstat_collocations.tokens works ok with zero-length documents (#940) (@test-textstat_collocations.R#265) 
  3. Failure: textstat_collocations.tokens works ok with zero-length documents (#940) (@test-textstat_collocations.R#275) 
  
  Error: testthat unit tests failed
  Execution halted

It is still a mystery why the segfault happens only on Solaris, but I upgraded the C++ code to make it more robust against corrupt objects. I suspect that that the errors is happening in other R packages or Solaris libraries. We should reduce quanteda's dependency on other packages.