Decouple scda - reorganise example and test data samples

[pawelru]

As a NEST packages maintainer
In order to have my packages more encapsulated (in particular: independent of scda packages)
I want to move example data into the package while still keeping extensive data testing somewhere else (i.e. new package)

This is the target state:

package	tern / rlistings / rtables	scda.tests	chevron	tmg / tmc / tg / to	goshawk / osprey
test scope	- unit - (general) regression	- (Roche-specific) regression	- unit - (Roche specific) regresion	- unit - shiny	- unit - (general) regression
size of data	small	bigger (scda)	small	small	small

I have discussed this with @shajoezhu and here's the summary:

• For each of the repo (starting from tern and rlistings): please analyse if scda dependency is really required. If not: let's create example and testing data (it could be the same data sample) and save it into the respective package. This is assumed to be much lighter than scda produces. For some of the cases (e.g. internals to teal) it could be non CDISC.
  As a result, a given package would become independent of scda. This is relevant for individual package release model.
  This needs to be done for many packages (basically all downstream dependencies of scda) starting from tern and rlistings.
• Create a new (private) repo which would just implement data intensive regression tests based on scda. Those exist currently in tern and it need to be moved from there.

List of tasks:

• create a new (private) repository scda.tests (name TBD)
• tern Decouping scda from tern tern#777
• rlistings
  • creating minimal data and regression test
  • adding Roche listing regression tests to scda.tests (new)
• chevron
  • this is very similar to: efficient way of storing synthetic test data chevron#216
• teal Decouple scda from teal teal#834
• teal.data Decouple scda from teal.data teal.data#145
• teal.slice Decouple scda from teal.slice teal.slice#299
• teal.transform Decouple scda from teal.transform teal.transform#133
• teal.widgets Decouple scda from teal.widgets teal.widgets#157
• teal.modules.general Decouple scda from the dependencies teal.modules.general#532
• teal.modules.clinical Decoupling scda from tmc teal.modules.clinical#707
• osprey and teal.osprey Decouple scda teal.osprey#213
• goshawk and teal.goshawk Decouple scda in goshawk and teal.goshawk teal.goshawk#222
• teal.modules.hermes Decouple scda teal.modules.hermes#311
• teal.modules.helios https://github.com/insightsengineering/teal.modules.helios/issues/88

[shajoezhu]

how about we name the new repo as "tlg.tests.scda"

[Melkiades]

I widely agree with this issue. Today, I have been thinking about table tests that are currently in tern and I found their repetition with the real TLG-c unnecessary and dangerous. Dangerous because it is very easy at the moment to find discrepancies and even tables that are not tested. I would structure scda.tests (or tlg.tests) so to have all tlgs in only one place (the repo we currently have) while tests just take the output and compare it to saved values generated from the tables. I am strongly against copy-paste the tlg catalog for testing. I think it is prone to errors and updating times take longer and it is possible we forget something, either the current catalog or the testing. Examples of problems:

• Missing checks for table structure -> Update table (and all) tests to check also correct indentation  tern#784
• Missing tests for some tables -> Missing tests for pdt02? tern#786
• Missing variants and some are fixed on one end but not on the other

[gogonzo]

Done!