Local testing #94

DLBPointon · 2023-05-03T15:25:18Z

Updates to allow for local testing by ToL-IT.

Also fixed an error where gene_alignment had the incorrect number of channels for input compared to those declared in the gene alignment subworkflow.

github-actions · 2023-05-03T15:27:06Z

`nf-core lint` overall result: Passed ✅ ⚠️

Posted for pipeline commit dd21d15

+| ✅ 134 tests passed       |+
#| ❔  15 tests were ignored |#
!| ❗   9 tests had warnings |!

❗ Test warnings:

files_exist - File not found: assets/multiqc_config.yml
files_exist - File not found: pyproject.toml
pipeline_todos - TODO string in README.md: Add full-sized test dataset and amend the paragraph below if applicable
pipeline_todos - TODO string in README.md: Update the example "typical command" below used to run the pipeline
pipeline_todos - TODO string in WorkflowMain.groovy: Add Zenodo DOI for pipeline after first release
pipeline_todos - TODO string in base.config: Customise requirements for specific processes.
pipeline_todos - TODO string in output.md: Write this documentation describing your workflow's output
pipeline_todos - TODO string in usage.md: Add documentation about anything specific to running your pipeline. For general topics, please point to (and add to) the main nf-core website.
pipeline_todos - TODO string in awsfulltest.yml: You can customise AWS full pipeline tests as required

❔ Tests ignored:

nextflow_config - Config variable ignored: manifest.name
nextflow_config - Config variable ignored: manifest.homePage
files_unchanged - File ignored due to lint config: .gitattributes
files_unchanged - File ignored due to lint config: LICENSE or LICENSE.md or LICENCE or LICENCE.md
files_unchanged - File ignored due to lint config: .github/CONTRIBUTING.md
files_unchanged - File ignored due to lint config: .github/ISSUE_TEMPLATE/bug_report.yml
files_unchanged - File ignored due to lint config: .github/PULL_REQUEST_TEMPLATE.md
files_unchanged - File ignored due to lint config: .github/workflows/linting_comment.yml
files_unchanged - File ignored due to lint config: .github/workflows/linting.yml
files_unchanged - File ignored due to lint config: assets/email_template.txt
files_unchanged - File ignored due to lint config: assets/sendmail_template.txt
files_unchanged - File ignored due to lint config: lib/NfcoreSchema.groovy
files_unchanged - File ignored due to lint config: lib/NfcoreTemplate.groovy
files_unchanged - File ignored due to lint config: .gitignore or .prettierignore or pyproject.toml
multiqc_config - 'assets/multiqc_config.yml' not found

✅ Tests passed:

files_exist - File found: .gitattributes
files_exist - File found: .gitignore
files_exist - File found: .nf-core.yml
files_exist - File found: .editorconfig
files_exist - File found: .prettierignore
files_exist - File found: .prettierrc.yml
files_exist - File found: CHANGELOG.md
files_exist - File found: CITATIONS.md
files_exist - File found: CODE_OF_CONDUCT.md
files_exist - File found: CODE_OF_CONDUCT.md
files_exist - File found: LICENSE or LICENSE.md or LICENCE or LICENCE.md
files_exist - File found: nextflow_schema.json
files_exist - File found: nextflow.config
files_exist - File found: README.md
files_exist - File found: .github/.dockstore.yml
files_exist - File found: .github/CONTRIBUTING.md
files_exist - File found: .github/ISSUE_TEMPLATE/bug_report.yml
files_exist - File found: .github/ISSUE_TEMPLATE/config.yml
files_exist - File found: .github/ISSUE_TEMPLATE/feature_request.yml
files_exist - File found: .github/PULL_REQUEST_TEMPLATE.md
files_exist - File found: .github/workflows/branch.yml
files_exist - File found: .github/workflows/ci.yml
files_exist - File found: .github/workflows/linting_comment.yml
files_exist - File found: .github/workflows/linting.yml
files_exist - File found: assets/email_template.html
files_exist - File found: assets/email_template.txt
files_exist - File found: assets/sendmail_template.txt
files_exist - File found: assets/nf-core-treeval_logo_light.png
files_exist - File found: conf/modules.config
files_exist - File found: conf/test.config
files_exist - File found: conf/test_full.config
files_exist - File found: docs/images/nf-core-treeval_logo_light.png
files_exist - File found: docs/images/nf-core-treeval_logo_dark.png
files_exist - File found: docs/output.md
files_exist - File found: docs/README.md
files_exist - File found: docs/README.md
files_exist - File found: docs/usage.md
files_exist - File found: lib/nfcore_external_java_deps.jar
files_exist - File found: lib/NfcoreSchema.groovy
files_exist - File found: lib/NfcoreTemplate.groovy
files_exist - File found: lib/Utils.groovy
files_exist - File found: lib/WorkflowMain.groovy
files_exist - File found: main.nf
files_exist - File found: conf/base.config
files_exist - File found: conf/igenomes.config
files_exist - File found: .github/workflows/awstest.yml
files_exist - File found: .github/workflows/awsfulltest.yml
files_exist - File found: lib/WorkflowTreeval.groovy
files_exist - File found: modules.json
files_exist - File not found check: Singularity
files_exist - File not found check: parameters.settings.json
files_exist - File not found check: .nf-core.yaml
files_exist - File not found check: bin/markdown_to_html.r
files_exist - File not found check: conf/aws.config
files_exist - File not found check: .github/workflows/push_dockerhub.yml
files_exist - File not found check: .github/ISSUE_TEMPLATE/bug_report.md
files_exist - File not found check: .github/ISSUE_TEMPLATE/feature_request.md
files_exist - File not found check: docs/images/nf-core-treeval_logo.png
files_exist - File not found check: .markdownlint.yml
files_exist - File not found check: .yamllint.yml
files_exist - File not found check: lib/Checks.groovy
files_exist - File not found check: lib/Completion.groovy
files_exist - File not found check: lib/Workflow.groovy
files_exist - File not found check: .travis.yml
nextflow_config - Config variable found: manifest.nextflowVersion
nextflow_config - Config variable found: manifest.description
nextflow_config - Config variable found: manifest.version
nextflow_config - Config variable found: timeline.enabled
nextflow_config - Config variable found: trace.enabled
nextflow_config - Config variable found: report.enabled
nextflow_config - Config variable found: dag.enabled
nextflow_config - Config variable found: process.cpus
nextflow_config - Config variable found: process.memory
nextflow_config - Config variable found: process.time
nextflow_config - Config variable found: params.outdir
nextflow_config - Config variable found: params.input
nextflow_config - Config variable found: params.show_hidden_params
nextflow_config - Config variable found: params.schema_ignore_params
nextflow_config - Config variable found: manifest.mainScript
nextflow_config - Config variable found: timeline.file
nextflow_config - Config variable found: trace.file
nextflow_config - Config variable found: report.file
nextflow_config - Config variable found: dag.file
nextflow_config - Config variable (correctly) not found: params.nf_required_version
nextflow_config - Config variable (correctly) not found: params.container
nextflow_config - Config variable (correctly) not found: params.singleEnd
nextflow_config - Config variable (correctly) not found: params.igenomesIgnore
nextflow_config - Config variable (correctly) not found: params.name
nextflow_config - Config variable (correctly) not found: params.enable_conda
nextflow_config - Config timeline.enabled had correct value: true
nextflow_config - Config report.enabled had correct value: true
nextflow_config - Config trace.enabled had correct value: true
nextflow_config - Config dag.enabled had correct value: true
nextflow_config - Config dag.file ended with .html
nextflow_config - Config variable manifest.nextflowVersion started with >= or !>=
nextflow_config - Config manifest.version ends in dev: '1.0dev'
nextflow_config - Config params.custom_config_version is set to master
nextflow_config - Config params.custom_config_base is set to https://raw.githubusercontent.com/nf-core/configs/master
nextflow_config - Lines for loading custom profiles found
files_unchanged - .prettierrc.yml matches the template
files_unchanged - CODE_OF_CONDUCT.md matches the template
files_unchanged - .github/.dockstore.yml matches the template
files_unchanged - .github/ISSUE_TEMPLATE/config.yml matches the template
files_unchanged - .github/ISSUE_TEMPLATE/feature_request.yml matches the template
files_unchanged - .github/workflows/branch.yml matches the template
files_unchanged - assets/email_template.html matches the template
files_unchanged - assets/nf-core-treeval_logo_light.png matches the template
files_unchanged - docs/images/nf-core-treeval_logo_light.png matches the template
files_unchanged - docs/images/nf-core-treeval_logo_dark.png matches the template
files_unchanged - docs/README.md matches the template
files_unchanged - lib/nfcore_external_java_deps.jar matches the template
actions_ci - '.github/workflows/ci.yml' is triggered on expected events
actions_ci - '.github/workflows/ci.yml' checks minimum NF version
actions_awstest - '.github/workflows/awstest.yml' is triggered correctly
actions_awsfulltest - .github/workflows/awsfulltest.yml is triggered correctly
actions_awsfulltest - .github/workflows/awsfulltest.yml does not use -profile test
readme - README Nextflow minimum version badge matched config. Badge: 22.04.0, Config: 22.04.0
readme - README Nextflow minimum version in Quick Start section matched config. README: 22.04.0, Config: 22.04.0
pipeline_name_conventions - Name adheres to nf-core convention
template_strings - Did not find any Jinja template strings (179 files)
schema_lint - Schema lint passed
schema_lint - Schema title + description lint passed
schema_lint - Input mimetype lint passed: 'text/yaml'
schema_params - Schema matched params returned from nextflow config
actions_schema_validation - Workflow validation passed: linting_comment.yml
actions_schema_validation - Workflow validation passed: ci.yml
actions_schema_validation - Workflow validation passed: awstest.yml
actions_schema_validation - Workflow validation passed: linting.yml
actions_schema_validation - Workflow validation passed: fix-linting.yml
actions_schema_validation - Workflow validation passed: awsfulltest.yml
actions_schema_validation - Workflow validation passed: branch.yml
merge_markers - No merge markers found in pipeline files
modules_json - Only installed modules found in modules.json
modules_structure - modules directory structure is correct 'modules/nf-core/TOOL/SUBTOOL'

Run details

nf-core/tools version 2.7.2
Run at 2023-05-18 13:20:36

…it local, pipeline runs in sub 3 mins)

DLBPointon · 2023-05-11T11:54:35Z

This included a couple of updated to the gap_finder subworkflow too (adding gziptabix) and a few other housekeeping bits.

Unit test run with: nextflow run main.nf -profile test,singularity,sanger --outdir OscSUBSET

Locally, completes within 3 minutes.

DLBPointon · 2023-05-16T15:35:03Z

~~Currently waiting on my pr for~~ nf-core/modules#3423
This will complete the gap subworkflow bug mentioned in ToL-pipelines.

Merged in

priyanka-surana · 2023-05-18T08:15:44Z

~~I tried running the pipeline, but nothing actually runs. It says pipeline completed successfully, but no process is actually executed.~~

Pipeline execution works for me. I used the following command.

bsub -R 'select[mem>4000] rusage[mem=4000] span[hosts=1]' -n 2 -M 4000 -q oversubscribed -J treeval -o treeval.%J.o -e treeval.%J.e "nextflow run main.nf -profile test,singularity,sanger --outdir OscSUBSET -entry FULL -ansi-log false"

priyanka-surana

Minor changes to make things clear.

conf/modules.config

conf/test.config

priyanka-surana · 2023-05-18T12:20:06Z

bin/cut_size.sh

Why not add directly in the module?

bin/gff_to_bed.sh

bin/paf_to_bed12.sh

conf/test.config

conf/test_full.config

subworkflows/local/gap_finder.nf

Suggested Changes Co-authored-by: Priyanka Surana <surana.priyanka@gmail.com>

priyanka-surana

The test works. There are some formatting issues which need to be resolved and hopefully will be addressed in the template update planned.

Local testing

DLBPointon added 4 commits May 3, 2023 13:51

Files for local testing

a22b70e

new file: assets/local_testing/RunTime.md

629ff8f

Updates to directories in local folders

b847ac2

Updating for local testing, texts extra channel in treeval.nf

8c29e83

DLBPointon added bug Something isn't working Data Issue Issue related to test data labels May 3, 2023

DLBPointon requested a review from priyanka-surana May 3, 2023 15:25

DLBPointon self-assigned this May 3, 2023

DLBPointon added 6 commits May 3, 2023 16:37

linting fix

20df899

Update linting

ebd3495

User request to get rid of assembly name from output file

6c7d3f8

Addition to local testing, adding a representative dataset (all data …

3b3a520

…it local, pipeline runs in sub 3 mins)

Small changes

e6c7317

Linting Fix

ea78699

DLBPointon added 4 commits May 15, 2023 14:09

Changes to allow for multiple entry points

47fd460

Updates to fix naming by scripts

3087ce4

Updates to Gap subworkflow, correcting error in workflow

f170683

Adding tabix configurations to modules.config, allowing csi output

e0250ab

DLBPointon added 3 commits May 17, 2023 09:56

Updates from the nf-core modules repo

5975dc0

Fixing modules.json prettier lint

810358b

Update config

55b6519

priyanka-surana suggested changes May 18, 2023

View reviewed changes

conf/modules.config Outdated Show resolved Hide resolved

conf/test.config Show resolved Hide resolved

DLBPointon added 4 commits May 18, 2023 11:48

Finally fixed gap

9327a63

Minor changes including pipeline_into to treeval_info

a64f0ef

Linting

eadd195

Updates

3125748

DLBPointon added 3 commits May 18, 2023 12:58

Updates

1afe379

Linting update

b4722e2

Linting

c2ccf69

priyanka-surana suggested changes May 18, 2023

View reviewed changes

DLBPointon and others added 5 commits May 18, 2023 14:17

Update conf/test.config

330898c

Suggested Changes Co-authored-by: Priyanka Surana <surana.priyanka@gmail.com>

Update conf/test.config

dd21d15

Suggested Changes Co-authored-by: Priyanka Surana <surana.priyanka@gmail.com>

Update conf/test.config

7acf444

Suggested Changes Co-authored-by: Priyanka Surana <surana.priyanka@gmail.com>

Update conf/test_full.config

1754a34

Suggested Changes Co-authored-by: Priyanka Surana <surana.priyanka@gmail.com>

Update subworkflows/local/gap_finder.nf

3071064

Suggested Changes Co-authored-by: Priyanka Surana <surana.priyanka@gmail.com>

priyanka-surana approved these changes May 18, 2023

View reviewed changes

DLBPointon merged commit 204d303 into dev May 18, 2023

muffato deleted the local_testing branch May 18, 2023 21:24

DLBPointon added a commit that referenced this pull request Sep 7, 2023

Merge pull request #94 from sanger-tol/local_testing

766aefe

Local testing

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Local testing #94

Local testing #94

DLBPointon commented May 3, 2023

github-actions bot commented May 3, 2023 •

edited

Loading

❗ Test warnings:

❔ Tests ignored:

✅ Tests passed:

Run details

DLBPointon commented May 11, 2023 •

edited

Loading

DLBPointon commented May 16, 2023 •

edited

Loading

priyanka-surana commented May 18, 2023 •

edited

Loading

priyanka-surana left a comment

priyanka-surana May 18, 2023

priyanka-surana left a comment

Local testing #94

Local testing #94

Conversation

DLBPointon commented May 3, 2023

github-actions bot commented May 3, 2023 • edited Loading

nf-core lint overall result: Passed ✅ ⚠️

❗ Test warnings:

❔ Tests ignored:

✅ Tests passed:

Run details

DLBPointon commented May 11, 2023 • edited Loading

DLBPointon commented May 16, 2023 • edited Loading

priyanka-surana commented May 18, 2023 • edited Loading

priyanka-surana left a comment

Choose a reason for hiding this comment

priyanka-surana May 18, 2023

Choose a reason for hiding this comment

priyanka-surana left a comment

Choose a reason for hiding this comment

github-actions bot commented May 3, 2023 •

edited

Loading

`nf-core lint` overall result: Passed ✅ ⚠️

DLBPointon commented May 11, 2023 •

edited

Loading

DLBPointon commented May 16, 2023 •

edited

Loading

priyanka-surana commented May 18, 2023 •

edited

Loading