BSJ_DETECTION ADD_INTRONS error

[mariaUCL]

Description of the bug

I'm not sure what exactly is happening but currently I am running the pipeline as below

nextflow run nf-core/circrna \
-profile ucl_myriad \
-c ~/my_extra_config.txt \
--input ~/samplesheet.csv \
--outdir ~/Scratch/rnaseq/results/circRNA \
--phenotype ~/phenotype.csv \
--tools 'circexplorer2,ciriquant,find_circ' \
--genome GRCh38 \
--fasta ~/Scratch/GCA_000001405.15_GRCh38_no_alt_analysis_set.fna \
--gtf ~/Scratch/gencode.v47.annotation.gtf \
--mature ~/Scratch/rnaseq/reference_files/mature.fa \
-resume

When it gets the ADD_INTRONS, I get the below:

------------------------------------------------------------------------------
|   Another GFF Analysis Toolkit (AGAT) - Version: v1.4.0                      |
|   https://github.com/NBISweden/AGAT                                          |
|   National Bioinformatics Infrastructure Sweden (NBIS) - www.nbis.se         |
 ------------------------------------------------------------------------------
=> Using standard /usr/local/lib/perl5/site_perl/auto/share/dist/AGAT/agat_config.yaml config file

                                       
                          ------ Start parsing ------                           
-------------------------- parse options and metadata --------------------------
=> Accessing the feature_levels YAML file
Using standard /usr/local/lib/perl5/site_perl/auto/share/dist/AGAT/feature_levels.yaml file
=> Attribute used to group features when no Parent/ID relationship exists (i.e common tag):
	* locus_tag
	* gene_id
=> merge_loci option deactivated
=> Machine information:
	This script is being run by perl v5.32.1
	Bioperl location being used: /usr/local/lib/perl5/site_perl/Bio/
	Operating system being used: linux 
=> Accessing Ontology
	No ontology accessible from the gff file header!
	We use the SOFA ontology distributed with AGAT:
		/usr/local/lib/perl5/site_perl/auto/share/dist/AGAT/so.obo
	Read ontology /usr/local/lib/perl5/site_perl/auto/share/dist/AGAT/so.obo:
		4 root terms, and 2596 total terms, and 1516 leaf terms
	Filtering ontology:
		We found 1861 terms that are sequence_feature or is_a child of it.
--------------------------------- parsing file ---------------------------------
=> Number of line in file: 4026761
=> Number of comment lines: 0
=> Fasta included: No
=> Number of features lines: 4026761
=> Number of feature type (3rd column): 7
	* Level1: 0 => 
	* level2: 1 => transcript
	* level3: 6 => exon start_codon UTR stop_codon CDS Selenocysteine
	* unknown: 0 => 
=>Check because only level3 features:
 * Number of feature with Parent attribute:0
 * Number of feature with a common attribute:4026761
  => Some common attributes and some Parent attributes missing.
  /!\ For features where both are missing A single Level2 features (e.g. mRNA) and a single level1 (e.g. gene) will be created by AGAT, and all level3 feautres (e,g, CDS,exon) will be attached to them. This is probably not what you want...
  see B. 2.2 and 3. at https://agat.readthedocs.io/en/latest/agat_how_does_it_work.html 
  /!\ For features where the common attribute or the parent attribute is missing, it would be fine as long as you do not expect isoforms in your annotation (Eukaryote).  see B. 4. at https://agat.readthedocs.io/en/latest/agat_how_does_it_work.html 
  !! You might try to fix the issue by choosing a common tag attribute to use in order to group the features correctly (parameter locus_tag in agat config file agat_config.yaml (to access it type: agat config --expose)).
=> Version of the Bioperl GFF parser selected by AGAT: 2

and then it starts parsing, gets to 20% and the process gets killed every time at that exact point.

There isn’t much detail on the error just:
ProcessNFCORE_CIRCRNA:CIRCRNA:BSJ_DETECTION:ADD_INTRONS (gtf) terminated with an error exit status (1)
I have tried to increase the memory but it does nothing. This is what I added to my config file:

process {
	withName: ADD_INTRONS {
     time = '24h'
     memory = '100 GB'
     clusterOptions = [ '-l h_vmem=100G',
      '-l mem_free=100G',
      '-l h_rss=100G',
      '-l h_fsize=200G']
    }
}

I’m quite new to this, so I would really appreciate any feedback
Can someone explain what is happening? How do I fix this?

Command used and terminal output

nextflow run nf-core/circrna \
-profile ucl_myriad \
-c ~/my_extra_config.txt \
--input ~/samplesheet.csv \
--outdir ~/Scratch/rnaseq/results/circRNA \
--phenotype ~/phenotype.csv \
--tools 'circexplorer2,ciriquant,find_circ' \
--genome GRCh38 \
--fasta ~/Scratch/GCA_000001405.15_GRCh38_no_alt_analysis_set.fna \
--gtf ~/Scratch/gencode.v47.annotation.gtf \
--mature ~/Scratch/rnaseq/reference_files/mature.fa \
-resume

Relevant files

No response

System information

Next flow version: version 24.10.4
Executor: SGE
Hardware: HPC
Container: Singularity
OS: macOS
nf-core/circrna version: nf-core/circrna v0.0.1dev-gabdc4cb

[mariaUCL]

Update: I managed to solve this issue with the ADD_INTRON by using this gtf instead Homo_sapiens.GRCh38.dna_sm.primary_assembly.fa
I also managed to resolve a similar issue I was having with BWA_INDEX by using the below config file

params {
  config_profile_description = 'University College London Myriad cluster'
  config_profile_contact   = 'Chris Wyatt ([ucbtcdr@ucl.ac.uk](mailto:ucbtcdr@ucl.ac.uk))'
  config_profile_url     = 'https://www.rc.ucl.ac.uk/docs/Clusters/Myriad/'
}
executor {
  name = 'sge'
}
apptainer.runOptions = "-B ${HOME},${PWD}"
process {
  //NEED TO SET PARALLEL ENVIRONMENT TO SMP SO MULTIPLE CPUS CAN BE SUBMITTED
  penv      = 'smp'
  //PROVIDE EXTRA PARAMETERS AS CLUSTER OPTIONS
  clusterOptions = "-S /bin/bash"
}
singularity {
  enabled  = true
  autoMounts = true
  cacheDir  = "${HOME}/Scratch/.apptainer/pull"
}
process {
 withLabel:process_low {
  memory = 64.GB
  cpus = 15
  time = 24.h
 }
}
process {
   withName:BWA_INDEX {
  memory = 64.GB
  cpus = 15
  time = 24.h
 }
}
process {
   withName:STAR_GENOMEGENERATE {
   memory = 64.GB
  cpus = 15
  time = 24.h
 }
}
memory = { check_max( 64.GB * task.attempt, 'memory' ) }
// Function to ensure that resource requirements don't go beyond
// a maximum limit
def check_max(obj, type) {
 if (type == 'memory') {
  try {
   if (obj.compareTo(params.max_memory as nextflow.util.MemoryUnit) == 1)
    return params.max_memory as nextflow.util.MemoryUnit
   else
    return obj
  } catch (all) {
   println "  ### ERROR ###  Max memory '${params.max_memory}' is not valid! Using default value: $obj"
   return obj
  }
 } else if (type == 'time') {
  try {
   if (obj.compareTo(params.max_time as nextflow.util.Duration) == 1)
    return params.max_time as nextflow.util.Duration
   else
    return obj
  } catch (all) {
   println "  ### ERROR ###  Max time '${params.max_time}' is not valid! Using default value: $obj"
   return obj
  }
 } else if (type == 'cpus') {
  try {
   return Math.min( obj, params.max_cpus as int )
  } catch (all) {
   println "  ### ERROR ###  Max cpus '${params.max_cpus}' is not valid! Using default value: $obj"
   return obj
  }
 }
}

But now this job: nf-NFCORE_CIRCRNA_CIRCRNA_PREPARE_GENOME_STAR_GENOMEGENERATE_(Homo_sapiens.GRCh38.dna_sm.primary_assembly.fa) just causes the “pipeline broken” error. Any tips?

[nictru]

Hey, your config looks overly complicated - I would recommend you using the resourceLimits config instead. Also, did you use the latest version of the pipeline and nextflow?

[mariaUCL]

Hi,
I tried the resourceLimits config. This is my new config below. It should be more than enough memory for STAR_GENOMEGENERATE but I still get this error: NOTE: Process NFCORE_CIRCRNA:CIRCRNA:PREPARE_GENOME:STAR_GENOMEGENERATE (Homo_sapiens.GRCh38.dna_sm.primary_assembly.fa) terminated with an error exit status (137) -- Execution is retried (1)

params {
    config_profile_description = 'University College London Myriad cluster'
    config_profile_contact     = 'Chris Wyatt (ucbtcdr@ucl.ac.uk)'
    config_profile_url         = 'https://www.rc.ucl.ac.uk/docs/Clusters/Myriad/'
}

executor {
    name = 'sge'
}

apptainer.runOptions = "-B ${HOME},${PWD}"

nextflow.jvmArgs = '-Xss8M'

process {
    //NEED TO SET PARALLEL ENVIRONMENT TO SMP SO MULTIPLE CPUS CAN BE SUBMITTED
    penv           = 'smp'
    //PROVIDE EXTRA PARAMETERS AS CLUSTER OPTIONS
    clusterOptions = "-S /bin/bash"
}

process {
    resourceLimits = [ cpus: 24, memory: 768.GB, time: 72.h ]
}

I think I am using the latest nextflow version. I am using version 24.10.4. This is my command:

nextflow run nf-core/circrna -r 925eb02043 \
-profile ucl_myriad \
--input ~/samplesheet.csv \
--outdir ~/Scratch/rnaseq/results/circRNA \
--phenotype ~/phenotype.csv \
--tools 'circexplorer2,ciriquant,find_circ' \
--gtf ~/Scratch/gencode.v47.primary_assembly.annotation.gtf \
--fasta ~/Scratch/Homo_sapiens.GRCh38.dna_sm.primary_assembly.fa \
--mature ~/Scratch/rnaseq/reference_files/mature.fa \
-c ~/new_config.txt \
-resume

Any suggestions?

[nictru]

Hey, STAR_GENOMEGENERATE comes with the resource label process_high, which requests up to 72GB of memory on the first attempt. On subsequent attempts, the request is increased to attempt * 72GB.
By default, the maximum number of retries is 1 (so 2 attempts).

terminated with an error exit status (137) -- Execution is retried (1)

What you see is that the first attempt failed, and that the pipeline starts making a second attempt. So the pipeline did encounter an error, but did not fail. The dynamic resource upscaling should tackle this.

If you want to adjust memory requests for the process (one of the two parameter changes will probably be enough):

process {
    withName: STAR_GENOMEGENERATE {
        maxRetries = 5
        memory = { 200.GB * task.attempt }
    }
}

Note that 5*200GB -> 1000GB - this will be capped by the resourceLimits to 768GB

[mariaUCL]

Hi, I will give this a go. I noticed it was trying to retry it with double the CPUs, but on the second try the pipeline broke (no other error message).

[nictru]

This sounds like a different problem then. Could you provide the .nextflow.log file of a run where this happened?

[mariaUCL]

nextflow.log.txt

[nictru]

Okay so my interpretation of the log is:

1. STAR_GENOMEGENERATE is submitted, picked up and fails due to memory issues (yesterday, 13:00)
2. STAR_GENOMEGENERATE is retried and submitted
3. The new task is not picked up
4. Failure due to SIGHUP at 9:25 today

So there are two things that need to be investigated:

1. Why did the task not get picked up?
2. Why did the SIGHUP occur?

For (1): This highly depends on your scheduler. All I can tell you is that you requested 24 cores and 144GB of memory for a maximum of 32 hours. When looking into /lustre/scratch/scratch/zchams6/work/16/f373a4d739c07c569d93f0fd92564e you will find the configuration for your scheduler in .command.run.
Possibly the maximum time that your scheduler allows is below 32h, or something similar.
Another reason could be that the queue was just full.

For (2): This can either be manually triggered, or if your session was automatically closed. If you did not trigger this manually, I recommend using something like tmux to keep the session persistent.