Skip to content
New issue

Have a question about this project? # for a free GitHub account to open an issue and contact its maintainers and the community.

#

By clicking “#”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? # to your account

Jump to bottom

Fetus reference fails to build due to a missing file #16

Open
volkansevim opened this issue Mar 24, 2023 · 0 comments
Open

Fetus reference fails to build due to a missing file #16

volkansevim opened this issue Mar 24, 2023 · 0 comments

Comments

@volkansevim
Copy link

Snakelike fails on rule download_asp because this file on AWS is missing: 'https://md-datasets-cache-zipfiles-prod.s3.eu-west-1.amazonaws.com/mbvhhf8m62-2.zip'

Below is the complete log.

The flag 'directory' used in rule all is only valid for outputs, not inputs.
Building DAG of jobs...
Pulling singularity image docker://satijalab/azimuth-references:vitessce.
Pulling singularity image docker://satijalab/azimuth-references:latest.
Using shell: /usr/bin/bash
Provided cores: 8
Rules claiming more threads will be scaled down.
Job stats:
job                               count    min threads    max threads
------------------------------  -------  -------------  -------------
all                                   1              1              1
download_BBI_individual_organs       14              1              1
download_BBI_metadata                 1              1              1
download_BBI_subsampled               1              1              1
download_asp                          1              1              1
download_enge                         1              1              1
download_lind                         1              1              1
export_zarr                           1              1              1
make_BBI_query                        1              1              1
make_BBI_reference                    1              1              1
preprocess_BBI_reference              1              1              1
setup_asp                             1              1              1
setup_enge                            1              1              1
setup_lind                            1              1              1
total                                27              1              1

Select jobs to execute...

[Fri Mar 17 18:53:04 2023]
rule download_BBI_individual_organs:
    output: data/Stomach_gene_count.RDS
    jobid: 18
    reason: Missing output files: data/Stomach_gene_count.RDS
    wildcards: sample=Stomach
    resources: tmpdir=/tmp

Activating singularity image /prj/refs/azimuth-references/human_fetus/.snakemake/singularity/e4faa2430c925e8c50307f5d2609b777.simg

[Fri Mar 17 18:53:04 2023]
rule download_BBI_individual_organs:
    output: data/Liver_gene_count.RDS
    jobid: 11
    reason: Missing output files: data/Liver_gene_count.RDS
    wildcards: sample=Liver
    resources: tmpdir=/tmp

Activating singularity image /prj/refs/azimuth-references/human_fetus/.snakemake/singularity/e4faa2430c925e8c50307f5d2609b777.simg

[Fri Mar 17 18:53:04 2023]
rule download_BBI_subsampled:
    output: data/gene_count_sampled.RDS
    jobid: 3
    reason: Missing output files: data/gene_count_sampled.RDS
    resources: tmpdir=/tmp

Activating singularity image /prj/refs/azimuth-references/human_fetus/.snakemake/singularity/e4faa2430c925e8c50307f5d2609b777.simg

[Fri Mar 17 18:53:04 2023]
rule download_BBI_individual_organs:
    output: data/Thymus_gene_count.RDS
    jobid: 19
    reason: Missing output files: data/Thymus_gene_count.RDS
    wildcards: sample=Thymus
    resources: tmpdir=/tmp

Activating singularity image /prj/refs/azimuth-references/human_fetus/.snakemake/singularity/e4faa2430c925e8c50307f5d2609b777.simg

[Fri Mar 17 18:53:04 2023]
rule download_BBI_metadata:
    output: data/df_cell.RDS, data/df_gene.RDS
    jobid: 4
    reason: Missing output files: data/df_gene.RDS, data/df_cell.RDS
    resources: tmpdir=/tmp

Activating singularity image /prj/refs/azimuth-references/human_fetus/.snakemake/singularity/e4faa2430c925e8c50307f5d2609b777.simg

[Fri Mar 17 18:53:04 2023]
rule download_lind:
    output: data/lind_data/counts.tsv
    jobid: 23
    reason: Missing output files: data/lind_data/counts.tsv
    resources: tmpdir=/tmp

Activating singularity image /prj/refs/azimuth-references/human_fetus/.snakemake/singularity/e4faa2430c925e8c50307f5d2609b777.simg

[Fri Mar 17 18:53:04 2023]
rule download_enge:
    output: data/enge_data, data/enge_data/enge.tar
    jobid: 25
    reason: Missing output files: data/enge_data/enge.tar
    resources: tmpdir=/tmp

Activating singularity image /prj/refs/azimuth-references/human_fetus/.snakemake/singularity/e4faa2430c925e8c50307f5d2609b777.simg

[Fri Mar 17 18:53:04 2023]
rule download_BBI_individual_organs:
    output: data/Heart_gene_count.RDS
    jobid: 10
    reason: Missing output files: data/Heart_gene_count.RDS
    wildcards: sample=Heart
    resources: tmpdir=/tmp

Activating singularity image /prj/refs/azimuth-references/human_fetus/.snakemake/singularity/e4faa2430c925e8c50307f5d2609b777.simg
[Fri Mar 17 18:53:06 2023]
Finished job 18.
1 of 27 steps (4%) done
Select jobs to execute...

[Fri Mar 17 18:53:06 2023]
rule download_BBI_individual_organs:
    output: data/Lung_gene_count.RDS
    jobid: 12
    reason: Missing output files: data/Lung_gene_count.RDS
    wildcards: sample=Lung
    resources: tmpdir=/tmp

Activating singularity image /prj/refs/azimuth-references/human_fetus/.snakemake/singularity/e4faa2430c925e8c50307f5d2609b777.simg
[Fri Mar 17 18:53:08 2023]
Finished job 19.
2 of 27 steps (7%) done
Select jobs to execute...

[Fri Mar 17 18:53:08 2023]
rule download_BBI_individual_organs:
    output: data/Pancreas_gene_count.RDS
    jobid: 14
    reason: Missing output files: data/Pancreas_gene_count.RDS
    wildcards: sample=Pancreas
    resources: tmpdir=/tmp

Activating singularity image /prj/refs/azimuth-references/human_fetus/.snakemake/singularity/e4faa2430c925e8c50307f5d2609b777.simg
[Fri Mar 17 18:53:11 2023]
Finished job 4.
3 of 27 steps (11%) done
Select jobs to execute...

[Fri Mar 17 18:53:11 2023]
rule download_BBI_individual_organs:
    output: data/Cerebrum_gene_count.RDS
    jobid: 8
    reason: Missing output files: data/Cerebrum_gene_count.RDS
    wildcards: sample=Cerebrum
    resources: tmpdir=/tmp

Activating singularity image /prj/refs/azimuth-references/human_fetus/.snakemake/singularity/e4faa2430c925e8c50307f5d2609b777.simg
[Fri Mar 17 18:53:21 2023]
Finished job 12.
4 of 27 steps (15%) done
Select jobs to execute...

[Fri Mar 17 18:53:21 2023]
rule download_BBI_individual_organs:
    output: data/Adrenal_gene_count.RDS
    jobid: 6
    reason: Missing output files: data/Adrenal_gene_count.RDS
    wildcards: sample=Adrenal
    resources: tmpdir=/tmp

Activating singularity image /prj/refs/azimuth-references/human_fetus/.snakemake/singularity/e4faa2430c925e8c50307f5d2609b777.simg
[Fri Mar 17 18:53:30 2023]
Finished job 11.
5 of 27 steps (19%) done
Select jobs to execute...

[Fri Mar 17 18:53:30 2023]
rule download_asp:
    output: data/asp_data/counts_filtered.tsv, data/asp_data/md_filtered.tsv
    jobid: 21
    reason: Missing output files: data/asp_data/md_filtered.tsv, data/asp_data/counts_filtered.tsv
    resources: tmpdir=/tmp

Activating singularity image /prj/refs/azimuth-references/human_fetus/.snakemake/singularity/e4faa2430c925e8c50307f5d2609b777.simg
[Fri Mar 17 18:53:31 2023]
Error in rule download_asp:
    jobid: 21
    output: data/asp_data/counts_filtered.tsv, data/asp_data/md_filtered.tsv
    shell:
        
        mkdir -p data/asp_data
        cd data/asp_data
        wget 'https://md-datasets-cache-zipfiles-prod.s3.eu-west-1.amazonaws.com/mbvhhf8m62-2.zip'
        unzip mbvhhf8m62-2.zip
        unzip Filtered/'Developmental_heart_filtered_scRNA-seq_and_meta_data.zip'
        gunzip share_files/*
        mv share_files/all_cells_count_matrix_filtered.tsv counts_filtered.tsv
        mv share_files/all_cells_meta_data_filtered.tsv md_filtered.tsv
        echo "ASP data downloaded on: $(date)" > ../../logs/download_asp.log
        
        (one of the commands exited with non-zero exit code; note that snakemake uses bash strict mode!)

[Fri Mar 17 18:53:34 2023]
Finished job 10.
6 of 27 steps (22%) done
[Fri Mar 17 18:53:36 2023]
Finished job 3.
7 of 27 steps (26%) done
[Fri Mar 17 18:53:37 2023]
Finished job 14.
8 of 27 steps (30%) done
[Fri Mar 17 18:53:52 2023]
Finished job 6.
9 of 27 steps (33%) done
[Fri Mar 17 18:55:04 2023]
Finished job 8.
10 of 27 steps (37%) done
[Fri Mar 17 18:55:09 2023]
Finished job 23.
11 of 27 steps (41%) done
[Fri Mar 17 18:55:59 2023]
Finished job 25.
12 of 27 steps (44%) done
Shutting down, this might take some time.
Exiting because a job execution failed. Look above for error message
Complete log: .snakemake/log/2023-03-17T184850.349820.snakemake.log
# for free to join this conversation on GitHub. Already have an account? # to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@volkansevim