Is it possible to define again rule all in snakemake? - workflow

Question

I run in parallel processing of 8 files (fastq) with snakemake. Then each of these file is demultiplexed and I run in parallel processing of demultiplexed files generated by each of these files with snakemake again.
My first attempt (which is working well) was to use 2 snakefiles.
a snakefile for processing 8 files in parallel
a snakefile for processing generated demultiplexed files in parallel
I would like to use only one snakefile.
Here the solution with 2 snakefiles:
snakefile #1 for processing 8 files (wildcard {run}) in parallel
configfile: "config.yaml"
rule all:
input:
expand("{folder}{run}_R1.fastq.gz", run=config["fastqFiles"],folder=config["fastqFolderPath"]),
expand('assembled/{run}/{run}.fastq', run=config["fastqFiles"]),
expand('assembled/{run}/{run}.ali.fastq', run=config["fastqFiles"]),
expand('assembled/{run}/{run}.ali.assigned.fastq', run=config["fastqFiles"]),
expand('assembled/{run}/{run}.unidentified.fastq', run=config["fastqFiles"]),
expand('log/remove_unaligned/{run}.log',run=config["fastqFiles"]),
expand('log/illuminapairedend/{run}.log',run=config["fastqFiles"]),
expand('log/assign_sequences/{run}.log',run=config["fastqFiles"]),
expand('log/split_sequences/{run}.log',run=config["fastqFiles"])
include: "00-rules/assembly.smk"
include: "00-rules/demultiplex.smk"
snakefile #2 for processing generated demultiplexed files in parallel
SAMPLES, = glob_wildcards('samples/{sample}.fasta')
rule all:
input:
expand('samples/{sample}.uniq.fasta',sample=SAMPLES),
expand('samples/{sample}.l.u.fasta',sample=SAMPLES),
expand('samples/{sample}.r.l.u.fasta',sample=SAMPLES),
expand('samples/{sample}.c.r.l.u.fasta',sample=SAMPLES),
expand('log/dereplicate_samples/{sample}.log',sample=SAMPLES),
expand('log/goodlength_samples/{sample}.log',sample=SAMPLES),
expand('log/clean_pcrerr/{sample}.log',sample=SAMPLES),
expand('log/rm_internal_samples/{sample}.log',sample=SAMPLES)
include: "00-rules/filtering.smk"
This solution is working well.
Is it possible to merge these 2 snakefiles into one this way ?
configfile: "config.yaml"
rule all:
input:
expand("{folder}{run}_R1.fastq.gz", run=config["fastqFiles"],folder=config["fastqFolderPath"]),
expand('assembled/{run}/{run}.fastq', run=config["fastqFiles"]),
expand('assembled/{run}/{run}.ali.fastq', run=config["fastqFiles"]),
expand('assembled/{run}/{run}.ali.assigned.fastq', run=config["fastqFiles"]),
expand('assembled/{run}/{run}.unidentified.fastq', run=config["fastqFiles"]),
expand('log/remove_unaligned/{run}.log',run=config["fastqFiles"]),
expand('log/illuminapairedend/{run}.log',run=config["fastqFiles"]),
expand('log/assign_sequences/{run}.log',run=config["fastqFiles"]),
expand('log/split_sequences/{run}.log',run=config["fastqFiles"])
include: "00-rules/assembly.smk"
include: "00-rules/demultiplex.smk"
SAMPLES, = glob_wildcards('samples/{sample}.fasta')
rule all:
input:
expand('samples/{sample}.uniq.fasta',sample=SAMPLES),
expand('samples/{sample}.l.u.fasta',sample=SAMPLES),
expand('samples/{sample}.r.l.u.fasta',sample=SAMPLES),
expand('samples/{sample}.c.r.l.u.fasta',sample=SAMPLES),
expand('log/dereplicate_samples/{sample}.log',sample=SAMPLES),
expand('log/goodlength_samples/{sample}.log',sample=SAMPLES),
expand('log/clean_pcrerr/{sample}.log',sample=SAMPLES),
expand('log/rm_internal_samples/{sample}.log',sample=SAMPLES)
include: "00-rules/filtering.smk"
So i have to define again rule all.
and i got the following message error:
The name all is already used by another rule
Is they're a way to have many rule all or the solution "using many snakefiles" is the only one possible ?
I would like to use snakemake in the most appropriate way possible.

Answer 1

You are not limited in the naming of the top level rule. You may call it all, or you may rename it: the only thing that matters is the order of their definition. By default Snakemake takes the first rule as the target one and then constructs the graph of dependencies.
Taking that in consideration you have several options. First, you can merge both top-level rules from the workflows into one. At the end of the day your all rules do nothing except of the definition of the target files. Next, you may rename your rules into all1 and all2 (making it possible to run a single workflow if you specify that in command line), and provide the all rule with merged input. Finally you can use subworkflows, but as long as your intention is to squash two scripts into one, that would be an overkill.
One more hint that could help: you don't need to specify the pattern expand('filename{sample}',sample=config["fastqFiles"]) for each file if you define a distinct output for each run. For example:
rule sample:
input:
'samples/{sample}.uniq.fasta',
'samples/{sample}.l.u.fasta',
'samples/{sample}.r.l.u.fasta',
'samples/{sample}.c.r.l.u.fasta',
'log/dereplicate_samples/{sample}.log',
'log/goodlength_samples/{sample}.log',
'log/clean_pcrerr/{sample}.log',
'log/rm_internal_samples/{sample}.log'
output:
temp('flag_sample_{sample}_complete')
In this case the all rule becomes trivial:
rule all:
input: expand('flag_sample_{sample}_complete', sample=SAMPLES)
Or, as I advised before:
rule all:
input: expand('flag_run_{run}_complete', run=config["fastqFiles"]),
input: expand('flag_sample_{sample}_complete', sample=SAMPLES)
rule all1:
input: expand('flag_run_{run}_complete', run=config["fastqFiles"])
rule all2:
input: expand('flag_sample_{sample}_complete', sample=SAMPLES)