I am trying to test the workflow https://github.com/avsastry/modulome-workflow but getting an error. I am using a windows 10 laptop with Debian installed. I followed the instruction of setting up nextflow on windows 10 (https://www.nextflow.io/blog/2021/setup-nextflow-on-windows.html) but got an error when trying the test example
nextflow run main.nf -profile local --organism bacillus_subtilis --metadata ../test/test_metadata.tsv --sequence_dir ../test/sequence_files/ --outdir ../test/nf_results/
from here https://github.com/avsastry/modulome-workflow/tree/main/2_process_data. The output is:
N E X T F L O W ~ version 22.10.2
Launching `main.nf` [sharp_franklin] DSL1 - revision: ef90b5fca3
executor > local (10)
[aa/71ecd5] process > bowtie_build [ 0%] 0 of 1
[7d/feb977] process > gff2bed [100%] 1 of 1 ✔
[e2/1485b4] process > download_fastq (3) [100%] 6 of 6, failed: 6, retries: 3 ✔
[- ] process > stage_fastq_single -
[- ] process > stage_fastq_paired -
[- ] process > trim_galore -
[- ] process > bowtie_align -
[- ] process > sam2bam -
[- ] process > get_read_direction -
[- ] process > featureCounts -
[9b/b1d618] process > multiqc (1) [ 0%] 0 of 1
[8b/b66a85] process > assemble_tpm [ 0%] 0 of 1
[b7/16ad73] NOTE: Process `download_fastq (2)` terminated with an error exit status (3) -- Execution is retried (1)
[96/7fda2e] NOTE: Process `download_fastq (1)` terminated with an error exit status (3) -- Execution is retried (1)
[e8/737f05] NOTE: Process `download_fastq (3)` terminated with an error exit status (3) -- Execution is retried (1)
[2d/659ece] NOTE: Process `download_fastq (2)` terminated with an error exit status (3) -- Error is ignored
[1a/948c92] NOTE: Process `download_fastq (1)` terminated with an error exit status (3) -- Error is ignored
[e2/1485b4] NOTE: Process `download_fastq (3)` terminated with an error exit status (3) -- Error is ignored
Error executing process > 'assemble_tpm'
executor > local (10)
[aa/71ecd5] process > bowtie_build [ 0%] 0 of 1
[7d/feb977] process > gff2bed [100%] 1 of 1 ✔
[e2/1485b4] process > download_fastq (3) [100%] 6 of 6, failed: 6, retries: 3 ✔
[- ] process > stage_fastq_single -
[- ] process > stage_fastq_paired -
[- ] process > trim_galore -
[- ] process > bowtie_align -
[- ] process > sam2bam -
[- ] process > get_read_direction -
[- ] process > featureCounts -
[9b/b1d618] process > multiqc (1) [ 0%] 0 of 1
[8b/b66a85] process > assemble_tpm [100%] 1 of 1, failed: 1 ✘
Execution cancelled -- Finishing pending tasks before exit
[b7/16ad73] NOTE: Process `download_fastq (2)` terminated with an error exit status (3) -- Execution is retried (1)
[96/7fda2e] NOTE: Process `download_fastq (1)` terminated with an error exit status (3) -- Execution is retried (1)
[e8/737f05] NOTE: Process `download_fastq (3)` terminated with an error exit status (3) -- Execution is retried (1)
[2d/659ece] NOTE: Process `download_fastq (2)` terminated with an error exit status (3) -- Error is ignored
[1a/948c92] NOTE: Process `download_fastq (1)` terminated with an error exit status (3) -- Error is ignored
[e2/1485b4] NOTE: Process `download_fastq (3)` terminated with an error exit status (3) -- Error is ignored
Error executing process > 'assemble_tpm'
executor > local (10)
[aa/71ecd5] process > bowtie_build [100%] 1 of 1 ✔
[7d/feb977] process > gff2bed [100%] 1 of 1 ✔
[e2/1485b4] process > download_fastq (3) [100%] 6 of 6, failed: 6, retries: 3 ✔
[- ] process > stage_fastq_single -
[- ] process > stage_fastq_paired -
[- ] process > trim_galore -
[- ] process > bowtie_align -
[- ] process > sam2bam -
[- ] process > get_read_direction -
[- ] process > featureCounts -
[9b/b1d618] process > multiqc (1) [ 0%] 0 of 1
[8b/b66a85] process > assemble_tpm [100%] 1 of 1, failed: 1 ✘
Execution cancelled -- Finishing pending tasks before exit
[b7/16ad73] NOTE: Process `download_fastq (2)` terminated with an error exit status (3) -- Execution is retried (1)
[96/7fda2e] NOTE: Process `download_fastq (1)` terminated with an error exit status (3) -- Execution is retried (1)
executor > local (10)
[aa/71ecd5] process > bowtie_build [100%] 1 of 1 ✔
[7d/feb977] process > gff2bed [100%] 1 of 1 ✔
[e2/1485b4] process > download_fastq (3) [100%] 6 of 6, failed: 6, retries: 3 ✔
[- ] process > stage_fastq_single -
[- ] process > stage_fastq_paired -
[- ] process > trim_galore -
[- ] process > bowtie_align -
[- ] process > sam2bam -
[- ] process > get_read_direction -
[- ] process > featureCounts -
[9b/b1d618] process > multiqc (1) [100%] 1 of 1, failed: 1 ✘
[8b/b66a85] process > assemble_tpm [100%] 1 of 1, failed: 1 ✘
Execution cancelled -- Finishing pending tasks before exit
[b7/16ad73] NOTE: Process `download_fastq (2)` terminated with an error exit status (3) -- Execution is retried (1)
[96/7fda2e] NOTE: Process `download_fastq (1)` terminated with an error exit status (3) -- Execution is retried (1)
[e8/737f05] NOTE: Process `download_fastq (3)` terminated with an error exit status (3) -- Execution is retried (1)
[2d/659ece] NOTE: Process `download_fastq (2)` terminated with an error exit status (3) -- Error is ignored
[1a/948c92] NOTE: Process `download_fastq (1)` terminated with an error exit status (3) -- Error is ignored
[e2/1485b4] NOTE: Process `download_fastq (3)` terminated with an error exit status (3) -- Error is ignored
executor > local (10)
[aa/71ecd5] process > bowtie_build [100%] 1 of 1 ✔
[7d/feb977] process > gff2bed [100%] 1 of 1 ✔
[e2/1485b4] process > download_fastq (3) [100%] 6 of 6, failed: 6, retries: 3 ✔
[- ] process > stage_fastq_single -
[- ] process > stage_fastq_paired -
[- ] process > trim_galore -
[- ] process > bowtie_align -
[- ] process > sam2bam -
[- ] process > get_read_direction -
[- ] process > featureCounts -
[9b/b1d618] process > multiqc (1) [100%] 1 of 1, failed: 1 ✘
[8b/b66a85] process > assemble_tpm [100%] 1 of 1, failed: 1 ✘
Execution cancelled -- Finishing pending tasks before exit
[b7/16ad73] NOTE: Process `download_fastq (2)` terminated with an error exit status (3) -- Execution is retried (1)
[96/7fda2e] NOTE: Process `download_fastq (1)` terminated with an error exit status (3) -- Execution is retried (1)
[e8/737f05] NOTE: Process `download_fastq (3)` terminated with an error exit status (3) -- Execution is retried (1)
[2d/659ece] NOTE: Process `download_fastq (2)` terminated with an error exit status (3) -- Error is ignored
[1a/948c92] NOTE: Process `download_fastq (1)` terminated with an error exit status (3) -- Error is ignored
[e2/1485b4] NOTE: Process `download_fastq (3)` terminated with an error exit status (3) -- Error is ignored
Error executing process > 'assemble_tpm'
Caused by:
Process `assemble_tpm` terminated with an error exit status (1)
Command executed:
assemble_tpm.py -d featureCounts -o .
Command exit status:
1
Command output:
(empty)
Command error:
Traceback (most recent call last):
File "/home/wpi/temp/mycode/modulome-workflow/2_process_data/bin/assemble_tpm.py", line 55, in <module>
main(args.results_dir, args.outdir)
File "/home/wpi/temp/mycode/modulome-workflow/2_process_data/bin/assemble_tpm.py", line 15, in main
for filename in os.listdir(results_dir):
FileNotFoundError: [Errno 2] No such file or directory: 'featureCounts'
Work dir:
/home/wpi/temp/mycode/modulome-workflow/2_process_data/work/8b/b66a85df906013da59bf2940bb15a1
Tip: view the complete command output by changing to the process work dir and entering the command `cat .command.out`
This error has been spotted on two MacBooks and one windows laptop. Could anyone help me on why download_fastq() is having an issue here?
I tried on multiple computers and got the same error. Nextflow seems to work fine as I ran some tests from nextflow.io successfully.
The cause of the error above is obfuscated by the process' errorStrategy, which becomes 'ignore' on the second task attempt1. This is less than ideal. If we clone the repo and run the test command, we can take a look at a dot file called .command.err in the process' working directory to see what was written to stderr. For example:
$ cat work/61/1ac660e79ce6f598ad291dfa1151d1/.command.err
2022-11-17T23:58:33 prefetch.2.9.6 sys: connection failed while opening file within cryptographic module - mbedtls_ssl_handshake returned -9984 ( X509 - Certificate verification failed, e.g. CRL, CA or signature check failed )
2022-11-17T23:58:33 prefetch.2.9.6 sys: mbedtls_ssl_get_verify_result returned 0x4008 ( !! The certificate is not correctly signed by the trusted CA !! The certificate is signed with an unacceptable hash. )
2022-11-17T23:58:33 prefetch.2.9.6 sys: connection failed while opening file within cryptographic module - ktls_handshake failed while accessing '...' from '...'
2022-11-17T23:58:33 prefetch.2.9.6 sys: connection failed while opening file within cryptographic module - Failed to create TLS stream for 'www.ncbi.nlm.nih.gov' (...) from '...'
It looks cryptic, but can just be resolved with a newer version of the SRA toolkit. Since you're using Docker, you can simply provide your own image with the necessary dependencies and then point to it in the appropriate config file. For example, with the following in a file called Dockerfile:
FROM ncbi/sra-tools:3.0.0
RUN apk add --no-cache bash pigz
We can build the image and tag it:
docker build -t pigz-sra-tools .
Then apply the following changes:
diff --git a/2_process_data/conf/docker.config b/2_process_data/conf/docker.config
index 2c84f8e..8570268 100644
--- a/2_process_data/conf/docker.config
+++ b/2_process_data/conf/docker.config
## -6,7 +6,7 ## docker {
process {
withLabel: fastq {
- container = "avsastry/fasterq-dump:1.2"
+ container = "pigz-sra-tools:latest"
}
withLabel: trim_galore {
diff --git a/2_process_data/main.nf b/2_process_data/main.nf
index 72ed754..0ce8be4 100644
--- a/2_process_data/main.nf
+++ b/2_process_data/main.nf
## -220,7 +220,7 ## Channel
process download_fastq {
maxRetries 1
- errorStrategy { task.attempt <= maxRetries ? 'retry' : 'ignore' }
+ errorStrategy 'retry'
label 'fastq'
label 'medium'
Then give it a test run. Results:
N E X T F L O W ~ version 22.04.5
Launching `main.nf` [deadly_venter] DSL1 - revision: f687a666aa
executor > local (22)
[6c/72e3ae] process > bowtie_build [100%] 1 of 1 ✔
[57/659741] process > gff2bed [100%] 1 of 1 ✔
[02/d33f87] process > download_fastq (3) [100%] 3 of 3 ✔
[- ] process > stage_fastq_single -
[- ] process > stage_fastq_paired -
[80/9207fb] process > trim_galore (3) [100%] 3 of 3 ✔
[94/296bf0] process > bowtie_align (3) [100%] 3 of 3 ✔
[59/b735ee] process > sam2bam (3) [100%] 3 of 3 ✔
[b4/d63389] process > get_read_direction (3) [100%] 3 of 3 ✔
[b4/dc83ab] process > featureCounts (3) [100%] 3 of 3 ✔
[b6/9daa2c] process > multiqc (1) [100%] 1 of 1 ✔
[29/cbe58d] process > assemble_tpm [100%] 1 of 1 ✔
Completed at: 18-Nov-2022 11:03:41
Duration : 49m 46s
CPU hours : 0.8
Succeeded : 22
Related
I installing perl-5.28.1 via perlbrew. Some tests fail:
../cpan/Tie-RefHash/t/storable.t ................................... ok
../cpan/Tie-RefHash/t/threaded.t ................................... skipped: -- threads aren't enabled in your perl, or Scalar::Util::weaken is missing
# Failed test 'timelocal year for 1970 1 2 0 0 0'
# at t/Local.t line 109.
# got: '170'
# expected: '70'
# Failed test 'timegm year for 1970 1 2 0 0 0'
# at t/Local.t line 124.
# got: '170'
# expected: '70'
# Looks like you failed 2 tests of 187.
../cpan/Time-Local/t/Local.t .......................................
Dubious, test returned 2 (wstat 512, 0x200)
Failed 2/187 subtests
(less 8 skipped subtests: 177 okay)
../cpan/Time-Piece/t/01base.t ...................................... ok
...
Test Summary Report
-------------------
../cpan/Time-Local/t/Local.t (Wstat: 512 Tests: 187 Failed: 2)
Failed tests: 6, 12
Non-zero exit status: 2
Files=2653, Tests=1157584, 928 wallclock secs (116.85 usr 11.01 sys + 603.60 cusr 55.01 csys = 786.47 CPU)
Result: FAIL
makefile:835: recipe for target 'test_harness' failed
make: *** [test_harness] Error 2
##### Brew Failed #####
How to rerun only this failed test?
To run a single one of perl's test file, you can use
t/TEST ../cpan/Time-Local/t/Local.t
This would be run from the following directory:
cd "${PERLBREW_ROOT:-$HOME/perl5/perlbrew}"
cd build/perl-5.x.y/perl-5.x.y
See the What if make test doesn't work? section of the INSTALL file in the same folder for more on testing perl.
I can't install nodejs using the meta-nodejs library on qemux86-64.
bitbake nodejs gives the following error
Initialising tasks: 100%
|########################################################################################################################################################################|
Time: 0:00:05 Sstate summary: Wanted 7 Found 0 Missed 7 Current 780
(0% match, 99% complete) NOTE: Executing Tasks ERROR:
nodejs-7.10.0-r1.4 do_configure: Execution of
'/home/user/poky/build/tmp/work/core2-64-poky-linux/nodejs/7.10.0-r1.4/temp/run.do_configure.68465'
failed with exit code 127: /usr/bin/env: ‘python’: No such file or
directory WARNING: exit code 127 from a shell command.
ERROR: Logfile of failure stored in:
/home/user/poky/build/tmp/work/core2-64-poky-linux/nodejs/7.10.0-r1.4/temp/log.do_configure.68465
Log data follows: | DEBUG: Executing shell function do_configure |
/usr/bin/env: ‘python’: No such file or directory | WARNING: exit code
127 from a shell command. | ERROR: Execution of
'/home/user/poky/build/tmp/work/core2-64-poky-linux/nodejs/7.10.0-r1.4/temp/run.do_configure.68465'
failed with exit code 127: | /usr/bin/env: ‘python’: No such file or
directory | WARNING: exit code 127 from a shell command. | ERROR: Task
(/home/user/poky/meta-openembedded/meta-nodejs/recipes-devtools/nodejs/nodejs_7.10.0.bb:do_configure)
failed with exit code '1' NOTE: Tasks Summary: Attempted 2022 tasks of
which 2016 didn't need to be rerun and 1 failed.
Summary: 1 task failed:
/home/user/poky/meta-openembedded/meta-nodejs/recipes-devtools/nodejs/nodejs_7.10.0.bb:do_configure
Summary: There was 1 WARNING message shown. Summary: There was 1 ERROR
message shown, returning a non-zero exit code.
I installed python on both the host and on the target
can someone help me?
meta-nodejs is outdated, use nodejs from meta-oe
On job failure (exit code > 0) Rundeck automatically add detailed status informations to the notification attachment:
Failed: NonZeroResultCode: Remote command failed with exit status 1
Execution failed: 3709 in project test_project_1: [Workflow result: , step failures: {1=Dispatch failed on 1 nodes: [host1: NonZeroResultCode: Remote command failed with exit status 1 + {dataContext=MultiDataContextImpl(map={ContextView(step:1, node:host1)=BaseDataContext{{exec={exitCode=0}}}, ContextView(node:host1)=BaseDataContext{{exec={exitCode=0}}}}, base=null)} ]}, Node failures: {host1=[NonZeroResultCode: Remote command failed with exit status 1 + {dataContext=MultiDataContextImpl(map={ContextView(step:1, node:host1)=BaseDataContext{{exec={exitCode=0}}}, ContextView(node:host1)=BaseDataContext{{exec={exitCode=0}}}}, base=null)} ]}, status: failed]
Can this message by disabled / hiden to only send the script output like in attachment on a success job run?
You can force the "exit 0" in your step wrapping it on some inline script like this
#!/bin/bash
touch /root/test 2> /dev/null
if [ $? -eq 1 ]
then
# whatever you want
echo "Successfully created file"
exit 0
else
echo "Could not create file" >&2
exit 1
fi
I have a python script on server_A that connects to server_B via SSH and calls a local rsync command to reset a directory B with a fresh set of files. Then the script on A proceeds to rsync over additional set of files to B. My hope was to run this on a schedule in Rundeck. However, it is erroring on me every time during run with this output. What am I doing wrong?
Remote command failed with exit status 1
Failed: NonZeroResultCode: Remote command failed with exit status 1
Execution failed: 9 in project Test: [Workflow result: , step failures: {1=Dispatch failed on 1 nodes: [server_A: NonZeroResultCode: Remote command failed with exit status 1]}, Node failures: {server_A=[NonZeroResultCode: Remote command failed with exit status 1]}, flow control: Continue, status: failed]
Exit status 1 was returned by the command you called. What are you running?
I have the following piece code (extracted from larger script):
Write-Output "Syncing $directory"
Push-Location $directory
git pull origin $branch
$directoryName = [IO.Path]::GetFileName($directory)
git log -n 1 --pretty=format:"%H %cd %aN%n%B" --date=short > "..\$directoryName.lastcommit.txt"
Pop-Location
Which occasionally (~50% chance) produces an error when run in a Windows Azure WebJob:
[05/06/2014 22:20:43 > e5e3ee: INFO] Syncing D:\home\site\!roslyn-sources\DeclarationExpressions
[05/06/2014 22:20:45 > e5e3ee: ERR ] From https://git01.codeplex.com/roslyn
[05/06/2014 22:20:45 > e5e3ee: ERR ] * branch DeclarationExpressions -> FETCH_HEAD
[05/06/2014 22:20:45 > e5e3ee: INFO] Already up-to-date.
[05/06/2014 22:20:45 > e5e3ee: INFO] [ERROR] Window title cannot be longer than 1023 characters.
[05/06/2014 22:20:45 > e5e3ee: INFO] Returning exit code 1
[05/06/2014 22:20:45 > e5e3ee: SYS INFO] Status changed to Failed
[05/06/2014 22:20:45 > e5e3ee: SYS ERR ] Job failed due to exit code 1
I never get this error when running locally.
What might be the reason?
Maybe the exe you are running are changin the Window Title.
So, instead of call git directly in your powershell script run it with the cmdlet Start-Process