MissingOutputException in snakemake - workflow

I'm trying to run a peak calling tool within a conda environment using snakemake.
The script looks as such (I only added the rows connect to the problem):
rule all:
input:
expand('{project}/{organism}/{mapper}/seacr/{pattern}.auc.threshold.bed', pattern = PATTERN, sample = IDS, organism = config['org'], project = config['project'], mapper = config['mapper']) # SEACR - run the peak calling
rule seacr_run:
input:
IP = '{project}/{organism}/{mapper}/seacr/IP_{PATTERN}.bedgraph',
IgG = '{project}/{organism}/{mapper}/seacr/IgG_{PATTERN}.bedgraph',
output:
bed1 = '{project}/{organism}/{mapper}/seacr/{PATTERN}.auc.threshold.bed',
shell:
'''
bash /fs/home/yeroslaviz/SEACR/SEACR_1.3.sh {input.IP} 0.01 non stringent {output.bed1}
'''
When running the -nps dryrun of the snamemake command I get the correct command printed to STDOUT
> snakemake -nps /fs/pool/pool-bcfngs/scripts/P193.ChipSeq.Snakemake -j 100
...
Building DAG of jobs...
Job counts:
count jobs
1 all
1 seacr_run
2
[Tue Mar 3 13:56:19 2020]
rule seacr_run:
input: P193/Mmu.GrCm38/bowtie2/seacr/IP_H3K4m3.bedgraph, P193/Mmu.GrCm38/bowtie2/seacr/IgG_H3K4m3.bedgraph
output: P193/Mmu.GrCm38/bowtie2/seacr/H3K4m3.auc.threshold.bed
jobid: 22
wildcards: project=P193, organism=Mmu.GrCm38, mapper=bowtie2, PATTERN=H3K4m3
bash /fs/home/yeroslaviz/SEACR/SEACR_1.3.sh P193/Mmu.GrCm38/bowtie2/seacr/IP_H3K4m3.bedgraph 0.01 non stringent P193/Mmu.GrCm38/bowtie2/seacr/H3K4m3.auc.threshold.bed
[Tue Mar 3 13:56:19 2020]
localrule all:
...
Job counts:
count jobs
1 all
1 seacr_run
2
This was a dry-run (flag -n). The order of jobs does not reflect the order of execution.
When running the command above in the command line the tool works without problems. But hwhen I try to run it within the snakemake workflow I get the following error:
Waiting at most 5 seconds for missing files.
MissingOutputException in line 67 of /fs/pool/pool-bcfngs/scripts/P193.ChipSeq.Snakemake:
Missing files after 5 seconds:
P193/Mmu.GrCm38/bowtie2/seacr/H3K4m3.auc.threshold.bed
This might be due to filesystem latency. If that is the case, consider to increase the wait time with --latency-wait.
Shutting down, this might take some time.
Exiting because a job execution failed. Look above for error message
Can anyone explain what is happening?
Thanks

Related

Azure Rest API CI task completes but stuck on non-closing STDIO lingering process

As part of CI pipeline on ADO, I make REST API GET calls to get a list of requirement work items objects and a list of test results object. I sort and match the list and then I do multiple POST call to add information from that list as a attachment of my ADO test item object. Everything is done thought the ADO Matlab plugin task by using system function to execute curl through the command line via Matlab. Everything seem to work, I see the attachment on every test uploaded well and it even prints ''Done'' after all curl POST request to indicate, I'm done with my Matlab script.
I would expect the ADO CI job task to complete and pass to the next task in YAML file. But after running for 5 mins and completing everything, the job task stall and keep running for another 10h(Max ADO pipeline time)
The STDIO streams did not close within 10 seconds of the exit event from process 'C:\agent_work_tasks\RunMATLABCommand_28fdff80-51b4-4b6e-83e1-cfcf3f3b25a6\0.6.3\bin\run_matlab_command.bat'. This may indicate a child process inherited the STDIO streams and has not yet exited.
Any Ideas how to resolve this bug?
What I Tried:
I tried to run my script locally, I saw all my curl outputs via my matlab command window, but I saw that upon completion it opened a empty command prompt window.
looking for similar incidents and tried closing all processes(#4) as indicated here: https://developercommunity.visualstudio.com/t/the-stdio-streams-did-not-close-within-10-seconds/523146
I tried adding pause/wait in matlab script or via system command call to give time for asynchronous process to complete.
I added quit to my matlab script, and I tried using Taskkill/IM cmd.exe to kill all open command windows when my scripts ends.
Both didn't work on the CI pipeline. it still runs forever until i manually stop it or it reaches max time.
Error message & partial log:
2022-04-04T18:39:45.6947848Z curl -u :**ADOPAT** -X POST -H "Content-Type: application/json" -d "{\"stream\":\"***B64 encoded content***\", \"fileName\": \"requirementPath.txt\", \"comment\":\"Testattachmentupload\",\"attachmentType\":\"GeneralAttachment\"}" http://dev.azure.com/{organization}/{project}/_apis/test/Runs/**runID**/Results/**testResultID**/attachments?api-version=6.0-preview
2022-04-04T18:39:46.3140720Z
2022-04-04T18:39:46.3141125Z status =
2022-04-04T18:39:46.3141240Z
2022-04-04T18:39:46.3141367Z 0
2022-04-04T18:39:46.3141523Z
2022-04-04T18:39:46.3141598Z
2022-04-04T18:39:46.3141697Z cmdout =
2022-04-04T18:39:46.3141781Z
2022-04-04T18:39:46.3141992Z ' % Total % Received % Xferd Average Speed Time Time Time Current
2022-04-04T18:39:46.3142311Z Dload Upload Total Spent Left Speed
2022-04-04T18:39:46.3142510Z
2022-04-04T18:39:46.3142760Z 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0
2022-04-04T18:39:46.3143126Z 100 354 100 139 100 215 269 417 --:--:-- --:--:-- --:--:-- 692
2022-04-04T18:39:46.3143506Z 100 354 100 139 100 215 269 416 --:--:-- --:--:-- --:--:-- 692
2022-04-04T18:39:46.3143948Z {"id":**attachementID**,"url":"http://dev.azure.com/{organization}/{project}/_apis/test/Runs/**RunID**/Results/**testResultID**/Attachments/**attachmentID**"}'
2022-04-04T18:39:46.3144224Z
2022-04-04T18:43:06.5720937Z done
2022-04-04T18:43:29.0053832Z ##[debug]Exit code 0 received from tool 'C:\agent\_work\_tasks\RunMATLABCommand_28fdff80-51b4-4b6e-83e1-cfcf3f3b25a6\0.6.3\bin\run_matlab_command.bat'
2022-04-04T18:43:39.0075174Z The STDIO streams did not close within 10 seconds of the exit event from process 'C:\agent\_work\_tasks\RunMATLABCommand_28fdff80-51b4-4b6e-83e1-cfcf3f3b25a6\0.6.3\bin\run_matlab_command.bat'. This may indicate a child process inherited the STDIO streams and has not yet exited.
2022-04-04T18:43:39.0076952Z ##[debug]The STDIO streams did not close within 10 seconds of the exit event from process 'C:\agent\_work\_tasks\RunMATLABCommand_28fdff80-51b4-4b6e-83e1-cfcf3f3b25a6\0.6.3\bin\run_matlab_command.bat'. This may indicate a child process inherited the STDIO streams and has not yet exited.
2022-04-05T04:10:04.2626714Z ##[debug]Re-evaluate condition on job cancellation for step: 'Matlab Rest API Test Update Action'.
2022-04-05T04:10:04.2919553Z ##[error]The operation was canceled.
2022-04-05T04:10:04.2932116Z ##[debug]System.OperationCanceledException: The operation was canceled.

Snakemake: Singularity parameters --home and --bind set by default but disallowed on HPC

I have already posted this as an issue on Github at https://github.com/snakemake/snakemake/issues/279 but haven't got any response yet. I hope to find help here.
Version
I am using the following versions on our HPC cluster:
Snakemake c5.4.4
singularity version 3.5.3
Minimal example
singularity: "docker://bash"
rule test:
shell: "echo test"
Describe the bug
snakemake --use-singularity --debug
returns this message:
Building DAG of jobs...
Pulling singularity image docker://bash.
Using shell: /bin/bash
Provided cores: 1
Rules claiming more threads will be scaled down.
Job counts:
count jobs
1 test
1
[Fri Mar 13 15:59:30 2020]
rule test:
jobid: 0
Activating singularity image /data/nanopore/test/.snakemake/singularity/36b22e49e8a03fd08160e9345dd1034e.simg
FATAL: container creation failed: not mounting user requested home: user bind control is disallowed
[Fri Mar 13 15:59:30 2020]
Error in rule test:
jobid: 0
RuleException:
CalledProcessError in line 4 of /data/nanopore/test/Snakefile:
Command ' singularity exec --home /data/nanopore/test --bind /opt/snakemake/v5.4.4/lib/python3.5/site-packages/snakemake-5.4.4-py3.5.egg:/mnt/snakemake /data/nanopore/test/.snakemake/singularity/36b22e49e8a03fd08160e9345dd1034e.simg bash -c 'set -euo pipefail; echo test'' returned non-zero exit status 255
File "/data/nanopore/test/Snakefile", line 4, in __rule_test
File "/usr/lib/python3.5/concurrent/futures/thread.py", line 55, in run
Shutting down, this might take some time.
Exiting because a job execution failed. Look above for error message
Complete log: /data/nanopore/test/.snakemake/log/2020-03-13T155917.601627.snakemake.log
Apparently, snakemake runs singularity with default values for --home and --bind. These were disallowed by the administrator, however.
Executing
singularity exec --home /data/nanopore/test --bind /opt/snakemake/v5.4.4/lib/python3.5/site-packages/snakemake-5.4.4-py3.5.egg:/mnt/snakemake /data/nanopore/test/.snakemake/singularity/36b22e49e8a03fd08160e9345dd1034e.simg bash -c 'set -euo pipefail;'
returns:
FATAL: container creation failed: not mounting user requested home: user bind control is disallowed
Additional context
Is there a way to disable the Singularity default parameter setting in snakemake? Inside the singularity container the /data directory is still writeable and readable anyway.
Thanks a lot

Running Hive Using Scala As a Script

I was going through the processBuilder API of Scala, In order to run shell commands like we run in Shell Script, I could run few scripts but Having an issue with one type of Hive query execution.
When I run the below commands it was running success fully, but one type of format fails:
Running Shell Command(Successful):
scala> import sys.process._
scala> "ls -lrt /home/cloudera/Desktop".!
total 164
-rwxrwxr-x 1 cloudera cloudera 237 Apr 5 2016 Parcels.desktop
-rwxrwxr-x 1 cloudera cloudera 238 Apr 5 2016 Kerberos.desktop
-rwxrwxr-x 1 cloudera cloudera 259 Apr 5 2016 Express.desktop
Running Hive Query With File Option(Successful):
scala> "hive -f /home/cloudera/hi.hql" !!
warning: there was one feature warning; re-run with -feature for
details
ls: cannot access /usr/lib/spark/lib/spark-assembly-*.jar: No such
file or directory
2017-09-03 23:20:34,392 WARN [main] mapreduce.TableMapReduceUtil: The
hbase-prefix-tree module jar containing PrefixTreeCodec is not
present. Continuing without it.
Logging initialized using configuration in
file:/etc/hive/conf.dist/hive-log4j.properties
OK
Time taken: 0.913 seconds, Fetched: 2 row(s)
res20: String =
"100 Amit 12000 10
101 Allen 22000 20 .
"
Running Hive Query With -e Option(Failed):
If i had the run the below query on the terminal I could run in the below given Format.
bash$ hive -e "select * staging.from employee_canada;"
The problem while running the same query in scala terminal fails because of the double quotes ("") # the Select query. How can I escape those and run successfully. Tried using triple quotes and as well as escape "\" sequence but still failed to execute.
scala> "hive -e select * staging.from employee_canada; "!!
Below is the Error:
FAILED: ParseException line 1:6 cannot recognize input near '<EOF>'
'<EOF>' '<EOF>' in select clause
java.lang.RuntimeException: Nonzero exit value: 64
at scala.sys.package$.error(package.scala:27)
at scala.sys.process.ProcessBuilderImpl$AbstractBuilder.slurp
at scala.sys.process.ProcessBuilderImpl$AbstractBuilder.$bang$bang(
ProcessBuilderImpl.scala:102)
... 32 elided

JMeter Command Line Output

I'm running a JMeter test plan from command line and it's currently outputting something along the lines of:
Created the tree successfully using C:\*****\TestPlan.jmx
Starting the test # Thu Oct 11 10:20:43 EDT 2012 (1349965243947)
Waiting for possible shutdown message on port 4445
Tidying up ... # Thu Oct 11 10:20:46 EDT 2012 (1349965246384)
... end of run
Is there any way to turn off this output and have the plan execute 'silently'?
Found a way to do this, by following this article http://www.robvanderwoude.com/battech_redirection.php
and appending > NUL to the command
jmeter -n -t C:\***\TestPlan.jmx -Jhostname=%1 > NUL

Solaris CPU run queue

Is there a command which can tell me whats in the Solaris run queue?
I can get a count using vmstat, but I need to know what processes/threads are in there.
The run-queue is always changing, so it's almost impossible to get the set of processes in the current run-queue.
That said, you can get an approximation by looking at the STAT (state) field of the process list from ps. When running the command below:
$ ps aux
...the if the STAT field begins with R, then the process is marked RUNNABLE by the kernel, which on most operating systems means that it is in the run-queue. Here's what a runnable process looks like on my machine:
USER PID %CPU %MEM VSZ RSS TT STAT STARTED TIME COMMAND
root 78179 0.0 0.0 599828 480 s003 R+ 7:51AM 0:00.00 ps aux
On solaris, you can also use the prstat command and look at the STATE column. The value run indicates that the process is on the run-queue. (Also note that the value cpuN indicates that the process is currently running on processor N.
For example:
$ prstat -s cpu -n 5
PID USERNAME SIZE RSS STATE PRI NICE TIME CPU PROCESS/NLWP
13974 kincaid 888K 432K run 40 0 36:14.51 67% cpuhog/1
27354 kincaid 2216K 1928K run 31 0 314:48.51 27% server/5
14690 root 136M 46M sleep 59 0 0:00.59 2.3% Xsun/1
14797 kincaid 9192K 7496K sleep 59 0 0:00.10 0.9% dtwm/8
14851 kincaid 24M 14M sleep 48 0 0:00.03 0.3% netscape/1
Total: 97 processes, 190 lwps, load averages: 2.18, 2.15, 2.11
I was about to correct 0xfe answer when I saw you already did it. The run queue is containing theads not processes so the -L option is mandatory with the prstat command if you want to have the number of "state run" lines more or less matching the run queue. Beware that sampling artifacts will probably prevent to get accurate matches.
In any case, if you want to precisely know what processes/threads are sitting in the run queue you'd rather go the dtrace way assuming you are running Solaris 10 or newer.
The whoqueue.d script which might already been in /usr/demo/dtrace directory on your machine will be a good start:
# dtrace -s /usr/demo/dtrace/whoqueue.d
Run queue of length 1:
24349/1 (dtrace)
Run queue of length 3:
0/0 (sched)
0/0 (sched)
0/0 (sched)
Run queue of length 4:
22468/30 (java)
22468/17 (java)
22468/23 (java)
22468/10 (java)
Have a look at this page for details.