Consider first script_0.sh:
# script_0.sh
foo
bar
baz
I can run this script via LSF like this, for example:
bsub -q myqueue -J myjob_0 -o path/to/log_0.out -e path/to/log_0.err -- /bin/sh ./script_0.sh
Now consider a second, very similar script:
# script_1.sh
#BSUB -q myqueue
#BSUB -J myjob_1
#BSUB -o path/to/log_1.out
#BSUB -e path/to/log_1.err
foo
bar
baz
I can also run this script, via LSF, as shown above (after changing the command-line options appropriately), but, in addition, I can do it like this:
bsub < script_1.sh
In this second form, bsub figures out the options to use by parsing the lines beginning with #BSUB .
As far as I understand, all the forms described above would entail running exactly the same commands (foo, etc.), since the two scripts are identical, if we exclude comments and blank lines.
What are the pros/cons of this second form vis-à-vis the first form? In particular, does either form make possible job scheduling strategies that are difficult or impossible to implement with the other form?
Related
I want to implement dynamic tab completion of options to a Fish command. This is easy for the -a switch, but I can't figure out how to do it for the -l switch.
Consider the following lines:
$ complete -c foo -a '(echo bar\nbaz\nbiz)' -f
$ complete -c foo -l '(echo bar\nbaz\nbiz)' -f
The behavior of my shell is then as follows:
$ foo b<tab>
bar baz biz
$ foo --<tab>
foo --\(echo\ bar\\nbaz\\nbiz\)
Instead I'd like it to suggest three options --bar, --baz and --biz. Is this possible?
edit: Now I understand better. You can do this by just making your "arguments" start with dashes. Here's an example that uses a function for clarity:
function get_foo_completions
echo --bar
echo --baz
echo --biz
set prev_arg (commandline -pco)[-1]
test "$prev_arg" = print
and echo --conditional
end
complete -c foo -a '(get_foo_completions)' -f
The --conditional argument will only be printed if the previous argument is print which illustrates that these can be dynamic.
To my knowledge this isn't yet possible. Options are declarative, and only arguments to those options may be dynamic.
If you give more details about your use case I might be able to suggest other approaches.
I would like to pass LSB_JOBINDEX to as an argument to my script instead of using an environment variable.
This makes my script more LSF agnostic and avoids creating a helper script that uses the environment variable.
However, I was not able to use LSB_JOBINDEX in arguments: it only works as part of the initial command string.
For example, from a bash shell, I use the test command:
bsub -J 'myjobname[1-4]' -o bsub%I.log \
'echo $LSB_JOBINDEX' \
'$LSB_JOBINDEX' \
\$LSB_JOBINDEX \
'$LSB_JOBINDEX' \
"\$LSB_JOBINDEX"
and the output of say bsub2.log is:
2 $LSB_JOBINDEX $LSB_JOBINDEX $LSB_JOBINDEX $LSB_JOBINDEX
So in this case, only the first $LSB_JOBINDEX got expanded, but not any of the following ones.
But I would rather not pass the entire command as a single huge string as the 'echo $LSB_JOBINDEX' in this example. I would prefer to just use separate arguments as in a regular bash command.
I've also tried to play around with %I but it only works for -o and related bsub options, not for the command itself.
Related: Referencing job index in LSF job array
Tested in LSF 10.1.0. Related documentation: https://www.ibm.com/support/knowledgecenter/en/SSWRJV_10.1.0/lsf_admin/job_array_cl_args.html
bsub will add single quotes around the arguments if the argument starts with $. For example. If the bsub command line is
bsub command -a $ARG1 -b $ARG2
Then bsub will add quotes to the arguments to the 2nd and 4th parameters. The command is stored like this
command -a '$ARG1' -b '$ARG2'
One way to prevent this is to put the commands in a script. Like this:
$ cat cmd
echo $LSB_JOBINDEX
echo "line 2"
echo $LSB_JOBINDEX
Then run your job like this:
$ bsub -I < cmd
Job <2669> is submitted to default queue <normal>.
<<Waiting for dispatch ...>>
<<Starting on hostA>>
0
line 2
0
Note that the -I is not needed. Its just so you can see the job output on the bsub's stdout.
EDIT
OK. Looks like this works. But its not really a serious answer since it's so ugly. The thing is that bsub will surround the argument with single quotes if the argument starts with $. So the strategy is to find some way to make sure that the first character in the argument isn't a $. One way is to put any character other than $ as the first character of the argument. Follow it by a backspace literal, followed by the $. Note that it needs to be the actual backspace character, not ^ followed by H. Use ctrl-v followed by a ctrl-h to get the literal appended to the command line.
$ bsub -I echo "x^H\$LSB_JOBINDEX" "x^H\$LSB_JOBINDEX"
Job <2686> is submitted to default queue <normal>.
<<Waiting for dispatch ...>>
<<Starting on hostA>>
0 0
EDIT2
A tab literal also works. Not that its much better.
$ bsub -I echo " \$LSB_JOBINDEX" " \$LSB_JOBINDEX"
Job <2687> is submitted to default queue <normal>.
<<Waiting for dispatch ...>>
<<Starting on hostA>>
0 0
I am searching for swipl the similar feature as perl -e
In particular, I want to run prolog code in this fashion:
swipl --wanted-flag "fact(a). message:-writeln('hello')." -g "message" -t halt
This is possible to do with
swipl -f file -g "message" -t halt
where the prolog clauses are written in file
I am running swipl on the server side that takes user input as prolog clauses, therefore writing a file on the server is not a good idea.
One thing you can do is to use load_files/2 with the option stream, and load from standard input, not from an argument (you can still pass the entry point as an argument, I guess):
Say in a file fromstdin.pl you have:
main :-
load_files(stdin, [stream(user_input)]),
current_prolog_flag(argv, [Goal|_]),
call(Goal),
halt.
main :- halt(1).
and with this you can do:
$ echo 'message :- format("hello~n").' | swipl -q -t main fromstdin.pl -- message
|: hello
The comments by #false to this answer and the question will tell you what this |: is, if you are wondering, but if it annoys you, just do:
$ echo 'message :- format("hello~n").' \
| swipl -q -t main fromstdin.pl -- message \
| cat
hello
instead.
This will let you read any Prolog from standard input and call an arbitrary predicate from it. Whether this is a clever thing to do, I don't know. I would also not be surprised if there is a much easier way to achieve the same.
I'm looking for a way to log information to a file about a submitted job immediately after it starts.
Normally all the job status is appended to the log file after a job has completed, but I'd like to know the information it has when it starts.
I know there's the -B flag but I want it in a file, and I could also do something like:
bsub -J jobby -o run_job.log bjobs -l -J jobby > jobby.log; run_job
but maybe someone knows of a funkier way of doing this.
There are some subtle variations that essentially accomplish the same thing:
You can use a pre-exec to do a similar thing instead of doing the
bjobs as part of the command:
bsub -J jobby -E "bjobs -l -J jobby > jobby.log" run_job
You can use the job's environment to get your own jobid instead of
using -J if you write your submission as a script:
#!/bin/sh
#BSUB -o run_job.log
bjobs -l $LSB_JOBID > $LSB_JOBID.log
run_job
Then submit your job like this:
bsub < jobscript.sh
You can do some combination of the above: use $LSB_JOBID in a
pre-execution script.
That's about as 'funky' as it gets AFAIK :)
Say I submit a job using something like bsub pwd. Now I would like to get the job ID of that job in order to build a dependency for the next job. Is there some way I can get bsub to return the job ID?
Nils and Andrey have the answers to this specific question in shell and C/C++ environments respectively. For the purposes of building dependencies, you can also name your job with -J then build the dependency based on the job name:
bsub -J "job1" <cmd1>
bsub -J "job2" <cmd2>
bsub -w "done(job1) && done(job2)" <cmd>
There's a bit more info here.
This also works with job arrays:
bsub -J "ArrayA[1-10]" <cmd1>
bsub -J "ArrayB[1-10]" <cmd2>
bsub -w "done(ArrayA[3]) && done(ArrayB[5])" <cmd>
You can even do element-by-element dependency. The following job's i-th element will only run when the corresponding element in ArrayB reaches DONE status:
bsub -w "done(ArrayB[*])" -J "ArrayC[1-10]" <cmd3>
You can find more info on the various things you can specify in -w here.
Just as a reference, this is the best solution I could come up with so far. It takes advantage of the fact that bsub write a line containing the ID to STDOUT.
function nk_jobid {
output=$($*)
echo $output | head -n1 | cut -d'<' -f2 | cut -d'>' -f1
}
Usage:
jobid=$(nk_jobid bsub pwd)
In case you are using C++, you can use the lsblib, LSF C API to submit jobs. The input and the output are structs. In particular, the output struct contains the job id.
#include <lsf/lsbatch.h>
LS_LONG_INT lsb_submit (struct submit *jobSubReq, struct submitReply *jobSubReply)
$jobid = "0"
bsub pwd > $jobid
cat $jobid
If you just want to view the JOBID after submission, most of the time I will just use bhist or bhist -l to view the running jobs and details.
$ bhist
Summary of time in seconds spent in various states:
JOBID USER JOB_NAME PEND PSUSP RUN USUSP SSUSP UNKWN TOTAL
8664 F14r3 sample 2 0 187954 0 0 0 187956