Imagine I have a lsf file as fllows:
#!/bin/sh
#BSUB -J X
#BSUB -o X.out
#BSUB -e X.err
...
Once it is run the output will appear in the current folder.
Now imagine I am in
~/code
I need the files to appear in
../cluster/
basically go one folder back and from there go to folder cluster.
How should I write do it within the lsf file?
You can put any relative or absolute path in #BSUB -[eo] <file>. e.g. #BSUB -e ../cluster/X.err. If using a relative path, its relative to the job CWD. By default the job CWD is the job submission directory, but can be changed by a bunch of different parameters. bjobs -l <jobid> shows the actual CWD.
What happens is that while the job is running, the stdout and stderr goes to a file under LSF_TMPDIR (default is $HOME/.lsbatch). After the job finishes, the contents of those files is copied to the pathnames specified in -[eo]. The copying is done on the execution host.
Related
I've submitted my job by the following command:
bsub -e error.log -o output.log ./myScript.sh
I have a question: why are the output and errors logs available only once the job ended?
Thanks
LSF doesn't steam the output back to the submission host. If the submission host and the execution host have a shared file system, and the JOB_SPOOL_DIR is in that shared file system (the spool directory is $HOME/.lsbatch by default) then you should see the stdout and stderr there. After the job finishes, the files there are copied back to the location specified by bsub.
Check bparams -a | grep JOB_SPOOL_DIR to see if the admin has changed the location of the spool dir. With or without the -o/-e options, while the job is running its stdout/err will be captured in the job's spool directory. When the job is finished, the stdout/stderr is copied to the filenames specified by bsub -o/-e. The location of the files in the spool dir is $JOB_SPOOL_DIR/<jobsubmittime>.<jobid>.out or $JOB_SPOOL_DIR/<jobsubmittime>.<jobid>.err
[user1#beta ~]$ cat log.sh
LINE=1
while :
do
echo "line $LINE"
LINE=$((LINE+1))
sleep 1
done
[user1#beta ~]$ bsub -o output.log -e error.log ./log.sh
Job <930> is submitted to default queue <normal>.
[user1#beta ~]$ tail -f .lsbatch/*.930.out
line 1
line 2
line 3
...
According to the LSF documentation the behaviour is configurable:
If LSB_STDOUT_DIRECT is not set and you use the bsub -o option, the standard output of a job is written to a temporary file and copied to the file you specify after the job finishes.
I have small perl script test.pl. This perl script uses input as YAML file database.yml
To convert these perl script into executable, I run following command shown below.
pp -o -x teste.exe test.pl.
It creates executable but when i try to run this executable it throws error that database.yml file not found in that directory.
How can i ensure that executable includes YAML file while creating the exe of perl script?
Note :- I am using perl verison 5.8 in Unix.
By using the -a switch :
pp -a database.yml -o -x teste.exe test.pl
From pp documentation :
-a, --addfile=FILE|DIR
Add an extra file into the package. If the file is a directory, recursively add all files inside that directory, with links turned into actual files.
By default, files are placed under "/" inside the package with their original names. You may override this by appending the target filename after a ";", like this:
% pp -a "old_filename.txt;new_filename.txt"
% pp -a "old_dirname;new_dirname"
You may specify "-a" multiple times.
I tried '-N' and '--no-clobber' but the only result that I get is to retrieve a new copy of the existing example.exe with number a number added using this synax 'example.exe.1'. This is not what I'd like to get. I just need to download and overwrite the file example.exe in the same folder where I already saved a copy of example.com without that wget verifies if the mine is older or newer respect the on example.exe file already present in my download folder. Do you think is i possible or I need to create a script that delete the example.exe file or maybe something that change his modification date etc?
If you specify the output file using the -O option it will overwrite any existing file.
For example:
wget -O index.html bbc.co.uk
Run multiple times will keep over-writting index.html.
wget doesn't let you overwrite an existing file unless you explicitly name the output file on the command line with option -O.
I'm a bit lazy and I don't want to type the output file name on the command line when it is already known from the downloaded file. Therefore, I use curl like this:
curl -O http://ftp.vim.org/vim/runtime/spell/fr.utf-8.spl
Be careful when downloading files like this from unsafe sites. The above command will write a file named as the connected web site wishes to name it (inside the current directory though). The final name may be hidden through redirections and php scripts or be obfuscated in the URL. You might end up overwriting a file you don't want to overwrite.
And if you ever find a file named ls or any other enticing name in the current directory after using curl that way, refrain from executing the downloaded file. It may be a trojan downloaded from a rogue or corrupted web site!
wget --backups=1 google.com
renames original file with .1 suffix and writes new file to the intended filename.
Not exactly what was requested, but could be handy in some cases.
-c or --continue
From the manual:
If you use ‘-c’ on a non-empty file, and the server does not support
continued downloading, Wget will restart the download from scratch and
overwrite the existing file entirely.
I like the -c option. I started with the man page then the web but I've searched for this several times. Like if you're relaying a webcam so the image needs to always be named image.jpg. Seems like it should be more clear in the man page.
I've been using this for a couple years to download things in the background, sometimes combined with "limit-rate = " in my wgetrc file
while true
do
wget -c -i url.txt && break
echo "Restarting wget"
sleep 2
done
Make a little file called url.txt and paste the file's URL into it. Set this script up in your path or maybe as an alias and run it. It keeps retrying the download until there's no error. Sometimes at the end it gets into a loop displaying
416 Requested Range Not Satisfiable
The file is already fully retrieved; nothing to do.
but that's harmless, just ctrl-c it. I think it's always gotten the file I wanted even if wget runs out of retries or the connection temporarily goes away. I've downloaded things for days at a time with it. A CD image on dialup, yes, always with wget.
My use case involves two different URLs, sometimes the second one doesn't exist, but if it DOES exist, I want it to overwrite the first file.
The problem of using wget -O is that, when the second file DOESN'T exist, it will overwrite the first file with a BLANK file.
So the only way I could find is with an if statement:
--spider checks if a file exists, and returns 0 if it does
--quiet fail quietly, with no output
-nv is quiet, but still reports errors
wget -nv https://example.com/files/file01.png -O file01.png
# quietly check if a different version exists
wget --quiet --spider https://example.com/custom-files/file01.png
if [ $? -eq 0 ] ; then
# A different version exists, so download and overwrite the first
wget -nv https://example.com/custom-files/file01.png -O file01.png
fi
It's verbose, but I found it necessary. I hope this is helpful for someone.
Here is an easy way to get it done with parameter trimming
url=https://example.com/example.exe ; wget -nv $url -O ${url##*/}
Or you can use basename
url=https://example.com/example.exe ; wget -nv $url -O $( basename $url )
For those who do not want to use -O and want to specify the output directory only, the following command can be used.
wget \
--directory-prefix "$dest" \
--backups 0 \
-- "$link"
the first command will download from the source with the wget command
the second command will remove the older file
wget \
--directory-prefix "$dest" \
--backups 0 \
-- "$link"; \
rm '$file.1' -f;
I am installing a program which has a file "drown.bin" (Bourne shell script text executable).
When I execute this file as part of the process, it gives error
gzip: stdin: unexpected end of file
tar: Child returned status 1
tar: Error exit delayed from previous errors
Below are file contents (pasted only Bash portion, rest is machine language)
dir_tmp=/tmp/.$(date +"%y%m%d%H%M%S")
mkdir /$dir_tmp >/dev/null
sed -n -e '1,/^exit 0$/!p' $0 > "${dir_tmp}/.make-3000.tar.gz" 2>/dev/null
cd $dir_tmp >/dev/null
tar xvfz .make*.tar.gz >/dev/null
./.make
rm -rf $dir_tmp >/dev/null
exit 0
Can someone please advise what goes wrong in "sed" command to create a corrupted .tar.gz file. I already tried 3 systems with different CentOS versions.
It's not the sed command that fails, but the tar command.
This is a "self-extracting" tar file. The script that sits in front attempts to unpack the rest of the file, starting after the line exit 0. Likely the rest of the file is somehow corrupt.
If you downloaded it, try that again. If you copied it from somewhere else (especially FTP) make sure you used binary mode.
If that didn't work, what you could try to do:
copy the script-file to a file with extension .tgz, e.g. cp drown.bin mycopy.tgz
edit the copy, removing all script lines up to and including the exit 0 line. The file now should be a pure tar.gz file.
on the command line, do a tar tzvf mycopy.tgz to see the contents. Try tar xzvf mycopy.tgz to actually unpack. This likely will fail with just the same error you got first, but at least now you can see the extracted content, at least the part that didn't fail.
I have a relative path
$base_path = "input/myMock.TGZ";
myMock.TGZ is the file name located in input folder.
The filename can change. But the path is always stored in $base_path.
I need to check if the file exists in $base_path.
Test whether something exists at given path using the -e file-test operator.
print "$base_path exists!\n" if -e $base_path;
However, this test is probably broader than you intend. The code above will generate output if a plain file exists at that path, but it will also fire for a directory, a named pipe, a symlink, or a more exotic possibility. See the documentation for details.
Given the extension of .TGZ in your question, it seems that you expect a plain file rather than the alternatives. The -f file-test operator asks whether a path leads to a plain file.
print "$base_path is a plain file!\n" if -f $base_path;
The perlfunc documentation covers the long list of Perl's file-test operators that covers many situations you will encounter in practice.
-r
File is readable by effective uid/gid.
-w
File is writable by effective uid/gid.
-x
File is executable by effective uid/gid.
-o
File is owned by effective uid.
-R
File is readable by real uid/gid.
-W
File is writable by real uid/gid.
-X
File is executable by real uid/gid.
-O
File is owned by real uid.
-e
File exists.
-z
File has zero size (is empty).
-s
File has nonzero size (returns size in bytes).
-f
File is a plain file.
-d
File is a directory.
-l
File is a symbolic link (false if symlinks aren’t supported by the file system).
-p
File is a named pipe (FIFO), or Filehandle is a pipe.
-S
File is a socket.
-b
File is a block special file.
-c
File is a character special file.
-t
Filehandle is opened to a tty.
-u
File has setuid bit set.
-g
File has setgid bit set.
-k
File has sticky bit set.
-T
File is an ASCII or UTF-8 text file (heuristic guess).
-B
File is a “binary” file (opposite of -T).
-M
Script start time minus file modification time, in days.
-A
Same for access time.
-C
Same for inode change time (Unix, may differ for other platforms)
You might want a variant of exists ... perldoc -f "-f"
-X FILEHANDLE
-X EXPR
-X DIRHANDLE
-X A file test, where X is one of the letters listed below. This unary operator takes one argument,
either a filename, a filehandle, or a dirhandle, and tests the associated file to see if something is
true about it. If the argument is omitted, tests $_, except for "-t", which tests STDIN. Unless
otherwise documented, it returns 1 for true and '' for false, or the undefined value if the file
doesn’t exist. Despite the funny names, precedence is the same as any other named unary operator.
The operator may be any of:
-r File is readable by effective uid/gid.
-w File is writable by effective uid/gid.
-x File is executable by effective uid/gid.
-o File is owned by effective uid.
-R File is readable by real uid/gid.
-W File is writable by real uid/gid.
-X File is executable by real uid/gid.
-O File is owned by real uid.
-e File exists.
-z File has zero size (is empty).
-s File has nonzero size (returns size in bytes).
-f File is a plain file.
-d File is a directory.
-l File is a symbolic link.
-p File is a named pipe (FIFO), or Filehandle is a pipe.
-S File is a socket.
-b File is a block special file.
-c File is a character special file.
-t Filehandle is opened to a tty.
-u File has setuid bit set.
-g File has setgid bit set.
-k File has sticky bit set.
-T File is an ASCII text file (heuristic guess).
-B File is a "binary" file (opposite of -T).
-M Script start time minus file modification time, in days.
if (-e $base_path)
{
# code
}
-e is the 'existence' operator in Perl.
You can check permissions and other attributes using the code on this page.
Use:
if (-f $filePath)
{
# code
}
-e returns true even if the file is a directory. -f will only return true if it's an actual file
You can use: if(-e $base_path)
if(-e $base_path)
{
print "Something";
}
would do the trick.
#!/usr/bin/perl -w
$fileToLocate = '/whatever/path/for/file/you/are/searching/MyFile.txt';
if (-e $fileToLocate) {
print "File is present";
}
Use the below code. Here -f checks if it's a file or not:
print "File $base_path is exists!\n" if -f $base_path;