POSIX shell (sh) redirect stderr to stdout and capture stderr and stdout into variables - sh

I would like to redirect stderr to stdout so a terminal prints both of them during a command execution but I would like also to capture both of them into separate variables. I managed to achieve that in Bash (version 4.4.20(1)-release):
#!/bin/bash
echo "terminal:"
{ err="$(find / -maxdepth 2 -iname 'tmp' -type d 2>&1 1>&3 3>&- | tee /dev/stderr)"; ec="$?"; } 3>&1 | tee /dev/fd/4 2>&1; out=$(cat /dev/fd/4)
echo "stdout:" && echo "$out"
echo "stderr:" && echo "$err"
that gives desired:
terminal:
find: ‘/root’: Permission denied
/tmp
/var/tmp
find: ‘/lost+found’: Permission denied
stdout:
/tmp
/var/tmp
stderr:
find: ‘/root’: Permission denied
find: ‘/lost+found’: Permission denied
but I have a problem converting that script into POSIX sh /bin/sh
#!/bin/sh
echo "terminal:"
{ err="$(find / -maxdepth 2 -iname 'tmp' -type d 2>&1 1>&3 3>&- | tee /dev/stderr)"; ec="$?"; } 3>&1 | tee /dev/fd/4 2>&1; out=$(cat /dev/fd/4)
echo "stdout:" && echo "$out"
echo "stderr:" && echo "$err"
gives:
terminal:
tee: /dev/fd/4: No such file or directory
find: ‘/root’: Permission denied
/tmp
/var/tmp
find: ‘/lost+found’: Permission denied
cat: /dev/fd/4: No such file or directory
stdout:
stderr:
/dev/fd/4 does not exist, and there is no /proc/self/fd/4 either.
How to make that script working as a POSIX shell script?

Well I'm doing this in busybox's ash shell and my options are rather limited, but this is what I came up with that works.
tmpfile=/tmp/some_prefix_stderr.$$
out=$(my_cmd) 2>$tmpfile
result=$?
err=$(cat $tmpfile)
rm $tmpfile
It's not the prettiest, but it works. In general, replace some_prefix with something specific to help avoid collisions, but just having the PID on there will solve that unless you are running multiple threads.
If you can rely upon your program exiting non-zero when there's something to show in stderr, you can simplify it a little:
tmpfile=/tmp/some_prefix_stderr.$$
if ! out=$(my_cmd) 2>$tmpfile; then
# specific behavior here
cat $tmpfile >&2
fi
rm $tmpfile
This is all POSIX compliant. Just be sure to clean up your temp file.

Related

How can I check whether a piped content is text with perl

I've written a svn-hook for text files. The content test looks like this:
svnlook cat -t $txn $repos $file 2>/dev/null | file - | egrep -q 'text$'
and I was wondering if this could be done with Perl. However something like this doesn't work:
svnlook cat -t $txn $repos $file 2>/dev/null | perl -wnl -e '-T' -
I'm testing the exit status of this invocation ($?) to see if the given file was text or binary. Since I'm getting the content out of svn. I can't use perl's normal file check.
I've done a simulation with the file program and perl with a text and binary file (text.txt, icon.png):
find -type f | xargs -i /bin/bash -c 'if $(cat {} | file - | egrep -q "text$"); then echo "{}: text"; else echo "{}: binary"; fi'
./text.txt: text
./icons.png: binary
find -type f | xargs -i /bin/bash -c 'if $(cat {} | perl -wln -e "-T;"); then echo "{}: text"; else echo "{}: binary"; fi'
./text.txt: text
./icons.png: text
You're testing perl's exit code, but you never set it. You need
perl -le'exit(-T STDIN ?0:1)' < file

tail and grep log and mail (linux)

i want to tail log file with grep and sent it via mail
like:
tail -f /var/log/foo.log | grep error | mail -s subject name#example.com
how can i do this?
You want to send an email when emailing errors occur? That might fail ;)
You can however try something like this:
tail -f $log |
grep --line-buffered error |
while read line
do
echo "$line" | mail -s subject "$email"
done
Which for every line in the grep output sends an email.
Run above shell script with
nohup ./monitor.sh &
so it will keep running in the background.
I'll have a go at this. Perhaps I'll learn something if my icky bash code gets scrutinised. There is a chance there are already a gazillion solutions to do this, but I am not going to find out, as I am sure you have trawled the depths and widths of the cyberocean. It sounds like what you want can be separated into two bits: 1) at regular intervals obtain the 'latest tail' of the file, 2) if the latest tail actually exists, send it by e-mail. For the regular intervals in 1), use cron. For obtaining the latest tail in 2), you'll have to keep track of the file size. The bash script below does that - it's a solution to 2) that can be invoked by cron. It uses the cached file size to compute the chunk of the file it needs to mail. Note that for a file myfile another file .offset.myfile is created. Also, the script does not allow path components in the file name. Rewrite, or fix it in the invocation [e.g. (cd /foo/bar && segtail.sh zut), assuming it is called segtail.sh ].
#!/usr/local/bin/bash
file=$1
size=0
offset=0
if [[ $file =~ / ]]; then
echo "$0 does not accept path components in the file name" 2>&1
exit 1
fi
if [[ -e .offset.$file ]]; then
offset=$(<".offset.$file")
fi
if [[ -e $file ]]; then
size=$(stat -c "%s" "$file") # this assumes GNU stat, possibly present as gstat. CHECK!
# (gstat can also be Ganglias Status tool - careful).
fi
if (( $size < $offset )); then # file might have been reduced in size
echo "reset offset to zero" 2>&1
offset=0
fi
echo $size > ".offset.$file"
if [[ -e $file && $size -gt $offset ]]; then
tail -c +$(($offset+1)) "$file" | head -c $(($size - $offset)) | mail -s "tail $file" foo#bar
fi
How about:
mail -s "catalina.out errors" blah#myaddress.com < grep ERROR catalina.out

Converting relative path into absolute path?

I'm not sure if these paths are duplicates. Given the relative path, how do I determine absolute path using a shell script?
Example:
relative path: /x/y/../../a/b/z/../c/d
absolute path: /a/b/c/d
The most reliable method I've come across in unix is readlink -f:
$ readlink -f /x/y/../../a/b/z/../c/d
/a/b/c/d
A couple caveats:
This also has the side-effect of resolving all symlinks. This may or may not be desirable, but usually is.
readlink will give a blank result if you reference a non-existant directory. If you want to support non-existant paths, use readlink -m instead. Unfortunately this option doesn't exist on versions of readlink released before ~2005.
From this source comes:
#!/bin/bash
# Assume parameter passed in is a relative path to a directory.
# For brevity, we won't do argument type or length checking.
ABS_PATH=`cd "$1"; pwd` # double quotes for paths that contain spaces etc...
echo "Absolute path: $ABS_PATH"
You can also do a Perl one-liner, e.g. using Cwd::abs_path
Using bash
# Directory
relative_dir="folder/subfolder/"
absolute_dir="$( cd "$relative_dir" && pwd )"
# File
relative_file="folder/subfolder/file"
absolute_file="$( cd "${relative_file%/*}" && pwd )"/"${relative_file##*/}"
${relative_file%/*} is same result as dirname "$relative_file"
${relative_file##*/} is same result as basename "$relative_file"
Caveats: Does not resolve symbolic links (i.e. does not canonicalize path ) => May not differentiate all duplicates if you use symbolic links.
Using realpath
Command realpath does the job. An alternative is to use readlink -e (or readlink -f). However realpath is not often installed by default. If you cannot be sure realpath or readlink is present, you can substitute it using perl (see below).
Using perl
Steven Kramer proposes a shell alias if realpath is not available in your system:
$ alias realpath="perl -MCwd -e 'print Cwd::realpath(\$ARGV[0]),qq<\n>'"
$ realpath path/folder/file
/home/user/absolute/path/folder/file
or if you prefer using directly perl:
$ perl -MCwd -e 'print Cwd::realpath($ARGV[0]),qq<\n>' path/folder/file
/home/user/absolute/path/folder/file
This one-line perl command uses Cwd::realpath. There are in fact three perl functions. They take a single argument and return the absolute pathname. Below details are from documentation Perl5 > Core modules > Cwd.
abs_path() uses the same algorithm as getcwd(). Symbolic links and relative-path components (. and ..) are resolved to return the canonical pathname, just like realpath.
use Cwd 'abs_path';
my $abs_path = abs_path($file);
realpath() is a synonym for abs_path()
use Cwd 'realpath';
my $abs_path = realpath($file);
fast_abs_path() is a more dangerous, but potentially faster version of abs_path()
use Cwd 'fast_abs_path';
my $abs_path = fast_abs_path($file);
These functions are exported only on request => therefore use Cwd to avoid the "Undefined subroutine" error as pointed out by arielf. If you want to import all these three functions, you can use a single use Cwd line:
use Cwd qw(abs_path realpath fast_abs_path);
Take a look at 'realpath'.
$ realpath
usage: realpath [-q] path [...]
$ realpath ../../../../../
/data/home
Since I've run into this many times over the years, and this time around I needed a pure bash portable version that I could use on OSX and linux, I went ahead and wrote one:
The living version lives here:
https://github.com/keen99/shell-functions/tree/master/resolve_path
but for the sake of SO, here's the current version (I feel it's well tested..but I'm open to feedback!)
Might not be difficult to make it work for plain bourne shell (sh), but I didn't try...I like $FUNCNAME too much. :)
#!/bin/bash
resolve_path() {
#I'm bash only, please!
# usage: resolve_path <a file or directory>
# follows symlinks and relative paths, returns a full real path
#
local owd="$PWD"
#echo "$FUNCNAME for $1" >&2
local opath="$1"
local npath=""
local obase=$(basename "$opath")
local odir=$(dirname "$opath")
if [[ -L "$opath" ]]
then
#it's a link.
#file or directory, we want to cd into it's dir
cd $odir
#then extract where the link points.
npath=$(readlink "$obase")
#have to -L BEFORE we -f, because -f includes -L :(
if [[ -L $npath ]]
then
#the link points to another symlink, so go follow that.
resolve_path "$npath"
#and finish out early, we're done.
return $?
#done
elif [[ -f $npath ]]
#the link points to a file.
then
#get the dir for the new file
nbase=$(basename $npath)
npath=$(dirname $npath)
cd "$npath"
ndir=$(pwd -P)
retval=0
#done
elif [[ -d $npath ]]
then
#the link points to a directory.
cd "$npath"
ndir=$(pwd -P)
retval=0
#done
else
echo "$FUNCNAME: ERROR: unknown condition inside link!!" >&2
echo "opath [[ $opath ]]" >&2
echo "npath [[ $npath ]]" >&2
return 1
fi
else
if ! [[ -e "$opath" ]]
then
echo "$FUNCNAME: $opath: No such file or directory" >&2
return 1
#and break early
elif [[ -d "$opath" ]]
then
cd "$opath"
ndir=$(pwd -P)
retval=0
#done
elif [[ -f "$opath" ]]
then
cd $odir
ndir=$(pwd -P)
nbase=$(basename "$opath")
retval=0
#done
else
echo "$FUNCNAME: ERROR: unknown condition outside link!!" >&2
echo "opath [[ $opath ]]" >&2
return 1
fi
fi
#now assemble our output
echo -n "$ndir"
if [[ "x${nbase:=}" != "x" ]]
then
echo "/$nbase"
else
echo
fi
#now return to where we were
cd "$owd"
return $retval
}
here's a classic example, thanks to brew:
%% ls -l `which mvn`
lrwxr-xr-x 1 draistrick 502 29 Dec 17 10:50 /usr/local/bin/mvn# -> ../Cellar/maven/3.2.3/bin/mvn
use this function and it will return the -real- path:
%% cat test.sh
#!/bin/bash
. resolve_path.inc
echo
echo "relative symlinked path:"
which mvn
echo
echo "and the real path:"
resolve_path `which mvn`
%% test.sh
relative symlinked path:
/usr/local/bin/mvn
and the real path:
/usr/local/Cellar/maven/3.2.3/libexec/bin/mvn
I wanted to use realpath but it is not available on my system (macOS), so I came up with this script:
#!/bin/sh
# NAME
# absolute_path.sh -- convert relative path into absolute path
#
# SYNOPSYS
# absolute_path.sh ../relative/path/to/file
echo "$(cd $(dirname $1); pwd)/$(basename $1)"
Example:
./absolute_path.sh ../styles/academy-of-management-review.csl
/Users/doej/GitHub/styles/academy-of-management-review.csl
May be this helps:
$path = "~user/dir/../file"
$resolvedPath = glob($path); # (To resolve paths with '~')
# Since glob does not resolve relative path, we use abs_path
$absPath = abs_path($path);

How to redirect stdout and stderr from csh script

For the following script
install.csh:
#!/bin/csh -f
tar -zxf Python-3.1.1.tgz
cd Python-3.1.1
./configure
make
make install
cd ..
rm -rf Python-3.1.1
Intended use:
./install.csh |& tee install.log
How can I change the script so that I still get a install.log and the output on console without asking the user to do the redirecting?
Some simple solutions:
Solution 1:
tee every line you want to log independently, make use of -a switch of tee to append
#!/bin/csh -f
tar -zxf Python-3.1.1.tgz |& tee -a install.log
cd Python-3.1.1 |& tee -a install.log
./configure |& tee -a install.log
make |& tee -a install.log
make install |& tee -a install.log
cd .. |& tee -a install.log
rm -rf Python-3.1.1 |& tee -a install.log
Solution 2: Add a second script.
For example, rename current install.csh to install_commands,
then add a new install.csh script:
#!/bin/csh -f
/bin/csh install_commands |& tee install.log
G'day,
I highly recommend moving away from csh towards something like bash or zsh.
stdio manipulation is not possible in csh. Have a read of "csh programming considered harmful". An elegant treatise on this topic.
Sorry it's not a direct answer but you'll find that you'll keep banging your head against the constraints of csh the longer you stick with it.
A lot of csh syntax is already available in bash so your learning curve won't be too steep.
Here's a quick suggestion for the same thing written in bash. It's not elegant though.
#!/bin/bash
TO_LOGFILE= "| tee -a ./install.log"
tar -zxf Python-3.1.1.tgz 2>&1 ${TO_LOGFILE}
if [ $? -ne 0 ];then
echo "Untar of Python failed. Exiting..."; exit 5
fi
cd Python-3.1.1 2>&1 ${TO_LOGFILE}
if [ $? -ne 0 ];then
echo "Can't change into Python dir. Exiting..."; exit 5
fi
echo "============== configure ================"
./configure 2>&1 ${TO_LOGFILE}
if [ $? -ne 0 ];then
echo "Configure failed. Exiting..."; exit 5
fi
echo "================ make ==================="
make 2>&1 ${TO_LOGFILE}
if [ $? -ne 0 ];then
echo "Compile of Python failed. Exiting..."; exit 5
fi
echo "================ install ================"
make install 2>&1 ${TO_LOGFILE}
if [ $? -ne 0 ];then
echo "Install of Python failed. Exiting..."; exit 5
fi
cd ..
rm -rf Python-3.1.1 2>&1 ${TO_LOGFILE}
exit 0
I've added a bit more checking and reporting so that if there's a problem in an earlier step the log file will just contain up until the error was uncovered rather than a stack of pretty useless error messages from the later phases that wouldn't complete anyway.
cheers,
You can run it in a subshell and redirect all the output of that. Don't remember if this works in csh, it has been a long, long time since I used that.
#!/bin/csh -f
(
tar -zxf Python-3.1.1.tgz
cd Python-3.1.1
./configure
make
make install
cd ..
rm -rf Python-3.1.1
) |& tee install.log

Is `xargs -t` output stderr or stdout, and can you control it?

say i have a directory with hi.txt and blah.txt and i execute the following command on a linux-ish command line
ls *.* | xargs -t -i{} echo {}
the output you will see is
echo blah.txt
blah.txt
echo hi.txt
hi.txt
i'd like to redirect the stderr output (say 'echo blah.txt' fails...), leaving only the output from the xargs -t command written to std out, but it looks as if it's stderr as well.
ls *.* | xargs -t -i{} echo {} 2> /dev/null
Is there a way to control it, to make it output to stdout?
Use:
ls | xargs -t -i{} echo {} 2>&1 >/dev/null
The 2>&1 sends the standard error from xargs to where standard output is currently going; the >/dev/null sends the original standard output to /dev/null. So, the net result is that standard output contains the echo commands, and /dev/null contains the file names. We can debate about spaces in file names and whether it would be easier to use a sed script to put 'echo' at the front of each line (with no -t option), or whether you could use:
ls | xargs -i{} echo echo {}
(Tested: Solaris 10, Korn Shell ; should work on other shells and Unix platforms.)
If you don't mind seeing the inner workings of the commands, I did manage to segregate the error output from xargs and the error output of the command executed.
al * zzz | xargs -t 2>/tmp/xargs.stderr -i{} ksh -c "ls -dl {} 2>&1"
The (non-standard) command al lists its arguments one per line:
for arg in "$#"; do echo "$arg"; done
The first redirection (2>/tmp/xargs.stderr) sends the error output from xargs to the file /tmp/xargs.stderr. The command executed is 'ksh -c "ls -dl {} 2>&1"', which uses the Korn shell to run ls -ld on the file name with any error output going to standard output.
The output in /tmp/xargs.stderr looks like:
ksh -c ls -dl x1 2>&1
ksh -c ls -dl x2 2>&1
ksh -c ls -dl xxx 2>&1
ksh -c ls -dl zzz 2>&1
I used 'ls -ld' in place of echo to ensure I was testing errors - the files x1, x2, and xxx existed, but zzz does not.
The output on standard output looked like:
-rw-r--r-- 1 jleffler rd 1020 May 9 13:05 x1
-rw-r--r-- 1 jleffler rd 1069 May 9 13:07 x2
-rw-r--r-- 1 jleffler rd 87 May 9 20:42 xxx
zzz: No such file or directory
When run without the command wrapped in 'ksh -c "..."', the I/O redirection was passed as an argument to the command ('ls -ld'), and it therefore reported that it could not find the file '2>&1'. That is, xargs did not itself use the shell to do the I/O redirection.
It would be possible to arrange for various other redirections, but the basic problem is that xargs makes no provision for separating its own error output from that of the commands it executes, so it is hard to do.
The other rather obvious option is to use xargs to write a shell script, and then have the shell execute it. This is the option I showed before:
ls | xargs -i{} echo echo {} >/tmp/new.script
You can then see the commands with:
cat /tmp/new.script
You can run the commands to discard the errors with:
sh /tmp/new.script 2>/dev/null
And, if you don't want to see the standard output from the commands either, append 1>&2 to the end of the command.
So I believe what you want is to have as stdout is
the stdout from the utility that xargs executes
the listing of commands generated by xargs -t
You want to ignore the stderr stream generated by the
executed utility.
Please correct me if I'm wrong.
First, let's create a better testing utility:
% cat myecho
#!/bin/sh
echo STDOUT $#
echo STDERR $# 1>&2
% chmod +x myecho
% ./myecho hello world
STDOUT hello world
STDERR hello world
% ./myecho hello world >/dev/null
STDERR hello world
% ./myecho hello world 2>/dev/null
STDOUT hello world
%
So now we have something that actually outputs to both stdout and stderr, so we
can be sure we're only getting what we want.
A tangential way to do this is not to use xargs, but rather, make. Echoing a command
and then doing it is kind of what make does. That's its bag.
% cat Makefile
all: $(shell ls *.*)
$(shell ls): .FORCE
./myecho $# 2>/dev/null
.FORCE:
% make
./myecho blah.txt 2>/dev/null
STDOUT blah.txt
./myecho hi.txt 2>/dev/null
STDOUT hi.txt
% make >/dev/null
%
If you're tied to using xargs, then you need to modify your utility that
xargs uses so it surpresses stderr. Then you can use the 2>&1 trick others
have mentioned to move the command listing generated by xargs -t from stderr
to stdout.
% cat myecho2
#!/bin/sh
./myecho $# 2>/dev/null
% chmod +x myecho2
% ./myecho2 hello world
STDOUT hello world
% ls *.* | xargs -t -i{} ./myecho2 {} 2>&1
./myecho blah.txt 2>/dev/null
STDOUT blah.txt
./myecho hi.txt 2>/dev/null
STDOUT hi.txt
% ls *.* | xargs -t -i{} ./myecho2 {} 2>&1 | tee >/dev/null
%
So this approach works, and collapses everything you want to stdout (leaving out what you don't want).
If you find yourself doing this a lot, you can write a general utility to surpress stderr:
% cat surpress_stderr
#!/bin/sh
$# 2>/dev/null
% ./surpress_stderr ./myecho hello world
STDOUT hello world
% ls *.* | xargs -t -i{} ./surpress_stderr ./myecho {} 2>&1
./surpress_stderr ./myecho blah.txt 2>/dev/null
STDOUT blah.txt
./surpress_stderr ./myecho hi.txt 2>/dev/null
STDOUT hi.txt
%
xargs -t echos the commands to be executed to stderr before executing them. If you want them to instead echo to stderr, you can pipe stderr to stdout with the 2>&1 construct:
ls *.* | xargs -t -i{} echo {} 2>&1
It looks like xargs -t goes to stderr, and there's not much you can do about it.
You could do:
ls | xargs -t -i{} echo "Foo: {}" >stderr.txt | tee stderr.txt
to display only the stderr data on your terminal as your command runs, and then grep through stderr.txt after to see if anything unexpected occurred, along the lines of grep -v Foo: stderr.txt
Also note that on Unix, ls *.* isn't how you display everything. If you want to see all the files, just run ls on its own.
As I understand your problem using GNU Parallel http://www.gnu.org/software/parallel/ would do the right thing:
ls *.* | parallel -v echo {} 2> /dev/null