say i have a directory with hi.txt and blah.txt and i execute the following command on a linux-ish command line
ls *.* | xargs -t -i{} echo {}
the output you will see is
echo blah.txt
blah.txt
echo hi.txt
hi.txt
i'd like to redirect the stderr output (say 'echo blah.txt' fails...), leaving only the output from the xargs -t command written to std out, but it looks as if it's stderr as well.
ls *.* | xargs -t -i{} echo {} 2> /dev/null
Is there a way to control it, to make it output to stdout?
Use:
ls | xargs -t -i{} echo {} 2>&1 >/dev/null
The 2>&1 sends the standard error from xargs to where standard output is currently going; the >/dev/null sends the original standard output to /dev/null. So, the net result is that standard output contains the echo commands, and /dev/null contains the file names. We can debate about spaces in file names and whether it would be easier to use a sed script to put 'echo' at the front of each line (with no -t option), or whether you could use:
ls | xargs -i{} echo echo {}
(Tested: Solaris 10, Korn Shell ; should work on other shells and Unix platforms.)
If you don't mind seeing the inner workings of the commands, I did manage to segregate the error output from xargs and the error output of the command executed.
al * zzz | xargs -t 2>/tmp/xargs.stderr -i{} ksh -c "ls -dl {} 2>&1"
The (non-standard) command al lists its arguments one per line:
for arg in "$#"; do echo "$arg"; done
The first redirection (2>/tmp/xargs.stderr) sends the error output from xargs to the file /tmp/xargs.stderr. The command executed is 'ksh -c "ls -dl {} 2>&1"', which uses the Korn shell to run ls -ld on the file name with any error output going to standard output.
The output in /tmp/xargs.stderr looks like:
ksh -c ls -dl x1 2>&1
ksh -c ls -dl x2 2>&1
ksh -c ls -dl xxx 2>&1
ksh -c ls -dl zzz 2>&1
I used 'ls -ld' in place of echo to ensure I was testing errors - the files x1, x2, and xxx existed, but zzz does not.
The output on standard output looked like:
-rw-r--r-- 1 jleffler rd 1020 May 9 13:05 x1
-rw-r--r-- 1 jleffler rd 1069 May 9 13:07 x2
-rw-r--r-- 1 jleffler rd 87 May 9 20:42 xxx
zzz: No such file or directory
When run without the command wrapped in 'ksh -c "..."', the I/O redirection was passed as an argument to the command ('ls -ld'), and it therefore reported that it could not find the file '2>&1'. That is, xargs did not itself use the shell to do the I/O redirection.
It would be possible to arrange for various other redirections, but the basic problem is that xargs makes no provision for separating its own error output from that of the commands it executes, so it is hard to do.
The other rather obvious option is to use xargs to write a shell script, and then have the shell execute it. This is the option I showed before:
ls | xargs -i{} echo echo {} >/tmp/new.script
You can then see the commands with:
cat /tmp/new.script
You can run the commands to discard the errors with:
sh /tmp/new.script 2>/dev/null
And, if you don't want to see the standard output from the commands either, append 1>&2 to the end of the command.
So I believe what you want is to have as stdout is
the stdout from the utility that xargs executes
the listing of commands generated by xargs -t
You want to ignore the stderr stream generated by the
executed utility.
Please correct me if I'm wrong.
First, let's create a better testing utility:
% cat myecho
#!/bin/sh
echo STDOUT $#
echo STDERR $# 1>&2
% chmod +x myecho
% ./myecho hello world
STDOUT hello world
STDERR hello world
% ./myecho hello world >/dev/null
STDERR hello world
% ./myecho hello world 2>/dev/null
STDOUT hello world
%
So now we have something that actually outputs to both stdout and stderr, so we
can be sure we're only getting what we want.
A tangential way to do this is not to use xargs, but rather, make. Echoing a command
and then doing it is kind of what make does. That's its bag.
% cat Makefile
all: $(shell ls *.*)
$(shell ls): .FORCE
./myecho $# 2>/dev/null
.FORCE:
% make
./myecho blah.txt 2>/dev/null
STDOUT blah.txt
./myecho hi.txt 2>/dev/null
STDOUT hi.txt
% make >/dev/null
%
If you're tied to using xargs, then you need to modify your utility that
xargs uses so it surpresses stderr. Then you can use the 2>&1 trick others
have mentioned to move the command listing generated by xargs -t from stderr
to stdout.
% cat myecho2
#!/bin/sh
./myecho $# 2>/dev/null
% chmod +x myecho2
% ./myecho2 hello world
STDOUT hello world
% ls *.* | xargs -t -i{} ./myecho2 {} 2>&1
./myecho blah.txt 2>/dev/null
STDOUT blah.txt
./myecho hi.txt 2>/dev/null
STDOUT hi.txt
% ls *.* | xargs -t -i{} ./myecho2 {} 2>&1 | tee >/dev/null
%
So this approach works, and collapses everything you want to stdout (leaving out what you don't want).
If you find yourself doing this a lot, you can write a general utility to surpress stderr:
% cat surpress_stderr
#!/bin/sh
$# 2>/dev/null
% ./surpress_stderr ./myecho hello world
STDOUT hello world
% ls *.* | xargs -t -i{} ./surpress_stderr ./myecho {} 2>&1
./surpress_stderr ./myecho blah.txt 2>/dev/null
STDOUT blah.txt
./surpress_stderr ./myecho hi.txt 2>/dev/null
STDOUT hi.txt
%
xargs -t echos the commands to be executed to stderr before executing them. If you want them to instead echo to stderr, you can pipe stderr to stdout with the 2>&1 construct:
ls *.* | xargs -t -i{} echo {} 2>&1
It looks like xargs -t goes to stderr, and there's not much you can do about it.
You could do:
ls | xargs -t -i{} echo "Foo: {}" >stderr.txt | tee stderr.txt
to display only the stderr data on your terminal as your command runs, and then grep through stderr.txt after to see if anything unexpected occurred, along the lines of grep -v Foo: stderr.txt
Also note that on Unix, ls *.* isn't how you display everything. If you want to see all the files, just run ls on its own.
As I understand your problem using GNU Parallel http://www.gnu.org/software/parallel/ would do the right thing:
ls *.* | parallel -v echo {} 2> /dev/null
Related
When running the following command with xargs (GNU findutils) 4.7.0
xargs -n1 <<<"-d -e -n -o"
I get this output
-d
-o
Why is -e and -n not present in the output?
From man xargs:
[...] and executes the command (default is /bin/echo) [...]
So it runs:
echo -d
echo -e
echo -n
echo -o
But from man echo:
-n do not output the trailing newline
-e enable interpretation of backslash escapes
And echo -n outputs nothing, and echo -e outputs one empty newlines that you see in the output.
I would like to redirect stderr to stdout so a terminal prints both of them during a command execution but I would like also to capture both of them into separate variables. I managed to achieve that in Bash (version 4.4.20(1)-release):
#!/bin/bash
echo "terminal:"
{ err="$(find / -maxdepth 2 -iname 'tmp' -type d 2>&1 1>&3 3>&- | tee /dev/stderr)"; ec="$?"; } 3>&1 | tee /dev/fd/4 2>&1; out=$(cat /dev/fd/4)
echo "stdout:" && echo "$out"
echo "stderr:" && echo "$err"
that gives desired:
terminal:
find: ‘/root’: Permission denied
/tmp
/var/tmp
find: ‘/lost+found’: Permission denied
stdout:
/tmp
/var/tmp
stderr:
find: ‘/root’: Permission denied
find: ‘/lost+found’: Permission denied
but I have a problem converting that script into POSIX sh /bin/sh
#!/bin/sh
echo "terminal:"
{ err="$(find / -maxdepth 2 -iname 'tmp' -type d 2>&1 1>&3 3>&- | tee /dev/stderr)"; ec="$?"; } 3>&1 | tee /dev/fd/4 2>&1; out=$(cat /dev/fd/4)
echo "stdout:" && echo "$out"
echo "stderr:" && echo "$err"
gives:
terminal:
tee: /dev/fd/4: No such file or directory
find: ‘/root’: Permission denied
/tmp
/var/tmp
find: ‘/lost+found’: Permission denied
cat: /dev/fd/4: No such file or directory
stdout:
stderr:
/dev/fd/4 does not exist, and there is no /proc/self/fd/4 either.
How to make that script working as a POSIX shell script?
Well I'm doing this in busybox's ash shell and my options are rather limited, but this is what I came up with that works.
tmpfile=/tmp/some_prefix_stderr.$$
out=$(my_cmd) 2>$tmpfile
result=$?
err=$(cat $tmpfile)
rm $tmpfile
It's not the prettiest, but it works. In general, replace some_prefix with something specific to help avoid collisions, but just having the PID on there will solve that unless you are running multiple threads.
If you can rely upon your program exiting non-zero when there's something to show in stderr, you can simplify it a little:
tmpfile=/tmp/some_prefix_stderr.$$
if ! out=$(my_cmd) 2>$tmpfile; then
# specific behavior here
cat $tmpfile >&2
fi
rm $tmpfile
This is all POSIX compliant. Just be sure to clean up your temp file.
"ls" behaves differently when its output is being piped:
> ls ???
bar foo
> ls ??? | cat
bar
foo
How does it know, and how would I do this in Perl?
In Perl, the -t file test operator indicates whether a filehandle
(including STDIN) is connected to a terminal.
There is also the -p test operator to indicate whether a filehandle
is attached to a pipe.
$ perl -e 'printf "term:%d, pipe:%d\n", -t STDIN, -p STDIN'
term:1, pipe:0
$ perl -e 'printf "term:%d, pipe:%d\n", -t STDIN, -p STDIN' < /tmp/foo
term:0, pipe:0
$ echo foo | perl -e 'printf "term:%d, pipe:%d\n", -t STDIN, -p STDIN'
term:0, pipe:1
File test operator documentation at perldoc -f -X.
use IO::Interactive qw(is_interactive);
is_interactive() or warn "Being piped\n";
I've written a svn-hook for text files. The content test looks like this:
svnlook cat -t $txn $repos $file 2>/dev/null | file - | egrep -q 'text$'
and I was wondering if this could be done with Perl. However something like this doesn't work:
svnlook cat -t $txn $repos $file 2>/dev/null | perl -wnl -e '-T' -
I'm testing the exit status of this invocation ($?) to see if the given file was text or binary. Since I'm getting the content out of svn. I can't use perl's normal file check.
I've done a simulation with the file program and perl with a text and binary file (text.txt, icon.png):
find -type f | xargs -i /bin/bash -c 'if $(cat {} | file - | egrep -q "text$"); then echo "{}: text"; else echo "{}: binary"; fi'
./text.txt: text
./icons.png: binary
find -type f | xargs -i /bin/bash -c 'if $(cat {} | perl -wln -e "-T;"); then echo "{}: text"; else echo "{}: binary"; fi'
./text.txt: text
./icons.png: text
You're testing perl's exit code, but you never set it. You need
perl -le'exit(-T STDIN ?0:1)' < file
If I have input file containing
statementes
asda
rertte
something
nothing here
I want to grep / extract (without using awk) every line from starting till I get the string "something". How can I do this? grep -B does not work since it needs the exact number of lines.
Desired output:
statementes
asda
rertte
something
it's not completely robust, but sure -B works... just make the -B count huge:
grep -B `wc -l <filename>` -e 'something' <filename>
You could use a bash while loop and exit early when you hit the string:
$ cat file | while read line; do
> echo $line
> if echo $line | grep -q something; then
> exit 0
> fi
> done
head -n `grep -n -e 'something' <filename> | cut -d: -f1` <filename>