I am trying to use the tee command on Solaris to route output of 1 command to 2 different steams each of which comprises multiple statements. Here is the snippet of what I coded, but does not work. This iteration throws errors about unexpected end of files. If I change the > to | it throws an error Syntax Error near unexpected token do.
todaydir=/some/path
baselen=${#todaydir}
grep sometext $todaydir/somefiles*
while read iline
tee
>(
# this is the first block
do ojob=${iline:$baselen+1:8}
echo 'some text here' $ojob
done > firstoutfile
)
>(
# this is the 2nd block
do ojob=${iline:$baselen+1:8}
echo 'ls -l '$todaydir'/'$ojob'*'
done > secondoutfile
)
Suggestions?
The "while" should begin (and end) inside each >( ... ) substitution, not outside. Thus, I believe what you want is:
todaydir=/some/path
baselen=${#todaydir}
grep sometext $todaydir/somefiles* | tee >(
# this is the first block
while read iline
do ojob=${iline:$baselen+1:8}
echo 'some text here' $ojob
done > firstoutfile
) >(
# this is the 2nd block
while read iline
do ojob=${iline:$baselen+1:8}
echo 'ls -l '$todaydir'/'$ojob'*'
done > secondoutfile
)
I don't think the tee command will do that. The tee command will write stdin to one or more files as well as spit it back out to stdout. Plus I'm not sure the shell can fork off two sub-processes in the command pipeline like you are trying. You'd probably be better off to use something like Perl to fork off a couple of sub-process and write stdin to each.
Related
RHEL6
I have a c-shell script that runs a perl script. After dumping tons of stuff to stdout, it determines where (what dir) the parent shell should cd to when the perl script finishes. But that's a string, not an int which is all I can pass back with "exit()".
Storing the name of the dir in a file which the c-shell script can read is what I have now. It works, but is not elegant. Is there a better way to do this ? Maybe a little chunk of memory that I can share with the perl script ?
Short:
Redirect Perl's streams and restore in the end to print that info, taken by the shell script
Or, print that last and the shell script can pass output to the console and take the last line
Or, use a named pipe (either shell) or specific file descriptors (not csh) for that print
When the Perl script prints out that name you can assign it to a variable
in the shell script
#!/bin/csh
set DIR `perl -e'print "dir_name"'`
while in bash
#!/bin/bash
DIR="$(perl -e'print "dir_name"')"
where $(...) is preferred for the command substitution.
But those other prints to console from the Perl script then need be handled
One way is to redirect all output in Perl script other than that one print, what can be controlled by a command-line option (filename to which to redirect, which shell script can print out)
Or, take all Perl's output and pass it to console, the last line being the needed "return." This puts the burden on the Perl script to print that last (perhaps in an END block). The program's output can be printed from the shell script after it completes or line by line as it is emitted.
Or, use a named pipe (both shells) or a specific file descriptor (bash only) to which the Perl script can print that information. In this case its streams go straight to the console.
The question explicitly mentions csh so it is given below. But I must repeat the old and worn fact that shell scripting is far better done in bash than in csh. I strongly recommend to reconsider.
bash
If you need the program's output on the console as it goes, take and print it line by line
#!/bin/bash
while read line; do
echo "$line"
DIR=$line
done < <(perl script.pl)
echo "$DIR"
Or, if you don't need output on the console before the script is finished
#!/bin/bash
mapfile -t lines < <(perl script.pl)
DIR="${lines[-1]}"
printf '%s\n' "${lines[#]}" # print script.pl's output
Or, use file descriptors for that particular print
F=$(mktemp) # safe filename
exec 3> "$F" # open fd 3 to write to it
exec 4< "$F" # open fd 4 to read from it
rm -f "$F" # remove file(name) for safety; opened fd's can still access
perl -E'$fd=shift; say "...normal prints to STDOUT...";
open(FH, ">&=$fd") or die $!;
say FH "dirname";
close FH
' 3
read dir_name <&4
exec 3>&- # close them
exec 4<&-
echo "$dir_name"
I couldn't get it to work with a single file descriptor for both reading and writing (exec 3<> ...), I think because the read can't rewind after the write, thus separate descriptors are used.
With a Perl script (and not the demo one-liner above) pass the fd number as a command-line option. The script can then do this only if it's invoked with that option.
Or, use a named pipe very similarly to how it's done for csh below. This is probably best here, if the manipulation of the program's STDOUT isn't to your liking.
csh
Iterate over the program's (completed) output line by line
#!/bin/csh
foreach line ( "`perl script.pl`" )
echo "$line"
set dir_name = "$line"
end
echo "Directory name: $dir_name"
or extract the last line first and then print the whole output
#!/bin/csh
set lines = ( "`perl script.pl`" )
set dir_name = $lines[$#]
# Print program's output
while ( $#lines )
echo "$lines[1]"
shift lines
end
or use a named pipe
set fifo_name = "/tmp/fifo$$" # or use mktemp
mkfifo "$fifo_name"
( perl script.pl --fifo $fifo_name [other args] & )
set dir_name = `cat "$fifo_name"`
rm -f $fifo_name
echo "dir name from FIFO: $dir_name"
The Perl command is in the background since FIFO blocks until written and read. So if the shell script were to wait for perl ... to complete the Perl script would block as it's writing to FIFO (since that's not being read) so shell would never get to read it; we would deadlock. It is also in a subshell, with ( ), so to avoid the informational prints about the background job.
The --fifo NAME command-line option is needed so that Perl script knows what special file to use (and not to do this if the option is not there).
For an in-line example replace ( perl script ...) with this one-liner, used above as well
( perl -E'$ff = shift; say qq(\t...normal prints to STDOUT...);
open FF, ">$ff" or die $!;
say FF "dir_name_$$";
close FF
' $fifo_name
& )
(broken over lines for readability)
The follwoing code is Perl script, grep lines with 'Stage' from hostlog. and then line by line match the content with regex, if find add the count by 1:
$command = 'grep \'Stage \' '. $hostlog;
#stage_info = qx($command);
foreach (#stage_info) {
if ( /Stage\s(\d+)\s(.*)/ ) {
$stage_number = $stage_number+1;
}
}
so how to do this in linux shell? Based on my test, the we can not loop line by line, since there is space inside.
That is a horrible piece of Perl code you've got there. Here's why:
It looks like you are not using use strict; use warnings;. That is a huge mistake, and will not prevent errors, it will just hide them.
Using qx() to grep lines from a file is a completely redundant thing to do, as this is what Perl does best itself. "Shelling out" a process like that most often slows your program down.
Use some whitespace to make your code readable. This is hard to read, and looks more complicated than it is.
You capture strings by using parentheses in your regex, but you never use these strings.
Re: $stage_number=$stage_number+1, see point 3. And also, this can be written $stage_number++. Using the ++ operator will make your code clearer, will prevent the uninitialized warnings, and save you some typing.
Here is what your code should look like:
use strict;
use warnings;
open my $fh, "<", $hostlog or die "Cannot open $hostlog for reading: $!";
while (<$fh>) {
if (/Stage\s\d+/) {
$stage_number++;
}
}
You're not doing anything with the internal captures, so why bother? You could do everything with a grep:
$ stage_number=$(grep -E 'Stage\s\d+\s' | wc -l)
This is using extended regular expressions. I believe the GNU version takes these without a -E parameter, and in Solaris, even the egrep command might not quite allow for this regular expression.
If there's something more you have to do, you've got to explain it in your question.
If I understand the issue correctly, you should be able to do this just fine in the shell:
while read; do
if echo ${REPLY} | grep -q -P "'Stage' "; then
# Do what you need to do
fi
done < test.log
Note that if your grep command supports the -P option you may be able to use the Perl regular expression as-is for the second test.
this is almost it. bash has no expression for multiple digits.
#!/bin/bash
command=( grep 'Stage ' "$hostlog" )
while read line
do
[ "$line" != "${line/Stage [0-9]/}" ] && (( ++stage_number ))
done < <( "${command[#]}" )
On the other hand taking the function of the perl script into account rather than the operations it performs the whole thing could be rewritten as
(( stage_number += ` grep -c 'Stage \d\+\s' "$hostlog" ` ))
or this
stage_number=` grep -c 'Stage \d\+\s' "$hostlog" `
if, in the original perl, stage_number is uninitialised, or is initalised to 0.
I would like to use a small script to do some cosmetic work to the output of my gcc.
So I use this command:
mygcc foo.c 2>&1 | myscript.pl
Basically my script does things like this:
$error = 0;
while(<>)
{
s/^"(.*)"\s*,\s*line\s*(\d+)\s*:\s*(cc\d+)\s*:/colored("[$3]", 'bold red').colored(" $1", 'red').":".colored("$2", 'yellow')/ge;
s/ \^/colored(" ^", 'yellow')/e;
s/(error:.*$)/colored($1, 'red')/ge;
s/(warning.*$)/colored($1, 'yellow')/ge;
print;
$error = -1;
}
Unfortunately the exit code from gcc is not correctly propagated through the pipe. What I need to do is to get the exit code from gcc and write it back from my script.
Without this, make won't correctly stop the build process in case of an error.
How can I achieve this?
Try using a sub shell:
( mygcc foo.c; echo "gcc returned $?" ) |& myscript.pl
The ( cmd ) construct is used to launch cmd in a sub-shell. Your current shell will fork itself, and the commands will be executed by the child shell. It's an easy way to run multiple commands and have the output fed to a pipe.
The $? variable is the exit status of the last command.
The cmd1 |& cmd1 construct is equivalent to cmd1 2>&1 | cmd2
Take a look at this. You can then use the %ENV variable to access the gcc return status and return that value from your perl script.
Is there a way to filter stdout (or stderr) before being redirected to a file?
"redirecting to a pipe" is probably not the best way to put it but I'm looking for the easiest way to achieve something with that effect.
The usage scenario is the following. I'm using gawk --lint-invalid by principle to detect possible errors in my scripts and want to filter out spurious ones. Instead of redirecting errors to a file and grepping them out when examining the file, I would like the filtering to take place before writing to the file.
Example: this script prints every second line to stderr.
echo -ne 'a\nb\nc\nd\n' | gawk --lint=invalid 'BEGIN {b = 1;} // {if (b) print; else print > "/dev/stderr"; b = !b;}' 1>/dev/null 2>errors
cat errors | less
gawk: warning: regexp constant `//' looks like a C++ comment, but is not
b
d
gawk: (FILENAME=- FNR=4) warning: no explicit close of file `/dev/stderr' provided
But you can see the spurious gawk warnings (they are not of concern). They could be filtered for example, using
filter-gawk-output.sh
---------------------
grep -Ev 'looks like a|explicit close'
Is there an elegant way of doing that in-line when redirecting to errors file?
Right now when examining error files I always do
cat errors | ./filter-gawk-output.sh | less
What about:
gawk --lint=invalid 'whatever' INPUTFILE 2> GAWK_ERRORS.LOG
This way STDERR will be redirected to the error log.
I am not aware of gawk having facility to change the output of warnings. So I think this is more a question about shell syntax.
Given
filter_warnings() { grep -v '^gawk:'; }
awkprog='BEGIN {b = 1;} // {if (b) print; else print > "/dev/stderr"; b = !b;}'
where filter_warnings is for filtering out the gawk warnings and assuming bash as your shell, we can direct stderr to pipe command using |& syntax:
echo -ne 'a\nb\nc\nd\n' | gawk --lint=invalid "$awkprog" |& filter_warnings
If you want to outputs to file, then need to use parenthesis:
(echo -ne 'a\nb\nc\nd\n' | gawk --lint=invalid "$awkprog" > output.1) |& filter_warnings > output.2
Here output.1 will contain the gawk program output to stdout and output.2 the program output to to stderr.
So, I have a bash script inside of which I'd like to have a conditional which depends on what a perl script returns. The idea behind my code is as follows:
for i in $(ls); do
if $(perl -e "if (\$i =~ /^.*(bleh|blah|bluh)/) {print 'true';}"); then
echo $i;
fi;
done
Currently, this always returns true, and when I tried it with [[]] around the if statement, I got errors. Any ideas anyone?
P.s. I know I can do this with grep, but it's just an example. I'd like to know how to have Bash use Perl output in general
P.p.s I know I can do this in two lines, setting the perl output to a variable and then testing for that variables value, but I'd rather avoid using that extra variable if possible. Seems wasteful.
If you use exit, you can just use an if directly. E.g.
if perl -e "exit 0 if (successful); exit 1"; then
echo $i;
fi;
0 is success, non-zero is failure, and 0 is the default if you don't call exit.
To answer your question, you want perl to exit 1 for failure and exit 0 for success. That being said, you're doing this the wrong way. Really. Also, don't parse the output of ls. You'll cause yourself many headaches.
for file in *; do
if [[ $file = *bl[eau]h ]]; then
echo "$file matches"
fi
done
for file in * ; do
perl -e "shift =~ /^.*(bleh|blah|bluh)/ || exit 1" "$file" && echo $file: true
done
You should never parse the output of ls. You will have, at least, problems with file names containing spaces. Plus, why bother when your shell can glob on its own?
Quoting $file when passing to the perl script avoids problems with spaces in file names (and other special characters). Internally I avoided expanding the bash $file variable so as to not run afoul of quoting problems if the file name contained ", ' or \
Perl seems to (for some reason) always return 0 if you don't exit with an explicit value, which seems weird to me. Since this is the case I test for failure inside the script and return nonzero in that case.
The return value of the previous command is stored in the bash variable $?. You can do something like:
perl someargs script.pl more args
if [ $? == 0 ] ; then
echo true
else
echo false
fi
It's a good question, my advice is: keep it simple and go Posix (avoid Bashisms1) where possible..
so ross$ if perl -e 'exit 0'; then echo Good; else echo Bad; fi
Good
so ross$ if perl -e 'exit 1'; then echo Good; else echo Bad; fi
Bad
1. Sure, the OP was tagged bash, but others may want to know the generic-Posix form.