awk command in Perl's system does not work - perl

I am writing a small Perl script that executes an Awk command :
I try to swap two columns in a file, the file is like this :
domain1,ip1
domain2,ip2
domain3,ip3
the result should be
ip1,domain1
ip2,domain2
ip3,domain3
The Perl command invoking awk is like this:
system("ssh -p 22 root\#$mainip 'awk -F, '{print $2,$1}' OFS=, /root/archive/ipdomain.txt > /root/ipdom.txt'");
This is the error I get :
awk: cmd. line:1: {print
awk: cmd. line:1: ^ unexpected newline or end of string
any suggestions, please?

With the layered commands and all that multi-level quoting and escaping that need be done right,† no wonder it fails. A complex command like that will always be tricky, but libraries help a lot.
A properly quoted string to run through a shell can be formed with String::ShellQuote ‡
use warnings;
use strict;
use feature 'say';
use String::ShellQuote qw(shell_quote);
die "Usage: $0 file outfile\n" if #ARGV != 2;
my ($file, $out) = #ARGV;
my #cmd_words =
( 'ssh', 'hostname', 'awk', q('{print $2 $1}'), $file, '>', $out );
my $cmd = shell_quote #cmd_words;
system($cmd);
Note how the q() operator from of single quotes enables us to pass single quotes nicely.
This swaps the first two words on each line of a file and prints them, using awk, and redirects the output to a file, on a remote host. It works as expected in my tests (with a real hostname). Please adjust as needed.
Another possible improvement would be to use a library for ssh, like Net::OpenSSH.
A complete command, as the one in the question, to use in the above program
my #cmd_words = (
'ssh', '-p', '22', "root\#$mainip",
'awk', '-F,', q('{print $2,$1}'), 'OFS=,', $file, '>', $out );
Tested with a file from the question.
The makeVoiceBot answer is informative and it got half way there but I find the need for
system("ssh hostname \"awk '{print \\\$2 \\\$1}' $path\"");
This works in my tests (on systems I ssh to). I try to avoid needing to deal with such quoting and escaping.
† This is a shell command which runs ssh, and then executes a command on the remote system which runs a shell (there) as well, in order to run awk and redirect its output to a file.
A bit more than an "awk command" as the title says.
‡ The library can prepare a command for bash (as of this writing), but one can look at the source for it and adjust it for their own shell, at least. There is also Win32::ShellQuote

I am using a shortened example here
system("ssh localhost 'awk '{print $2,$1}' file.txt'")
system() sees:
ssh localhost 'awk '{print $2,$1}' file.txt'
local shell expands:
ssh
localhost
awk
{print
$2,$1}
file.txt
local shell replaces $1 and $2 (positional args) with empty strings:
ssh
localhost
awk
{print
,}
file.txt
ssh executes:
ssh localhost awk {print ,} file.txt
remote shell gets:
awk
{print
,}
file.txt
So the remote shell runs awk with {print as its program argument, resulting in the described error. To prevent this, the invocation of system() can be changed to;
system("ssh localhost \"awk '{print \$2,\$1}' file.txt\"")
system() sees:
ssh localhost "awk '{print \$2,\$1}' file.txt"
local shell expands:
ssh
localhost
awk '{print \$2,\$1}' file.txt
ssh executes
ssh localhost awk '{print \$2,\$1}' file.txt
remote shell gets
awk
{print \$2,\$1}
file.txt
remote shell expands \ escapes
awk
{print $2,$1}
file.txt
Remote awk now gets {print $2,$1} as its program argument, and executes successfully.

Related

How to use both pipes and prevent shell expansion in perl system function?

If multiple arguments are passed to perl's system function then the shell expansion will not work:
# COMMAND
$ perl -e 'my $s="*"; system("echo", "$s" )'
# RESULT
*
If the command is passed as an one argument then the expansion will work:
# COMMAND
$ perl -e 'my $s="echo *"; system("$s")'
# RESULT
Desktop Documents Downloads
The system function also allows to using multiple commands and connect them using pipes. This only works when argument is passed as an one command:
# COMMAND
$ perl -e 'my $s="echo * | cat -n"; system("$s")'
# RESULT
1 Desktop Documents Downloads
How can I combine mentioned commands and use both pipes and prevent shell expansion?
I have tried:
# COMMAND
$ perl -e 'my $s="echo"; system("$s", "* | cat -n")'
# RESULT
* | cat -n
but this did not work because of reasons that I've described above (multiple arguments are not expanded). The result that I want is:
1 *
EDIT:
The problem that I'm actually facing is that when I use following command:
system("echo \"$email_message\" | mailx -s \"$email_subject\" $recipient");
Then the $email_message is expanded and it will break mailx if it contains some characters that are further expanded by shell.
system has three calling conventions:
system($SHELL_CMD)
system($PROG, #ARGS) # #ARGS>0
system( { $PROG } $NAME, #ARGS ) # #ARGS>=0
The first passes a command to the shell. It's equivalent to
system('/bin/sh', '-c', $SHELL_CMD)
The other two execute the program $PROG. system never prevents shell expansion or performs any escaping. There's simply no shell involved.
So your question is about building a shell command. If you were at the prompt, you might use
echo \* | cat -n
or
echo '*' | cat -n
to pass *. You need a function that performs the job of escaping * before interpolating it. Fortunately, one already exists: String::ShellQuote's shell_quote.
$ perl -e'
use String::ShellQuote qw( shell_quote );
my $s = "*";
my $cmd1 = shell_quote("printf", q{%s\n}, $s);
my $cmd2 = "cat -n";
my $cmd = "$cmd1 | $cmd2";
print("Executing <<$cmd>>\n");
system($cmd);
'
Executing <<printf '%s\n' '*' | cat -n>>
1 *
I used printf instead of echo since it's very hard to handle arguments starting with - in echo. Most programs accept -- to separate options from non-options, but not my echo.
All these complications beg the question: Why are you shelling out to send an email? It's usually much harder to handle errors from external programs than from libraries.
You can use open to pipe directly to mailx, without your content being interpreted by the shell:
open( my $mail, "|-", "mailx", "-s", $email_subject, $recipient );
say $mail $email_message;
close $mail;
More details can be found in open section of perlipc.

Executing shell command with pipe in perl

I want the output of the shell command captured in variable of a perl script, only the first section of the command before the pipe "|" is getting executed, and there is no error while executing the script
File.txt
Error input.txt got an error while parsing
Info output.txt has no error while parsing
my $var = `grep Error ./File.txt | awk '{print $2}'`;
print "Errored file $var";
Errored file Error input.txt got an error while parsing
I want just the input.txt which gets filtered by awk command but not happening. Please help
The $ in $2 is interpolated by Perl, so the command that the shell receives looks like:
grep Error ./File.txt | awk '{print }'
(or something else if you have recently matched a regular expression with capture groups). The workaround is to escape the dollar sign:
my $var = `grep Error ./File.txt | awk '{print \$2}'`
Always include use strict; and use warnings; in EVERY perl script.
If you had, you'd have gotten the following warning:
Use of uninitialized value $2 in concatenation (.) or string at scratch.pl line 4.
This would've alerted you to the problem in your command, namely that the $2 variable is being interpolated instead of being treated like a literal.
There are three ways to avoid this.
1) You can do what mob suggested and just escape the $2
my $var = `grep Error ./File.txt | awk '{print \$2}'`
2) You can use the qx form of backticks with single quotes so that it doesn't interpolate, although that is less ideal because you are using single quotes inside your command:
my $var = qx'grep Error ./File.txt | awk \'{print $2}\''
3) You can just use a pure perl solution.
use strict;
use warnings;
my ($var) = do {
local #ARGV = 'File.txt';
map {(split ' ')[1]} grep /Error/, <>;
};

Perl command line search and replace with multiple expressions

I am using Perl to search and replace multiple regular expressions:
When I execute the following command, I get an error:
prompt> find "*.cpp" | xargs perl -i -pe 's/##(\W)/\1/g' -pe 's/(\W)##/\1/g'
syntax error at -e line 2, near "s/(\W)##/\1/g"
Execution of -e aborted due to compilation errors.
xargs: perl: exited with status 255; aborting
Having multiple -e is valid in Perl, then why is this not working? Is there a solution to this?
Several -e's are allowed.
You are missing the ';'
find "*.cpp" | xargs perl -i -pe 's/##(\W)/\1/g;' -pe 's/(\W)##/\1/g;'
Perl statements has to end with ;.
Final statement in a block doesn't need a terminating semicolon.
So a single -e without ; will work, but you will have to add ; when you have multiple -e statements.
Having multiple -e values are valid, but is it useful? The values from the multiple -e are merely combined into one program, and it's up to you to ensure that together they make a syntactically correct program. The B::Deparse program can show you what perl thinks the program is:
$ perl -MO=Deparse -e 'print' -e 'q(Hello' -e ')'
print "Hello\n";
-e syntax OK
A curious thing to note is that a newline snuck in there. Think about how it got there to see what else perl is doing to combine multiple -e values.
In your program, you are substituting on the current line, then taking the modified line and substituting again. That's better written as:
prompt> find "*.cpp" | xargs perl -i -pe 's/##(\W)/\1/g; s/(\W)##/\1/g'
Now, if you are building up this command line by adding more and more -e through some automated process and you don't know ahead of time what you get, maybe those -e make sense. However, you might consider that you can do the same thing to build up the string you give to -e. I don't know what might be better because you didn't explain why you are doing it that way.
But, I suspect that in some cases, people are actually thinking about having only one substitution work. They want to try one and if its pattern doesn't work, try a different one until one succeeds. In that case you don't want to separate the substitutions by semicolons. Use the short-circuiting || instead. The s/// returns the number of substitutions it made and || will stop (short circuit) when it finds a true value:
prompt> find "*.cpp" | xargs perl -i -pe 's/##(\W)/\1/g || s/(\W)##/\1/g'
And note, you only need one -p. It only does its job once. Here's the program with multiple -p deparsed:
$ perl -MO=Deparse -i -pe 's/##(\W)/\1/g;' -pe 's/(\W)##/\1/g;'
BEGIN { $^I = ""; }
LINE: while (defined($_ = readline ARGV)) {
s/##(\W)/$1/g;
s/(\W)##/$1/g;
}
continue {
die "-p destination: $!\n" unless print $_;
}
-e syntax OK
It's the same thing as having only one -p:
$ perl -MO=Deparse -pi -e 's/##(\W)/\1/g;' -e 's/(\W)##/\1/g;'
BEGIN { $^I = ""; }
LINE: while (defined($_ = readline ARGV)) {
s/##(\W)/$1/g;
s/(\W)##/$1/g;
}
continue {
die "-p destination: $!\n" unless print $_;
}
-e syntax OK
Thanks so much! You helped me reduce my ascii / decimal / 8-bit binary table printer enough to fit in a tweet:
for i in {32..126};do printf "'\x$(printf %x $i)'(%3i) = " $i; printf '%03o\n' $i | perl \
-pe 's#0#000#g;' -pe 's#1#001#g;' -pe 's#2#010#g;' -pe 's#3#011#g;' \
-pe 's#4#100#g;' -pe 's#5#101#g;' -pe 's#6#110#g;' -pe 's#7#111#g' ; done | \
perl -pe 's#= 0#= #'

How can I store the result of a system command in a Perl variable?

$ cat test.pl
my $pid = 5892;
my $not = system("top -H -p $pid -n 1 | grep myprocess | wc -l");
print "not = $not\n";
$ perl test.pl
11
not = 0
$
I want to capture the result i.e. 11 into a variable. How can I do that?
From Perlfaq8:
You're confusing the purpose of system() and backticks (``). system() runs a command and returns exit status information (as a 16 bit value: the low 7 bits are the signal the process died from, if any, and the high 8 bits are the actual exit value). Backticks (``) run a command and return what it sent to STDOUT.
$exit_status = system("mail-users");
$output_string = `ls`;
There are many ways to execute external commands from Perl. The most commons with their meanings are:
system() : you want to execute a command and don't want to capture its output
exec: you don't want to return to the
calling perl script
backticks : you want to capture the
output of the command
open: you want to pipe the command (as
input or output) to your script
Also see How can I capture STDERR from an external command?
The easiest way is to use the `` feature in Perl. This will execute what is inside and return what was printed to stdout:
my $pid = 5892;
my $var = `top -H -p $pid -n 1 | grep myprocess | wc -l`;
print "not = $var\n";
This should do it.
Try using qx{command} rather than backticks. To me, it's a bit better because: you can do SQL with it and not worry about escaping quotes and such. Depending on the editor and screen, my old eyes tend to miss the tiny back ticks, and it shouldn't ever have an issue with being overloaded like using angle brackets versus glob.
Using backtick or qx helps, thanks everybody for the answers. However, I found that if you use backtick or qx, the output contains trailing newline and I need to remove that. So I used chomp.
chomp($host = `hostname`);
chomp($domain = `domainname`);
$fqdn = $host.".".$domain;
More information here:
http://irouble.blogspot.in/2011/04/perl-chomp-backticks.html
Use backticks for system commands, which helps to store their results into Perl variables.
my $pid = 5892;
my $not = ``top -H -p $pid -n 1 | grep myprocess | wc -l`;
print "not = $not\n";
Also for eg. you can use IPC::Run:
use IPC::Run qw(run);
my $pid = 5892;
run [qw(top -H -n 1 -p), $pid],
'|', sub { print grep { /myprocess/ } <STDIN> },
'|', [qw(wc -l)],
'>', \my $out;
print $out;
processes are running without bash subprocess
can be piped to perl subs
very similar to shell

How can I grep for a value from a shell variable?

I've been trying to grep an exact shell 'variable' using word boundaries,
grep "\<$variable\>" file.txt
but haven't managed to; I've tried everything else but haven't succeeded.
Actually I'm invoking grep from a Perl script:
$attrval=`/usr/bin/grep "\<$_[0]\>" $upgradetmpdir/fullConfiguration.txt`
$_[0] and $upgradetmpdir/fullConfiguration.txt contains some matching "text".
But $attrval is empty after the operation.
#OP, you should do that 'grepping' in Perl. don't call system commands unnecessarily unless there is no choice.
$mysearch="pattern";
while (<>){
chomp;
#s = split /\s+/;
foreach my $line (#s){
if ($line eq $mysearch){
print "found: $line\n";
}
}
}
I'm not seeing the problem here:
file.txt:
hello
hi
anotherline
Now,
mala#human ~ $ export GREPVAR="hi"
mala#human ~ $ echo $GREPVAR
hi
mala#human ~ $ grep "\<$GREPVAR\>" file.txt
hi
What exactly isn't working for you?
Not every grep supports the ex(1) / vi(1) word boundary syntax.
I think I would just do:
grep -w "$variable" ...
Using single quotes works for me in tcsh:
grep '<$variable>' file.txt
I am assuming your input file contains the literal string: <$variable>
If variable=foo are you trying to grep for "foo"? If so, it works for me. If you're trying to grep for the variable named "$variable", then change the quotes to single quotes.
On a recent linux it works as expected. Do could try egrep instead
Say you have
$ cat file.txt
This line has $variable
DO NOT PRINT ME! $variableNope
$variable also
Then with the following program
#! /usr/bin/perl -l
use warnings;
use strict;
system("grep", "-P", '\$variable\b', "file.txt") == 0
or warn "$0: grep exited " . ($? >> 8);
you'd get output of
This line has $variable
$variable also
It uses the -P switch to GNU grep that matches Perl regular expressions. The feature is still experimental, so proceed with care.
Also note the use of system LIST that bypasses shell quoting, allowing the program to specify arguments with Perl's quoting rules rather than the shell's.
You could use the -w (or --word-regexp) switch, as in
system("grep", "-w", '\$variable', "file.txt") == 0
or warn "$0: grep exited " . ($? >> 8);
to get the same result.
Using single quote it wont work. You should go for double quote
For example:
this wont work
--------------
for i in 1
do
grep '$i' file
done
this will work
--------------
for i in 1
do
grep "$i" file
done