I am trying to execute below awk command inside a perl script, but it is failing.
#!/usr/bin/perl
print `awk -F, '{print $NF}' f1.txt > f2.txt`
This is the error:
syntax error at ./MO.pl line 3, near "print"
Execution of ./MO.pl aborted due to compilation errors.
Can anyone please help what I am doing wrong here?
This is a Perl error and has nothing to do with your awk script itself. The error is usually seen when the previous statement doesn't have a semicolon at the end.
Here's a very simple program (which should include use strict; and use warnings;, but I wanted to emulate what you have).
#! /usr/bin/env perl
#
print "Hello, World\n" # Line 4
print "Hello, Back\n"; # Line 5
And the error message is:
syntax error at test.pl line 5, near "print"
Execution of test.pl aborted due to compilation errors.
Note the error is near the print in Line #5, but the error is actually at the end of Line #4 where I'm missing a semicolon.
Running your exact program works on my system (although doesn't quite produce the results you want). I am assuming this isn't your exact program, but instead a simplification of your program. Is there a statement before that print?
Several other things:
You're redirecting your awk output, so there's nothing to print.
Use strict and warnings.
Better to use qx(....) than backticks (grave accent). It's more readable and allows you to do quoted executable in quoted executable.
Watch for Perlisms in your code. The $NF is interpreted by Perl, and without the use strict;, doesn't give you an error. Instead, the print in your Awk statement is a null print which prints the entire line.
Why do you use print if nothing is printing out? You're better off in this position to use system which allows you to put single quotes around your entire statement:
system q(awk -F, '{print $NF}' f1.txt > f2.txt);
This way, $NF doesn't have to be quoted.
Why are you doing Awk in a Perl program? Perl will do anything Awk will do and do it better:
Here's a version of your program using plain ol' Perl:
#! /usr/bin/env perl
use strict;
use warnings;
use autodie;
while ( my $line = <> ) {
my #array = split /\s*,\s*/, $line;
print $array[-1];
}
To run this program:
$ test.pl f1.txt > f2.txt
Yes, it's longer, but half of the program is taken up by pragmas that all Perl programs should use.
I'm sure people smarter than me can turn this into a Perl one-liner.
Since you're redirecting the awk output, there's nothing for perl to print. You might as well use system and the quoting operator q():
system q(awk -F, '{print $NF}' f1.txt > f2.txt)
Or, of course, do it in perl, which saves you from having to spawn a shell and then spawn awk:
open my $in, '<', 'f1.txt';
open my $out, '>', 'f1.txt';
while (<$in>) {
print $out (split " ")[-1], "\n";
}
close $in;
close $out;
If there are more lines in the script, you need a semi-colon at the end of the print statement.
Related
first of all, i'm totally new to Perl and i need to remove a string from a file.
Code i have:
#!/usr/bin/perl -w
use warnings;
use strict;
my $server_a = 'xxx.test.ch';
print "$server_a\n";
perl -pli -e "s/$server_a//" '/Users/user/Downloads/exports';
When i execute this script i get this error:
syntax error at nfs_test.sh line 9, near "pli -e "
Execution of nfs_test.sh aborted due to compilation errors.
i cant find whats wrong, i hope you can help me.
#! /usr/bin/perl
use warnings;
use strict;
#ARGV = '/Users/user/Downloads/exports';
$^I = '~';
my $server_a = quotemeta 'xxx.test.ch';
s/$server_a//, print while <>;
The #ARGV array contains the arguments, it's iterated by the diamond operator.
$^I is equivalent to the -i option.
You probably need quotemeta, as you don't mean "anything but newline" by dots in the server name.
I'm trying to run a tail command from within a perl script using the usual backticks.
The section in my perl script is as follows:
$nexusTime += nexusUploadTime(`tail $log -n 5`);
So I'm trying to get the last 5 lines of this file but I'm getting the following error when the perl script finishes:
sh: line 1: -n: command not found
Even though when I run the command on the command line it is indeed successful and I can see the 5 lines from that particular.
Not sure what is going on here. Why it works from command line but through perl it won't recognize the -n option.
Anybody have any suggestions?
$log has an extraneous trailing newline, so you are executing
tail file.log
-n 5 # Tries to execute a program named "-n"
Fix:
chomp($log);
Note that you will run into problems if log $log contains shell meta characters (such as spaces). Fix:
use String::ShellQuote qw( shell_quote );
my $tail_cmd = shell_quote('tail', '-n', '5', '--', $log);
$nexusTime += nexusUploadTime(`$tail_cmd`);
ikegami pointed out your error, but I would recommend avoiding external commands whenever possible. They aren't portable and debugging them can be a pain, among other things. You can simulate tail with pure Perl code like this:
use strict;
use warnings;
use File::ReadBackwards;
sub tail {
my ($file, $num_lines) = #_;
my $bw = File::ReadBackwards->new($file) or die "Can't read '$file': $!";
my ($lines, $count);
while (defined(my $line = $bw->readline) && $num_lines > $count++) {
$lines .= $line;
}
$bw->close;
return $lines;
}
print tail('/usr/share/dict/words', 5);
Output
ZZZ
zZt
Zz
ZZ
zyzzyvas
Note that if you pass a file name containing a newline, this will fail with
Can't read 'foo
': No such file or directory at tail.pl line 10.
instead of the more cryptic
sh: line 1: -n: command not found
that you got from running the tail utility in backticks.
The answer to this question is to place the option -n 5 before the target file
When i am using "sed" in command line it is working but not when included in perl script.
An example is sed 's/\s+//g' aaa > bbb
but say when i am trying to call the same command through perl script
$gf = `sed 's/\s\+//g' aaa > bbb` ;
the output file remains same as the input file!!!! Please suggest.
In Perl, backticks have the same escape and interpolation rules as double quoted strings: A backslash forming an unknown escape code forgets the backslash, e.g. "\." eq ".".
Therefore, the Perl code
print `echo \"1\"`;
print `echo \\"1\\"`;
outputs
1
"1"
If you want to embed that sed command into Perl, you have to escape the backslashes so that they even reach the shell:
$gf = `sed 's/\\s\\+//g' aaa > bbb`;
Actually, you won't get any output into $gf as you redirect the output to a file. We could just do
use autodie;
system "sed 's/\\s\\+//g' aaa > bbb";
or with single quotes:
use autodie;
system q{ sed 's/\s\+//g' aaa > bbb };
which keeps the backslashes.
Still, this is quite unneccessary as Perl could apply the substitution itself.
use autodie; # automatic error handling
open my $out, ">", "bbb";
open my $in, "<", "aaa";
while (<$in>) {
s/\s\+//g; # remove all spaces followed by a plus
print {$out} $_;
}
In these weird situations, I ensure that I'm running the right command. I'll construct it, store it, and output the command so I can see exactly what I created:
my $command = '....';
print "Command is [$command]\n";
my $output = `$command`;
If you're running sed from Perl, you might be doing it wrong since Perl can already do all that.
have you got
use strict;
use warnings;
at the top of your file?
you could need backticks to execute the command
$gf = `sed 's/\s\+//g' aaa > bbb`;
I find a lot of Perl one-liners online. Sometimes I want to convert these one-liners into a script, because otherwise I'll forget the syntax of the one-liner.
For example, I'm using the following command (from nagios.com):
tail -f /var/log/nagios/nagios.log | perl -pe 's/(\d+)/localtime($1)/e'
I'd to replace it with something like this:
tail -f /var/log/nagios/nagios.log | ~/bin/nagiostime.pl
However, I can't figure out the best way to quickly throw this stuff into a script. Does anyone have a quick way to throw these one-liners into a Bash or Perl script?
You can convert any Perl one-liner into a full script by passing it through the B::Deparse compiler backend that generates Perl source code:
perl -MO=Deparse -pe 's/(\d+)/localtime($1)/e'
outputs:
LINE: while (defined($_ = <ARGV>)) {
s/(\d+)/localtime($1);/e;
}
continue {
print $_;
}
The advantage of this approach over decoding the command line flags manually is that this is exactly the way Perl interprets your script, so there is no guesswork. B::Deparse is a core module, so there is nothing to install.
Take a look at perlrun:
-p
causes Perl to assume the following loop around your program, which makes it iterate over filename arguments somewhat like sed:
LINE:
while (<>) {
... # your program goes here
} continue {
print or die "-p destination: $!\n";
}
If a file named by an argument cannot be opened for some reason, Perl warns you about it, and moves on to the next file. Note that the lines are printed automatically. An error occurring during printing is treated as fatal. To suppress printing use the -n switch. A -p overrides a -n switch.
BEGIN and END blocks may be used to capture control before or after the implicit loop, just as in awk.
So, simply take this chunk of code, insertyour code at the "# your program goes here" line, and viola, your script is ready!
Thus, it would be:
#!/usr/bin/perl -w
use strict; # or use 5.012 if you've got newer perls
while (<>) {
s/(\d+)/localtime($1)/e
} continue {
print or die "-p destination: $!\n";
}
That one's really easy to store in a script!
#! /usr/bin/perl -p
s/(\d+)/localtime($1)/e
The -e option introduces Perl code to be executed—which you might think of as a script on the command line—so drop it and stick the code in the body. Leave -p in the shebang (#!) line.
In general, it's safest to stick to at most one "clump" of options in the shebang line. If you need more, you could always throw their equivalents inside a BEGIN {} block.
Don't forget chmod +x ~/bin/nagiostime.pl
You could get a little fancier and embed the tail part too:
#! /usr/bin/perl -p
BEGIN {
die "Usage: $0 [ nagios-log ]\n" if #ARGV > 1;
my $log = #ARGV ? shift : "/var/log/nagios/nagios.log";
#ARGV = ("tail -f '$log' |");
}
s/(\d+)/localtime($1)/e
This works because the code written for you by -p uses Perl's "magic" (2-argument) open that processes pipes specially.
With no arguments, it transforms nagios.log, but you can also specify a different log file, e.g.,
$ ~/bin/nagiostime.pl /tmp/other-nagios.log
Robert has the "real" answer above, but it's not very practical. The -p switch does a bit of magic, and other options have even more magic (e.g. check out the logic behind the -i flag). In practice, I'd simply just make a bash alias/function to wrap around the oneliner, rather than convert it to a script.
Alternatively, here's your oneliner as a script: :)
#!/usr/bin/bash
# takes any number of arguments: the filenames to pipe to the perl filter
tail -f $# | perl -pe 's/(\d+)/localtime($1)/e'
There are some good answers here if you want to keep the one-liner-turned-script around and possibly even expand upon it, but the simplest thing that could possibly work is just:
#!/usr/bin/perl -p
s/(\d+)/localtime($1)/e
Perl will recognize parameters on the hashbang line of the script, so instead of writing out the loop in full, you can just continue to do the implicit loop with -p.
But writing the loop explicitly and using -w and "use strict;" are good if plan to use it as a starting point for writing a longer script.
#!/usr/bin/env perl
while(<>) {
s/(\d+)/localtime($1)/e;
print;
}
The while loop and the print is what -p does automatically for you.
By default, perl prints \r\n in a win32 environment. How can I override this? I'm using perl to make some changes to some source code in a repository, and I don't want to change all the newline characters.
I tried changing the output record separator but with no luck.
Thanks!
Edit: Wanted to include a code sample - I'm doing a search and replace over some files that follow a relatively straightforward pattern like this:
#!/usr/bin/perl
# test.pl
use strict;
use warnings;
$/ = undef;
$\ = "\n";
$^I=".old~";
while (<>) {
while (s/hello/world/) {
}
print;
}
This should replace any instances of "hello" with "world" for any files passed on the cmd line.
Edit 2: I tried the binmode as suggested without any luck initially. I delved a bit more and found that $^I (the inplace edit special variable) was overriding binmode. Any work around to still be able to use the inplace edit?
Edit 3: As Sinan points out below, I needed to use binmode ARGVOUT with $^I instead of binmode STDOUT in my example. Thanks.
Printing "\n" to a filehandle on Windows emits, by default, a CARRIAGE RETURN ("\015") followed by a LINE FEED ("\012") character because that the standard newline sequence on Windows.
This happens transparently, so you need to override it for the special filehandle ARGVOUT (see perldoc perlvar):
#!/usr/bin/perl -i.bak
use strict; use warnings;
local ($\, $/);
while (<>) {
binmode ARGVOUT;
print;
}
Output:
C:\Temp> xxd test.txt
0000000: 7465 7374 0d0a 0d0a test....
C:\Temp> h test.txt
C:\Temp> xxd test.txt
0000000: 7465 7374 0a0a test..
See also perldoc open, perldoc binmode and perldoc perliol (thanks daotoad).
Does binmode( STDOUT ) work?
Re: your question about the binmode being lost when $^I opens a new output handle, you could solve this with the open pragma:
use open OUT => ':raw';
which will force all filehandles opened for writing to have the ':raw' PerlIO layer (equivalent to binmode with no argument) to apply to them. Just take care if you're opening anything else for output that you apply :crlf or any other layer as needed.
The data you are reading in contains line endings, so you're getting them back out again. You can strip them off yourself with chomp, then add your own ending back, provided you have set binmode as Sinan describes::
while (<>) {
binmode;
chomp; # strip off \r\n
while (s/search/replace/) {
# ...
}
print;
print "\n"; # add your own line ending back
}
By default, perl prints \r\n in a win32 environment. How can I override this?
I ended up creating my own file and setting binmode(fh) specifically. I could not get STDOUT (or ARGVOUT) to work reliably under both Windows 10 using perl 5.8.8 and Windows 7 with perl 5.14.4.
perl -e 'open(fh, ">x"); binmode(fh); print fh "\n";' ; od -c x
0000000 \n
Sometimes the binmode(fh) was needed here and sometimes it seemed to be the default.
I could not get binmode(STDOUT) to be work reliably. Some of the following did output just \n under Windows:
perl -e 'binmode(ARGVOUT); print "\n";' | od -c
perl -e 'binmode(STDOUT); print "\n";' | od -c
perl -e 'binmode(STDOUT); syswrite(STDOUT, "\n");' | od -c
... but then not when the output was going to a file. The following still spat out \r \n.
perl -e 'binmode(STDOUT); print "\n";' > x ; od -c x
perl -e 'binmode(ARGVOUT); print "\n";' > x ; od -c x
Interestingly, the following worked when piping to cat which then writes to a file. Perl must be seeing if STDOUT is a terminal, file, or pipe and enabling the cr-lf layer or not. Why a pipe is binary but a file is not is an interesting decision. There are also differences between running perl interactively from the command-line and running it from a script with the same args and redirects.
perl -e 'binmode(STDOUT); print "\n";' | cat > x ; od -c x
Noticed that I tried print and syswrite. I was surprised that syswrite didn't give me a direct layer to the file-handle. I also tried to copy the STDOUT file-handle and set binmode on that new file-handle but that didn't work either. PERLIO environmental variable didn't help either. The use out => ":raw"; worked under Windows 10 perl 5.8.8 but not Windows 7 perl 5.14.4 when redirected to an output file.
Btw, I wasn't doing a print "\n"; in my code when I stumbled over this problem. I was doing a print of pack("c", $num); where $num happened to be 10. Imagine my surprise when my binary file was corrupted by \rs.
Porting sucks!
A unix newline is a LINEFEED character, which is ASCII code 10.
print "\012";