I find a lot of Perl one-liners online. Sometimes I want to convert these one-liners into a script, because otherwise I'll forget the syntax of the one-liner.
For example, I'm using the following command (from nagios.com):
tail -f /var/log/nagios/nagios.log | perl -pe 's/(\d+)/localtime($1)/e'
I'd to replace it with something like this:
tail -f /var/log/nagios/nagios.log | ~/bin/nagiostime.pl
However, I can't figure out the best way to quickly throw this stuff into a script. Does anyone have a quick way to throw these one-liners into a Bash or Perl script?
You can convert any Perl one-liner into a full script by passing it through the B::Deparse compiler backend that generates Perl source code:
perl -MO=Deparse -pe 's/(\d+)/localtime($1)/e'
outputs:
LINE: while (defined($_ = <ARGV>)) {
s/(\d+)/localtime($1);/e;
}
continue {
print $_;
}
The advantage of this approach over decoding the command line flags manually is that this is exactly the way Perl interprets your script, so there is no guesswork. B::Deparse is a core module, so there is nothing to install.
Take a look at perlrun:
-p
causes Perl to assume the following loop around your program, which makes it iterate over filename arguments somewhat like sed:
LINE:
while (<>) {
... # your program goes here
} continue {
print or die "-p destination: $!\n";
}
If a file named by an argument cannot be opened for some reason, Perl warns you about it, and moves on to the next file. Note that the lines are printed automatically. An error occurring during printing is treated as fatal. To suppress printing use the -n switch. A -p overrides a -n switch.
BEGIN and END blocks may be used to capture control before or after the implicit loop, just as in awk.
So, simply take this chunk of code, insertyour code at the "# your program goes here" line, and viola, your script is ready!
Thus, it would be:
#!/usr/bin/perl -w
use strict; # or use 5.012 if you've got newer perls
while (<>) {
s/(\d+)/localtime($1)/e
} continue {
print or die "-p destination: $!\n";
}
That one's really easy to store in a script!
#! /usr/bin/perl -p
s/(\d+)/localtime($1)/e
The -e option introduces Perl code to be executed—which you might think of as a script on the command line—so drop it and stick the code in the body. Leave -p in the shebang (#!) line.
In general, it's safest to stick to at most one "clump" of options in the shebang line. If you need more, you could always throw their equivalents inside a BEGIN {} block.
Don't forget chmod +x ~/bin/nagiostime.pl
You could get a little fancier and embed the tail part too:
#! /usr/bin/perl -p
BEGIN {
die "Usage: $0 [ nagios-log ]\n" if #ARGV > 1;
my $log = #ARGV ? shift : "/var/log/nagios/nagios.log";
#ARGV = ("tail -f '$log' |");
}
s/(\d+)/localtime($1)/e
This works because the code written for you by -p uses Perl's "magic" (2-argument) open that processes pipes specially.
With no arguments, it transforms nagios.log, but you can also specify a different log file, e.g.,
$ ~/bin/nagiostime.pl /tmp/other-nagios.log
Robert has the "real" answer above, but it's not very practical. The -p switch does a bit of magic, and other options have even more magic (e.g. check out the logic behind the -i flag). In practice, I'd simply just make a bash alias/function to wrap around the oneliner, rather than convert it to a script.
Alternatively, here's your oneliner as a script: :)
#!/usr/bin/bash
# takes any number of arguments: the filenames to pipe to the perl filter
tail -f $# | perl -pe 's/(\d+)/localtime($1)/e'
There are some good answers here if you want to keep the one-liner-turned-script around and possibly even expand upon it, but the simplest thing that could possibly work is just:
#!/usr/bin/perl -p
s/(\d+)/localtime($1)/e
Perl will recognize parameters on the hashbang line of the script, so instead of writing out the loop in full, you can just continue to do the implicit loop with -p.
But writing the loop explicitly and using -w and "use strict;" are good if plan to use it as a starting point for writing a longer script.
#!/usr/bin/env perl
while(<>) {
s/(\d+)/localtime($1)/e;
print;
}
The while loop and the print is what -p does automatically for you.
Related
What would be an example of how I can call a shell command, say 'ls -a' in a Perl script and the way to retrieve the output of the command as well?
How to run a shell script from a Perl program
1. Using system system($command, #arguments);
For example:
system("sh", "script.sh", "--help" );
system("sh script.sh --help");
System will execute the $command with
#arguments and return to your script when finished. You may check $!
for certain errors passed to the OS by the external application. Read
the documentation for system for the nuances of how various
invocations are slightly different.
2. Using exec
This is very similar to the use of system, but it will
terminate your script upon execution. Again, read the documentation
for exec for more.
3. Using backticks or qx//
my $output = `script.sh --option`;
my $output = qx/script.sh --option/;
The backtick operator and it's equivalent qx//, excute the command and options inside the operator and return that commands output to STDOUT when it finishes.
There are also ways to run external applications through creative use of open, but this is advanced use; read the documentation for more.
From Perl HowTo, the most common ways to execute external commands from Perl are:
my $files = `ls -la` — captures the output of the command in $files
system "touch ~/foo" — if you don't want to capture the command's output
exec "vim ~/foo" — if you don't want to return to the script after executing the command
open(my $file, '|-', "grep foo"); print $file "foo\nbar" — if you want to pipe input into the command
Examples
`ls -l`;
system("ls -l");
exec("ls -l");
Look at the open function in Perl - especially the variants using a '|' (pipe) in the arguments. Done correctly, you'll get a file handle that you can use to read the output of the command. The back tick operators also do this.
You might also want to review whether Perl has access to the C functions that the command itself uses. For example, for ls -a, you could use the opendir function, and then read the file names with the readdir function, and finally close the directory with (surprise) the closedir function. This has a number of benefits - precision probably being more important than speed. Using these functions, you can get the correct data even if the file names contain odd characters like newline.
As you become more experienced with using Perl, you'll find that there are fewer and fewer occasions when you need to run shell commands. For example, one way to get a list of files is to use Perl's built-in glob function. If you want the list in sorted order you could combine it with the built-in sort function. If you want details about each file, you can use the stat function. Here's an example:
#!/usr/bin/perl
use strict;
use warnings;
foreach my $file ( sort glob('/home/grant/*') ) {
my($dev,$ino,$mode,$nlink,$uid,$gid,$rdev,$size,$atime,$mtime,$ctime,$blksize,$blocks)
= stat($file);
printf("%-40s %8u bytes\n", $file, $size);
}
There are a lot of ways you can call a shell command from a Perl script, such as:
back tick
ls which captures the output and gives back to you.
system
system('ls');
open
Refer #17 here: Perl programming tips
You might want to look into open2 and open3 in case you need bidirectional communication.
I have been using system and qq to run linux programs inside perl. And it has worked well.
#!/usr/bin/perl # A hashbang line in perl
use strict; # It can save you a lot of time and headache
use warnings; # It helps you find typing mistakes
# my keyword in Perl declares the listed variable
my $adduser = '/usr/sbin/adduser';
my $edquota = '/usr/sbin/edquota';
my $chage = '/usr/bin/chage';
my $quota = '/usr/bin/quota';
my $nomeinteiro;
my $username;
my $home;
# system() function executes a system shell command
# qq() can be used in place of double quotes
system qq($adduser --home $home --gecos "$fullname" $username);
system qq($edquota -p john $username);
system qq($chage -E \$(date -d +180days +%Y-%m-%d) $username);
system qq($chage -l $username);
system qq($quota -s $username);
I am having some trouble of getting the argument passed in in the following script
echo "abc"|perl <<'EOF'
#how to get "abc". it seems not $ARGV[0] nor in <STDIN>
EOF
Thank you.
The precise command line you have there may be your problem, if that is what you're actually executing. What you are saying there is "put 'abc' on the standard input of the next thing in the pipeline. Now run a Perl script consisting of a single comment."
This will do nothing, because there's nothing executable in that Perl script. Try this:
echo "abc" | perl -e 'print <STDIN>'
If you have a short Perl script the -e option is the way to go.
Your example is not using argument, it's using standard input. You can read standard input with the I/O operators. If you actually mean that you want an argument like myscript.pl --arg then I would recommend using Getopt::Long.
You have not passed any argument to the Perl script.
You redirected the Perl script itself so it comes from standard input; that means that the piped output goes nowhere and cannot be seen by Perl.
Reconsider how you're invoking your script. Maybe:
perl script.pl "abc"
where script.pl is a file that contains the Perl script you used as a here-document. Or simply make that script executable (perhaps without the .pl suffix).
Your problem is that both the pipe and the here-document redirect the STDIN. And the here-document wins, so the perl process never sees the pipe; it gets the script on STDIN (and has read to EOF before running the script, so that will see STDIN at EOF).
Observe:
$ echo "abc" | perl <<'EOF'
print "[What have we here?]\n";
seek(STDIN, 0, 0);
print <STDIN>;
print "[Well, what do you know ...]\n";
EOF
[What have we here?]
print "[What have we here?]\n";
seek(STDIN, 0, 0);
print <STDIN>;
print "[Well, what do you know ...]\n";
[Well, what do you know ...]
$
Moral: Don't try to mix pipes and here-documents in the shell. :)
The follwoing code is Perl script, grep lines with 'Stage' from hostlog. and then line by line match the content with regex, if find add the count by 1:
$command = 'grep \'Stage \' '. $hostlog;
#stage_info = qx($command);
foreach (#stage_info) {
if ( /Stage\s(\d+)\s(.*)/ ) {
$stage_number = $stage_number+1;
}
}
so how to do this in linux shell? Based on my test, the we can not loop line by line, since there is space inside.
That is a horrible piece of Perl code you've got there. Here's why:
It looks like you are not using use strict; use warnings;. That is a huge mistake, and will not prevent errors, it will just hide them.
Using qx() to grep lines from a file is a completely redundant thing to do, as this is what Perl does best itself. "Shelling out" a process like that most often slows your program down.
Use some whitespace to make your code readable. This is hard to read, and looks more complicated than it is.
You capture strings by using parentheses in your regex, but you never use these strings.
Re: $stage_number=$stage_number+1, see point 3. And also, this can be written $stage_number++. Using the ++ operator will make your code clearer, will prevent the uninitialized warnings, and save you some typing.
Here is what your code should look like:
use strict;
use warnings;
open my $fh, "<", $hostlog or die "Cannot open $hostlog for reading: $!";
while (<$fh>) {
if (/Stage\s\d+/) {
$stage_number++;
}
}
You're not doing anything with the internal captures, so why bother? You could do everything with a grep:
$ stage_number=$(grep -E 'Stage\s\d+\s' | wc -l)
This is using extended regular expressions. I believe the GNU version takes these without a -E parameter, and in Solaris, even the egrep command might not quite allow for this regular expression.
If there's something more you have to do, you've got to explain it in your question.
If I understand the issue correctly, you should be able to do this just fine in the shell:
while read; do
if echo ${REPLY} | grep -q -P "'Stage' "; then
# Do what you need to do
fi
done < test.log
Note that if your grep command supports the -P option you may be able to use the Perl regular expression as-is for the second test.
this is almost it. bash has no expression for multiple digits.
#!/bin/bash
command=( grep 'Stage ' "$hostlog" )
while read line
do
[ "$line" != "${line/Stage [0-9]/}" ] && (( ++stage_number ))
done < <( "${command[#]}" )
On the other hand taking the function of the perl script into account rather than the operations it performs the whole thing could be rewritten as
(( stage_number += ` grep -c 'Stage \d\+\s' "$hostlog" ` ))
or this
stage_number=` grep -c 'Stage \d\+\s' "$hostlog" `
if, in the original perl, stage_number is uninitialised, or is initalised to 0.
context: I'm a beginner in Perl and struggling, please be patient, thanks.
the question: there is a one-liner that seems to do the job I want (in a cygwin console it does fine on my test file). So now I would need to turn it into a script, but I can't manage that unfortunately.
The one-liner in question is provided in the answer by Aki here Delete lines in perl
perl -ne 'print unless /HELLO/../GOODBYE/' <file_name>
Namely I would like to have a script that opens my file "test.dat" and removes the lines between some strings HELLO and GOODBYE. Here is what I tried and which fails (the path is fine for cygwin):
#!/bin/perl
use strict;
use warnings;
open (THEFILE, "+<test.dat") || die "error opening";
my $line;
while ($line =<THEFILE>){
next if /hello/../goodbye/;
print THEFILE $line;
}
close (THEFILE);
Many thanks in advance!
Your one-liner is equivalent to the following
while (<>) {
print unless /HELLO/../GOODBYE/;
}
Your code does something quite different. You should not attempt to read and write to the same file handle, that usually does not do what you think. When you want to quickly edit a file, you can use the -i "in-place edit" switch:
perl -ni -e 'print unless /HELLO/../GOODBYE/' file
Do note that changes to the file are irreversible, so you should make backups. You can use the backup option for that switch, e.g. -i.bak, but be aware that it is not flawless, as running the same command twice will still overwrite your backup (by saving to the same file name twice).
The simplest and safest way to do it, IMO, is to simply use shell redirection
perl script.pl file.txt > newfile.txt
While using the script file I showed at the top.
As I understand (Perl is new to me) Perl can be used to script against a Unix command line. What I want to do is run (hardcoded) command line calls, and search the output of these calls for RegEx matches. Is there a way to do this simply in Perl? How?
EDIT: Sequence here is:
-Call another program.
-Run a regex against its output.
my $command = "ls -l /";
my #output = `$command`;
for (#output) {
print if /^d/;
}
The qx// quasi-quoting operator (for which backticks are a shortcut) is stolen from shell syntax: run the string as a command in a new shell, and return its output (as a string or a list, depending on context). See perlop for details.
You can also open a pipe:
open my $pipe, "$command |";
while (<$pipe>) {
# do stuff
}
close $pipe;
This allows you to (a) avoid gathering the entire command's output into memory at once, and (b) gives you finer control over running the command. For example, you can avoid having the command be parsed by the shell:
open my $pipe, '-|', #command, '< single argument not mangled by shell >';
See perlipc for more details on that.
You might be able to get away without Perl, as others have mentioned. However, if there is some Perl feature you need, such as extended regex features or additional text manipulation, you can pipe your output to perl then do what you need. Perl's -e switch let's you specify the Perl program on the command line:
command | perl -ne 'print if /.../'
There are several other switches you can pass to perl to make it very powerful on the command line. These are documented in perlrun. Also check out some of the articles in Randal Schwartz's Unix Review column, especially his first article for them. You can also google for Perl one liners to find lots of examples.
Do you need Perl at all? How about
command -I use | grep "myregexp" && dosomething
right in the shell?
#!/usr/bin/perl
sub my_action() {
print "Implement some action here\n";
}
open PROG, "/path/to/your/command|" or die $!;
while (<PROG>) {
/your_regexp_here/ and my_action();
print $_;
}
close PROG;
This will scan output from your command, match regexps and do some action (which now is printing the line)
In Perl you can use backticks to execute commands on the shell. Here is a document on using backticks. I'm not sure about how to capture the output, but I'm sure there's more than a way to do it.
You indeed use a one-liner in a case like this. I recently coded up one that I use, among other ways, to produce output which lists the directory structure present in a .zip archive (one dir entry per line). So using that output as an example of command output that we'd like to filter, we could put a pipe in and then use perl with the -n -e flags to filter the incoming data (and/or do other things with it):
[command_producing_text_output] | perl -MFile::Path -n -e \
"BEGIN{#PTM=()} if (m{^perl/(bin|lib(?!/site))}) {chomp;push #PTM,$_}" ^
-e "END{#WDD=mkpath (\#PTM,1);" ^
-e "printf qq/Created %u dirs to reflect part of structure present in the .ZIP file\n/, scalar(#WDD);}"
the shell syntax used, including: quoting of perl code and escaping of newlines, reflects CMD.exe usage in Windows NT-like consoles. If you need to, mentally replace
"^" with "\" and " with ' in the appropriate places.
The one-liner above adds only the directory names that start with "perl/bin" or
"perl/lib (not followed by "/site"); it then creates those directories. You wind
up with a (empty) tree that you can use for whatever evil purposes you desire.
The main point is to illustrate that there are flags available (-n, -p) to
allow perl to loop over each input record (line), and that what you can do is unlimited in terms of complexity.