The follwoing code is Perl script, grep lines with 'Stage' from hostlog. and then line by line match the content with regex, if find add the count by 1:
$command = 'grep \'Stage \' '. $hostlog;
#stage_info = qx($command);
foreach (#stage_info) {
if ( /Stage\s(\d+)\s(.*)/ ) {
$stage_number = $stage_number+1;
}
}
so how to do this in linux shell? Based on my test, the we can not loop line by line, since there is space inside.
That is a horrible piece of Perl code you've got there. Here's why:
It looks like you are not using use strict; use warnings;. That is a huge mistake, and will not prevent errors, it will just hide them.
Using qx() to grep lines from a file is a completely redundant thing to do, as this is what Perl does best itself. "Shelling out" a process like that most often slows your program down.
Use some whitespace to make your code readable. This is hard to read, and looks more complicated than it is.
You capture strings by using parentheses in your regex, but you never use these strings.
Re: $stage_number=$stage_number+1, see point 3. And also, this can be written $stage_number++. Using the ++ operator will make your code clearer, will prevent the uninitialized warnings, and save you some typing.
Here is what your code should look like:
use strict;
use warnings;
open my $fh, "<", $hostlog or die "Cannot open $hostlog for reading: $!";
while (<$fh>) {
if (/Stage\s\d+/) {
$stage_number++;
}
}
You're not doing anything with the internal captures, so why bother? You could do everything with a grep:
$ stage_number=$(grep -E 'Stage\s\d+\s' | wc -l)
This is using extended regular expressions. I believe the GNU version takes these without a -E parameter, and in Solaris, even the egrep command might not quite allow for this regular expression.
If there's something more you have to do, you've got to explain it in your question.
If I understand the issue correctly, you should be able to do this just fine in the shell:
while read; do
if echo ${REPLY} | grep -q -P "'Stage' "; then
# Do what you need to do
fi
done < test.log
Note that if your grep command supports the -P option you may be able to use the Perl regular expression as-is for the second test.
this is almost it. bash has no expression for multiple digits.
#!/bin/bash
command=( grep 'Stage ' "$hostlog" )
while read line
do
[ "$line" != "${line/Stage [0-9]/}" ] && (( ++stage_number ))
done < <( "${command[#]}" )
On the other hand taking the function of the perl script into account rather than the operations it performs the whole thing could be rewritten as
(( stage_number += ` grep -c 'Stage \d\+\s' "$hostlog" ` ))
or this
stage_number=` grep -c 'Stage \d\+\s' "$hostlog" `
if, in the original perl, stage_number is uninitialised, or is initalised to 0.
Related
I would like to inplace edit ssd_config file where i need to replace the #Port to a custom port.
Before:
#Port <portnum>
ex: #Port 22
After:
Port <customport>
ex: Port 2022
Here the custom port is coming in a variable $port.
I tried the below script but does nothing.
my $prt = "Port 2022";
my $cmd = "sed -i \'s/#Port [0-9]\+/$prt/g\' sshd_config";
system($cmd);
Tried even with tick operator.
`sed -i \"s/#Port [0-9]\+/\$prt/g\" sshd_config`;
I'd suggest to do all that in Perl, once you are running a Perl program. That way one doesn't have to run external programs, not to mention the benefits of not having to mess with all the quoting and escaping if you actually need a shell (not in the shown example though).
Then we need a few more lines of code, to read the file and and edit its content and write it back out. That shouldn't matter, given all the benefits -- but there are libraries that cut out even that as well, for example the very handy Path::Tiny
use Path::Tiny;
...
path($filename)->edit_lines( sub { s/#Port [0-9]+/Port $prt/ } );
I take it that $filename and $prt have been introduced earlier in the program.
There is also a edit method, which slurps the whole file.
Anything sed can do, Perl can do.
If this is your entire Perl program:
my $prt = "Port 2022";
my $cmd = "sed -i \'s/#Port [0-9]\+/$prt/g\' sshd_config";
system($cmd);
Then you can do it all in Perl itself from the command line.
perl -i -p -e's/#Port [0-9]+/Port 2022/g' sshd_config
system("sed ....") is invoking a shell to parse the command line, which means that everything needs to be properly escaped according to the rules of the shell. This together with string escaping inside Perl makes it hard to get right. A much simpler way is to skip the shell and call sed directly and also use string concatenation to add the $prt at the right place (and don't forget to also include the "Port" string itself since you want to have it in the output):
system('sed','-i','s/#Port [0-9]+/Port ' . $prt . '/', 'sshd_config');
Alternatively one could do it in Perl directly, i.e. something like this:
open(my $fh,'<','sshd_config') or die $!;
my #lines = map { s/#Port \d+/Port $prt/ } <$fh>;
open($fh,'>','sshd_config') or die $!;
print $fh #lines;
close($fh);
This is a bit longer but does not rely on starting an external program and is thus faster. And if there is more to do than a simple replacement it is also more flexible.
I'm having an issue with some code and I'm wondering if anyone can assist.
Basically I'm trying to execute an isql query against a database and assign it to a scalar variable. The isql command makes use of the column seperator which is defined as the pipe symbol.
So I have it set-up as such:
my $command = "isql -S -U -s| -i";
my $isql_output = `$command`;
The isql command works in isolation but when issued as a backtick it stops at the pipe. I've tried concatenating the $command string using sub-strings, using single quotes and backslash escaping items such as -s\"\|\" to no avail. I've also tried using qx instead of backticks.
Unfortunately I'm currently using an older version of perl (v5.6.1) with limited scope for upgrade so I'm not sure if I can resolve this.
You have to quote the | in a way that the shell does not recognize it as a special character. Two ways:
Put the -s| into single quotes: '-s|'. Perl will leave single quotes inside double quoted strings alone and pass them to the shell unmodified.
Escape the | with two backslashes: -s\\|. Why two? The first one is seen by Perl telling it to pass the next character through unmodified. Therefore the shell sees -s\| and does something very similar: it sees the single \ and knows not to treat the next char, |, special.
The problem is that the command is being executed through a shell.
You can avoid this by passing the command and arguments in a list rather than a single string.
The backtick construct does not support that, so you would need to use the open() function instead.
I haven't tested the following code but it gives the idea:
my #command = (qw(isql -Sserver -Uuser -Ppassword -s| -w4096), '–i' . $file);
print join(' ', #command), "\n";
open(my $fh, '-|', #command)
or die "failed to run isql command: $#\n";
my #isql_output = <$fh>;
close($fh);
my $isql_output = $isql_output[0]; chomp($isql_output);
If you're working with a 15 year old version of Perl (which Oracle users tend to do) I'm not sure this will all be supported. For instance, you may need to write chop instead of chomp.
UPDATE: the problem is not the Perl version, but this construct not being supported on Windows, according to the documentation. This must be qualified: I use Perl on Cygwin and it works fine there, but I don't know whether you can use Cygwin.
Single quotes should work. Try to run test perl script:
my $cmd = "./test.sh -S -U -s '|' -i";
print `$cmd`;
With test.sh:
#!/bin/sh
echo $#
Output should be -S -U -s | -i
On Unix, all these three generate the same result
system("top -H -p $pid -n 1"); #ver1
system("top", "H", "p $pid", "n 1"); #ver2
system("top", "-H", "-p $pid", "-n 1"); #ver3
What is the difference between ver2 and ver3?
Is there any reason I should use ver2 and ver3, and not ver1?
They do not even support piping the results, for example, are there any ver2 and ver3 equivalents of the following call?
system("top -H -p $pid -n 1 | grep myprocess | wc -l");
Even it looks same it is not same:
$ perl -e 'system("./test.pl -H -p $$ -n 1");system("./test.pl", "H", "p $$", "n 1");system("./test.pl", "-H", "-p $$", "-n 1");'
-H,-p,10497,-n,1
H,p 10497,n 1
-H,-p 10497,-n 1
$ cat ./test.pl
#!/usr/bin/perl
$\="\n";
$,=",";
print #ARGV;
It is up to top implementation that it works same. Other applications may not work same.
Quoth perlfunc for system:
Note that argument processing varies depending on the number of arguments. If there is more than one argument
in LIST, or if LIST is an array with more than one value, starts the program given by the first element of
the list with arguments given by the rest of the list. If there is only one scalar argument, the argument is checked for shell metacharacters, and if there are any, the entire argument is passed to the system's command shell for parsing (this is /bin/sh -c on Unix platforms, but varies on other platforms). If there are no shell metacharacters in the argument, it is split into words and passed directly to execvp , which is more efficient.
So if $pid is just digits, all are equivalent.
To interpolate results of an arbitrary shell command including pipes use qx and friends.
As a practical reason for using LIST, sometimes your command-line arguments contain spaces or other characters that would confuse your shell.
system("mplayer.exe", "--volume", "75",
q[C:/Program Files/My Music Player/Music Library/The "Music" Song.mp3]);
What is the difference between ver2 and ver3?
Just in what arguments you're passing to top. I don't know of a version of top that will take switches without dashes like some versions of ps do, so you should use version 3.
Is there any reason I should use ver2 and ver3, and not ver1?
If you pass a single string to system it will run it via your shell. This means it will be shell interpreted. Any stray spaces or shell meta characters (quotes, dollar signs, etc...) in the arguments would be interpreted and possibly mess things up. It's also a potential security hole.
For example, if $pid was something like '10; echo pwnd; echo ' then you'd run top -H -p 10 then echo pwnd then echo -n1.
So for both safety and security, unless you need shell processing (see below) you should pass system a list.
Are there any ver2 and ver3 equivalents which allow pipes?
No, piping and redirection is done by the shell. You have to use something other than system. You can do it with open, but it's a pain in the ass. Easiest way is to use IPC::Run.
use IPC::Run;
my $out;
run ["echo", "foo\nbar\nbaz"], "|",
["grep", "ba"], "|",
["wc", "-l"],
\$out;
print $out; # 2
But really if you're just grepping and counting a handful of lines, use Perl.
my $out;
run ["echo", "foo\nbar\nbaz"], '>', \$out;
my $count = grep { /ba/ } split /\n/, $out;
print $count;
I find a lot of Perl one-liners online. Sometimes I want to convert these one-liners into a script, because otherwise I'll forget the syntax of the one-liner.
For example, I'm using the following command (from nagios.com):
tail -f /var/log/nagios/nagios.log | perl -pe 's/(\d+)/localtime($1)/e'
I'd to replace it with something like this:
tail -f /var/log/nagios/nagios.log | ~/bin/nagiostime.pl
However, I can't figure out the best way to quickly throw this stuff into a script. Does anyone have a quick way to throw these one-liners into a Bash or Perl script?
You can convert any Perl one-liner into a full script by passing it through the B::Deparse compiler backend that generates Perl source code:
perl -MO=Deparse -pe 's/(\d+)/localtime($1)/e'
outputs:
LINE: while (defined($_ = <ARGV>)) {
s/(\d+)/localtime($1);/e;
}
continue {
print $_;
}
The advantage of this approach over decoding the command line flags manually is that this is exactly the way Perl interprets your script, so there is no guesswork. B::Deparse is a core module, so there is nothing to install.
Take a look at perlrun:
-p
causes Perl to assume the following loop around your program, which makes it iterate over filename arguments somewhat like sed:
LINE:
while (<>) {
... # your program goes here
} continue {
print or die "-p destination: $!\n";
}
If a file named by an argument cannot be opened for some reason, Perl warns you about it, and moves on to the next file. Note that the lines are printed automatically. An error occurring during printing is treated as fatal. To suppress printing use the -n switch. A -p overrides a -n switch.
BEGIN and END blocks may be used to capture control before or after the implicit loop, just as in awk.
So, simply take this chunk of code, insertyour code at the "# your program goes here" line, and viola, your script is ready!
Thus, it would be:
#!/usr/bin/perl -w
use strict; # or use 5.012 if you've got newer perls
while (<>) {
s/(\d+)/localtime($1)/e
} continue {
print or die "-p destination: $!\n";
}
That one's really easy to store in a script!
#! /usr/bin/perl -p
s/(\d+)/localtime($1)/e
The -e option introduces Perl code to be executed—which you might think of as a script on the command line—so drop it and stick the code in the body. Leave -p in the shebang (#!) line.
In general, it's safest to stick to at most one "clump" of options in the shebang line. If you need more, you could always throw their equivalents inside a BEGIN {} block.
Don't forget chmod +x ~/bin/nagiostime.pl
You could get a little fancier and embed the tail part too:
#! /usr/bin/perl -p
BEGIN {
die "Usage: $0 [ nagios-log ]\n" if #ARGV > 1;
my $log = #ARGV ? shift : "/var/log/nagios/nagios.log";
#ARGV = ("tail -f '$log' |");
}
s/(\d+)/localtime($1)/e
This works because the code written for you by -p uses Perl's "magic" (2-argument) open that processes pipes specially.
With no arguments, it transforms nagios.log, but you can also specify a different log file, e.g.,
$ ~/bin/nagiostime.pl /tmp/other-nagios.log
Robert has the "real" answer above, but it's not very practical. The -p switch does a bit of magic, and other options have even more magic (e.g. check out the logic behind the -i flag). In practice, I'd simply just make a bash alias/function to wrap around the oneliner, rather than convert it to a script.
Alternatively, here's your oneliner as a script: :)
#!/usr/bin/bash
# takes any number of arguments: the filenames to pipe to the perl filter
tail -f $# | perl -pe 's/(\d+)/localtime($1)/e'
There are some good answers here if you want to keep the one-liner-turned-script around and possibly even expand upon it, but the simplest thing that could possibly work is just:
#!/usr/bin/perl -p
s/(\d+)/localtime($1)/e
Perl will recognize parameters on the hashbang line of the script, so instead of writing out the loop in full, you can just continue to do the implicit loop with -p.
But writing the loop explicitly and using -w and "use strict;" are good if plan to use it as a starting point for writing a longer script.
#!/usr/bin/env perl
while(<>) {
s/(\d+)/localtime($1)/e;
print;
}
The while loop and the print is what -p does automatically for you.
As I understand (Perl is new to me) Perl can be used to script against a Unix command line. What I want to do is run (hardcoded) command line calls, and search the output of these calls for RegEx matches. Is there a way to do this simply in Perl? How?
EDIT: Sequence here is:
-Call another program.
-Run a regex against its output.
my $command = "ls -l /";
my #output = `$command`;
for (#output) {
print if /^d/;
}
The qx// quasi-quoting operator (for which backticks are a shortcut) is stolen from shell syntax: run the string as a command in a new shell, and return its output (as a string or a list, depending on context). See perlop for details.
You can also open a pipe:
open my $pipe, "$command |";
while (<$pipe>) {
# do stuff
}
close $pipe;
This allows you to (a) avoid gathering the entire command's output into memory at once, and (b) gives you finer control over running the command. For example, you can avoid having the command be parsed by the shell:
open my $pipe, '-|', #command, '< single argument not mangled by shell >';
See perlipc for more details on that.
You might be able to get away without Perl, as others have mentioned. However, if there is some Perl feature you need, such as extended regex features or additional text manipulation, you can pipe your output to perl then do what you need. Perl's -e switch let's you specify the Perl program on the command line:
command | perl -ne 'print if /.../'
There are several other switches you can pass to perl to make it very powerful on the command line. These are documented in perlrun. Also check out some of the articles in Randal Schwartz's Unix Review column, especially his first article for them. You can also google for Perl one liners to find lots of examples.
Do you need Perl at all? How about
command -I use | grep "myregexp" && dosomething
right in the shell?
#!/usr/bin/perl
sub my_action() {
print "Implement some action here\n";
}
open PROG, "/path/to/your/command|" or die $!;
while (<PROG>) {
/your_regexp_here/ and my_action();
print $_;
}
close PROG;
This will scan output from your command, match regexps and do some action (which now is printing the line)
In Perl you can use backticks to execute commands on the shell. Here is a document on using backticks. I'm not sure about how to capture the output, but I'm sure there's more than a way to do it.
You indeed use a one-liner in a case like this. I recently coded up one that I use, among other ways, to produce output which lists the directory structure present in a .zip archive (one dir entry per line). So using that output as an example of command output that we'd like to filter, we could put a pipe in and then use perl with the -n -e flags to filter the incoming data (and/or do other things with it):
[command_producing_text_output] | perl -MFile::Path -n -e \
"BEGIN{#PTM=()} if (m{^perl/(bin|lib(?!/site))}) {chomp;push #PTM,$_}" ^
-e "END{#WDD=mkpath (\#PTM,1);" ^
-e "printf qq/Created %u dirs to reflect part of structure present in the .ZIP file\n/, scalar(#WDD);}"
the shell syntax used, including: quoting of perl code and escaping of newlines, reflects CMD.exe usage in Windows NT-like consoles. If you need to, mentally replace
"^" with "\" and " with ' in the appropriate places.
The one-liner above adds only the directory names that start with "perl/bin" or
"perl/lib (not followed by "/site"); it then creates those directories. You wind
up with a (empty) tree that you can use for whatever evil purposes you desire.
The main point is to illustrate that there are flags available (-n, -p) to
allow perl to loop over each input record (line), and that what you can do is unlimited in terms of complexity.