I am using webmin and I am trying to change some settings in a file. I am having problems if the person uses any weird characters that might trip up sed or Perl using the following code:
&execute_command("sed -i 's/^$Pref.*\$/$Pref \"$in{$Pref}\"/g' $DIR/pserver.prefs.cache");
Where execute_command is a webmin function to basically run a special system call. $pref is the preference name such as "SERVERNAME", "OPTION2", etc. and $in{Pref} is going to be the option I want set for the PREF. For example here is a typical pserver.prefs:
SERVERNAME "Test Name"
OWNERPASSWORD "Hd8sdH&3"
Therefore, if we wanted to change SERVERNAME to say Tes"t#&^"#'"##& and OWNERPASSWORD to *#(&'"#$"(')29 then they would be passed in as $in{Pref}. What is the easiest way to escape the $in{} variables so that they can work OK with sed, or better yet, what is a way I can convert my sed command to a strictly Perl command so that it doesn't have errors?
Update:
Awesome, now I'm just trying to get it to work with and I get this error:
**/bin/sh: -c: line 0: unexpected EOF while looking >for matching `"' /bin/sh: -c: line 1: syntax error: unexpected end of file**
This does not work:
my $Pref = "&*())(*&'''''^%$##!";
&execute_command("perl -pi -e 's/^SERVERNAME.*\$/SERVERNAME \"\Q$Pref\E\"/g' $DIR/pserver.prefs");
This does:
my $Pref = "&*())(*&^%$##!";
&execute_command("perl -pi -e 's/^SERVERNAME.*\$/SERVERNAME \"\Q$Pref\E\"/g' $DIR/pserver.prefs");
Perl's regex support includes the \Q and \E operators, which will cause it to avoid interpreting regex symbols within their scope, yet they allow variable interpolation.
This works:
$i = '(*&%)*$£(*';
if ($i =~ /\Q$i\E/){
print "matches!\n";
}
Without the \Q and \E, you'd get an error because of the regex symbols in $i.
The most trivial part is simply to stop executing a command as a single string. Get the shell out of it. Assuming your execute_command function just calls system under the covers, try:
execute_command(qw/perl -pi -e/, 's/^SERVERNAME.*$/SERVERNAME "\Q$Pref\E"/g', "$DIR/pserver.prefs");
That's better, but not perfect. After all, the user could put in something silly like "#[system qw:rm -rf /:]" and then silly things would happen. I think there are ways around this, too, but the most trivial might be to simply do the work inside your code. How to do that? Maybe starting with what perl is doing with the "-pi" flags might help. Let's take a peek:
$ perl -MO=Deparse -pi -e 's/^SERVERNAME.*$/SERVERNAME "\Qfoo\E"/'
BEGIN { $^I = ""; }
LINE: while (defined($_ = <ARGV>)) {
s/^SERVERNAME.*$/SERVERNAME "foo"/;
}
continue {
print $_;
}
Maybe you can do the same thing in your code? Not sure how easy that is to replicate, especially that $^I bit. Worst case scenario, read the file, write to a new file, delete the original file, rename the new file to the original name. That'll help get rid of all the exposures of passing dangerous junk around.
Related
I have the following Perl 'one-liner' script (found it online, so not mine):
perl -lsne '
/$today.* \[([0-9.]+)\]:.+dovecot_(?:login|plain):([^\s]+).* for (.*)/
and $sender{$2}{r}+=scalar (split / /,$3)
and $sender{$2}{i}{$1}=1;
END {
foreach $sender(keys %sender){
printf"Recip=%05d Hosts=%03d Auth=%s\n",
$sender{$sender}{r},
scalar (keys %{$sender{$sender}{i}}),
$sender;
}
}
' -- -today=$(date +%F) /var/log/exim_mainlog | sort
I've been trying to understand its innards, because I would like to modify it to re-use its functionality.
Some questions I got:
What does the flag -lsne does? (From what I know, it's got to be, at least, 3 different flags in one)
Where does $sender gets its value from?
What about that (?:login|plain) segment, are they 'variables'? (I get that's ReGex, I'm just not familiarized with it)
What I'm trying to achieve:
Get the number of emails sent by each user in a SMTP relay periodically (cron job)
If there's an irregular number of emails (say, 500 in a 1-hour timespan), do something (like shutting of the service, or send a notification)
Why I'm trying to achieve this:
Lately, someone has been using my SMTP server to send spam, so I would like to monitor email activity so they stop abusing the SMTP relay resources. (Security-related suggestions are always welcomed, but out of topic for this question. Trying to focus on the script for now)
What I'm NOT trying to achieve:
To get the script done by third-parties. (Just try and point me in the right direction, maybe an example)
So, any suggestions, guidance,and friendly comments are welcomed. I understand this may be an out-of-topic question, yet I've been struggling with this for almost a week and my background with Perl is null.
Thanks in advance.
What does the flag -lsne does? (From what I know, it's got to be, at least, 3 different flags in one)
-l causes lines of input read in to be auto-chomped, and lines of
output printed out to have "\n" auto-appended
-s enables switch
parsing. This is what creates the variable $today, because a
command-line switch of --today=$(date +%F) was passed.
-n surrounds the entire "one-liner" in a while (<>) { ... } loop.
Effectively reading every line from standard input and running the
body of the one liner on that line
-e is the switch that tells
perl to execute the following code from the command line, rather
than running a file containing Perl code
Where does $sender gets its value from?
I suspect you are confusing $sender with %sender. The code uses $sender{$2}{r} without explicitly mentioning %sender. This is a function of Perl called "auto-vivification". Basically, because we used $sender{$2}{r}, perl automatically created a variable %sender, and added a key whose name is whatever is in $2, and set the value of that key in %sender to be a reference to a new hash. It then set that new hash to have a key 'r' and a value of scalar (split / /,$3)
What about that (?:login|plain) segment, are they 'variables'? (I get that's ReGex, I'm just not familiarized with it)
It's saying that this portion of the regular expression will match either 'login' or 'plain'. The ?: at the beginning tells Perl that these parentheses are used only for clustering, not capturing. In other words, the result of this portion of the pattern match will not be stored in the $1, $2, $3, etc variables.
-MO=Deparse is your friend for understanding one-liners (and one liners that wrap into five lines on your terminal):
$ perl -MO=Deparse -lsne '/$today.* \[([0-9.]+)\]:.+dovecot_( ...
BEGIN { $/ = "\n"; $\ = "\n"; }
LINE:
while ( defined($_ = <ARGV>) ) {
chomp $_;
$sender{$2}{'i'}{$1} = 1 if
/$today.* \[([0-9.]+)\]:.+dovecot_(?:login|plain):([^\s]+).* for (.*)/
and $sender{$2}{'r'} += scalar split(/ /, $3, 0);
sub END {
foreach $sender (keys %sender) {
printf "Recip=%05d Hosts=%03d Auth=%s\n",
$sender{$sender}{'r'},
scalar keys %{$sender{$sender}{'i'};}, $sender;
}
}
}
-e syntax OK
[newlines and indentation added for clarity]
What does the flag -lsne does? (From what I know, it's got to be, at least, 3 different flags in one)
You can access a summary of the available perl command line options by running '~$ perl -h' in the terminal. Below are filtered out the specific command line options you were asking about.
~$ perl -h|perl -ne 'print if /^\s+(-l|-s|-n|-e)/'
-e program one line of program (several -e's allowed, omit programfile)
-l[octal] enable line ending processing, specifies line terminator
-n assume "while (<>) { ... }" loop around program
-s enable rudimentary parsing for switches after programfile
Two examples of the '-s' command line option in use.
~$ perl -se 'print "Todays date is $today\n"' -- -today=`date +%F`
Todays date is 2016-10-17
~$ perl -se 'print "The sky is $color.\n"' -- -color='blue'
The sky is blue.
For detailed explanations of those command line options read the online documentation below.
http://perldoc.perl.org/perlrun.html
Or run the command below from your terminal.
~$ perldoc perlrun
Unrelated to the questions of the OP, I'm aware that this is not a complete answer (added as much as I was able to at the moment), so if this post/answer violates any SO rules, the moderators should remove it. Thx.
I know of Perl compiler back-end that allows you to convert any one-liner into script on following matter:
perl -MO=Deparse -pe 's/(\d+)/localtime($1)/e'
Which would give the following output
LINE: while (defined($_ = <ARGV>)) {
s/(\d+)/localtime($1);/e;
}
continue {
print $_;
}
Is there possibly a reverse tool, usable from command-line, which provided full script will generate one-liner version of it?
Note: The above example was taken from https://stackoverflow.com/a/2822721/4313369.
Perl is a free-form syntax language with clear statement and block separators, so there is nothing preventing you from simply folding a normal script up into a single line.
To use your example in reverse, you could write it as:
$ perl -e 'LINE: while (defined($_ = <ARGV>)) { s/(\d+)/localtime($1);/e; } continue { print $_; }'
This is a rather contrived example, since there is a shorter and clearer way to write it. Presumably you're starting with scripts that are already as short and clear as they should be.
Any use statements in your program can be turned into -M flags.
Obviously you have to be careful about quoting and other characters that are special to the shell. You mention running this on a remote system, by which I assume you mean SSH, which means you now have two levels of shell to sneak any escaping through. It can be tricky to work out the proper sequence of escapes, but it's always doable.
This method may work for automatically translating a Perl script on disk into a one-liner:
$ perl -e "$(tr '\n' ' ' < myscript.pl)"
The Perl script can't have comments in it, since that would comment out the entire rest of the script. If that's a problem, a bit of egrep -v '\w+#' type hackery could solve the problem.
I am trying to run the following command in perl script :
#!/usr/bin/perl
my $cmd3 =`sed ':cycle s/^\(\([^,]*,\)\{0,13\}[^,|]*\)|[^,]*/\1/;t cycle' file1 >file2`;
system($cmd3);
but is not producing any output nor any error.
Although when I am running the command from command line it is working perfectly and gives desired output. Can you guys please help what I am doing wrong here ?
Thanks
system() doesn't return the output, just the exit status.
To see the output, print $cmd3.
my $cmd3 = `sed ':cycle s/^\(\([^,]*,\)\{0,13\}[^,|]*\)|[^,]*/\1/;t cycle' file1 >file2`;
print "$cmd3\n";
Edit:
If you want to check for exceptional return values, use CPAN module IPC::System::Simple:
use IPC::System::Simple qw(capture);
my $result = capture("any-command");
Running sed from inside Perl is just insane.
#!/usr/bin/perl
open (F, '<', "file1") or die "$O: Could not open file1: $!\n";
while (<F>) {
1 while s/^(([^,]*,){0,13}[^,|]*)\|[^,]*/$1/;
print;
}
Notice how Perl differs from your sed regex dialect in that grouping parentheses and alternation are unescaped, whereas a literal round parenthesis or pipe symbol needs to be backslash-escaped (or otherwise made into a literal, such as by putting it in a character class). Also, the right-hand side of the substitution prefers $1 (you will get a warning if you use warnings and have \1 in the substitution; technically, at this level, they are equivalent).
man perlrun has a snippet explaining how to implement the -i option inside a script if you really need that, but it's rather cumbersome. (Search for the first occurrence of "LINE:" which is part of the code you want.)
However, if you want to modify file1 in-place, and you pass it to your Perl script as its sole command-line argument, you can simply say $^I = 1; (or with use English; you can say $INPLACE_EDIT = 1;). See man perlvar.
By the way, the comment that your code "isn't producing any output" isn't entirely correct. It does what you are asking it to; but you are apparently asking for the wrong things.
Quoting a command in backticks executes that command. So
my $cmd3 = `sed ... file1 >file2`;
runs the sed command in a subshell, there and then, with input from file1, and redirected into file2. Because of the redirection, the output from this pipeline is nothing, i.e. an empty string "", which is assigned to $cmd3, which you then completely superfluously attempt to pass to system.
Maybe you wanted to put the sed command in regular quotes instead of backticks (so that the sed command line would be the value of $cmd3, which it then makes sense to pass to system). But because of the redirection, it would still not produce any visible output; it would create file2 containing the (possibly partially substituted) text from file1.
I have a huge pile of log files constantly being updated on HP-UX server.I have created the Perl code to find out the name of log file in which the string i'm using resides inside.
Perl gets the file name using split and passes it into a variable.Using the userinput i create the start and stop strings as two variables.Such as:
my $ssh = Net::OpenSSH->new($host, user => $user,
master_opts => [ -o => 'NumberOfPasswordPrompts=1',
-o => 'PreferredAuthentications=keyboard-interactive,password'],
login_handler => \&login_handler);
$ssh-> error and die "Unable to connect" . $ssh->error;
my $output=$ssh->capture("grep .$userinput1. /app/bea/user_projects/domains/granite/om_ni.log*");
my $array = (split ":", $output)[0];
print "$array"."\n";
[EDIT]: As you guys requested,above is the beginning of how the $array got filled in.Below is where the awk sequence starts:
my $a= "INFO - $userinput1";print $a;
my $b= "INFO - ProcessNode terminated... [$userinput1]";print $b;
Using the awk as part of ssh capture command,it will search through the whole log file and capture every line between the string $a and string $b,then get everything inside another array.Such as:
my $output2=$ssh->capture("awk -v i=$array '$a,$b' i");
Here $array is where the log file's full path is held and it work completely fine as a passing variable.
I tried using the awk without -v parameter as well,didn't matter at all.
[EDIT 2]:this is the result of print "$array"."\n";
/app/bea/user_projects/domains/granite/om_ni.log.2
When I run the perl script,I get the result:
INFO - 28B26AD1-E959-4F5F-BD89-A7A6E601BE18INFO - ProcessNode terminated... [28B26AD1-E959-4F5F-BD89-A7A6E601BE18] syntax error The source line is 1.
The error context is
INFO - 28B26AD1-E959-4F5F-BD89-A7A6E601BE18,INFO - ProcessNode >>> terminated. <<< .. [28B26AD1-E959-4F5F-BD89-A7A6E601BE18]
awk: Quitting
The source line is 1.
Error pointing at the "terminate" word somehow but even when I use escape characters all over the strings,it just doesn't care and returns the same error.
Any help on this issue is highly appreciated.Thanks a lot in advance.
While I don't really know awk, the way you are invoking it does not appear to be correct. Here is the manual for awk on HP-UX.
The part in single quotes ($a,$b) should be the program. However, you are passing it two text strings, which are not even quoted to separate them. This is not a valid awk program; hence the syntax error.
I think what you want is something like '/$a/, /$b/' for the program (but again, I am not an awk expert).
Also, you are setting the filename to variable i, then using i in the place of the filename when you invoke the command. I don't know why you are doing this, and I don't think it will even work to use a variable in the filename. Just use $array (which you should rename to something like $file for clarity) in the filename position.
So your whole command should look something like:
"awk '/$a/,/$b/' $file"
In this single command, you are dealing with three different tools: Perl, SSH, and awk. This is very hard to debug, because if there is a problem, it can be hard to tell where the problem is. It is essential that you break down the task into smaller parts in order to get something like this working.
In this case, that means that you should manually SSH into the server and play around with awk until you get the command right. Only when you are sure that you have the awk command right should you try incorporating it into Perl. It will be much easier if you break down the task in that way.
I've been trying to do some perl regex's and have hit the wall.
I'm trying to do some data analysis of a log file and I'm running into the following problem:
I have a file, test.csv, that is comprised of multiple single line entries from another program that produces the following layout format:
d:\snow\dir.txt
d:\snow\history\dir.tff
d:\snow\history\help.jar
d:\winter\show\help.txt
d:\summer\beach\ocean\swimming.txt
What I want would like to do is delete the file names from the path listing, so the resulting file would contain:
d:\snow\
d:\snow\history\
d:\snow\history\
d:\winter\show\
d:\summer\beach\ocean\
I've banged my head against the wall on this one and have tried various perl regex's in an attempt to drop the file names out without much luck. Since the paths to the directories are of varying length, I'm hitting a wall, I'm not sure if this is something that I can do within perl or python.
You can do this with one line in Perl:
perl -pe 's/[^\\]+$/\n/' <infile.txt >outfile.txt
Taking this in pieces:
-p causes Perl to wrap the statement (supplied with -e) in a while loop, apply the statement to each line of the input file, and print the result.
-e gives Perl a statement to run against every line.
s/[^\\]+$/\n/ is a substitution statement that uses a regular expression to change any sequence of characters not including a backslash at the end of the line, to just a newline \n.
[^\\] is a regular expression that matches any single character that is not a backslash
[^\\]+ is a regular expression that matches one or more characters that are not a backslash
[^\\]+$ is a regular expression that matches one or more characters that are not a backslash followed by the end of the line
Using regexes might work, but using a module designed for this purpose is generally speaking a better idea. File::Basename or File::Spec are suitable core modules for this purpose:
Code:
use strict;
use warnings;
use v5.10;
use File::Basename;
say dirname($_) for <DATA>;
__DATA__
d:\snow\dir.txt
d:\snow\history\dir.tff
d:\snow\history\help.jar
d:\winter\show\help.txt
d:\summer\beach\ocean\swimming.txt
Output:
d:\snow
d:\snow\history
d:\snow\history
d:\winter\show
d:\summer\beach\ocean
Of course, if you want ending backslashes, you'll have to add them.
And for File::Spec:
my ($volume, $dir, $file) = File::Spec->splitpath($path);
my $wanted_path = $volume . $dir; # what you want
These two modules have been part of the core distribution for a long time, which is a nice benefit.
You can do with this one liner also
perl -pe s /\\\\\w+\.\w+$// test.csv > Output.txt
\w+\.\w+$ matches for the filename with the extension which is at the end of the path
Here's one way to do it in Python:
python -c 'import sys,re;[sys.stdout.write(re.sub("[^\\\]+$","\n",l))for l in sys.stdin]' < in.txt > out.txt
I'll admit it's a bit more verbose than a Perl solution.