Simple PERL script to loop very quickly - perl

I'm trying to get a perl script to loop very quickly (in Solaris).
I have something like this:
#! /bin/perl
while ('true')
{
use strict;
use warnings;
use Time::HiRes;
system("sh", "shell script.sh");
Time::HiRes::usleep(10);
}
I want the perl script to execute a shell script every 10 microseconds. The script doesn't fail but no matter how much I change the precision of usleep within the script, the script is still only being executed approx 10 times per second. I need it to loop much faster than that.
Am I missing something fundamental here? I've never used perl before but I can't get the sleep speed I want in Solaris so I've opted for perl.
TIA
Huskie.
EDIT:
Revised script idea thanks to user comments - I'm now trying to do it all within perl and failing miserably!
Basically I'm trying to run the PS command to capture processes - if the process exists I want to capture the line and output to a text file.
#! /bin/perl
while ('true')
{
use strict;
use warnings;
use Time::HiRes;
open(PS,"ps -ef | grep <program> |egrep -v 'shl|grep' >> grep_out.txt");
Time::HiRes::usleep(10);
}
This returns the following error:
Name "main::PS" used only once: possible typo at ./ps_test_loop.pl line 9.

This is a pure perl program (not launching any external process) that looks for processes running some particular executable:
#!/usr/bin/perl
use strict;
use warnings;
my $cmd = 'lxc-start';
my $cmd_re = qr|/\Q$cmd\E$|;
$| = 1;
while (1) {
opendir PROC, "/proc" or die $!;
while (defined(my $pid = readdir PROC)) {
next unless $pid =~ /^\d+$/;
if (defined(my $exe = readlink "/proc/$pid/exe")) {
if ($exe =~ $cmd_re) {
print "pid: $pid\n";
}
}
}
closedir PROC;
# sleep 1;
}
On my computer this runs at 250 times/second.

The bottleneck is the creation of processes, pipes, and opening the output file. You should be doing that at most once, instead of doing it in each iteration. That's why you need to do everything in Perl if you want to make this faster. Which means: don't call the ps command, or any other command. Instead, read from /proc or use Proc::ProcessTable, as the comments suggest.
Incidentally: the use statement is executed only once (it is essentially a shorthand for a require statement wrapped in a BEGIN { } clause), so you might as well put that at the top of the file for clarity.

Related

Writing into a pipe in perl

I am passing the commands to some application through the Perl script using the pipe.
So I write the commands on pipe. But my problem is that the pipe is not waiting till the command execution is over from the application side and it takes the next command. So it is not blocking the inputs till the command execution is over. I need my Perl script work like UNIX shell. But it happens like the process is running in to background. I use readling to read the inputs.
#!/usr/bin/perl -w
use strict;
use Term::ReadLine;
open (GP, "|/usr/bin/gnuplot -noraise") or die "no gnuplot";
use FileHandle;
GP->autoflush(1);
# define readline
$term = new Term::ReadLine 'ProgramName';
while (defined( $_ = $term->readline('plot>'))) {
printf GP ("$_\n");
}
close(GP);
I recommend use of a concrete CPAN module, like Chart::Gnuplot, so you can have a high level of control

Perl 'system' failure messages

Say I have this perl "program" called simple.pl:
#!/usr/bin/perl
use xyz; # xyz is bogus and doesn't exist
And I also have this "program", called simple2.pl:
#!/usr/bin/perl
system("simple.pl");
my $abc = `simple.pl`;
printf("abc %s\n", $abc);
for both system and backtick, I get this message:
Can't exec "simple.pl": No such file or directory at scripts/perl/simple2.pl line 7.
Can't exec "simple.pl": No such file or directory at scripts/perl/simple2.pl line 9.
Not very useful for the user calling simple2.pl. Is there a way to get a more useful message?
Note. simple.pl does exist in the current directory. The real problem is that simple.pl doesn't compile. simple2 responds by saying simple doesn't exist. it's a misleading message.
If I had a way to even capture the compile message that would be a start.
This means system couldn't find an executable named "simple.pl" on your PATH. If your simple.pl is in the current directory, you could try to change "simple.pl" to "./simple.pl".
Actually, I don't see how to make this message more descriptive. If you were perl, how would you report this error?
BTW, I wouldn't try to run "simple2.pl" from inside of simple2.pl :)
Yes, check to see if the file exists and is executable, and if it isn't, print a more descriptive message.
unless (-ex $filename) {
print "I am unable to execute file $filename.";
}
If perl say it can't find the file, then it can't find the file. And the problem is more your code. Look at this example.
sidburn#sid:~/perl$ cat test.pl
#!/usr/bin/env perl
use strict;
use warnings;
use xyz;
sidburn#sid:~/perl$ cat test2.pl
#!/usr/bin/env perl
use strict;
use warnings;
system('test.pl');
sidburn#sid:~/perl$ cat test3.pl
#!/usr/bin/env perl
use strict;
use warnings;
system('./test.pl');
If you execute test2.pl you get:
sidburn#sid:~/perl$ ./test2.pl
Can't exec "test.pl": No such file or directory at ./test2.pl line 4.
If you execute test3.pl you get:
sidburn#sid:~/perl$ ./test3.pl
Can't locate xyz.pm in #INC (#INC contains: /home/sidburn/perl510/lib/5.10.1/i686-linux /home/sidburn/perl510/lib/5.10.1 /home/sidburn/perl510/lib/site_perl/5.10.1/i686-linux /home/sidburn/perl510/lib/site_perl/5.10.1 .) at ./test.pl line 4.
BEGIN failed--compilation aborted at ./test.pl line 4.
If you don't provide a relative or absolute path then perl lookup the command in your $PATH environment variable. If it is not there it can't find the file.
You need to provide "./" if it is in the current directory. But note "current directory" doesn't mean the directory where your script relies.
If you want the later then you probably want to do a
use FindBin;
with this you can do something like this:
#!/usr/bin/env perl
use strict;
use warnings;
use FindBin;
use File::Spec::Functions;
my $exe = catfile($FindBin::RealBin, 'test.pl');
print $exe, "\n";
system($exe);
if you want to check if system returns correctly, you need to check the return value from the system() command or $? later that holds the value.
if ( $? != 0 ) {
die "Cannot execute $exe.\n";
}
if you want to suppress messages from your program you need to redirect STDOUT, STDERR before starting your program with system().
Or use something like IPC::System::Simple
Or IPC::Open3 (in the core).
Bonus points for enabling the warnings pragma! Have an upvote!
You want to use backticks or qx// to capture the output of an external program, not system. To substitute your own error message that will make sense to your users (more points for you!), then you might do something as in
#! /usr/bin/perl
use strict;
use warnings;
no warnings 'exec';
chomp(my $abc = `simple2.pl`);
if ($? == 0) {
printf("abc %s\n", $abc);
}
else {
die "$0: unable to calculate abc\n";
}
In case you're unfamiliar, $? is
$CHILD_ERROR
$?
The status returned by the last pipe close, backtick command, successful call to wait or waitpid, or from the system operator.
When $? is zero, it indicates success.
Remember that the warnings pragma is lexical, so rather than disabling the warning for the whole program, you might do it for just one sub:
sub calculate_abc {
no warnings 'exec';
# ...
}
If you are trying to execute something you know is a Perl script, why not invoke the interpreter directly rather than dealing with the system knowing how to execute the file?
my $file = 'simple.pl';
-e $file or die "file '$file' not found";
system "perl $file";
# or
print `perl $file`;
to run with the same installation of perl that is running your current script:
system "$^X $file"; # or `$^X $file`
$^X is a special Perl variable that contains the file name of the running interpreter.
I had the exact same issue and figured out that perl wasn't installed. So the bash script was trying to execute the perl without an interpreter.
ls /usr/bin/perl
Try specifying the full path to the "simple.pl" file.

Can I run a Perl script from stdin?

Suppose I have a Perl script, namely mytest.pl. Can I run it by something like cat mytest.pl | perl -e?
The reason I want to do this is that I have a encrypted perl script and I can decrypt it in my c program and I want to run it in my c program. I don't want to write the decrypted script back to harddisk due to secruity concerns, so I need to run this perl script on-the-fly, all in memory.
This question has nothing to do with the cat command, I just want to know how to feed perl script to stdin, and let perl interpreter to run it.
perl < mytest.pl
should do the trick in any shell. It invokes perl and feeds the script in via the shell redirection operator <.
As pointed out, though, it seems a little unnecessary. Why not start the script with
#!/usr/bin/perl
or perhaps
#!/usr/bin/env perl
? (modified to reflect your Perl and/or env path)
Note the Useless Use of Cat Award. Whenever I use cat I stop and think whether the shell can provide this functionality for me instead.
Sometimes one needs to execute a perl script and pass it an argument. The STDIN construction perl input_file.txt < script.pl won't work. Using the tip from How to assign a heredoc value to a variable in Bash we overcome this by using a "here-script":
#!/bin/bash
read -r -d '' SCRIPT <<'EOS'
$total = 0;
while (<>) {
chomp;
#line = split "\t";
$total++;
}
print "Total: $total\n";
EOS
perl -e "$SCRIPT" input_file.txt
perl mytest.pl
should be the correct way. Why are you doing the unnecessary?
cat mytest.pl | perl
…is all you need. The -e switch expects the script as a command line argument.
perl will read the program from STDIN if you don't give it any arguments.
So you could theoretically read an encrypted file, decrypt it, and run it, without saving the file anywhere.
Here is a sample program:
#! /usr/bin/perl
use strict;
use warnings;
use 5.10.1;
use Crypt::CBC;
my $encrypted = do {
open my $encrypted_file, '<', 'perl_program.encrypted';
local $/ = undef;
<$encrypted_file>;
};
my $key = pack("H16", "0123456789ABCDEF");
my $cipher = Crypt::CBC->new(
'-key' => $key,
'-cipher' => 'Blowfish'
);
my $plaintext = $cipher->decrypt($encrypted);
use IPC::Run qw'run';
run [$^X], \$plaintext;
To test this program, I first ran this:
perl -MCrypt::CBC -e'
my $a = qq[print "Hello World\n"];
my $key = pack("H16", "0123456789ABCDEF");
my $cipher = Crypt::CBC->new(-key=>$key,-cipher=>"Blowfish");
my $encrypted = $cipher->encrypt($a);
print $encrypted;
' > perl_program.encrypted
This still won't stop dedicated hackers, but it will prevent most users from looking at the unencrypted program.

Why does my jzip process hang when I call it with Perl's system?

I am definitely new to Perl, and please forgive me if this seem like a stupid question to you.
I am trying to unzip a bunch of .cab file with jzip in Perl (ActivePerl, jzip, Windows XP):
#!/usr/bin/perl
use strict;
use warnings;
use File::Find;
use IO::File;
use v5.10;
my $prefix = 'myfileprefix';
my $dir = '.';
File::Find::find(
sub {
my $file = $_;
return if -d $file;
return if $file !~ /^$prefix(.*)\.cab$/;
my $cmd = 'jzip -eo '.$file;
system($cmd);
}, $dir
);
The code decompresses the first .cab files in the folder and hangs (without any errors). It hangs in there until I press Ctrl+c to stop. Anyone know what the problem is?
EDIT: I used processxp to inspect the processes, and I found that there are correct number of jzip processes fired up (per the number of cab files resides at the source folder). However, only one of them is run under cmd.exe => perl, and none of these process gets shut down after fired. Seems to me I need to shut down the process and execute it one by one, which I have no clue how to do so in perl. Any pointers?
EDIT: I also tried replacing jzip with notepad, it turns out it opens up notepad with one file at a time (in sequential order), and only if I manually close notepad then another instance is fired. Is this common behavior in ActivePerl?
EDIT: I finally solved it, and I am still not entire sure why. What I did was removing XML library in the script, which should not relevant. Sorry I removed "use XML::DOM" purposefully in the beginning as I thought it is completely irrelevant to this problem.
OLD:
use strict;
use warnings;
use File::Find;
use IO::File;
use File::Copy;
use XML::DOM;
use DBI;
use v5.10;
NEW:
#!/usr/bin/perl
use strict;
use warnings;
use File::Find;
use IO::File;
use File::Copy;
use DBI;
use v5.10;
my $prefix = 'myfileprefix';
my $dir = '.';
# retrieve xml file within given folder
File::Find::find(
sub {
my $file = $_;
return if -d $file;
return if $file !~ /^$prefix(.*)\.cab$/;
say $file;
#say $file or die $!;
my $cmd = 'jzip -eo '.$file;
say $cmd;
system($cmd);
}, $dir
);
This, however, imposes another problem, when the extracted file already exists, the script will hang again. I highly suspect this is a problem of jzip and an alternative of solving the problem is simply replacing jzip with extract, like #ghostdog74 pointed out below.
First off, if you are using commands via system() call, you should always redirect their output/error to a log or at least process within your program.
In this particular case, if you do that, you'd have a log of what every single command is doing and will see if/when any of them are stuck.
Second, just a general tip, it's a good idea to always use native Perl libraries - in this case, it may be impossible of course (I'm not that experienced with Windows Perl so no clue if there's a jzip module in Perl, but search CPAN).
UPDATE: Didn't find a Perl native CAB extractor, but found a jzip replacement that might work better - worth a try. http://www.cabextract.org.uk/ - there's a DOS version which will hopefully work on Windows
Based on your edit, this is what I suggest:
#!/usr/bin/perl
use strict;
use warnings;
use File::Find;
use IO::File;
use v5.10;
my $prefix = 'myfileprefix';
my $dir = '.';
my #commands;
File::Find::find(
sub {
my $file = $_;
return if -d $file;
return if $file !~ /^$prefix(.*)\.cab$/;
my $cmd = "jzip -eo $File::Find::name";
push #commands, $cmd;
}, $dir
);
#asynchronously kick off jzips
my $fresult;
for #commands
{
$fresult = fork();
if($fresult == 0) #child
{
`$_`;
}
elsif(! defined($fresult))
{
die("Fork failed");
}
else
{
#no-op, just keep moving
}
}
edit: added asynch. edit2: fixed scope issue.
What happens when you run the jzip command from the dos window? Does it work correctly? What happens if you add an end of line character (\n) to the command in the script? Does this prevent the hang?
here's an alternative, using extract.exe which you can download here or here
use File::Find;
use IO::File;
use v5.10;
my $prefix = 'myfileprefix';
my $dir = '.';
File::Find::find({wanted => \&wanted}, '.');
exit;
sub wanted {
my $destination = q(c:\test\temp);
if ( -f $_ && $_=~/^$prefix(.*)\.cab$/ ) {
$filename = "$File::Find::name";
$path = "$File::Find::dir";
$cmd = "extract /Y $path\\$filename /E /L $destination";
print $cmd."\n";
system($cmd);
}
} $dir;
Although no one has mentioned it explicitly, system blocks until the process finishes. The real problem, as people have noted, is figuring out why the process doesn't exit. Forking or any other parallelization won't help because you'll be left with a lot of hung processes.
Until you can figure out the issue, start small. Make the smallest Perl script that demonstrates the problem:
#!perl
system( '/path/to/jzip', '-eo', 'literal_file_name' ); # full path, list syntax!
print "I finished!\n";
Now the trick is to figure out why it hangs, and sometimes that means different solutions for different external programs. Sometimes you need to close STDIN before you run the external process or it sits there waiting for it to close, sometimes you do some other thing.
Instead of system, you might also try things such as IPC::System::Simple, which handles a lot of platform-specific details for you, or modules like IPC::Run or IPC::Open3.
Sometimes it just sucks, and this situation is one of those times.

How can I run a system command in Perl asynchronously?

I currently have a Perl script that runs an external command on the system, gathers the output, and performs some action based on what was returned. Right now, here is how I run this (where $cmd is a string with the command setup):
#output = `$cmd`;
I'd like to change this so if the command hangs and does not return a value after so much time then I kill the command. How would I go about running this asynchronously?
There's a LOT of ways to do this:
You can do this with a fork (perldoc -f fork)
or using threads (perldoc threads). Both of these make passing the returned information back to the main program difficult.
On systems that support it, you can set an alarm (perldoc -f alarm) and then clean up in the signal handler.
You can use an event loop like POE or Coro.
Instead of the backticks, you can use open() or respectively open2 or open3 (cf. IPC::Open2, IPC::Open3) to start a program while getting its STDOUT/STDERR via a file handle. Run non-blocking read operations on it. (perldoc -f select and probably google "perl nonblocking read")
As a more powerful variant of the openX()'s, check out IPC::Run/IPC::Cmd.
Probably tons I can't think of in the middle of the night.
If you really just need to put a timeout on a given system call that is a much simpler problem than asynchronous programming.
All you need is alarm() inside of an eval() block.
Here is a sample code block that puts these into a subroutine that you could drop into your code. The example calls sleep so isn't exciting for output, but does show you the timeout functionality you were interested in.
Output of running it is:
/bin/sleep 2 failure: timeout at
./time-out line 15.
$ cat time-out
#!/usr/bin/perl
use warnings;
use strict;
my $timeout = 1;
my #cmd = qw(/bin/sleep 2);
my $response = timeout_command($timeout, #cmd);
print "$response\n" if (defined $response);
sub timeout_command {
my $timeout = (shift);
my #command = #_;
undef $#;
my $return = eval {
local($SIG{ALRM}) = sub {die "timeout";};
alarm($timeout);
my $response;
open(CMD, '-|', #command) || die "couldn't run #command: $!\n";
while(<CMD>) {
$response .= $_;
}
close(CMD) || die "Couldn't close execution of #command: $!\n";
$response;
};
alarm(0);
if ($#) {
warn "#cmd failure: $#\n";
}
return $return;
}
If your external program doesn't take any input, look for the following words in the perlipc manpage:
Here's a safe backtick or pipe open for read:
Use the example code and guard it with an alarm (which is also explained in perlipc).
I coded below to run rsync on 20 directories simultaneously (in parallel instead of sequentially requiring me to wait hours for it to complete):
use threads;
for my $user ( keys %users ) {
my $host = $users{$user};
async {
system <<~ "SHELL";
ssh $host \\
rsync_user $user
SHELL
}
}
$ pgrep -lf rsync | wc -l
20
Not sure if it's best or even a good solution, but I was glad that it worked for my use case.
With this you get a mixed output on screen (what I ignored anyway), but it does its job successfully.
threads pragma exports the (very useful) async function by default.
rsync_user is my Perl script that wraps rsync command with options, and source and target directories set.
Ran on FreeBSD 13.1 with Perl 5.32.1