perl Win32 Signal Handling between perl processes - perl

I have a couple long running perl scripts in windows (strawberry perl) that I'm working on.
The first process is a parent monitoring process. It restarts the child process every 24 hours and will be always running.
The second is the child payment processing script. It is imperative that this process completes whatever it's doing before being shutdown.
It's my understanding that signal handling doesn't work in perl on win32 and that it shouldn't be relied on. Is there some other way that I can handle a signal? Win32::Process::Kill seems to kill the process without letting it safely shut down.
This is the signal handling that I've tried...
#Child
my $interrupted = 0;
$SIG{INT} = sub{$interrupted = 1;};
while(!$interrupted){
#keep doing your thing, man
}
#Parent
my $pid = open2(\*CHLD_OUT,\*CHLD_IN,'C:\\strawberry\\perl\\bin\\perl.exe','process.pl');
kill INT=>$pid;
waitpid($pid,0);
The only other thing I can think of is to open a socket between the two processes and write messages across the socket. But there must be something easier. Anyone know of any module that can do this?
Update
I've started working on creating a "signal" mechanism via IO::Socket::INET and IO::Select by opening a socket. This appears to work and I'm thinking of writing a module that is compatible with AnyEvent. But I'm still interested in an implementation that doesn't require opening a listening port and that doesn't require a server/client relationship. Is it possible to do this by subscribing to and firing custom events in windows?

Hmm, an interesting question. One thing I'd be wondering - how feasible is it to rewrite your code to thread?
When faced with a similar problem I found encapsulating the 'child' process as a thread meant I could better 'manage' it from the parent.
e.g.:
#!/usr/bin/perl
use strict;
use warnings;
use threads;
use threads::shared;
my $interrupted : shared;
sub child {
while ( not $interrupted ) {
#loop;
}
}
#main process
while ( 1 ) {
$interrupted = 0;
my $child = threads -> create ( \&child );
sleep 60;
$interrupted = 1;
$child -> join();
sleep ( 3600 );
}
But because you've got the IPCs from threading - you've got Thread::Queue, threads::shared, Thread::Semaphore and - I'm at least fairly sure you can send pseudo 'kill' signals within the script. This is because threads emulates 'kill' signals internally too.
http://www.perlmonks.org/?node_id=557328
Add to your thread:
$SIG{'TERM'} = sub { threads->exit(); };
And then then your 'main' can:
$thr->kill('TERM')->detach();

With ActiveState Perl I use windows native events through the Win32::Event module.
This way you don't need to implement anything fancy and you can even have your script interract with native applications.
I use this module in many applications.
Since it is part of Win32::IPC, which uses native code, it may not be available for Strawberry Perl. If that is the case you could try compiling it from the CPAN sources. It might be worth a try if you have lots of Windows perl-based software.

Related

Perl daemonize with child daemons

I have to employ daemons in my code. I need a control daemon that constantly checks the database for the tasks and supervises child daemons. The control daemon must assign tasks to the child daemons, control tasks, create new children if one of them dies, etc. The child daemons check database for tasks for them (by PID). How should I implement daemons for this purpose?
Daemon is just a code word for "background process that runs a long time". So the answer is 'it depends'. Perl has two major ways of doing multiprocessing:
Threading
You run a subroutine as a thread, in parallel with the main program code. (Which may then just monitor thread states).
The overhead of creating a thread is higher, but it's better suited for 'shared memory' style multiprocessing, e.g. when you're passing significant quantities of data back and forth. There's several libraries that make passing information between threads positively straightforward. Personally I quite like Thread::Queue, Thread::Semaphore and Storable.
In particular - Storable has freeze and thaw which lets you move complex data structures (e.g. objects/hashes) around in queues, which is very useful.
Basic threading example:
#!/usr/bin/perl
use strict;
use warnings;
use threads;
use Thread::Queue;
my $nthreads = 5;
my $process_q = Thread::Queue->new();
my $failed_q = Thread::Queue->new();
#this is a subroutine, but that runs 'as a thread'.
#when it starts, it inherits the program state 'as is'. E.g.
#the variable declarations above all apply - but changes to
#values within the program are 'thread local' unless the
#variable is defined as 'shared'.
#Behind the scenes - Thread::Queue are 'shared' arrays.
sub worker {
#NB - this will sit a loop indefinitely, until you close the queue.
#using $process_q -> end
#we do this once we've queued all the things we want to process
#and the sub completes and exits neatly.
#however if you _don't_ end it, this will sit waiting forever.
while ( my $server = $process_q->dequeue() ) {
chomp($server);
print threads->self()->tid() . ": pinging $server\n";
my $result = `/bin/ping -c 1 $server`;
if ($?) { $failed_q->enqueue($server) }
print $result;
}
}
#insert tasks into thread queue.
open( my $input_fh, "<", "server_list" ) or die $!;
$process_q->enqueue(<$input_fh>);
close($input_fh);
#we 'end' process_q - when we do, no more items may be inserted,
#and 'dequeue' returns 'undefined' when the queue is emptied.
#this means our worker threads (in their 'while' loop) will then exit.
$process_q->end();
#start some threads
for ( 1 .. $nthreads ) {
threads->create( \&worker );
}
#Wait for threads to all finish processing.
foreach my $thr ( threads->list() ) {
$thr->join();
}
#collate results. ('synchronise' operation)
while ( my $server = $failed_q->dequeue_nb() ) {
print "$server failed to ping\n";
}
Storable
When it comes to Storable, this is worth a separate example I think, because it's handy to move data around.
use Storable qw ( freeze thaw );
use MyObject; #home made object.
use Thread::Queue;
my $work_q = Thread::Queue->new();
sub worker_thread {
while ( my $packed_item = $work_q->dequeue ) {
my $object = thaw($packed_item);
$object->run_some_methods();
$object->set_status("processed");
#maybe return $object via 'freeze' and a queue?
}
}
my $thr = threads->create( \&worker_thread );
my $newobject = MyObject->new("some_parameters");
$work_q->enqueue( freeze($newobject) );
$work_q->end();
$thr->join();
Because you're passing the object around within the queue, you're effectively cloning it between threads. So bear in mind that you may need to freeze it and 'return' it somehow once you've done something to it's internal state. But it does mean you can do this asynchronously without needing to arbitrate locking or shared memory. You may also find it useful to be able to 'store' and 'retrieve' and object - this works as you might expect. (Although I daresay you might need to be careful about availability of module versions vs. defined attributes if you're retrieving a stored object)
Forking
Your script clones itself, leaving a 'parent' and 'child' - the child then generally diverges and does something different. This uses the Unix built in fork() which as a result is well optimised and generally very efficient - but because it's low level, means it's difficult to do lots of data transfer. You'll end up some slightly complicated things to do Interprocess communication - IPC. (See perlipc for more detail). It's efficient not least because most fork() implementations do a lazy data copy - memory space for your process is only allocated as it's needed e.g. when it's changed.
It's therefore really good if you want to delegate a lot of tasks that don't require much supervision from the parent. For example - you might fork a web server, because the child is reading files and delivering them to a particular client, and the parent doesn't much care. Or perhaps you would do this if you want to spend a lot of CPU time computing a result, and only pass that result back.
It's also not supported on Windows.
Useful libraries include Parallel::ForkManager
A basic example of 'forking' code looks a bit like this:
#!/usr/bin/perl
use strict;
use warnings;
use Parallel::ForkManager;
my $concurrent_fork_limit = 4;
my $fork_manager = Parallel::ForkManager->new($concurrent_fork_limit);
foreach my $thing ( "fork", "spoon", "knife", "plate" ) {
my $pid = $fork_manager->start;
if ($pid) {
print "$$: Fork made a child with pid $pid\n";
} else {
print "$$: child process started, with a key of $thing ($pid)\n";
}
$fork_manager->finish;
}
$fork_manager->wait_all_children();
Which is right for you?
So it's hard to say without a bit more detail about what you're trying to accomplish. This is why StacKOverflow usually likes to show some working, approaches you've tried, etc.
I would generally say:
if you need to pass data around, use threads. Thread::Queue especially when combined with Storable is very good for it.
if you don't, forks (on Unix) are generally faster/more efficient. (But fast alone isn't usually enough - write understandable stuff first, and aim for speed second. It doesn't matter much usually).
Avoid where possible spawning too many threads - they're fairly intensive on memory and creation overhead. You're far better off using a fixed number in a 'worker thread' style of programming, than repeatedly creating new, short lived threads. (On the other hand - forks are actually very good at this, because they don't copy your whole process).
I would suggest in the scenario you give - you're looking at threads and queues. Your parent process can track child threads via threads -> list() and join or create to keep the right number. And can feed data via a central queue to your worker threads. Or have multiple queues - one per 'child' and use that as a task assignment system.

perl how to mock thread behavior

I want to write unit tests for a subroutine in perl. The subroutine is using multiple threads to do its tasks. So, it first creates some threads and then it waits for them to join.
The problem is that our unit tests run on a server which is not able to run multi-threaded tests, so I need to somehow mock out the thread behavior. Basically I want to override the threads create and join functions such that its not threaded anymore. Any pointers how can I do that and test the code ?
Edit : The server fails to run the threaded code for the following reason:
Devel::Cover does not yet work with threads
Update: this answer doesn't solve the OP's problem as described in the edited question, but it might be useful to someone.
Perl threads are an interpreter emulation, not an operating system feature. So, they should work on any platform. If your testing server doesn't support threads, it's probably for one of these reasons:
Your version of Perl is very old.
Perl was compiled without thread support.
Your testing framework wasn't created with threaded code in mind.
The first two could be easily rectified by updating your environment. However, I suspect yours is the third issue.
I don't think you should solve this by mocking the thread behavior. This changes the original code too much to be a valid test. And it would be a significant amount of work anyway, so why not direct that effort toward getting a threaded test working?
The exact issues depend on your code, but probably the issue is that your subroutine starts a thread and then returns, with the thread still running. Then your test framework runs the sub over and over, accumulating a whole bunch of concurrent threads.
In that case, all you need is a wrapper sub that calls the sub you are testing, and then blocks until the threads are complete. This should be fairly simple. Take a look at threads->list() to see how you can detect running threads. Just have a loop that waits until the threads in question are no longer running before exiting the wrapper sub.
Here is a simple complete example demonstrating a wrapper sub:
#!usr/bin/perl
use strict;
use warnings;
use threads;
sub sub_to_test {
threads->create(sub { sleep 5; print("Thread done\n"); threads->detach() });
return "Sub done\n";
}
sub wrapper {
#Get a count of the running threads.
my $original_running_threads = threads->list(threads::running);
my #results = sub_to_test(#_);
#block until the number of running threads is the same as when we started.
sleep 1 while (threads->list(threads::running) > $original_running_threads);
return #results;
}
print wrapper;

Inline::Java conflicts with Parallel::ForkManager

I am having problem with calling both Parallel::ForkManager and Inline::Java at the same time. Specifically, if I call the Inline::Java with the JNI => 1 option (which I have to), then the fork process doesn't come back to the parent. Here are the codes:
use Parallel::ForkManager;
##### Calling Inline::Java #####
use Inline Java => <<END, JNI => 1;
END
###### End of Inline::Java #####
my $pm = Parallel::ForkManager->new(2);
for my $i (0..1) {
$pm->start and next;
print "Inside process $i\n";
$pm->finish;
}
$pm->wait_all_children;
print "Back to Parent.\n";
If I run this program, it goes into the child processes but never comes back to the parent. If I remove the 3 lines between the comments, it runs fine. If I change the JNI => 1 to JNI => 0 (not that I'm allowed to change that parameter for my purpose), then there is an error message of Lost connection with Java virtual machine at /usr/lib64/perl5/site_perl/5.8.8/x86_64-linux-thread-multi/Inline/Java.pm line 975.
Does anyone have a clue how to resolve the conflict? I also have to call the Inline::Java before the parallel process so using require after parallel is done is not an option. Thx!
Every child is talking over the same socket, which leads to the VM receiving jibberish.
You need to delay making a connection to the VM so that it's done in the children instead of it the parent.
You could move all thing Inline::Java-related into another module, then use require Child; (not use Child;) after start.
If you need to use Inline::Java before launching the child, do it in a different process.
Using forks with Inline::Java is going to be a problem. Your perl script needs to maintain a TCP connection with the JVM. When you fork a new process, the same file descriptors for communicating with the JVM are passed to the child process so the parent and all the child processes are using the same sockets. That won't work. You need to redesign your application.
One possibility (which you have already discounted) is to delay starting the JVM until after a fork, starting a new JVM in each child process.
Another approach is to forget about forking from Perl and to leverage Java's superior threading model to do parallelization. Design your Java code to perform its tasks in new threads, and start new threads from Perl:
my $java = ... entry point to JVM ...
for my $n (1 .. $num_threads) {
$java->startNewThread(#inputs)
}
$java->waitForThreadsToFinish();
$result = $java->getResults();
Perl also has its own threading model (see threads and threads::shared). I doubt that Perl's threads will work for this problem, but it still might be worth a try.
Update: another possibility that is mentioned in the Inline::Java docs is to use a shared JVM. Invoke Inline::Java with the option SHARED_JVM => 1, and when a new child process starts, call Inline::Java::reconnect_JVM() from the child to make a fresh connection. The downsides of this approach are
it keeps the JVM active after the program ends, so you have to remember to kill the JVM
it is incompatible with the option JNI => 1, which might be a dealbreaker to the OP.

Is there a way to have managed processes in Perl (i.e. a threads replacement that actually works)?

I have a multithreded application in perl for which I have to rely on several non-thread safe modules, so I have been using fork()ed processes with kill() signals as a message passing interface.
The problem is that the signal handlers are a bit erratic (to say the least) and often end up with processes that get killed in inapropriate states.
Is there a better way to do this?
Depending on exactly what your program needs to do, you might consider using POE, which is a Perl framework for multi-threaded applications with user-space threads. It's complex, but elegant and powerful and can help you avoid non-thread-safe modules by confining activity to a single Perl interpreter thread.
Helpful resources to get started:
Programming POE presentation by Matt Sergeant (start here to understand what it is and does)
POE project page (lots of cookbook examples)
Plus there are hundreds of pre-built POE components you can use to assemble into an application.
You can always have a pipe between parent and child to pass messages back and forth.
pipe my $reader, my $writer;
my $pid = fork();
if ( $pid == 0 ) {
close $reader;
...
}
else {
close $writer;
my $msg_from_child = <$reader>;
....
}
Not a very comfortable way of programming, but it shouldn't be 'erratic'.
Have a look at forks.pm, a "drop-in replacement for Perl threads using fork()" which makes for much more sensible memory usage (but don't use it on Win32). It will allow you to declare "shared" variables and then it automatically passes changes made to such variables between the processes (similar to how threads.pm does things).
From perl 5.8 onwards you should be looking at the core threads module. Have a look at http://metacpan.org/pod/threads
If you want to use modules which aren't thread safe you can usually load them with a require and import inside the thread entry point.

Perl - Win32 - How to do a non-blocking read of a filehandle from another process?

I'm writing some server code that talks to a client process via STDIN. I'm trying to write a snippet of perl code that asynchronously receives responses from the client's STDOUT. The blocking version of the code might look like this:
sub _read_from_client
{
my ($file_handle) = #_;
while (my $line = <$file_handle>) {
print STDOUT $line;
}
return;
}
Importantly, the snippet needs to work in Win32 platform. There are many solutions for *nix platforms that I'm not interested in. I'm using ActivePerl 5.10.
This thread on Perlmonks suggests you can make a socket nonblocking on Windows in Perl this way:
ioctl($socket, 0x8004667e, 1);
More details and resources in that thread
If you don't want to go the low-level route, you will have to look at the other more frameworked solutions.
You can use a thread to read from the input and have it stuff all data it reads into a Thread::Queue which you then handle in your main thread.
You can look at POE which implements an event based framework, especially POE::Wheel::Run::Win32. Potentially, you can also steal the code from it to implement the nonblocking reads yourself.
You can look at [Coro], which implements a cooperative multitasking system using coroutines. This is mostly similar to threads except that you get userspace threads, not system threads.
You haven't stated how far up you want to go, but your choice is between sysread and a framework, or writing said framework yourself. The easiest route to go is just to use threads or by going through the code of Poe::Wheel::Run::Win32.