Signalling problems when combining Net::OpenSSH and threads - perl

Question

I wrote a quite large program that executes commands on many remote hosts, but I ran into a serious problem, and I don't know how to fix it.
After a lot of trying around, I was able to extract the minimum code to reproduce the problem reliably on my machine:
use warnings;
use strict;
use threads;
use threads::shared;
use Data::Dumper;
use POSIX ":sys_wait_h";
use Net::OpenSSH;
use Time::HiRes qw( usleep );
my #LIST=qw(host038b host039a host039b host040a host040b host041a host041b host043a
host043b host044a host044b host045a host045b host046a host046b host047a host047b host049a
host049b host050a host050b host054a host054b host055a host055b host056a host056b host057a
host057b host058a host059a host059b host060a host060b host062a host062b host063a host068a
host068b host069a host069b host071a host071b host072a host073a host073b host075a host075b
host078a host078b host082a host082b host087a host087b host089a host089b host090a host090b
host091a host091b host092a host092b host096a host096b host097a host097b host098a host099a
host099b host100a);
my ($SSH, $CPID, %PIDS, #DONE);
sub _testthread {
# Read stdout pipe
my $SCROUT=shift;
while (<$SCROUT>) {
print $_; # I normally write that to a logfile
}
return (0);
}
foreach (#LIST) {
$SSH->{$_}=Net::OpenSSH->new($_, async => 1,
master_opts => [ -o => "PasswordAuthentication=no"]);
}
$SIG{CHLD} = sub { my $WPID;
push (#DONE, { 'PID' => $WPID, 'RC' => $?, 'ERR' => $!}) while (($WPID = waitpid(-1, WNOHANG)) > 0) };
foreach (#LIST) {
my ($SCRFH, $SCROUT, undef, $CPID) = $SSH->{$_}->open_ex({stdin_pipe => 1,
stdout_pipe => 1}, '/bin/bash -s');
$PIDS{$CPID}='ACTIVE';
threads->new('_testthread', $SCROUT);
print $SCRFH "sleep 2\n";
print $SCRFH "echo test `hostname`\n";
print $SCRFH "exit 0\n";
close $SCRFH;
usleep 10000;
}
while (grep(/^ACTIVE/, values(%PIDS)) > 0) {
print Dumper \%PIDS;
while (#DONE) {
my $DONE = shift (#DONE);
$PIDS{$DONE->{PID}}='DONE';
}
sleep 1;
}
$_->join foreach (threads->list);
With the preinstalled perl 5.10, this segfaults most of the time, even when taking out some more complicated constructs of redericting the open_ex output to a file descriptor.
With a newly compiled perl 5.18.2, this script hangs indefinitely most of the time, because it does not seem to receive every SIG{CHLD}, even though I am using safe signalling (as far as I understand).
To reproduce the problem, the following things seem to be necessary:
A sufficient number of hosts in #LIST
letting open_ex (or derived methods of Net::OpenSSH) fork
giving the STDOUT file handle of that fork to a thread
using a signal handler for SIG{CHLD}
As my bigger program which uses this structure is mostly unusable, I would be very happy if somebody could help me find a solution, maybe an alternative.
Thanks, and greetings,
Mazze

Answer 1

You are trying to mix signals and threads, and that's always going to be a bad idea. The solution is simply to stop using threads; whatever you're doing can be done nicer a different way.
Perhaps consider some kind of asynchronous/event-driven IO system, to perform these concurrent IO tasks with.