I have a Perl script that forks.
Each fork runs an external program, parses the output, and converts the output to a Storable file.
The Storable files are then read in by the parent and the total data from each of the children are analyzed before proceeding onto a repeat of the previous fork or else the parent stops.
What exactly happens when I issue a ^C while some of the children are still running the external program? The parent perl script was called in the foreground and, I presume, remained in the foreground despite the forking.
Is the SIGINT passed to all children, that is, the parent, the parent's children, and the external program called by the children??
UPDATE:
I should add, it appears that when I issue the SIGINIT, the external program called by the children of my script seem to acknowledge the signal and terminate. But the children, or perhaps the parent program, carry on. This is all unclear to me.
UPDATE 2:
With respect to tchrist's comment, the external program is called with Perl's system() command.
In fact, tchrist's comment also seems to contain the explanation I was looking for. After some more debugging, based on the behavior of my program, it appears that, indeed, SIGINT is being passed from the parent to all children and from all children to all of their children (the external program).
Thus, what appears to be happening, based on tchrist's comment, is that CTRL-C is killing the external program which causes the children to move out of the system() command - and nothing more.
Although I had my children check the exit status of what was called in system(), I was assuming that a CTRL-C would kill everything from the parent down, rather than lead to the creation of more rounds of processing, which is what was happening!!!
SOLUTION (to my problem):
I need to just create a signal handler for SIGINT in the parent. The signal handler would then send SIGTERM to each of the children (which I presume would also send a SIGTERM to the children's children), and then cause the parent to exit gracefully. Although this somewhat obvious solution likely would have fixed things, I wanted to understand my misconception about the behavior of SIGINT with respect to forking in Perl.
Perl’s builtin system function works just like the C system(3) function from the standard C library as far as signals are concerned. If you are using Perl’s version of system() or pipe open or backticks, then the parent — the one calling system rather than the one called by it — will IGNORE any SIGINT and SIGQUIT while the children are running. If you’ve you rolled your own using some variant of the fork?wait:exec trio, then you must think about these matters yourself.
Consider what happens when you use system("vi somefile") and hit ^C during a long search in vi: only vi takes a (nonfatal) SIGINT; the parent ignores it. This is correct behavior. That’s why C works this way, and that’s why Perl works this way.
The thing you have to remember is that just because a ^C sends a SIGINT to all processes in the foregrounded process group (even those of differing effective UID or GID), that does not mean that it causes all those processes to exit. A ^C is only a SIGINT, meant to interrupt a process, not a SIGKILL, meant to terminate with no questions asked.
There are many sorts of program that it would be wrong to just kill off with no warning; an editor is just one such example. A mailer might be another. Be exceedingly careful about this.
Many sorts of programs selectively ignore, trap, or block (means delay delivery of) various sorts of signals. Only the default behavior of SIGINT is to cause the process to exit. You can find out whether this happened, and in fact which signal caused it to happen (amongst other things), with this sort of code on traditional operating systems:
if ($wait_status = system("whatever")) {
$sig_killed = $wait_status & 127;
$did_coredump = $wait_status & 128;
$exit_status = $wait_status >> 8;
# now do something based on that...
}
Note carefully that a ^C’d vi, for example, will not have a wait status word indicating it died from an untrapped SIGINT, since there wasn’t one: it caught it.
Sometimes your kids will go and have kids of their own behind your back. Messy but true. I have therefore been known, on occasion, to genocide all progeny known and unknown this way:
# scope to temporize (save+restore) any previous value of $SIG{HUP}
{
local $SIG{HUP} = "IGNORE";
kill HUP => -$$; # the killpg(getpid(), SIGHUP) syscall
}
That of course doesn’t work with SIGKILL or SIGSTOP, which are not amenable to being IGNOREd like that.
Another matter you might want to be careful of is that before the 5.8 release, signal handling in Perl has not historically been a reliably safe operation. It is now, but this is a version-dependent issue. If you haven’t yet done so, then you should definitely read up on deferred signals in the perlipc manpage, and perhaps also on the PERL_SIGNALS envariable in the perlrun manpage.
When you hit ^C in a terminal window (or any terminal, for that matter), it will send a SIGINT to the foreground process GROUP in that terminal. Now when you start a program from the command line, its generally in its own process group, and that becomes the foreground process group. By default, when you fork a child, it will be in the same process group as the parent, so by default, the parent (top level program invoked from the command line), all its children, children's children, etc, as well as any external programs invoked by any of these (which are all just children) will all be in that same process group, so will all receive the SIGINT signal.
However, if any of those children or programs call setpgrp or setpgid or setsid, or any other call that causes the process to be a in a new progress group, those processes (and any children they start after leaving the foreground process group) will NOT receive the SIGINT.
In addition, when a process receives a SIGINT, it might not terminate -- it might be ignoring the signal, or it might catch it and do something completely different. The default signal handler for SIGINT terminates the process, but that's just the default which can be overridden.
edit
From your update, it sounds like everything is remaining in the same process group, so the signals are being delivered to everyone, as the grandchildren (the external programs) are exiting, but the children are catching and ignoring the SIGINTs. By tchrist's comment, it sounds like this is the default behavior for perl.
If you kill the parent process, the children (and external programs running) will still run until they terminate one way or another. This can be avoided if the parent has a signal handler that catches the SIGINT, and then kills the process group (often parent's pid). That way, once the parent gets the SIGINT, it will kill all of it's children.
So to answer your question, it all depends on the implementation of the signal handler. Based on your update, it seems that the parent indeed kills it's children and then instead of terminating itself, it goes back to doing something else (think of it as a reset instead of a complete shutdown/start combination).
Related
I'm trying to deal with a server that works as follows:
It has a parent process
It creates a "helper" child process to handles some special tasks
It opens the child process with a pipe; and uses the pipe to issue commands to the child.
It also spawns off many other child processes (the server's main goal is to execute various commands).
I would like to be able to detect when the write to the pipe to the child process fails; and issue a special notification.
Ordinarily, I would achieve that by creating a custom $SIG{PIPE} handler in the parent process.
However, what I'm concerned with is the fact that some of the processes the parent launches to execute commands might have their own pipes open to them; and if the write to THOSE pipes fails, I'd like to simply ignore the SIGPIPE.
Q1. Is there a way for me to tell from within SIGPIPE handler, which of the open pipes threw the signal? (I know every child's PID, so PID would be fine... or if there's a way to do it via file descriptor #s?).
Q2. Could I solve the problem using local $SIG{PIPE} somehow? My assumption is that I would need to:
Set helper-process-specific local $SIG{PIPE} right before writing to that pipe
do print $HELPER_PIPE (this happens in only one subroutine)
Reset $SIG{PIPE} to DEFAULT or IGNORE
Ensure that these 3 actions are within their own block scope.
The write syscall returns the error EPIPE in the same case when a SIGPIPE is triggered, assuming that the SIGPIPE doesn't succeed in killing the process. So your best bet is to set $SIG{PIPE} = 'IGNORE' (to avoid dying from the signal), to use $fh->autoflush (to avoid PerlIO buffering, ensuring that you're notified of any I/O errors immediately), and to check the return value of print whenever you call it. If print returns false and $!{EPIPE} is set, then you've tried to write to a closed pipe. If print returns false and $!{EPIPE} isn't set, you have some other issue to deal with.
Portably you can't tell. However, you might find your OS supports the SIG_INFO information, and if you can get that up to Perl somehow, the siginfo structure contains a field that gives the FD number on SIGPIPE.
I have a Perl script that spawns some children. They all take quite a while to run, creating directories and files along the way. I often notice things I'd like to change before the children die of natural causes, so then I have to shut everything down (which involves a few grep and kill calls) and delete any files the children created. It's not really a big deal, but a bit of a pain in the neck. I'd like to create a setup where the children are all monitored so when I start up the parent again, old children still running are reported.
My best idea so far is keeping a log file of children I've run with their PIDs, checking it and updating it at the start of the parent script. Kill and report any children still running, and die to let the user clean up directories and files by hand before a fresh run.
So my question is this: How might I go about adding children to the log file? Is there a way to set up a trigger which could automatically take care of it, or am I stuck having to remember to do it any place in the code where a new process starts?
p.s. I'm certainly open to suggestions of better ways to accomplish this!
How about signal handlers? The parent can track the pids of the jobs it has launched (and if you want, the pids that have been reaped with a SIGCHLD handler). When you want to terminate everything prematurely, signal the parent to kill off all the children and clean up after them. If the child processes are also Perl scripts, you could put signal handlers in the children and see if they will clean up after themselves.
I have Perl program based on IO::Async, and it sometimes just exits after a few hours/days without printing any error message whatsoever. There's nothing in dmesg or /var/log either. STDOUT/STDERR are both autoflush(1) so data shouldn't be lost in buffers. It doesn't actually exit from IO::Async::Loop->loop_forever - print I put there just to make sure of that never gets triggered.
Now one way would be to keep peppering the program with more and more prints and hope one of them gives me some clue. Is there better way to get information what was going on in a program that made it exit/silently crash?
One trick I've used is to run the program under strace or ltrace (or attach to the process using strace). Naturally that was under Linux. Under other operating systems you'd use ktrace or dtrace or whatever is appropriate.
A trick I've used for programs which only exhibit sparse issues over days or week and then only over handfuls among hundreds of systems is to direct the output from my tracer to a FIFO, and have a custom program keep only 10K lines in a ring buffer (and with a handler on SIGPIPE and SIGHUP to dump the current buffer contents into a file. (It's a simple program, but I don't have a copy handy and I'm not going to re-write it tonight; my copy was written for internal use and is owned by a former employer).
The ring buffer allows the program to run indefinitely with fear of running systems out of disk space ... we usually only need a few hundred, even a couple thousand lines of the trace in such matters.
If you are capturing STDERR, you could start the program as perl -MCarp::Always foo_prog. Carp::Always forces a stack trace on all errors.
A sudden exit without any error message is possibly a SIGPIPE. Traditionally SIGPIPE is used to stop things like the cat command in the following pipeline:
cat file | head -10
It doesn't usually result in anything being printed either by libc or perl to indicate what happened.
Since in an IO::Async-based program you'd not want to silently exit on SIGPIPE, my suggestion would be to put somewhere in the main file of the program a line something like
$SIG{PIPE} = sub { die "Aborting on SIGPIPE\n" };
which will at least alert you to this fact. If instead you use Carp::croak without the \n you might even be lucky enough to get the file/line number of the syswrite, etc... that caused the SIGPIPE.
I'm writing a Perl script that runs 4 simultaneous, identical processes with different input parameters (see background here - the rest of my question will make much more sense after reading that).
I am making a system() call to a program that generates data (XFOIL, again see above link). My single-core version of this program looks like this:
eval{
local $SIG{ALRM} = sub{die "TIMEOUT"};
alarm 250;
system("xfoil <command_list >xfoil_output");
alarm 0;
};
if ($#){
# read the output log and run timeout stuff...
system('killall xfoil') # Kill the hung XFOIL. now it's a zombie.
}
Essentially, XFOIL should take only about 100 seconds to run - so after 250 seconds the program is hanging (presumably waiting for user input that it's never going to get).
The problem now is, if I do a killall in the multi-core version of my program, I'm going to kill 3 other instances of XFOIL, and those processes are generating data. So I need to kill only the hung instance, and this requires getting a PID.
I don't know very much about forks and such. From what I can tell so far, I would run an exec('xfoil') inside the child process that I fork. But the PID of the exec() will be different than the PID of the child process (or is it? It's a separate process so I'd assume it is, but again I've no experience with this..), so this still doesn't help when I want to forcefully kill the process since I won't have the PID anyway. How do I go about doing this?
Thanks a ton for your help!
If you want the PID, fork the process yourself instead of using system. The system command is mostly designed as a "fire and forget" tool. If you want to interact with the process, use something else. See, for instance, the perlipc documentation.
I think you've already looked at Parallel::ForkManager based on answers to your question How can I make my Perl script use multiple cores for child processes?
There are so many possible errors in the POSIX environment. Why do some of them (like writing to an unconnected socket in particular) get special treatment in the form of signals?
This is by design, so that simple programs producing text (e.g. find, grep, cat) used in a pipeline would die when their consumer dies. That is, if you're running a chain like find | grep | sed | head, head will exit as soon as it reads enough lines. That will kill sed with SIGPIPE, which will kill grep with SIGPIPE, which will kill find with SEGPIPE. If there were no SIGPIPE, naively written programs would continue running and producing content that nobody needs.
If you don't want to get SIGPIPE in your program, just ignore it with a call to signal(). After that, syscalls like write() that hit a broken pipe will return with errno=EPIPE instead.
See this SO answer for a detailed explanation of why writing a closed descriptor / socket generates SIGPIPE.
Why is writing a closed TCP socket worse than reading one?
SIGPIPE isn't specific to sockets — as the name would suggest, it is also sent when you try to write to a pipe (anonymous or named) as well. I guess the reason for having separate error-handling behaviour is that broken pipes shouldn't always be treated as an error (whereas, for example, trying to write to a file that doesn't exist should always be treated as an error).
Consider the program less. This program reads input from stdin (unless a filename is specified) and only shows part of it at a time. If the user scrolls down, it will try to read more input from stdin, and display that. Since it doesn't read all the input at once, the pipe will be broken if the user quits (e.g. by pressing q) before the input has all been read. This isn't really a problem, though, so the program that's writing down the pipe should handle it gracefully.
it's up to the design.
at the beginning people use signal to control events notification which were sent to the user space, and later it is not necessary because there're more popular skeletons such as polling which don't require a system caller to make a signal handler.