I read in a book that in order to make a inter-process communication using pipes between two processes it is preferred to use two pipes, one which will be for the children to write in it and for the father to read from it and another one to do the opposite communication. Why is this a better way?Can't we use just one pipe so that both parent and children can read from and write to that?
You need a way to schronize communication between processes, other-wise a process will read/write what it wrote/read again and again. e.g. if you use one pipe:
//parent
while(1)
{
write(p1);
//need a logic to wait so as to read what child wrote back and also so
// that we may not end up reading back what we wrote.
read(p1);
}
//child
while(1)
{
read(p1);
//need a logic to wait so as to read what child wrote back and also so
// that we may not end up reading back what we wrote.
write(p1);
}
Find a fool proof logic to sync or use two pipes. I am saying fool-proof wait, cuz simple techniques like sleep() or signals are vulnerable to those scheduling problems which OS people have outlined in their works.
Pipes themselves are blocking constructs, so depend on them to sync.
//parent
while(1)
{
write(p1);
//pipe blocks until till child writes something in pipe
read(p2);
}
//child
while(1)
{
//pipe waits for parent to write.
read(p1);
//parent is waiting to read.
write(p2);
}
Related
I've read through the boost:asio documentation (which appears silent on async clients), and looked through here, but can't seem to find the forest for the trees here.
I've got a simulation that has a main loop that looks like this:
for(;;)
{
a = do_stuff1();
do_stuff2(a);
}
Easy enough.
What I'd like to do, is modify it so that I have:
for(;;)
{
a = do_stuff1();
check_for_new_received_udp_data(&b);
modify_a_with_data_from_b(a,b);
do_stuff2(a);
}
Where I have the following requirements:
I cannot lose data just because I wasn't actively listening. IE I don't want to lose packets because I was in do_stuff2() instead of check_for_new_received_udp_data() at the time the server sent the packet.
I can't have check_for_new_received_udp_data() block for more than about 2ms, since the main for loop needs to execute at 60Hz.
The server will be running elsewhere, and has a completely erratic schedule. Sometimes there will be no data, othertimes I may get the same packet repeatedly.
I've played with the async UDP, but that requires calling io_service.run(), which blocks indefinitely, so that doesn't really help me.
I thought about timing out a blocking socket read, but it seems you have to cheat and get out of the boost calls to do that, so that's a non-starter.
Is the answer going to involve threading? Either way, could someone kindly point me to an example that is somewhat similar? Surely this has been done before.
To avoid blocking in the io_service::run() you can use io_service::poll_one().
Regarding loosing UDP packets, I think you are out of luck. UDP does not guarantee delivery, and any part of the network may decide to drop UDP packets if there is much traffic. If you need to ensure delivery you need to have either implement some sort of flow control or just use TCP.
I think your problem is that you're still thinking synchronously. You need to think asynchronously.
Async read on UDP socket - will call handler when data arrives.
Within that handler do your processing on the incoming data. Keep in mind that while you're processing, if you have a single thread, nothing else dispatches. This can be perfectly OK (UDP messages will still be queued in the network stack...).
As a result of this you could be starting other asynchronous operations.
If you need to do work in parallel that is essentially unrelated or offline that will involve threads. Create a thread that calls io_service.run().
If you need to do periodic work in an asynch framework use timers.
In your particular example we can rearrange things like this (psuedo-code):
read_handler( ... )
{
modify_a_with_data_from_b(a,b);
do_stuff2(a);
a = do_stuff1();
udp->async_read( ..., read_handler );
}
periodic_handler(...)
{
// do periodic stuff
timer.async_wait( ..., periodic_handler );
}
main()
{
...
a = do_stuff1();
udp->async_read( ..., read_handler )
timer.async_wait( ..., periodic_handler );
io_service.run();
}
Now I'm sure there are other requirements that aren't evident from your question but you'll need to figure out an asynchronous answer to them, this is just an idea. Also ask yourself if you really need an asynchronous framework or just use the synchronous socket APIs.
Reading the man page for close, if it's interrupted by a signal then the fd's state is unspecified. Is there a best practice for handling this case, or is it assumed to become the OS's problem afterwards.
I assume that failure after EIO closes the fd appropriately.
If you want your program to run for a long time, a possible file descriptor leak is never only the operating system's problem. Short-lived programs which don't use many file descriptors of course have the option of terminating the the descriptors unclosed, and rely on the kernel closing them when the program terminates. So, for the rest of my answer I'll assume your program is long-running.
If your program is not multi-threaded, you have a very easy situation:
int close_some_fd(int fd, int *was_interrupted) {
*was_interrupted = 0;
/* this is point X to which I will draw your attention later. */
for (;;) {
if (0 == close(fd))
return 0; /* Success. */
if (EIO != errno)
return -1; /* Some failure, not interrupted by a signal. */
/* Our close attempt was interrupted. */
*was_interrupted = 1;
int fdflags = 0;
/* Just use the fd to find out if it is still open. */
if (0 != fcntl(fd, F_GETFD, &fdflags))
return 0; /* Interrupted, but file is also closed. So we are done. */
}
}
On the other hand, if your code is multi-threaded, some other thread (and perhaps one you don't control, such as a name service cache) may have called dup, dup2, socket, open, accept or some other similar function that makes the kernel allocate a file descriptor.
To make a similar approach work in such an environment you will need to be able to tell the difference between the file descriptor you started with and a file descriptor newly opened by another thread. Knowing that there is already a lower-numbered fd which still isn't open is enough to discount may of those, but in a multi-threaded environment you don't have a simple way of figuring out that this is still the case.
One option is to rely on some common aspect to all the file descriptors your program works with. For example if it never uses O_CLOEXEC, you can use fcntl to set the O_CLOEXEC flag at the point marked X in the code, and then you just change the existing call to fcntl like this:
if (0 = fcntl(fd, F_GETFD, &fdflags)) {
if (fdflags & O_CLOEXEC) {
/* open, and still marked with O_CLOEXEC, unused elsewhere. */
continue;
} else {
return 0; /* Interrupted, but file is also closed. So we are done. */
}
}
You can adjust this approach to use something other than O_CLOEXEC (perhaps fstat for example, recording st_dev and st_ino), but if you can't be sure what the rest of your multithreaded program is doing, this general idea is likely to be unsatisfying.
There's another approach which is also a bit problematic, but which might serve. This is that, instead of closing your file descriptor, you use sendmsg to pass the file descriptor across a Unix domain socket to a separate, single-threaded special-purpose server whose only job is to close the file descriptor. Yes, this is a bit icky. Some entertaining properties of this approach though are:
In order to avoid uncertainty over whether your fd was really passed to the server and closed successfully, you should probably read from a return channel of fd-closed-OK messages coming back from the server. This avoids needing to block signal delivery while you are executing sendmsg too. However, it means a user-space context-switch overhead for every file descriptor close (unless you batch them up to amortise the cost). You need to avoid a situation where thread A may be reading an fd-closed-OK report corresponding to a request made from thread B. You can avoid that problem by serialising close operations (which will limit performance) or demultiplexing the responses (which is complex). Alternatively, you could use some other IPC mechanism to avoid the need to serialise or demultiplex (SYSV semaphores for example).
For a high-volume process this will place an entertainingly high load on the kernel's file descriptor garbage collector apparatus, which is normally not highly stressed and may therefore give you some interesting symptoms.
As for what I'd do personally in the applications I work with most often, I'd figure out to what extent I could make assumptions about what kind of file descriptor I was closing. If, for example, they were normally sockets, I'd just try this out with a test program and figure out whether EIO normally leaves the file descriptor closed or not. That is, determine whether the theoretically unspecified state of the file descriptor on EIO is, in practice, predictable. This will not work well if the fd could be anything (e.g. disk file, socket, tty, ...). Of course, if you're using some open-source operating system you may just be able to read the kernel source and determine what the possible outcomes are.
Certainly I'd try the above experiment-based system before worrying about sharding the fd-close servers to scale out on fd-closing.
I've been writing some code that replaces some existing:
while(runEventLoop){
if(select(openSockets, readFDS, writeFDS, errFDS, timeout) > 0){
// check file descriptors for activity and dispatch events based on same
}
}
socket reading code. I'd like to change this to use a GCD queue, so that I can pop events on to the queue using dispatch_async instead of maintaining a "must be called on next iteration" array. I also am already using a GCD queue to /contain/ this particular action, hence wanting to devolve it to a more natural GCD dispatch form. ( not a while() loop monopolizing a serial queue )
However, when I tried to refactor this into a form that relied on dispatch sources fired from event handlers tied to DISPATCH_SOURCE_TYPE_READ and DISPATCH_SOURCE_TYPE_WRITE on the socket descriptors, the library code that depended on this scheduling stopped working. My first assumption is that I'm misunderstanding the use of DISPATCH_SOURCE_TYPE_READ and DISPATCH_SOURCE_TYPE_WRITE - I had assumed that they would yield roughly the same behavior as calling select() with those socket descriptors.
Do I misunderstand GCD dispatch sources? Or, regarding the refactor, am I using it in a situation where it is not best suited?
The short answer to your question is: none. There are no differences, both GCD dispatch sources and select() do the same thing: they notify the user that a specific kernel event happened or that a particular condition holds true.
Note that, on a mac or iOS device you should not use select(), but rather the more advanced kqueue() and kevent() (or kevent64()).
You may certainly convert the code to use GCD dispatch sources, but you need to be careful not to break other code relying on this. So, this needs a complete inspection of the whole code handling signals, file descriptors, socket and all of the other low level kernel events.
May be a simpler solution could be to maintain the original code, simply adding GCD code in the part that react to events. Here, you dispatch events on different queues depending on the particular type of event.
I have a multithreded application in perl for which I have to rely on several non-thread safe modules, so I have been using fork()ed processes with kill() signals as a message passing interface.
The problem is that the signal handlers are a bit erratic (to say the least) and often end up with processes that get killed in inapropriate states.
Is there a better way to do this?
Depending on exactly what your program needs to do, you might consider using POE, which is a Perl framework for multi-threaded applications with user-space threads. It's complex, but elegant and powerful and can help you avoid non-thread-safe modules by confining activity to a single Perl interpreter thread.
Helpful resources to get started:
Programming POE presentation by Matt Sergeant (start here to understand what it is and does)
POE project page (lots of cookbook examples)
Plus there are hundreds of pre-built POE components you can use to assemble into an application.
You can always have a pipe between parent and child to pass messages back and forth.
pipe my $reader, my $writer;
my $pid = fork();
if ( $pid == 0 ) {
close $reader;
...
}
else {
close $writer;
my $msg_from_child = <$reader>;
....
}
Not a very comfortable way of programming, but it shouldn't be 'erratic'.
Have a look at forks.pm, a "drop-in replacement for Perl threads using fork()" which makes for much more sensible memory usage (but don't use it on Win32). It will allow you to declare "shared" variables and then it automatically passes changes made to such variables between the processes (similar to how threads.pm does things).
From perl 5.8 onwards you should be looking at the core threads module. Have a look at http://metacpan.org/pod/threads
If you want to use modules which aren't thread safe you can usually load them with a require and import inside the thread entry point.
If I have a script that is a wrapper for another program (e.g., a daemonizer wrapper or a wrapper for mathematica), it is sometimes useful to trap signals in the wrapper program and pass them along to the subprogram.
For example, here's some Perl code to deal with the INT (interrupt) signal so that if you do ctrl-C after launching the wrapper the subprogram also gets interrupted:
my $subprogram = "foo args";
my $pid = open(F, "$subprogram |") or die "Can't open pipe from $subprogram: $!";
$SIG{INT} = sub { kill('INT', $pid);
close(F);
die "Passed along INT signal and now aborting.\n"; };
print while(<F>);
close(F);
Of [all the possible signals](http://en.wikipedia.org/wiki/Signal_(computing) that a program might handle, which ones should a wrapper script pass along?
Is there anything else that a good wrapper should do?
EDIT: Originally this question was asking how to pass along all possible signals. Thanks to the initial answers I learned that was not the right question.
EDIT: I figured out what threw me for a loop here. Mathematica apparently detaches itself from its parent process. So I have to pass along various termination signals explicitly:
$SIG{INT} = sub { kill('INT', $pid); }; # pass along SIGINT (eg, ctrl-C)
$SIG{TERM} = sub { kill('TERM', $pid); }; # pass along SIGTERM (kill's default)
$SIG{QUIT} = sub { kill('QUIT', $pid); }; # pass along SIGQUIT
$SIG{ABRT} = sub { kill('ABRT', $pid); }; # pass along SIGABRT
$SIG{HUP} = sub { kill('HUP', $pid); }; # pass along SIGHUP
Normally this would not be necessary as child processes automatically get these signals passed along (thanks for the answers that set me straight on this!). So now I'm wondering why (and how) mathematica detaches itself...
There are many signals that would be dangerous to just "pass through" in this way. "man 7 signal" and look at things like SIGPIPE, SIGILL, SIGCHILD, etc. It's very likely that you just don't want to touch those guys.
So, the real question becomes: What is the behavior you're looking for? I bet all you really want is SIGINT & SIGCONT to be passed through to the child. Does the subprogram do anything fancy with other signals, like HUP, USR1, or USR2?
I presume you're really just interested in SIGINT & SIGCONT, and the rest (i.e. SIGSEGV, SIGKILL, etc.) will take care of themselves, as the parent process termination will clean up the child as well.
Oh, and by the way, your example of:
print while(<F>);
Is fine, and if the parent perl process is suspended, it won't continue reading from F, once that pipe fills up, your subprogram will block writing into stdout, which is probably pretty much the behavior you want as well.
For some more interesting thoughts, take a look at "man bash" and the "trap" builtin to see what the shell developers have done to help this problem.
None.
The signals that you might generate at the keyboard are sent to the child process anyway, unless the child takes steps to avoid that (in which case, who are you to interfere).
The other signals that might kill the parent process would normally cause the child to disappear when it next writes to the pipe that was closed when the parent died.
So, unless you have reason to think that the child process mismanages signals, I would not worry about relaying any signals to the child. And if the child mismanages signals, maybe you should fix the child rather than hack the parent.
I really don't see why you would want to do that. For that matter, I don't see why you would want to send signals to the wrapper script.
ETA: Stop and restart signals are sent to the whole process group. If that's what you're after: just make sure your child process is part of that process group (it is by default AFAIK).
ETA2:
If mathematica really detaches itself (which can be done using setsid() and setpgid()), that means it explicitly does not want to receive such signals, so you shouldn't send them: it probably won't handle them anyway.
You can get a list of all signals with keys %SIG. So, to pass them all through, you set handlers to send the signal the child process ID:
for my $signal (keys %SIG) {
$SIG{$signal} = sub { kill($signal, $child_pid); };
}
You'll probably want to localize %SIG.
Not that I'm saying it's a good idea, but that's how to do it.