I am using a ZeroRPC server to implement a service. Some of the functions of the service are long CPU intensive tasks. Since I don't want the RPC call to block, they must be handled appropriately. Considering the GIL, threading does not seem like an appropriate choice. I've tried multiprocessing but the children processes are unable to connect as an RPC client to another (unrelated) RPC server. See the code below. It is important that I am able to spawn off multiple children processes, every time the spin_off_worker() is called. What is the best way of handling/designing these long-lasting CPU intensive tasks within the ZeroRPC server?
RPC server #1
import zerorpc
import multiprocessing
import time
import threading
class Foo(object):
def heavy_task(self):
# some long processing...
# this RPC call never gets made...
cli = zerorpc.Client(connect_to="tcp://0.0.0.0:3334", timeout=30, heartbeat=5)
cli.test()
cli.close()
return
def spin_off_worker(self):
s = multiprocessing.Process(target=self.heavy_task)
s.start()
s.join()
def start():
s = zerorpc.Server(Foo(), heartbeat=5)
s.bind("tcp://0.0.0.0:3333")
s.run()
return s
svr_proc = multiprocessing.Process(target=start)
svr_proc.start()
RPC Server 2
import zerorpc
import time
class Foo(object):
def test(self):
print "TIGR"
s = zerorpc.Server(Foo(), heartbeat=5)
s.bind("tcp://0.0.0.0:3334")
s.run()
You solution is in the right direction. It just needs few tweaks:
zerorpc requires gevent, and so, runs in a cooperative coroutine environment. Because your worker (server #2) process is going to hold on the CPU for a while, zerorpc will miss heartbeats. You can disable the heartbeat between your master (server #1) and worker (server #2) process. Set heartbeat=None on both side.
since you are doing local interprocess communication, you might as well use a local unix socket (if you have access to unix sockets on your system). Use something like unix://workers.sock. This might be more efficient than having the full TCP stack on the way.
Heart-beating is more important on the network than locally on the same machine anyway. If want to run your worker on a different machine across the network, do something like: client machine -> [zerorpc heartbeat] -> server machine -> [local zerorpc no heartbeat] -> worker process.
Note: You do not have to instantiate a new zerorpc client for every heavy_task call. You can reuse the same instance over and over.
Related
This is what i want to achieve
root> ./webserver start // Does not block the terminal after startup, runs in the background and the process is guarded
root>
My current implementation logic：
Logic running in the background
use std::process::Command;
use std::thread;
use std::env;
use std::time::Duration;
fn main() {
let args: Vec<String> = env::args().collect();
if args.len() == 2 {
if &args[1] == "start" {
// Main process start child process
let child = Command::new(&args[0])
.spawn().expect("Child process failed to start.");
println!("child pid: {}", child.id());
// Main process exit
}
} else {Is there any more elegant approach? Looking forward to your reply
// Main business logic
run webserver
}
}
In this way, rust will run in the background without blocking the terminal, but the information printed by rust in the background will still be displayed on the terminal, and the current rust program will exit when exiting the terminal
Process daemon logic
My idea is to monitor the exit signal of the system and not process the exit request
SIGHUP 1 /* Hangup (POSIX). */
SIGINT 2 /* Interrupt (ANSI). */
SIGQUIT 3 /* Quit (POSIX). */
SIGTERM 15 /* Termination (ANSI). */
code:
use signal_hook::{iterator::Signals, SIGHUP,SIGINT,SIGQUIT,SIGTERM};
use std::{thread, time::Duration};
pub fn process_daemon() {
let signals = match Signals::new(&[SIGHUP,SIGINT,SIGQUIT,SIGTERM]) {
Ok(t) => t,
Err(e) => panic!(e),
};Is there any more elegant approach? Looking forward to your reply
thread::spawn(move || {
for sig in signals.forever() {
println!("Received signal {:?}", sig);
}
});
thread::sleep(Duration::from_secs(2));
}
Is there any more elegant approach? Looking forward to your reply.
TLDR: if you really want your process to act like a service (and never quit), probably do the work to set up a service manager. Otherwise, just let it be a normal process.
Daemonizing a Process
One thing to notice right off the bat is that most of the considerations about daemonizing have nothing to do with Rust as a language and are more about:
The underlying system your processes are targeted for
The exact behavior of your daemon processes once spawned
By looking at your question, it seems you have realized most of this. Unfortunately to properly answer your question we have to delve a bit into the intricacies of processes and how they are managed. It should be noted that existing 'service' managers are a great solution if you are OK with significant platform dependence in your launching infrastructure.
Linux: systemd
FreeBSD: rc
MacOS: launchd
Windows: sc
As you can see, no simple feat if you want to have a simple deployment that just works (provided that it is compiled for the relevant system). These are just the bare metal service managers. If you want to support virtual environments you will have to think about Kubernetes services, dockerizing, etc.
These tools exist because there are many considerations to managing a long-running process on a system:
Should my daemon behave like a service and respawn if killed (or if the system is rebooted)? The tools above will allow for this.
If a service, should my daemon have status states associated with it to help with maintenance? This can help with managing outages and building tooling to scale horizontally.
If the daemon shouldn't be a service (unlikely in your case given your binary's name) there are even more questions: should it be attached to the parent process? Should it be attached to the login process group?
My guess for your process given how complex this can become, simply run the process directly. Don't daemonize at all.
For testing, (if you are in a unix-like environment) you can run your process in the background:
./webserver start &
This will spawn the new process in the background, but attach it to your shell's process list. This can be nice for testing, because if that shell goes away the system will clean up these attached processes along with it.
The above will direct stderr and stdout file descriptors back to your terminal and print them. If you wish to avoid that, you can always redirect the output somewhere else.
Disabling signals to a process like this doesn't seem like the right approach to me. Save these signals to gracefully exit your process once you need to save state or send a termination message to a client. If you do the above, your daemon will only be killable by a kill -9 <pid> or rebooting, or finding some non-standard signal you haven't overridden whose default behavior is to terminate.
Supervisor document mentions about remote command line.
Supervisor provides you with one place to start, stop, and monitor
your processes. Processes can be controlled individually or in groups.
You can configure Supervisor to provide a local or remote command line
and web interface.
Does it mean that supervisor can control and monitor processes running on a remote host?
For example,
There are three servers A, B and C.
Server A run supervisor.
The supervisor run processes in B and C.
The supervisor can monitor and control(stop and start) the processes in B and C.
If it is possible, could you give me a Supervisor config example for this purpose?
you can't use supervisor only to manage remote host's process.
but you can install supervisor on each host, and manage them on a control host by using supervisor XML-RPC API, which is official.
and many plugins are implemented in this way.
The meaning of the quote is not 100% clear. IMHO it means the control can happen remotely or local.
Anyway found this part of the supervisor documentation very helpful.
http://supervisord.org/plugins.html
It mentions tools that can help you control supervisor on multiple machine from single source.
use xml-rpc
import socket
from http.client import HTTPConnection
from xmlrpc import client
class UnixStreamHTTPConnection(HTTPConnection):
def connect(self):
self.sock = socket.socket(socket.AF_UNIX, socket.SOCK_STREAM)
self.sock.connect(self.host)
class UnixStreamTransport(client.Transport, object):
def __init__(self, socket_path):
self.socket_path = socket_path
super(UnixStreamTransport, self).__init__()
def make_connection(self, host):
return UnixStreamHTTPConnection(self.socket_path)
server = client.ServerProxy('http://127.0.0.1',transport=UnixStreamTransport("/path/to/supervisor.sock"))
print(proxy.supervisor.getState())
server.supervisor.stopProcess("test")
server.supervisor.startProcess("test")
I have a C# application that enables users to write a test and execute it (client). It also supports distributed execution over multiple machines using a central server and agents on said machines.
The agent is practically a duplication of the original execution ability but it is in a standalone solution.
We'd like to refactor that because:
Code duplication.
If a user will try to write and execute on a machine that runs an agent, there will be a problematic collision.
I'm considering 2 options:
Move this execution to a service, that both client and agent will use. I mean a service that will run locally, not a web service.
Merge client and agent - we'll have no agent, but the server will communicate with the client as an agent instead.
I have no experience in working with services. Are there any known advantages/disadvantages to either options?
A common library shared by both client and agent sounds more appropriate to allow simple cases such as just using the client and avoid the overhead of having to set up an extra service locally.
I am on an embedded platform (mipsel architecture, Linux 2.6 kernel) where I need to monitor IPC between two closed-source processes (router firmware) in order to react to a certain event (dynamic IP change because of DSL reconnect). What I found out so far via strace is that whenever the IP changes, the DSL daemon writes a special message into a UNIX domain socket bound to a specific file name. The message is consumed by another daemon.
Now here is my requirement: I want to monitor the data flow through that specific UNIX domain socket and trigger an event (call a shell script) if a certain message is detected. I tried to monitor the file name with inotify, but it does not work on socket files. I know I could run strace all the time, filtering its output and react to changes in the filtered log file, but that would be too heavy a solution because strace really slows down the system. I also know I could just poll for the IP address change via cron, but I want a watchdog, not a polling solution. And I am interested in finding out whether there is a tool which can specifically monitor UNIX domain sockets and react to specific messages flowing through in a predefined direction. I imagine something similar to inotifywait, i.e. the tool should wait for a certain event, then exit, so I can react to the event and loop back into starting the tool again, waiting for the next event of the same type.
Is there any existing Linux tool capable of doing that? Or is there some simple C code for a stand-alone binary which I could compile on my platform (uClibc, not glibc)? I am not a C expert, but capable of running a makefile. Using a binary from the shell is no problem, I know enough about shell programming.
It has been a while since I was dealing with this topic and did not actually get around to testing what an acquaintance of mine, Denys Vlasenko, maintainer of Busybox, proposed as a solution to me several months ago. Because I just checked my account here on StackOverflow and saw the question again, let me share his insights with you. Maybe it is helpful for somebody:
One relatively easy hack I can propose is to do the following:
I assume that you have a running server app which opened a Unix domain listening socket (say, /tmp/some.socket), and client programs connect to it and talk to the server.
rename /tmp/some.socket -> /tmp/some.socket1
create a new socket /tmp/some.socket
listen on it for new client connections
for every such connection, open another connection to /tmp/some.socket1 to original server process
pump data (client<->server) over resulting pairs of sockets (code to do so is very similar to what telnetd server does) until EOF from either side.
While you are pumping data, it's easy to look at it, to save it, and even to modify it if you need to.
The downside is that this sniffer program needs to be restarted every time the original server program is restarted.
This is similar to what Celada also answered. Thanks to him as well! Denys's answer was a bit more concrete, though.
I asked back:
This sounds hacky, yes, because of the restart necessity, but feasible.
Me not being a C programmer, I keep wondering though if you know a
command line tool which could do the pass-through and protocolling or
event-based triggering work for me. I have one guy from our project in
mind who could hack a little C binary for that, but I am unsure if he
likes to do it. If there is something pre-fab, I would prefer it. Can it
even be done with a (combination of) BusyBox applet(s), maybe?
Denys answered again:
You need to build busybox with CONFIG_FEATURE_UNIX_LOCAL=y.
Run the following as intercepting server:
busybox tcpsvd -vvvE local:/tmp/socket 0 ./script.sh
Where script.sh is a simple passthrough connection
to the "original server":
#!/bin/sh
busybox nc -o /tmp/hexdump.$$ local:/tmp/socket1 0
As an example, I added hex logging to file (-o FILE option).
Test it by running an emulated "original server":
busybox tcpsvd -vvvE local:/tmp/socket1 0 sh -c 'echo PID:$$'
and by connecting to "intercepting server":
echo Hello world | busybox nc local:/tmp/socket 0
You should see "PID:19094" message and have a new /tmp/hexdump.19093 file
with the dumped data. Both tcpsvd processes should print some log too
(they are run with -vvv verbosity).
If you need more complex processing, replace nc invocation in script.sh
with a custom program.
I don't think there is anything that will let you cleanly sniff UNIX socket traffic. Here are some options:
Arrange for the sender process to connect to a different socket where you are listening. Also connect to the original socket as a client. On receipt of data, notice the data you want to notice and also pass everything along to the original socket.
Monitor the system for IP address changes yourself using a netlink socket (RTM_NEWADDR, RTM_NEWLINK, etc...).
Run ip monitor as an external process and take action when it writes messages about added & removed IP addresses on its standard output.
I need to launch a server script which will not exit. and after the server is ready I need to start the client code to run some automated tests.
tried, not work, the server process is not in the background and the client code cannot be executed.
system ($server &)
is it possible to use Parallel::ForkManager to handle this, how? all the examples are repetitive tasks, while my case is server and client.
Parallel::ForkManager isn't really designed for this; there are various other distributions for supporting what a server needs to do; Daemon::Daemonize looks like it does the fewest other things besides just running your designated server code in the background.