I plan to have celery workers sending notifications to mobile devices (e.g. through GCM) and wish to avoid opening needlessly many TCP connections
I have a celery task defined like this:
#task()
def send_push_notification(
requests.post(...)
)
Suppose this task is executed by a worker on a single machine. Will each subprocess open only one TCP connection? Or will a new connection be created each time the task is executed?
If I want to reuse the same TCP connection for each subprocess, how would I go about doing that?
Thanks!
Related
Gunicorn is based on the pre-fork worker model. There is Master Process and Sync Workers. Each Sync Worker Process handles a single request at a time. How does Gunicorn manage TCP Connection ?
Does the client create socket connection directly with each worker, which is listening on defined port, but two processes cannot listen on the same port ?
Does client create socket connection with Master process and it forwards the request via IPC to worker ?
I got a standard rails app with 2 process types, the web process and the worker process. both running as tasks on marathon.
Is there a way to define health check to the worker process, the process does not listen on any port, whats recommended here?
Is it possible to transparently migrate an established TCP connection along with the Docker container from one node to another?
My use case is scaling/re-scheduling an web-app which relies on WebSockets but I believe there would be more use cases for other application protocols and plain tcp.
What I'm looking for is a way to do it completely transparently for client applications. I'm aware it's possible to reconnect upon disconnection but this is not what I need.
I've been looking at SockMI agent but it seems to be still in beta and missing documentation.
If I understand this correctly the migration would require the following at high-level:
Trigger scaling action (when it all needs to start)
Launch replacement container on new node
Freeze container's processes on original node
Put tcp connections on hold
Transfer the processes and their state across to new node
Migrate the TCP connection
Is it possible to transparently migrate an established TCP connection ... from one node to another?
No.
Is it possible to create Celery task that just waits for a signal? I have this scenario:
Scrapyd in one virtualenv on remote machine A
Django project with Celery worker node in another virtualenv on remote machine A
The same Django project with Celery, but in another virtualenv on local machine B
How I use this setup:
I would send a task chain from Django on machine B
Let the task chain be consumed by the worker node on machine A.
In the first subtask of the task chain, I would schedule a crawl using Scrapyd's JSON over HTTP API, and pass the Celery task ID to the crawler as an HTTP request parameter.
I then want this first subtask to just wait for some kind of signal.
Scrapyd does its thing and runs the spider.
Once the spider is done crawling, I want it to send a signal, maybe by JSON over HTTP or by a Django management command, to the subtask that has been waiting for the signal.
Is this doable?
I would just need code snippets to show me how to wait for a signal in a subtask, and how to restore a task from the task ID and send a signal to it.
Can a worker job in Heroku make socket (ex.pop3) connection to external server ?
I guess scaling worker process to 2 or more will run jobs in parallel and they all trying to connect to same server/port from a same client/port, am I right or missing something ?
Yes - Heroku workers can connect to the outside world - however, there is no built in provision for handling the sort of problems that you mention - you'd need to do that bit yourself.
Just look at the workers as a variety of separate EC2 instances.