I recently encountered a few cases where a server would distribute an event stream that contains the exact same data for all listeners, such as a 'recent activity' box.
It occurred to me that it is quite strange and inefficient to have a server like Apache run a thread processing and querying the database for every single comet stream containing the same data.
What I would do for those global(not per user) streams is run a single thread that continuously emits data, and a new (green)thread for every new request that outputs the headers and then 'merges' into the main thread.
Is it possible for one thread to serve multiple sockets, or for multiple clients to listen to the same socket?
An example
o = event
# threads received
| a b # 3
o / / # 3 -
|/_/
| # 1
o c # 2 a, b
| /
o/ # 2 a, b
o # 1 a, b, c
| # connection b closed
o # 1 a, c
Does something like this exist? Would it work? Is it possible to do?
Disclaimer: I'm not a server expert.
Check out node.js - single threaded, event driven server. Uses JavaScript as a bonus.
If you are using ASP.NET, the following post should be useful
http://beta.codeproject.com/KB/aspnet/CometAsync.aspx
By the way, it is possible to implement Comet to serve more than one client per thread, but only one thread for all client seems not enough?
You are talking about "asynchronous web requests" applied to Comet, some like "asynchronous Comet".
In my opinion this approach, so popular these days, is deeply flawed.
Related
I'm building a platform to run ML jobs.
Jobs will be started from an interface.
I'm making a service for each type of jobs. Some times, a service S1 might require to first make a request to another service S2 and get its output before running its own job.
Each service is split into 2 Kubernetes deployment:
one that will pull the message from a topic, check it and persist it to a database (D1)
one that will read request from the database, run the actual job, update the request state in the database and then answer to the client (D2)
Here is the flow:
interface generates a PubSub message to a topic T1
D1 pulls message from T1 and persist a request to a database
D2 sees the new request in the database and runs it then update its state in the database and answer to the client
To answer to the client, D2 has 2 options:
push a message to a pubsub topic T2 that will continiously be checked by the client. An id is passed in both request and response so that only the client can pull it from the topic.
use a callback provided by the client to make a POST request
What do you think abouut this architecture ? Does the usage of PubSub makes sense ? Also does it make sense to split each service into 2 deployment (1 that deals with request, 1 that runs the actual job ) ?
interface generates a PubSub message to a topic T1 D1 pulls message
from T1 and persist a request to a database
If there's only one database, I'm not sure I see much advantage in using a topic (implying pub/sub). Another approach would be to use a queue: the interface creates jobs into the queue, then you can have any number of workers processing it. Depending on the situation you may not even need the database at all - if all the data needed can be in the message in the queue.
use a callback provided by the client to make a POST request
That's better if you can do it, on the assumption that there's only one consumer for the event; pub/sub is more for broadcasting out to multiple consumers. Polling works but is really inefficient and has limits on how much it can scale.
Also does it make sense to split each service into 2 deployment (1
that deals with request, 1 that runs the actual job ) ?
Having separate deployables make sense if they are built by different teams and have a different release cadence or if you need to scale them out independently, otherwise it may not be necessary.
I'm using 0MQ to let multiple processes talk to each other (IPC sockets, but should also work via TCP across different nodes). My code is similar to a client/server pattern, but REQ/REP sockets are not enough. Here is a sample conversation. See below for further details.
Process A
Process B
open socket
not started
start process B
-
-
open socket, connect to A
-
send hello (successful start, socket information)
request work
-
-
do work
-
send response (work result 1)
-
send response (work result 2)
-
send unsolicited message
-
send response (work finished)
request termination
-
Actually, A is (even though doing all the requests) closer to be the server component, since it is constantly running. Based on external triggers, A starts a sort of plugin process B.
Every request needs to be answered by a finished response. Before that, N (between 0 and an arbitrary upper bound) responses can be sent from B.
A new request can be sent from A even when the current request is still ongoing (no finished message received). If relevant, the code could be updated to buffer the requests.
B sends an initial message which is not preceded by a request from A.
B can send other messages (logging) anywhere in between, also not preceded by a request.
Optional: A single socket in A should handle multiple plugin processes B, C, D...
A DEALER/ROUTER combination would probably match all requirements, but might be a bit too much. Process B will only ever connect to a single peer. And without the optional requirement above, the same would be true for process A as well. So I'm a bit hesitant to use DEALER and ROUTER sockets which are both able to handle multiple peers.
For a system powerful enough to serve a number of requests (not running out of threads), would there be a difference, from the users perspective, in terms of response time / speed?
Also, would database the only thing that is usually blocking the thread, and hence we need reactive db driver?
I mean, if a rest endpoint does not make calls to db, there would be no diff whether the endpoint is reactive or not?
First of all, you need to know what happens when you use project-reactor - webflux client.
Let's assume your endpoint (call it /demo) is responsible for making 5 async calls to other systems to return response from itself.
Example response times:
Service A: 5 ms
Service B: 50 ms
Service C: 100 ms
Service D: 250 ms
Service E: 400 ms
Typical non-weblux client way:
5 threads are consumed, the last one is blocked for 400ms.
Webflux client way:
Every call to services A, B, C, D, E consume one thread, make call, returns thread and when the response comes another thread is consumed to process response.
The final conclusion:
If your system will be overloaded by big amount of requests (let it be n) in the same time you will lock n threads for 400 ms.
Try to imagine the scale of the problem.
one connection send many request to server
How to process request concurrently.
Please use a simple example like timeserver or echoserver in netty.io
to illustrate the operation.
One way I could find out is to create a separate threaded handler that will be called as in a producer/consumer way.
The producer will be your "network" handler, giving message to the consumers, therefore not waiting for any wanswear and being able then to proceed with the next request.
The consumer will be your "business" handler, one per connection but possibly multi-threaded, consuming with multiple instances the messages and being able to answer using the Netty's context from the connection from which it is attached.
Another option for the consumer would be to have only one handler, still multi-threaded, but then message will come in with the original Netty's Context such that it can answear to the client, whatever the connection attached.
But the difficulties will come soon:
How to deal with an answear among several requests on client side: let say the client sends 3 requests A, B and C and the answears will come back, due to speed of the Business handler, as C, A, B... You have to deal with it, and knowing for which request the answer is.
You have to ensure all the ways the context given in parameter is still valid (channel active), if you don't want to have too many errors.
Perhaps the best way would be to however handle your request in order (as Netty does), and keep the answear's action as quick as possible.
I'm writing a server for an online game, that should be able to handle 1,000-2,000 clients in the end. The 3 ways I found to do this were basically:
1 thread/connection (blocking)
Making a list of clients, and loop through them (non-blocking)
Select (Basically a blocking statement for all clients at once with optional timeout?)
In the past I've used 1, but as we all know, it doesn't scale well. 2 is okay, but I have mixed feelings, about one client technically being able to make everyone else freeze. 3 sounds interesting (a bit better than 2), but I've heard it's not suitable for too many connections.
So, what would be the best way to do it (in D)? Are there other options?
The usual approach is closest to 3: asynchronous programming with a higher-performance select alternative, such as the poll or epoll system calls on Linux, IOCP on Windows, or higher-level libraries wrapping them. D does not support them directly, but you can find D bindings or 3rd-party D libraries (e.g. Tango) providing support for them.
Higher-performance servers (e.g. nginx) use one thread/process per CPU core, and use asynchronous event handling within that thread/process.
One option to consider is to have a single thread that runs the select/pole/epoll but not process the results. Rather it queues up connections known to have results and lets a thread pool feed from that. If checking that a full request has been read in is cheap, you might do that in the poll thread with non-blocking IO and only queue up full requests.
I don't know if D provides any support for any of that aside from (possibly) the inter-thread communication and queuing.