Majordomo broker throughput measurement - sockets

Question

I am testing the majordomo broker's throughput. The test_client.c that comes along with the majordomo code on github sends synchronous request. I am wanting to test the maximum throughput that the majordomo broker can achieve. The specifications (http://rfc.zeromq.org/spec:7) say that it can switch upto a million messages per second.
First, I changed the client code to send 100k requests asynchronously. Even after setting the HWM on all the sockets sufficiently high, and increasing the TCP buffers to 4 MB, I was observing packet loss with three clients running in parallel.
So I changed the client to send 10k requests at once, and then send two requests for every reply that it receives. I chose 10k because that allowed me to run up to ten clients (each sending 100k messages) in parallel without any packet loss. Here is the client code:
#include "../include/mdp.h"
#include <time.h>
int main (int argc, char *argv [])
{
int verbose = (argc > 1 && streq (argv [1], "-v"));
mdp_client_t *session = mdp_client_new (argv[1], verbose);
int count1, count2;
struct timeval start,end;
gettimeofday(&start, NULL);
for (count1 = 0; count1 < 10000; count1++) {
zmsg_t *request = zmsg_new ();
zmsg_pushstr (request, "Hello world");
mdp_client_send (session, "echo", &request);
}
for (count1 = 0; count1 < 45000; count1++) {
zmsg_t *reply = mdp_client_recv (session,NULL,NULL);
if (reply)
{
zmsg_destroy (&reply);
zmsg_t *request = zmsg_new ();
zmsg_pushstr (request, "Hello world");
mdp_client_send (session, "echo", &request);
request = zmsg_new ();
zmsg_pushstr (request, "Hello world");
mdp_client_send (session, "echo", &request);
}
else
break; // Interrupted by Ctrl-C
}
/* receiving the remaining 55k replies */
for(count1 = 45000; count1 < 100000; count1++)
{
zmsg_t *reply = mdp_client_recv (session,NULL,NULL);
if (reply)
{
zmsg_destroy (&reply);
}
else
break;
}
gettimeofday(&end, NULL);
long elapsed = (end.tv_sec - start.tv_sec) +((end.tv_usec - start.tv_usec)/1000000);
printf("time = %ld\n", elapsed);
printf ("%d replies received\n", count1);
mdp_client_destroy (&session);
return 0;
}
I ran the broker, worker, and the clients within the same machine. Here is the recorded time:
number of clients in parallel
(each client sends 100k ) Time elapsed (seconds)
1 4
2 9
3 12
4 16
5 21
10 43
So for every 100k requests, the broker is taking about 4 seconds. Is this the expected behavior? Am not sure how to achieve million messages per second.
LATEST UPDATE:
I came up with an approach to improve the throughput of the system:
Two brokers instead of one. One of the brokers (broker1) is responsible for sending the client requests to the workers, and the other broker (broker2) is responsible for sending the response of the workers to the clients.
The workers register with broker1.
The clients generate a unique id and register with broker2.
Along with the request, a client also sends its unique id to broker1.
Worker extracts the unique client id from the request, and sends its response (along with the client id to whom the response has to be sent) to broker2.
Now, every 100k requests take around 2 seconds instead of 4 seconds (when using a single broker). I added gettimeofday calls within the broker code to measure how much latency is added by the broker itself.
Here is what I have recorded
100k requests (total time: ~2 seconds) -> latency added by the brokers is 2 seconds
200k requests (total time: ~4 seconds) -> latency added by the brokers is 3 seconds
300k requests (total time: ~7 seconds) -> latency added by the brokers is 5 seconds
So the bulk of the time is being spent within the broker code. Could someone please suggest how to improve this.

Answer 1

The maximum throughput is bound by the maximum throughout of the broker, but it's also bound by the maximum throughout of the workers.
It seems to me like you're starting only one worker. If you read the majordomo protocol specification carefully, it says that the broker should be able to switch up to millions of messages per second, but it doesn't guarantee that a single worker can process millions of requests per second.
Given that the worker treats only one request at a time and uses a fully synchronous dialog with the workers (the broker will not send another request until it gets a reply), it's impossible to squeeze the most out of the broker with a single worker, or even a single worker for that matter: a full network round-trip with the broker is performed between the reply and the next request so the worker spends most of its time waiting.
For starters, you should try adding more workers :-)
EDIT: quick demonstration.
I ran an echo service using a Python implementation of the Majordomo pattern with ZeroMQ. Of course, this is Python code will full error checking and the C version should be much, much faster, but it shows the impact of having multiple workers responding to replies.
Test setup:
1 client
client sends a max of 1000 outstanding requests at once (initially, 1000 are sent, then the client sends one new request each time it receives a response);
the client sends a total of 100,000 requests;
Results:
Using 1 worker, the client gets 100,000 replies in 13.6 seconds (~7300 RPS).
Using 2 workers, the clients gets 100,000 replies in 8.0 seconds (~12500 RPS).
Using 3 workers, the client gets 100,000 replies in 7.8 seconds (~13000 RPS).
Using more clients will yield better throughput in the broker because it will be performing I/O on multiple sockets, but it's not as easy to compute the total broker throughput, so I'll leave you measure that.