what can be used for (reliably) testing Couchbase latency? - memcached

Question

So, testrunner is over-complicated, cbloadworkgen gives no request times, and mcsoda seems to work bad with membase protocol (thus doing memcached direct tests - totally avoiding cluster layer).
Is there anything else? Or any known way to really bench couchbase with one of that tools?
I need timings for 99%, 95% of requests.

Answer 1

Are you refering about development or production server scenario ? Couchbase latency itself won't be the bottleneck - its basically only network latency and your application stack (e.g. PHP) that causes latency.
I have used weighttp tool to mainly test the request/seconds possible in a development scenario - that tool should also give you the average latency per HTTP request. You can find my setup on GitHub. Its using real HTTP requests to call an gwan C-servlet making multiple CB-queries.
If you want to know about how to achieve best latency and throughput with couchbase:
Use native async code using libcouchbase. PHP, NodeJS, Java.. all of them are the true bottlenecks in production stacks.
Async allows to keep multiple couchbase queries in flight - this way latency can "happen in parallel" - end user will notice only slight improvements, server load will be
Make sure your app is using connection pools. You don't want to initialize a connection for each user request you are serving! The servlet above countains a rough implementation of one.
Keep couchbase instances local, if possible. I was running a cluster where each node was running my app server and a couchbase instance, and each cb node would have their own bucket for caching things that can be cached (like configurations). You can use XDCR if your backup needs are not so strict - dont need automatic failover to live replicas. If ur running 2 Nodes with 1 Replica, it will mean that 50% of your keyspace is on another machine.
If data set is large, make sure to use SSDs.
This way you can achieve incredible performance, up to 300.000 ops / second if using a decent local node (i7 4770k, 32 GB DDR3-1600), 240 GB SSD for ~75$/mo here in germany) and a bit less if in a cluster (not sure why). Here's a 8 node CouchBase setup (with 24 additional load generator nodes) I was running with the above test benchmark:
Conclusion is: don't worry about couchbase itself. Worry only about your own app architecture/code and its latency. One thing is, that CouchBase knows about magic - the other thing is even if its 'bad' - you can hardly get any better latency with other products out there.

Answer 2

Try cbc-pillowfight which ships as part of the libcouchbase-tools package. This supports both memcached and couchbase protocols, and gives timings similar to what you want. For example:
cbc pillowfight -h localhost -b my_bucket -i 10000 -T -d
-T displays the timings, and -d enables memcached (dumb) node. Ensure you change my_bucket to whatever your bucket is named. Full documentation is on the cbc man page.
Example timing output (snipped for brevity):
[1397065322.061025] Populate
+---------+---------+---------+---------+
[110 - 119]us |################ - 68
[120 - 129]us |######################################## - 163
[130 - 139]us |######################### - 105
[140 - 149]us |###################### - 93
[150 - 159]us |####################### - 94
...
+----------------------------------------
[1397065322.070389] Run
+---------+---------+---------+---------+
[110 - 119]us |################ - 69
[120 - 129]us |######################################## - 165
[130 - 139]us |########################## - 109
[140 - 149]us |###################### - 94
[150 - 159]us |####################### - 95
[160 - 169]us |################### - 81
[170 - 179]us |############### - 62
[180 - 189]us |############### - 63
[190 - 199]us |########### - 46
[200 - 209]us |######## - 35
[210 - 219]us |######### - 39
...
+----------------------------------------