Ways to improve Cassandra read performance in my scenario - nosql

Question

We have recently started using Cassandra database in production. We have a single cross colo cluster of 24 nodes meaning 12 nodes in PHX and 12 nodes in SLC colo. We have a replication factor of 4 which means 2 copies will be there in each datacenter.
Below is the way by which keyspace and column families have been created by our Production DBA's.
create keyspace profile with placement_strategy =
'org.apache.cassandra.locator.NetworkTopologyStrategy' and
strategy_options = {slc:2,phx:2};
create column family PROFILE_USER
with key_validation_class = 'UTF8Type'
and comparator = 'UTF8Type'
and default_validation_class = 'UTF8Type'
and gc_grace = 86400;
We are running Cassandra 1.2.2 and it has org.apache.cassandra.dht.Murmur3Partitioner, with KeyCaching, SizeTieredCompactionStrategy and Virtual Nodes enabled as well.
Machine Specifications for Cassandra production nodes-
16 cores, 32 threads
128GB RAM
4 x 600GB SAS in Raid 10, 1.1TB usable
2 x 10GbaseT NIC, one usable
Below is the result I am getting.
Read Latency(95th Percentile) Number of Threads Duration the program was running(in minutes) Throughput(requests/seconds) Total number of id's requested Total number of columns requested
9 milliseconds 10 30 1977 3558701 65815867
I am not sure what other things I should try it out with Cassandra to get much better read performance. I am assuming it is hitting the disk in my case. Should I try increasing the Replication Factor to some higher number? Any other suggestion?
I believe reading the data from HDD is around 6-12ms as compared to SSD's? In my case it is hitting the disk everytime I guess and enabling key cache is not working fine here. I cannot enable RowCache becasue it’s more efficient to use OS page cache. Maintaining row cache in JVM is very expensive, thus row cache is recommended for smaller number of rows, like <100K rows, only.
Is there any way I can verify whether keycaching is working fine in my case or not?
This is what I get when I do show schema for column family-
create column PROFILE
with column_type = 'Standard'
and comparator = 'UTF8Type'
and default_validation_class = 'UTF8Type'
and key_validation_class = 'UTF8Type'
and read_repair_chance = 0.1
and dclocal_read_repair_chance = 0.0
and populate_io_cache_on_flush = false
and gc_grace = 86400
and min_compaction_threshold = 4
and max_compaction_threshold = 32
and replicate_on_write = true
and compaction_strategy = 'org.apache.cassandra.db.compaction.SizeTieredCompactionStrategy'
and caching = 'KEYS_ONLY'
and compression_options = {'sstable_compression' : 'org.apache.cassandra.io.compress.SnappyCompressor'};
Is there anything I should make a change to get good read performance?

Answer 1

I am assuming it is hitting the disk in my case. Should I try increasing the Replication Factor to some higher number? Any other suggestion?
If your data is much larger than memory and your access is close to random you will be hitting disk. This is consistent with latencies of ~10ms.
Increasing the replication factor might help, although it will make your cache less efficient since each node will store more data. It is probably only worth doing if your read pattern is mostly random, your data is very large, you have low consistency requirements and your access is read heavy.
If you want to decrease read latency, you can use a lower consistency level. Reading at consistency level CL.ONE generally gives the lowest read latency at a cost of consistency. You will only get consistent reads at CL.ONE if writes are at CL.ALL. But if consistency is not required it is a good tradeoff.
If you want to increase read throughput, you can decrease read_repair_chance. This number specifies the probability that Cassandra performs a read repair on each read. Read repair involves reading from available replicas and updating any that have old values.
If reading at a low consistency level, read repair incurs extra read I/O so decreases throughput. It doesn't affect latency (for low consistency levels) since read repair is done asynchronously. Again, if consistency isn't important for your application, decrease read_repair_chance to maybe 0.01 to improve throughput.
Is there any way I can verify whether keycaching is working fine in my
case or not?
Look at the output of 'nodetool info' and it will output a line like:
Key Cache : size 96468768 (bytes), capacity 96468992 (bytes), 959293 hits, 31637294 requests, 0.051 recent hit rate, 14400 save period in seconds
This gives you the key cache hit rate, which is quite low in the example above.

Answer 2

Old post but incase someone else comes by this.
Don't use even RF. Your RF of 4 requires quorum of 3 nodes, this is no different than a RF of 5.
Your key cache is probably working fine, this only tells cassandra where on the disk it's located. This only reduces seek times.
You have a rather large amount of ram pre 3.0, likely you're not leveraging all of this. Try G1GC on newer cassandra nodes.
Row key cache, make sure that your partitions are ordered in the way you intend to access them. Ex: If you're picking up only recent data, make sure you order by timestamp ASC instead of timestamp DESC as it will cache from the START of the partition.
Parallelize and bucket queries. Use nodetool cfhistograms to evaluate the size of your partitions. Then try and break the partitions into smaller chunks if they exceed 100mb. From here you change your queries to SELECT x FROM table WHERE id = X and bucket in (1,2,3) if you need to scan. Significant performance can then be gained from removing the "in bucket" and moving that to 3 separate queries. Ex running: Select... WHERE id = X and bucket = 1, Select ... WHERE id = X and bucket = 2 and doing the aggregation at the application layer.