I currently have a Kafka Stream service:
{
val _ = metrics
val timeWindow = Duration.of(config.timeWindow.toMillis, ChronoUnit.MILLIS)
val gracePeriod = Duration.of(config.gracePeriod.toMillis, ChronoUnit.MILLIS)
val store = Materialized
.as[AggregateKey, AggMetricDocument, ByteArrayWindowStore](
config.storeNames.reducerStateStore
)
.withRetention(gracePeriod.plus(timeWindow))
.withCachingEnabled()
builder
.stream[AggregateKey, AggMetricDocument(config.topicNames.aggMetricDocumentsIntermediate)
.groupByKey
.windowedBy(TimeWindows.of(timeWindow).grace(gracePeriod))
.reduce { (metricDoc1, metricDoc2) =>
metricDoc1.copy(
metrics = metricDoc1.metrics
.merge(metricDoc2.metrics, config.metricDocumentsReducerSamplesReservoirSize),
docsCount = metricDoc1.docsCount + metricDoc2.docsCount
)
}(store)
.toStream
.to(config.topicNames.aggMetricDocuments)(
Produced.`with`(AggregateKey.windowedSerde, AggMetricDocument.flattenSerde)
)
}
While
timeWindow=1m
gracePeriod=39h
The stream works fine on normal cardinality but when it starts processing high cardinality data(more than 100 million different keys) the processing rate declines after some time.
By looking at the RocksDB metrics it looks like the avg fetch latency is rising from 30µs to 600µs, and some decreasing in the hit rate of the filters and index as seen in the following test(sending ~15K/sec messages with uniqe keys):
The disk throughput and io seems under the disk limits.
The cpu usage and Load Avg increasing(the limit is 5 cores):
I made some RocksDB config modification:
private val cache = new LRUCache(2147483648, -1, false, 0.9) // 2GB
private val writeBufferManager = new WriteBufferManager(2147483648, cache)
val tableConfig = options.tableFormatConfig.asInstanceOf[BlockBasedTableConfig]
tableConfig.setBlockCache(BoundedMemoryRocksDBConfig.cache)
tableConfig.setCacheIndexAndFilterBlocks(true) // Default false
tableConfig.setCacheIndexAndFilterBlocksWithHighPriority(true)
All other setting has the default values of Kafka Streams.
It seems that increasing the LRUCache helps for a while.
I am not sure what the core problem, Does someone have an idea of what causing this problem and which configuration should I tune to get better performance on high cardinality data.
Related
I define a receiver to read data from Redis.
part of receiver simplified code:
class MyReceiver extends Receiver (StorageLevel.MEMORY_ONLY){
override def onStart() = {
while(!isStopped) {
val res = readMethod()
if (res != null) store(res.toIterator)
// using res.foreach(r => store(r)) the performance is almost the same
}
}
}
My streaming workflow:
val ssc = new StreamingContext(spark.sparkContext, new Duration(50))
val myReceiver = new MyReceiver()
val s = ssc.receiverStream(myReceiver)
s.foreachRDD{ r =>
r.persist()
if (!r.isEmpty) {
some short operations about 1s in total
// note this line ######1
}
}
I have a producer which produce much faster than consumer so that there are plenty records in Redis now, I tested with number 10000. I debugged, and all records could quickly be read after they are in Redis by readMethod() above. However, in each microbatch I can only get 30 records. (If store is fast enough it should get all of 10000)
With this suspect, I added a sleep 10 seconds code Thread.sleep(10000) to ######1 above. Each microbatch still gets about 30 records, and each microbatch process time increases 10 seconds. And if I increase the Duration to 200ms, val ssc = new StreamingContext(spark.sparkContext, new Duration(200)), it could get about 120 records.
All of these shows spark streaming only generate RDD in Duration? After gets RDD and in main workflow, store method is temporarily stopped? But this is a great waste if it is true. I want it also generates RDD (store) while the main workflow is running.
Any ideas?
I cannot leave a comment simply I don't have enough reputation. Is it possible that propertyspark.streaming.receiver.maxRate is set somewhere in your code ?
I'm writing a spark streaming job that reads data from Kafka, makes some changes to the records and sends the results to another Kafka cluster.
The performance of the job seems very slow, the processing rate is about 70,000 records per second. The sampling shows that 30% of the time is spent on reading data and processing it and the remaining 70% spent on sending data to the Kafka.
I've tried to tweak the Kafka configurations, add memory, change batch intervals, but the only change that works is to add more cores.
profiler:
Spark job details:
max.cores 30
driver memory 6G
executor memory 16G
batch.interval 3 minutes
ingres rate 180,000 messages per second
Producer Properties (I've tried different varations)
def buildProducerKafkaProperties: Properties = {
val producerConfig = new Properties
producerConfig.put(ProducerConfig.BOOTSTRAP_SERVERS_CONFIG, destKafkaBrokers)
producerConfig.put(ProducerConfig.ACKS_CONFIG, "all")
producerConfig.put(ProducerConfig.BATCH_SIZE_CONFIG, "200000")
producerConfig.put(ProducerConfig.LINGER_MS_CONFIG, "2000")
producerConfig.put(ProducerConfig.COMPRESSION_TYPE_CONFIG, "gzip")
producerConfig.put(ProducerConfig.RETRIES_CONFIG, "0")
producerConfig.put(ProducerConfig.BUFFER_MEMORY_CONFIG, "13421728")
producerConfig.put(ProducerConfig.SEND_BUFFER_CONFIG, "13421728")
producerConfig
}
Sending code
stream
.foreachRDD(rdd => {
val offsetRanges = rdd.asInstanceOf[HasOffsetRanges].offsetRanges
rdd
.map(consumerRecord => doSomething(consumerRecord))
.foreachPartition(partitionIter => {
val producer = kafkaSinkBroadcast.value
partitionIter.foreach(row => {
producer.send(kafkaTopic, row)
producedRecordsAcc.add(1)
})
stream.asInstanceOf[CanCommitOffsets].commitAsync(offsetRanges)
})
Versions
Spark Standalone cluster 2.3.1
Destination Kafka cluster 1.1.1
Kafka topic has 120 partitions
Can anyone suggest how to increase sending throughput?
Update Jul 2019
size: 150k messages per second, each message has about 100 columns.
main settings:
spark.cores.max = 30 # the cores balanced between all the workers.
spark.streaming.backpressure.enabled = true
ob.ingest.batch.duration= 3 minutes
I've tried to use rdd.repartition(30), but it made the execution slower by ~10%
Thanks
Try to use repartition as below -
val numPartitons = ( Number of executors * Number of executor cores )
stream
.repartition(numPartitons)
.foreachRDD(rdd => {
val offsetRanges = rdd.asInstanceOf[HasOffsetRanges].offsetRanges
rdd
.map(consumerRecord => doSomething(consumerRecord))
.foreachPartition(partitionIter => {
val producer = kafkaSinkBroadcast.value
partitionIter.foreach(row => {
producer.send(kafkaTopic, row)
producedRecordsAcc.add(1)
})
stream.asInstanceOf[CanCommitOffsets].commitAsync(offsetRanges)
})
This will give you optimum performance.
Hope this will help.
I try to aggregate a large amount of data using time windows of different sizes using Kafka Streams.
I increased the cache size to 2 GB, but when I set the window size in 1 hour I get the CPU load of 100% and the application starts to slow down.
My code looks like this:
val tradeStream = builder.stream<String, Trade>(configuration.topicNamePattern, Consumed.with(Serdes.String(), JsonSerde(Trade::class.java)))
tradeStream
.groupBy(
{ _, trade -> trade.pair },
Serialized.with(JsonSerde(TokensPair::class.java), JsonSerde(Trade::class.java))
)
.windowedBy(TimeWindows.of(windowDuration).advanceBy(windowHop).until(windowDuration))
.aggregate(
{ Ticker(windowDuration) },
{ _, newValue, aggregate -> aggregate.add(newValue) },
Materialized.`as`<TokensPair, Ticker>(storeByPairs)
.withKeySerde(JsonSerde(TokensPair::class.java))
.withValueSerde(JsonSerde(Ticker::class.java))
)
.toStream()
.filter { tokensPair, _ -> filterFinishedWindow(tokensPair.window(), windowHop) }
.map { tokensPair, ticker -> KeyValue(
TickerKey(ticker.tokensPair!!, windowDuration, Timestamp(tokensPair.window().start())),
ticker.calcPrice()
)}
.to(topicName, Produced.with(JsonSerde(TickerKey::class.java), JsonSerde(Ticker::class.java)))
In addition, before sending the aggregated data to the kafka topic they are filtered by end time of the window in order send to topic just finished window.
Perhaps there are some better approaches for implementing this kind of aggregation?
With out a knowing a bit more of the system it’s hard to diagnose.
How many partitions are present in your cluster ?
How many stream applications are you running ?
Are the stream applications running on the same machine ?
Are you using compression for the payload ?
Does it work for smaller intervals?
Hope that helps.
iam new to kafka i want to implement a kafka message pipeline system for our play-scala project,i created a topic in which records are inserted, and i wrote a consumer code like this,
val recordBatch = new ListBuffer[CountryModel]
consumer.subscribe(Arrays.asList("session-queue"))
while (true) {
val records = consumer.poll(1000)
val iter = records.iterator()
while (iter.hasNext()) {
val record: ConsumerRecord[String, String] = iter.next()
recordBatch += Json.parse(record.value()).as[CountryModel]
}
processData(recordBatch)
// Thread.sleep(5 * 1000)
}
but after certain time the service is stopping, the processor is going to 100% after certain time machine is stopping, how can i handle this infinite loop.
in production environment i can not rely on this loop right i tried to sleep the thread for some time but it is not a elegant solution
I have a following algorithm with scala:
Do initial call to db to initialize cursor
Get 1000 entities from db (Returns Future)
For every entity process one additional request to database and get modified entity (returns future)
Transform original entity
Put transformed entity to Future call back from #3
Wait for all Futures
In scala it some thing like:
val client = ...
val size = 1000
val init:Future = client.firstSearch(size) //request over network
val initResult = Await(init, 30.seconds)
var cursorId:String = initResult.getCursorId
while (!cursorId.isEmpty) {
val futures:Seq[Future] = client.grabWithSize(cursorId).map{response=>
response.getAllResults.map(result=>
val grabbedOne:Future[Entity] = client.grabOneEntity(result.id) //request over network
val resultMap:Map[String,Any] = buildMap(result)
val transformed:Map[String,Any] = transform(resultMap) //no future here
grabbedOne.map{grabbedOne=>
buildMap(grabbedOne) == transformed
}
}
Futures.sequence(futures).map(_=> response.getNewCursorId)
}
}
def buildMap(...):Map[String,Any] //sync call
I noticed that if I increase size say two times, every iteration in while started working slowly ~1.5. But I do not see that my PC processor loaded more. It loaded near zero, but time increases in ~1.5. Why? I have setuped:
implicit val ec = ExecutionContext.fromExecutor(Executors.newFixedThreadPool(1024))
I think, that not all Futures executed in parallel. But why? And ho to fix?
I see that in your code, the Futures don't block each other. It's more likely the database that is the bottleneck.
Is it possible to do a SQL join for O(1) rather than O(n) in terms of database calls? (If you're using Slick, have a look under the queries section about joins.)
If the load is low, it's probably that the connection pool is maxed out, you'd need to increase it for the database and the network.