OutOfMemory Error on DruidDB index_kafka_histogram tasks - druid

Question

I am new to the DruidDB setup. I am trying to ingest data in DruidDB. Initially, it was working fine but after some time I am getting the following error.
Sample Config:
...
"metricsSpec": [
{
"type": "longMin",
"name": "min",
"fieldName": "min",
"expression": null
},
{
"type": "longMax",
"name": "max",
"fieldName": "max",
"expression": null
},
{
"type": "longSum",
"name": "count",
"fieldName": "count",
"expression": null
},
{
"type": "longSum",
"name": "sum",
"fieldName": "sum",
"expression": null
},
{
"type": "quantilesDoublesSketch",
"name": "quantilesDoubleSketch",
"fieldName": "sketch",
"k": 128
}
],
"granularitySpec": {
"type": "uniform",
"segmentGranularity": "HOUR",
"queryGranularity": "MINUTE",
"rollup": true,
"intervals": null
},
...
"tuningConfig": {
"type": "kafka",
"maxRowsInMemory": 10000,
"maxBytesInMemory": 200000,
"maxRowsPerSegment": 5000000,
"intermediatePersistPeriod": "PT10M",
"basePersistDirectory": "/tmp/druid-realtime-persist15059426147899962275",
"maxPendingPersists": 0,
"indexSpec": {
"bitmap": {
"type": "roaring",
"compressRunOnSerialization": true
},
"dimensionCompression": "lz4",
"metricCompression": "lz4",
"longEncoding": "longs"
},
"indexSpecForIntermediatePersists": {
"bitmap": {
"type": "roaring",
"compressRunOnSerialization": true
},
"dimensionCompression": "lz4",
"metricCompression": "lz4",
"longEncoding": "longs"
},
"buildV9Directly": true,
"reportParseExceptions": false,
"handoffConditionTimeout": 0,
"resetOffsetAutomatically": false,
"chatRetries": 8,
"httpTimeout": "PT10S",
"shutdownTimeout": "PT80S",
"offsetFetchPeriod": "PT30S",
"intermediateHandoffPeriod": "P2147483647D",
"logParseExceptions": true,
"maxParseExceptions": 2147483647,
"maxSavedParseExceptions": 0,
"skipSequenceNumberAvailabilityCheck": false,
"repartitionTransitionDuration": "PT120S"
}
...
Error
09:55:49.134 [task-runner-0-priority-0] ERROR org.apache.druid.indexing.overlord.SingleTaskBackgroundRunner - Uncaught Throwable while running task[AbstractTask{id='index_kafka_histogram_25c6328c09f15d7_nofamgdk', groupId='index_kafka_histogram', taskResource=TaskResource{availabilityGroup='index_kafka_histogram_25c6328c09f15d7', requiredCapacity=1}, dataSource='histogram', context={checkpoints={"0":{"0":0,"1":0,"2":0,"3":0,"4":0,"5":0,"6":0,"7":0,"8":0,"9":0,"10":0,"11":0,"12":0,"13":0,"14":0,"15":0,"16":0,"17":0,"18":0,"19":0,"20":0,"21":0,"22":0,"23":0,"24":0,"25":0,"26":0,"27":0,"28":0,"29":0,"30":0,"31":0,"32":0,"33":0,"34":0,"35":0,"36":0,"37":0,"38":0,"39":0,"40":0,"41":0,"42":0,"43":0,"44":0,"45":0,"46":0,"47":0,"48":0,"49":0}}, IS_INCREMENTAL_HANDOFF_SUPPORTED=true, forceTimeChunkLock=true}}]
java.lang.OutOfMemoryError: Java heap space
Error!
java.lang.RuntimeException: java.util.concurrent.ExecutionException: java.lang.OutOfMemoryError: Java heap space
at org.apache.druid.indexing.worker.executor.ExecutorLifecycle.join(ExecutorLifecycle.java:215)
at org.apache.druid.cli.CliPeon.run(CliPeon.java:295)
at org.apache.druid.cli.Main.main(Main.java:113)
Caused by: java.util.concurrent.ExecutionException: java.lang.OutOfMemoryError: Java heap space
at com.google.common.util.concurrent.AbstractFuture$Sync.getValue(AbstractFuture.java:299)
at com.google.common.util.concurrent.AbstractFuture$Sync.get(AbstractFuture.java:286)
at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:116)
at org.apache.druid.indexing.worker.executor.ExecutorLifecycle.join(ExecutorLifecycle.java:212)
... 2 more
Caused by: java.lang.OutOfMemoryError: Java heap space
I have tried hard reset option for the tasks and answers mentioned in the following.
Link: druid indexing task fails with OutOfMemory Exception

Answer 1

A few things you should check.
Your ingestion is configured for maxRowsInMemory of 10000 and maxBytesInMemory of 200000. These seem small. They will cause each task to write to disk often. All writes to disk must be merged when it comes time to publish segments. Depending on your message size and throughput there could be a large number of small files to be merged. The merge operation may be running into trouble because of this. You can control the maxColumnsToMerge in your ingestion task that will keep this under control.
From one of the Druid committers, “the way the setting works is, let's say you have 30 columns per segment, and for each segment you have 110 intermediate persists. during segment generation, we merge all of those intermediate persists. that's 110 * 30 = 3,300 total to merge. each column has memory requirements of about 5KB on heap and 100KB direct (off heap). so that'll require about 16.5MB on heap and 500MB off heap. if you want to limit it, you can set maxColumnsToMerge = 1500 and it will use only about half that.“
Additionally you should verify that you are not overcommitting memory, you should have enough memory for worker.capacity * ( heap + direct memory ) on the middle manager in addition to what the Middle manager process itself uses. Also assuming there is nothing else running on the node.