pyspark - kafka stream - out of memory - pyspark

Question

I'm trying to test kafka streaming with broker version 0.10 with this code. It's just a simple code to print the content of a topic. Not a big deal yet! But, for some reason memory is not enough(10GB of RAM in a VM)! The code:
# coding: utf-8
"""
kafka-test-003.py: test with broker 0.10(new Spark Stream API)
How to run this script?
spark-submit --jars jars/spark-sql-kafka-0-10_2.11-2.3.0.jar,jars/kafka-clients-0.11.0.0.jar kafka-test-003.py
"""
import pyspark
from pyspark import SparkContext
from pyspark.sql.session import SparkSession,Row
from pyspark.sql.types import *
from pyspark.sql.functions import *
from pyspark.streaming import StreamingContext
from pyspark.streaming.kafka import KafkaUtils
# starting spark session
spark = SparkSession.builder.appName("Kakfa-test").getOrCreate()
spark.sparkContext.setLogLevel('WARN')
# getting streaming context
sc = spark.sparkContext
ssc = StreamingContext(sc, 2) # batching duration: each 2 seconds
broker = "kafka.some.address:9092"
topic = "my.topic"
### Streaming
df = spark \
.readStream \
.format("kafka") \
.option("kafka.bootstrap.servers", broker) \
.option("startingOffsets", "earliest") \
.option("subscribe", topic) \
.load() \
.select(col('key').cast("string"),col('value').cast("string"))
query = df \
.selectExpr("CAST(key AS STRING)", "CAST(value AS STRING)") \
.writeStream \
.outputMode("append") \
.format("console") \
.start()
### End Streaming
query.awaitTermination()
Running spark submit:
spark-submit --master local[*] --driver-memory 5G --executor-memory 5G --jars jars/kafka-clients-0.11.0.0.jar,jars/spark-sql-kafka-0-10_2.11-2.3.0.jar kafka-test-003.py
Unfortunately, the result is:
java.lang.OutOfMemoryError: Java heap space
I'm assuming that Kafka should bring a little portions of data per time exactly to avoid this problem, right? So, what am I doing wrong ?

Answer 1

spark memory management is a complex process. The optimal solution depends not only on your data and type of operations and the system behavior
can you retry the following spark command:
spark-submit --master local[*] --driver-memory 4G --executor-memory 2G
--executor-cores 5 --num-executors 8 --jars jars/kafka-clients-0.11.0.0.jar,jars/spark-sql-kafka-0-10_2.11-2.3.0.jar
kafka-test-003.py
Can you adjust the above memory parameters as per the following link by tuning your performance? Using spark-submit, what is the behavior of the --total-executor-cores option?