Memory Errors with Iterative Spark DataFrame Creation - pyspark

Question

I am converting raw records that arrive to me in a single zlib compressed files into enriched parquet records for later processing in Spark. I don't control the zlib file, and I need the parquet consistent with other processing. I'm working in Pyspark and Spark 2.3. My approach works except when the zlib file is reasonably large (~300MB). It holds in memory fine, but Spark is running out of memory. If i shove my driver memory up (8g), it works. It feels like a memory leak from using function calls, as shown below.
The enrichment process explodes the data in Spark somewhat, so I'm using an iterative dataframe creation and decent size driver memory (4g default). I load all of the original decompressed file into memory and then pass a chunk of this to a simple routine to create a spark dataframe, add columns through the enrichment process, and then write to parquet. The chunks are set in size. The enrichment itself can vary, but overall the chunk size should not create too large of a dataframe in the driver, i believe. The output chunks, in parquet format, are around 100MB.
def process_file(gzfile, spark, chunk_size=2000000):
# load_data_and decompress
data = load_original_data(gzfile)
if len(data) == 0:
raise ValueError("No records loaded from file ", gzfile)
chunks = len(data) // chunk_size + 1
offset = 0
for chunk in range(chunks):
# convert the chunk into a spark dataframe
df = raw_to_spark(data[offset:offset+chunk_size], spark)
offset += chunk_size
# enrich the data while in a spark dataframe w/ more columns
df = extract_fields_from_raw(df)
save_to_parquet(df, parquet_output_path)
return
def raw_to_spark(events: List[str], spark: pyspark.sql.SparkSession) -> pyspark.sql.DataFrame:
"""
convert the list of raw strings into a spark dataframe so we can do all the processing. this list is large in
memory so we pop one list while building the new one. then throw the new list into spark.
"""
schema = StructType([StructField("event", StringType())])
rows = [] # make the list smaller as we create the row list for the dataframe
while events:
event = events.pop()
if event.count(",") >= 6: # make sure there are 7 fields at least
rows.append(pyspark.sql.Row(event))
rdd = spark.sparkContext.parallelize(rows, numSlices=2000) # we need to partition in order to pass to workers
return spark.createDataFrame(rdd, schema=schema)
def extract_fields_from_raw(df: pyspark.sql.DataFrame) -> pyspark.sql.DataFrame:
"""
this adds columns to the dataframe for enrichment before saving
"""
So, Spark dataframes of size <2M are being created as I loop through the decompressed data. Each of these dataframes should have no problem sitting in the 4g driver space. I get the same error, e.g., if I were to use a chunk of 1M. The Spark log shows that the out of memory error will have the same memory consumption before failure, e.g, 4.5gb of memory used.
I suspect what is happening is that memory isn't being released after each call to raw_to_spark, but I'm not sure how to show that... and I can't see logically how to push the Spark dataframe conversion into a function otherwise.
Am I missing a best practice? thanks.