pyspark.sql.utils.AnalysisException: 'writeStream' can be called only on streaming Dataset/DataFram - pyspark

Question

I am having glue streaming job, and I need to write the data as stream but after applying some processing, so I did the following:
data_frame_DataSource0 = glueContext.create_data_frame.from_catalog(
database=database_name,
table_name=kinesis_table_name,
transformation_ctx="DataSource0",
additional_options={"inferSchema": "true", "startingPosition": starting_position_of_kinesis_iterator}
)
glueContext.forEachBatch(
frame=data_frame_DataSource0,
batch_function=processBatch,
options={
"windowSize": window_size,
"checkpointLocation": s3_path_spark
}
)
and in processBatch I do some processing and at the end of it i do the following:
df.writeStream.format("hudi").options(**combinedConf).outputMode('append').start()
I am getting the following error:
pyspark.sql.utils.AnalysisException: 'writeStream' can be called only on streaming Dataset/DataFram
as far as I unserstand that the df I am trying to write is not streaming that's why it's giving the error, I am not so aware how can I change it from the glue context and how I can apply the processing on the streaming data then writeStream it?
any idea please?

Answer 1

forEachBatch method processes streaming Dataset/DataFrame on batches of Dataset/DataFrame, so when we call it on data_frame_DataSource0, the df passed to the method processBatch is a normal Dataset/DataFrame contains a batch of data.
You have two options to fix this:
deal with the df as normal dataframe:
df.write.format("hudi").options(**combinedConf).mode("append").save()
apply your stream processing directly on data_frame_DataSource0:
data_frame_DataSource0 = glueContext.create_data_frame.from_catalog(
database=database_name,
table_name=kinesis_table_name,
transformation_ctx="DataSource0",
additional_options={"inferSchema": "true", "startingPosition": starting_position_of_kinesis_iterator}
)
(
data_frame_DataSource0.writeStream.format("hudi")
.options(**combinedConf)
.option("inferSchema", "true")
.option("startingPosition", starting_position_of_kinesis_iterator)
.outputMode('append').start()
)