Spark - Read only error while trying to write to Redshift - pyspark

Question

I'm trying to write data to Redshift using PySpark.
When I created the session, I could read files from Redshift and S3, which is what I wanted.
However I got error when trying to write back to Redshift.
Do you know what should I change in the script?
Here is how I defined the session:
spark = SparkSession \
.builder \
.config("spark.jars.packages", "org.apache.hadoop:hadoop-aws:2.7.0") \
.config("spark.driver.extraClassPath", ":".join(jars)) \
.config("spark.hadoop.fs.s3a.impl","org.apache.hadoop.fs.s3a.S3AFileSystem") \
.config("spark.hadoop.fs.s3a.awsAccessKeyId", os.environ['AWS_ACCESS_KEY_ID']) \
.config("spark.hadoop.fs.s3a.awsSecretAccessKey", os.environ['AWS_SECRET_ACCESS_KEY']) \
.config("spark.hadoop.fs.s3a.path.style.access", True)\
.config("com.amazonaws.services.s3.enableV4", True)\
.config("spark.executor.extraJavaOptions", "-Dcom.amazonaws.services.s3.enableV4=true")\
.config("spark.driver.extraJavaOptions", "-Dcom.amazonaws.services.s3.enableV4=true")\
.getOrCreate()
I could read from Redshift alright:
test = spark.read.format("jdbc").option("url","jdbc:redshift://host.redshift.amazonaws.com:5439/db").option("driver","com.amazon.redshift.jdbc42.Driver").option("dbtable","schema.table").option("user", DWH_DB_USER).option("password", DWH_DB_PASSWORD).load()
But got error while trying to run this:
test.write.format("jdbc").option("url","jdbc:redshift://host.redshift.amazonaws.com:5439/db").option("driver","com.amazon.redshift.jdbc42.Driver").option("dbtable","schema.table").option("user", DWH_DB_USER).option("password", DWH_DB_PASSWORD).mode('append').save()
The error I've got is:
Py4JJavaError: An error occurred while calling o412.save.
: org.apache.spark.SparkException: Job aborted due to stage failure: Task 0 in stage 1.0 failed 1 times, most recent failure: Lost task 0.0 in stage 1.0 (TID 1, localhost, executor driver): java.sql.SQLException: [Amazon](500310) Invalid operation: The session is read-only;
at com.amazon.redshift.client.messages.inbound.ErrorResponse.toErrorException(Unknown Source)
at com.amazon.redshift.client.PGMessagingContext.handleErrorResponse(Unknown Source)
at com.amazon.redshift.client.PGMessagingContext.handleMessage(Unknown Source)
at com.amazon.jdbc.communications.InboundMessagesPipeline.getNextMessageOfClass(Unknown Source)
at com.amazon.redshift.client.PGMessagingContext.doMoveToNextClass(Unknown Source)
at com.amazon.
......

Answer 1

This seems to be a bug where the JDBC driver sets your session as read-only seemingly at random.
My workaround is to add the following at the beginning of the query:
set ReadOnly = 0;