Getting "File ready exists" error in AWS Glue when writing a dynamic frame to Redshift - amazon-redshift

Question

I have a system with over 20 ETL jobs. I'm adding a new job which is similar in concept to other jobs, so I copied and modified the code. At the end, I'm writing a dynamic frame to a Redshift table. The frame has about 130,000 records. Here is my error
2020-07-14 00:06:11,563 ERROR [Thread-9] datasources.FileFormatWriter (Logging.scala:logError(91)) - Aborting job 1e89bbc9-ffe6-41a9-9bbc-f015cf6da2a5.
org.apache.spark.SparkException: Job aborted due to stage failure: Task 114 in stage 55.0 failed 4 times, most recent failure: Lost task 114.3 in stage 55.0 (TID 2631, ip-10-204-19-115.us-west-2.compute.internal, executor 4): org.apache.hadoop.fs.FileAlreadyExistsException: File already exists:s3://c2l-dwh-temp/dwh_dev_file_download_fact/3c9d8ef3-c831-4642-9e54-2d2c6f24e155/part-00114-5aacdd19-3f62-4ee8-bf53-ab17d117dd0f-c000.csv
at com.amazon.ws.emr.hadoop.fs.s3.upload.plan.RegularUploadPlanner.checkExistenceIfNotOverwriting(RegularUploadPlanner.java:36)
at com.amazon.ws.emr.hadoop.fs.s3.upload.plan.RegularUploadPlanner.plan(RegularUploadPlanner.java:30)
at com.amazon.ws.emr.hadoop.fs.s3.upload.plan.UploadPlannerChain.plan(UploadPlannerChain.java:37)
at com.amazon.ws.emr.hadoop.fs.s3n.S3NativeFileSystem.create(S3NativeFileSystem.java:601)
at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:932)
at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:913)
at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:810)
at com.amazon.ws.emr.hadoop.fs.EmrFileSystem.create(EmrFileSystem.java:212)
at org.apache.spark.sql.execution.datasources.CodecStreams$.createOutputStream(CodecStreams.scala:81)
at org.apache.spark.sql.execution.datasources.CodecStreams$.createOutputStreamWriter(CodecStreams.scala:92)
at org.apache.spark.sql.execution.datasources.csv.CsvOutputWriter.<init>(CSVFileFormat.scala:177)
at org.apache.spark.sql.execution.datasources.csv.CSVFileFormat$$anon$1.newInstance(CSVFileFormat.scala:85)
at org.apache.spark.sql.execution.datasources.SingleDirectoryDataWriter.newOutputWriter(FileFormatDataWriter.scala:120)
at org.apache.spark.sql.execution.datasources.SingleDirectoryDataWriter.<init>(FileFormatDataWriter.scala:108)
at org.apache.spark.sql.execution.datasources.FileFormatWriter$.org$apache$spark$sql$execution$datasources$FileFormatWriter$$executeTask(FileFormatWriter.scala:236)
at org.apache.spark.sql.execution.datasources.FileFormatWriter$$anonfun$write$1.apply(FileFormatWriter.scala:170)
at org.apache.spark.sql.execution.datasources.FileFormatWriter$$anonfun$write$1.apply(FileFormatWriter.scala:169)
at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:90)
at org.apache.spark.scheduler.Task.run(Task.scala:121)
at org.apache.spark.executor.Executor$TaskRunner$$anonfun$10.apply(Executor.scala:408)
at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1360)
at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:414)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
The temp directory was empty at the start of the Glue job. Now it has a bunch of folders, some with lots of CSV files. I've never seen this error with my other similar jobs.
Any pointers?
Thanks

Answer 1

Found the problem. Further down in the traceback was a reference to my code, where I had a index error. The PySpark thread kept retrying, and that is why it got the "file already exists" error. The retry would not work because I had a coding error in my user-defined function.