Multiple threads writing to same partition - pyspark

Question

Say that I have a AWS Glue job that looks like this:
import threading
def thread_worker(df, id):
df.write.mode('overwrite') \
.save('./output_{0}'.format(id))
def main():
...
threads = [threading.Thread(target=thread_worker, args=(df, id) \
for id in range(2)]
for t in threads:
t.start()
for t in threads:
t.join()
But instead of having 2 I have 60,000 and the output goes to a single partition in S3. so like:
import threading
def thread_worker(df, id):
df.write.mode('overwrite') \
.save('s3://bucket/partition_name=x/')
def main():
...
threads = [threading.Thread(target=thread_worker, args=(df, id) \
for id in range(60000)]
for t in threads:
t.start()
for t in threads:
t.join()
This will fail with different FileNotFound Java exceptions. Which is caused by what I have come to learn is the _temporary directory created in S3. Each thread needs to have its own if they all write to the same partition.
So, my questions are:
Can i pass an argument somewhere on df.write to use a custom name that is not _temporary?
We are talking about a lot of data here, so it's either threading or several hours for the data to load. Is there a way to safely implement threads here?

Answer 1

It seems that your FileNotFound Java exception happens because you are passing a partition name (folder-like structure) in 's3://bucket/partition_name=x/'
My suggestion would to use the id argument in the thread_worker function to create a filename convention like filename-{id}.ext, so your function would be modified like this:
def thread_worker(df, id):
df.write.mode('overwrite') \
.save('s3://bucket/partition_name/filename-{0}.ext'.format(id))
Optionally you could create a filename based on uuid, if you think there is a chance of the same name getting repeated.