How to save dataframe as shp/geojson in PySpark/Databricks? - pyspark

Question

I have a DataFrame that has WKT in one of the columns. That column can be transformed to geojson if needed.
Is there a way to save (output to storage) this data as a geojson or shapefile in Databricks/PySpark?
Example of a DataFrame:
Id
Color
Wkt
1
Green
POINT (3 7)
2
Yellow
POLYGON ((30 10, 40 40, 20 40, 10 20, 30 10))
The DataFrame can have ~100K rows and more.
I've tried using Geopandas library, but it doesn't work:
import geopandas as gpd
# df is as PySpark dataframe
# Covert it to Pandas dataframe
pd_df = df.toPandas()
pd_df['geometry']=pd['point_wkt'].apply(wkt.loads)
# Convert it to GeoPandas dataframe
gdf = gpd.GeoDataFrame(pd, geometry='geometry')
# The following fails:
gdf.to_file(os.path.join(MOUNT_POINT,output_folder,"shapefile.shp"))
The error is:
Failed to create file /mnt/traces/output_folder/shapefile.shp: No such file or directory
The error makes no sense as the folder /mnt/traces/output_folder/ does exist, and I've successfully saved the PySpark dataframe as CSV to it.
df.write.csv(os.path.join(MOUNT_POINT,output_folder), sep='\t')
I'm able to save GeoPandas dataframe to shapefile with the above code when running locally, but not on Spark (Databricks).

Answer 1

If you are using Databricks then
dbutils.library.installPyPI("geopandas")
dbutils.library.installPyPI("shapely")
dbutils.library.installPyPI("geojsonio")
If you are using pyspark then it will be similar to Python Environment
pip3 install shapely
pip3 install geopandas
pip3 install geojsonio
Before writing to the path please check whether the path is mounted in databricks
display(dbutils.fs.ls('/mnt/traces'))

Answer 2

If you use dbutils.fs.ls("/mnt/traces/output_folder/") you'll see this path:
dbfs:/mnt/ traces/output_folder/shapefile.shp which takes us to our solution:
Solution: when writing we CAN use /dbfs/mnt/ for path instead of /mnt/lab/:
gdf.to_file("/dbfs/mnt/traces/output_folder/shapefile.shp")
Good luck!