Write dataframe to SQL dedicated database using Synapse Analytics - pyspark

Question

I would like to load a dataframe from my Azure Data Lake Storage Gen2 and write it to an SQL
dedicated database that I created in Synapse.
This is what I did:
df = spark.read.format("delta").load(BronzePath)
df.write.format("com.databricks.spark.sqldw").option("url", jdbcUrl).save()
And I have the following error:
java.lang.ClassNotFoundException: Failed to find data source: com.databricks.spark.sqldw.
Doing:
df.write.mode("overwrite").saveAsTable("MyTable")
creates the table in Spark default database (blue cross). That is not what I need. I want to have my table in the dedicated database (blue arrow):

Answer 1

Post more of your code including jdbc url, if it's different than this guide. I don't see the code to set the storage key in conf and you seem to be using different way to save as well.
# Otherwise, set up the Blob storage account access key in the notebook session conf.
spark.conf.set(
"fs.azure.account.key.<your-storage-account-name>.blob.core.windows.net",
"<your-storage-account-access-key>")
# Get some data from an Azure Synapse table.
df = spark.read \
.format("com.databricks.spark.sqldw") \
.option("url", "jdbc:sqlserver://<the-rest-of-the-connection-string>") \
.option("tempDir", "wasbs://<your-container-name>#<your-storage-account-name>.blob.core.windows.net/<your-directory-name>") \
.option("forwardSparkAzureStorageCredentials", "true") \
.option("dbTable", "<your-table-name>") \
.load()
# Load data from an Azure Synapse query.
df = spark.read \
.format("com.databricks.spark.sqldw") \
.option("url", "jdbc:sqlserver://<the-rest-of-the-connection-string>") \
.option("tempDir", "wasbs://<your-container-name>#<your-storage-account-name>.blob.core.windows.net/<your-directory-name>") \
.option("forwardSparkAzureStorageCredentials", "true") \
.option("query", "select x, count(*) as cnt from table group by x") \
.load()
# Apply some transformations to the data, then use the
# Data Source API to write the data back to another table in Azure Synapse.
df.write \
.format("com.databricks.spark.sqldw") \
.option("url", "jdbc:sqlserver://<the-rest-of-the-connection-string>") \
.option("forwardSparkAzureStorageCredentials", "true") \
.option("dbTable", "<your-table-name>") \
.option("tempDir", "wasbs://<your-container-name>#<your-storage-account-name>.blob.core.windows.net/<your-directory-name>") \
.save()
Also read
Supported save modes for batch writes and
Write semantics
in FAQ.
creates the table in Spark default database (blue cross). That is not what I need. I want to have my table in the dedicated database (blue arrow):
As described here "Spark will create a default local Hive metastore (using Derby) for you."
So when you don't give it a path/jdbcurl (df.write.mode("overwrite").saveAsTable("MyTable")), it'll save to local Hive.