JDBC dialect when writing Spark to Hive - scala

Question

I am trying to write a dataframe from Spark 2.2.0 to Hive 2.1 through a JDBC connection. I know this is not the recommended way and that a direct connection using hive-site.xml should be configured but that is not an option for me currently due to factors outside of my control...so I'm stuck with JDBC at the moment.
I can read from Hive using JDBC but have to override the JDBC dialect quoteIdentifier method and specify the fetchsize to actually see any output from the dataframe in Spark.
Although inconvenient, this is okay with me for now. However, I am having trouble writing back to Hive now. I think I need to make additional changes to the JDBC dialect to write back as I'm getting this error message:
Caused by: org.apache.hive.service.cli.HiveSQLException: Error while
compiling statement: FAILED: ParseException line 1:28 cannot recognize input
near '.' 'name' 'TEXT' in column type
This is my process for reading via JDBC:
1.) Created db-properties.flat file with my username, pass, url, and driver.
url=jdbc:hive2://xxxx.com:10000/default
driver=org.apache.hive.jdbc.HiveDriver
user=xxxxxx
password=xxxxxx
2.) Open Spark Shell and run the below code to read table:
import java.io.Fileimport java.util.Properties
import java.io.FileInputStream
import org.apache.spark.{SparkConf, SparkContext}
import org.apache.spark.sql.{SQLContext, DataFrame}
import org.apache.spark.sql.hive.HiveContext
import org.apache.spark.sql.jdbc.{JdbcDialects, JdbcType, JdbcDialect}
val conf = new SparkConf()
val sqlContext = new HiveContext(sc)
val dbProperties = new Properties
dbProperties.load(new FileInputStream(new File("/home/xxxxxx/db-properties.flat")))
val url = dbProperties.getProperty("url")
val jdbcDriver = dbProperties.getProperty("driver")
val jdbcFetchsize = dbProperties.setProperty("fetchsize","10")
/* Update JDBC dialect */
val HiveDialect = new JdbcDialect {
override def canHandle(url: String): Boolean = url.startsWith("jdbc:hive2") || url.contains("hive2")
override def quoteIdentifier(colName: String): String ={ s"$colName" }
}
JdbcDialects.registerDialect(HiveDialect)
val myTable = "xxxxxx"
val df = spark.read.jdbc(url,myTable,dbProperties)
df.show()
3.) I am able to read the data without issues in step 2, but I cannot write to Hive from Spark using JDBC. Below is the code:
df.write.mode("error").jdbc(url,"newtable",dbProperties)
...which results in this error:
Caused by: org.apache.hive.service.cli.HiveSQLException: Error while
compiling statement: FAILED: ParseException line 1:28 cannot recognize input
near '.' 'name' 'TEXT' in column type
Does anyone have recommendations as to how I can modify the dialect to write back to Hive from Spark JDBC or any other suggestions? Thank you!