How to create shared JDBC connection to use on executors? - scala

Question

I have created spark jdbc singleton connection in driver & planning to use connection in executors. I get below exception. org.apache.spark.SparkException: Task not serializable
Inside spark main class:
object ExecutorConnection {
private var connection: Connection = null
val url = prop.getProperty("url")
val user = prop.getProperty("user")
val pwd = prop.getProperty("password")
val driver = prop.getProperty("driver")
Class.forName(driver)
def getConnection(url: String, username: String, password: String): Connection = synchronized {
if (connection == null) {
connection = DriverManager.getConnection(url, username, password)
Class.forName(driver)
connection.setAutoCommit(false)
}
connection
}
lazy val createConnection = getConnection(url, user, pwd)
}
I have multiple dataframes(df1,df2,df3) with different schema , where im planning to create connection in driver level & serialize the connection & use it for all dataframes.
df1.rdd.repartition(2).mapPartitions((d) => Iterator(d)).foreach { partition =>
val conn = ExecutorConnection.createConnection
var ps: PreparedStatement = null
partition.grouped(1).foreach(batch => {
batch.foreach { x =>
{
ps = conn.prepareStatement(SqlString)
ps.addBatch()
conn.commit()
}
}
})
}

Answer 1

Use Dataset.foreachPartition:
foreachPartition(f: (Iterator[T]) ⇒ Unit): Unit
Applies a function f to each partition of this Dataset.
This trick with Scala object is exactly how you get the connection once per task (and I think per executor also).
df1.foreachPartition { vs =>
// use the connection here
}
Use Guava for a cache.

Answer 2

Re:
where im planning to create connection in driver level & serialize the connection
It does not work that way.
You have to create connections on executor else you will keep getting this exception.