Load PostgreSQL database to SchemaRDD - postgresql

Question

I have data source in PostgreSQL with 1 Million rows and 100+ columns, and I want to use Spark SQL so I want to transform this data source to get SchemaRDD.
Two approaches are introduced in Spark SQL Programming Guide,
one is through reflection, which means I need to define:
case class Row(Var1: Int, Var2: String, ...)
This is tedious because I have 100+ columns.
Another approach is "Programmatically Specifying the Schema", which means I need to define:
val schema =
StructType(
Seq(StructField("Var1", IntegerType), StructField("Var2", StringType), ...))
This is also tedious for me.
Actually, there's still another problem because I load my PostgreSQL database using JdbcRDD class but I found I also need to define the schema in the mapRow parameter of JdbcRDD constructor, which looks like:
def extractValues(r: ResultSet) = {
(r.getInt("Var1"), r.getString("Var2"), ...)
}
val dbRDD = new JdbcRDD(sc, createConnection,
"SELECT * FROM PostgreSQL OFFSET ? LIMIT ?",
0, 1000000, 1, extractValues)
This API still asks me to create the schema by myself, what's worse is that I need to redo the similar thing to transform this JdbcRDD to SchemaRDD, that would be really clumsy code.
So I want to know what's the best approach for this task?

Answer 1

There are only a limited number of data types that you need to support. Why not use the
java.sql.ResultSetMetaData
e.g.
val rs = jdbcStatement.executeQuery("select * from myTable limit 1")
val rmeta = rs.getMetaData
to read one row and then dynamically generate the required StructField for each of the columns.
You would need a case statement to handle
val myStructFields = for (cx <- 0 until rmeta.getColumnCount) {
val jdbcType = rmeta.getColumnType(cx)
} yield StructField(rmeta.getColumnName(cx),jdbcToSparkType(jdbcType))
val mySchema = StructType(myStructFields.toSeq)
Where jdbcToSparkType is along the following lines:
def jdbcToSparkType(jdbcType: Int) = {
jdbcType match {
case 4 => InteegerType
case 6 => FloatType
..
}
UPDATE To generate the RDD[Row] : you would follow a similar pattern. In this case you would
val rows = for (rs.next) {
row = jdbcToSpark(rs)
} yield row
val rowRDD = sc.parallelize(rows)
where
def jdbcToSpark(rs: ResultSet) = {
var rowSeq = Seq[Any]()
for (cx <- 0 to rs.getMetaData.getColumnCount) {
rs.getColumnType(cx) match {
case 4 => rowSeq :+ rs.getInt(cx)
..
}
}
Row.fromSeq(rowSeq)
}
then
val rows