Spark: Solrj exception on serialization - scala

Question

I am trying to import some csv file to Solr using Spark as "pipe", in this way:
trait SparkConnector {
def connectionWrapper[A](f: SparkContext => A): A = {
val sc: SparkContext = getSparkContext()
val res: A = f(sc)
sc.stop
res
}
}
object SolrIndexBuilder extends SparkConnector {
val solr = new ConcurrentUpdateSolrClient ("http://some-solr-url", 10000, 5)
def run(implicit fileSystem: FileSystem) = connectionWrapper[Unit] {
sparkContext =>
def build(): Unit = {
def toSolrDocument(person: Map[String, String], fieldNames: Array[String]): SolrInputDocument = {
val doc = new SolrInputDocument()
//add the values to the solr doc
doc
}
def rddToMapWithHeaderUnchecked(rdd: RDD[String], header: Array[String]): RDD[Map[String, String]] =
rdd.map { line =>
val splits = new CSVParser().parseLine(line)
header.zip(splits).toMap
}
val indexCsv = sparkContext.textFile("/somewhere/filename.csv")
val fieldNames = Vector("field1", "field2", "field3", ...)
val indexRows = rddToMapWithHeaderUnchecked(indexCsv, fieldNames)
indexRows.map(row => toSolrDocument(row, fieldNames)).foreach { doc => solr.add(doc) }
if (!indexRows.isEmpty()) {
solr.blockUntilFinished()
solr.commit()
}
}
build()
}
}
Running the code on a cluster (Spark-mode) I get the following error:
Caused by: java.io.NotSerializableException: org.apache.http.impl.client.SystemDefaultHttpClient
Serialization stack:
- object not serializable (class: org.apache.http.impl.client.SystemDefaultHttpClient, value: org.apache.http.impl.client.SystemDefaultHttpClient#3bdd79c7)
- field (class: org.apache.solr.client.solrj.impl.HttpSolrClient, name: httpClient, type: interface org.apache.http.client.HttpClient)
- object (class org.apache.solr.client.solrj.impl.HttpSolrClient, org.apache.solr.client.solrj.impl.HttpSolrClient#5d2c576a)
- field (class: org.apache.solr.client.solrj.impl.ConcurrentUpdateSolrClient, name: client, type: class org.apache.solr.client.solrj.impl.HttpSolrClient)
- object (class org.apache.solr.client.solrj.impl.ConcurrentUpdateSolrClient, org.apache.solr.client.solrj.impl.ConcurrentUpdateSolrClient#5da7593d)
- field (class: com.oneninetwo.andromeda.solr.SolrIndexBuilder$$anonfun$run$1$$anonfun$build$1$2, name: solr$1, type: class org.apache.solr.client.solrj.impl.ConcurrentUpdateSolrClient)
- object (class com.oneninetwo.andromeda.solr.SolrIndexBuilder$$anonfun$run$1$$anonfun$build$1$2, <function1>)
at org.apache.spark.serializer.SerializationDebugger$.improveException(SerializationDebugger.scala:40)
at org.apache.spark.serializer.JavaSerializationStream.writeObject(JavaSerializer.scala:47)
at org.apache.spark.serializer.JavaSerializerInstance.serialize(JavaSerializer.scala:101)
at org.apache.spark.util.ClosureCleaner$.ensureSerializable(ClosureCleaner.scala:301)
... 26 more
which clearly states that I cannot serialize classes related to Solrj (and relative dependencies).
I tried different thing but I am cannot find a way to make it work. Any help?
UPDATE
This is one solution I have found:
indexRows.map(row => toSolrDocument(row, fieldNames)).foreachPartition{ partition =>
val solr = new ConcurrentUpdateSolrClient(...)
partition.foreach { doc =>
solr.add(doc)
}
solr.commit()