Strange "Task not serializable" with Spark - scala

Question

In my program, I have a method which returns some RDD, let's call it myMethod which takes a non-serializable parameter and let the RDD be of the type Long (my real RDD is a Tuple type but only contains primitive types).
When I try something like this:
val x: NonSerializableThing = ...
val l: Long = ...
myMethod(x, l).map(res => res + l) // myMethod's RDD does NOT include the NonSerializableThing
I get Task not serializable.
When I replace res + l by res + 1L (i.e., some constant), it runs.
From the serialization trace, it tries to serialize the NonSerializableThing and chokes there, but I double-checked my method and this object never appears in an RDD.
When I try to collect output of myMethod directly, i.e. with
myMethod(x, l).take(1) foreach println
I also get no problems.
The method uses the NonSerializableThing to get a (local) Seq of values on which multiple Cassandra queries are made (this is needed because I need to construct the partition keys to query for), like this:
def myMethod(x: NonSerializableThing, l: Long): RDD[Long] = {
val someParam1: String = x.someProperty
x.getSomeSeq.flatMap(y: OtherNonSerializableThing => {
val someParam2: String = y.someOtherProperty
y.someOtherSeq.map(someParam3: String =>
sc.cassandraTable("fooKeyspace", "fooTable").
select("foo").
where("bar=? and quux=? and baz=? and l=?", someParam1, someParam2, someParam3, l).
map(_.getLong(0))
}.reduce((a, b) => a.union(b))
}
The getSomeSeq and someOtherSeq return plain non-spark Seqs
What I want to achieve is to "union" multiple Cassandra queries.
What could be the problem here?
EDIT, Addendum, as requested by Jem Tucker:
What I have in my class is something like this:
implicit class MySparkExtension(sc: SparkContext) {
def getThing(/* some parameters */): NonSerializableThing = { ... }
def myMethod(x: NonSerializableThing, l: Long): RDD[Long] = {
val someParam1: String = x.someProperty
x.getSomeSeq.flatMap(y: OtherNonSerializableThing => {
val someParam2: String = y.someOtherProperty
y.someOtherSeq.map(someParam3: String =>
sc.cassandraTable("fooKeyspace", "fooTable").
select("foo").
where("bar=? and quux=? and baz=? and l=?", someParam1, someParam2, someParam3, l).
map(_.getLong(0))
}.reduce((a, b) => a.union(b))
}
}
This is declared in a package object. The problem occurrs here:
// SparkContext is already declared as sc
import my.pkg.with.extension._
val thing = sc.getThing(/* parameters */)
val l = 42L
val rdd = sc.myMethod(thing, l)
// until now, everything is OK.
// The following still works:
rdd.take(5) foreach println
// The following causes the exception:
rdd.map(x => x >= l).take(5) foreach println
// While the following works:
rdd.map(x => x >= 42L).take(5) foreach println
I tested this entered "live" into a Spark shell as well as in an algorithm submitted via spark-submit.
What I now want to try (as per my last comment) is the following:
implicit class MySparkExtension(sc: SparkContext) {
def getThing(/* some parameters */): NonSerializableThing = { ... }
def myMethod(x: NonSerializableThing, l: Long): RDD[Long] = {
val param1 = x.someProperty
val partitionKeys =
x.getSomeSeq.flatMap(y => {
val param2 = y.someOtherProperty
y.someOtherSeq.map(param3 => (param1, param2, param3, l)
}
queryTheDatabase(partitionKeys)
}
private def queryTheDatabase(partitionKeys: Seq[(String, String, String, Long)]): RDD[Long] = {
partitionKeys.map(k =>
sc.cassandraTable("fooKeyspace", "fooTable").
select("foo").
where("bar=? and quux=? and baz=? and l=?", k._1, k._2, k._3, k._4).
map(_.getLong(0))
).reduce((a, b) => a.union(b))
}
}
I believe this could work because the RDD is constructed in the method queryTheDatabase now, where no NonSerializableThing exists.
Another option might be: The NonSerializableThing would indeed be serializable, but I pass in the SparkContext in there as an implicit constructor parameter. I think if I make this transient, it would (uselessly) get serialized but not cause any problems.

Answer 1

When you replace l with 1L Spark no longer tries to serialize the class with the method / variables in and so the error is not thrown.
You should be able to fix by marking val x: NonSerializableThing = ... as transient e.g.
#transient
val x: NonSerializableThing = ...
This means when the class is serialized this variable should be ignored.