scala insert to redis gives task not serializable - scala

Question

I have a following code :-
case class event(imei: String, date: String, gpsdt: String,
entrygpsdt: String,lastgpsdt: String)
val result = rdd.map(row => {
val imei = row.getString(0)
val date = row.getString(1)
val gpsdt = row.getString(2)
event(imei, date, gpsdt, lastgpsdt ,"2018-04-06 10:10:10")
}).collect()
val collection = sc.parallelize(result)
collection.saveToCassandra("db", "table", SomeColumns("imei", "date", "gpsdt", "lastgpsdt", "dt")
This works fine . So, now I'm inserting this result value into cassandra but I want to insert part of each rdd into Redis as well . When, I'm trying to use redis insert inside loop it gives an error that Task is not serializable
I want something like this :-
case class event(imei: String, date: String, gpsdt: String,
entrygpsdt: String,lastgpsdt: String)
val result = rdd.map(row => {
val imei = row.getString(0)
val date = row.getString(1)
val gpsdt = row.getString(2)
val zscore = Calendar.getInstance().getTimeInMillis
val value = row.getString(0) + ',' + row.getString(2)
val key = row.getString(1)
client.zadd(key , zscore, value)
event(imei, date, gpsdt, lastgpsdt ,"2018-04-06 10:10:10")
}).collect()
val collection = sc.parallelize(result)
collection.saveToCassandra("db", "table", SomeColumns("imei", "date", "gpsdt", "lastgpsdt", "dt")
So, How Can I do that , "client" is object of scala redis library.
Thanks,

Answer 1

Since no answer was provided by anyone. I found the solution for my case. Don't know whether the approach is good or not but it worked for me. So, idea is collect data by iterating over RDD . You'll be given a result of Array[event]. So, now again start a loop on result and insert each row in Redis. and finally "result" in cassandra. This flow is solving my both purpose that I was looking for.
Thanks,

Answer 2

The serializable exception is generally caused due to the connection object creation.
However your code does not include, I guess you have created the client object outside the foreachRDD
If so the client object is created in driver and foreach is executed in executor where it cannot find the client object and occurs exception task not serializable.
What you can do is create the client object inside foreach, But this creates a connection for each record, which is also not good for performance.
So what you can do is
rdd.foreachPartition(partition => {
//Create a connection here for redis
partition.foreach(record => {
//send the data here
})
})
Hope this helps!