Changing an attribute in an object that belongs to RDD - scala

Question

I have the following code :
def generateStoriesnew(outputPath: String, groupedRDD:RDD[(String,Iterable[String])], isInChurnMode: Boolean, isInChurnPeriod: Boolean) {
val windowedRDD = groupedRDD.map(SOME CODE)
var windowedRDD2 = windowedRDD.filter(r => r != null).map(a=>a.churnPeriod(isInChurnPeriod,isInChurnMode))
val prettyStringRDD = windowedRDD2.map(r => {
r.toString
})
prettyStringRDD.saveAsTextFile(outputPath)
}
and here is the code for ChurnPriod function:
def churnPeriod( churnPeriod:Boolean, churnMode: Boolean): Unit = {
if (churnMode && rootEventType.equalsIgnoreCase("c")){
var churnCustStory: CustStoryN = null
var nonChurnCustStory: CustStoryN = null
var churnPeriodEventStory: mutable.MutableList[StoryEventN] = null
var NonChurnEventstory: mutable.MutableList[StoryEventN] = null
churnPeriodEventStory = new mutable.MutableList[StoryEventN]
NonChurnEventstory = new mutable.MutableList[StoryEventN]
var lastEventChurnPeriod = true
var currentEventStory = eventStory
var max = currentEventStory.length
println(max);
if (currentEventStory.size > 0) {
for (i <- 0 until currentEventStory.length) {
var currentEvent = currentEventStory(i)
if (currentEvent.timeSenseRootEvent < 90) {
churnPeriodEventStory.+=(currentEvent)
//lastEventChurnPeriod = true
}
else {
NonChurnEventstory.+=(currentEvent)
lastEventChurnPeriod = false
}
}
}
if (churnPeriod)
eventStory = churnPeriodEventStory
else
eventStory=null
}
}
but churn period function does not change eventstory which is a member of a custstory class. what am I missing here ?
class CustStoryN (val custId:String,
var rootEventType:String,
var rootEventTime:Long,
var eventStory:mutable.MutableList[StoryEventN])
my hypothesis is either:
1.map is not the right transformation for the function that I have
2.churnPeriod function never get called
3.I can not change eventstory which is a member of cust story class
Does anyone have any idea how I can troubleshoot this problem?

Answer 1

This would be trivial to determine by debugging. Just put a few breakpoints and you can see if the program stops inside the function, and what transformations do occur.
My guess would be that the problem is this line: if (currentEventStory.size > 0), thus, a list that starts at size 0 remains at size 0 forever. Another option is that churnPeriod is never true, thus, you compute a lot but never assign to the eventStory variable.
Your code does need a good cleanup ;-)