Why isn't Spark textFile method reading the whole text file? - scala

Question

class Cleaner {
def getDocumentData() = {
val conf = new SparkConf()
.setAppName("linkin_spark")
.setMaster("local[2]")
.set("spark.executor.memory", "1g")
.set("spark.rdd.compress", "true")
.set("spark.storage.memoryFraction", "1")
val CorpusReader = new Corpus()
val files = CorpusReader.getListOfFiles("/home/DATA/doc_collection/")
val sc = new SparkContext(conf)
val temp = sc.textFile(files(0).toString())
println(files(0).toString())
var count = 0
val regex = """<TAG>""".r
for (line <- temp ) {
line match {
case regex(_*) => {
println(line)
count += 1
println(count)
}
case _ => null //Handle error - scala.MatchError
}
}
println(s"There are " + count + " documents.") // this comes out to be 0
}
}
I have a list of text file that I have to read. They are XML-like files so, I need to extract the relevant text. Since, they are not standard XML files I thought of using regex to get the text. Every document start with a <TAG> tag. So I tried to count no. of documents in a file which will be equal to no. of <TAG> matches in the file. The function above does the same thing. Originally the file has 264 docs but when I run the function I get 127 or 137, either of the numbers. It does not seem to reading the whole file. Also count at the end comes out to be 0.
I am Scala/Spark newbie.
UPDATE:
var count = sc.accumulator(0)
val regex = """<TAG>""".r
for (line <- temp ) {
println(line)
line match {
case regex(_*) => {
count += 1
println(s"$line # $count") //There is no "<TAG> # 264" in the output
}
case _ => null
}
}
println(s"There are " + count.value + " documents.")
This change in the program gives me the correct value of count i.e. 264 but the file id printed correctly!! It appears to start somewhere from the middle and end somewhere in the middle.
UPDATE II:
This has something to do with threads. The SparkConf() has been initialised using local[2] which means 2 threads, if I am not wrong. As soon as I changed it to local[1] I got the correct answer but I cannot use one thread.
File appears like this:
<TAG>
<DOCNO> AP890825-0001 </DOCNO>
<FILEID>AP-NR-08-25-89 0134EDT</FILEID>
<TEXT>
Some large text.
</TEXT>
</TAG>
<TAG> // new doc started
How should I correct this issue?

Answer 1

This is a closure problem. Each node gets its own version of the count variable. You want to use accumulators or simply perform a reduce
Created by:
val tagCounter = sc.accumulator(0, "tagCount")
Updated by: (not readable on the nodes)
tagCounter += 1
Readable on the driver by:
tagCounter.value
Following your update:
var count = sc.accumulator(0)
val regex = """<TAG>""".r
val output = for (line <- temp ) yield {
//println(line)
line match {
case regex(_*) => {
count += 1
//println(s"$line # $count") //There is no "<TAG> # 264" in the output
line
}
case _ => "ERROR"
}
}
println(s"output size:{output.count}")
println(s"There are " + count.value + " documents.")
UPDATE AFTER SEEING THE INPUT FORMAT:
You may end up at the whim of using wholeTextFiles to guarantee ordering. Otherwise the distributed nature means that ordering is often not guaranteed, but if you can guarantee ordering (possibly custom partitioner or custom InputFormat), then something like this should work:
sc.parallelize(list)
.aggregate(Nil : List[String])((accum, value) => {
value match {
case regex(_*) => accum :+ value
case _ => {
accum match {
case Nil => List(value)
case _ => accum.init ++ (accum.tail :+ value)
}
}
}, _ ++ _)