Count occurrences of word in file - scala

Question

Below code attempts to count the number of times "Apple" appears in an HTML file.
object Question extends App {
def validWords(fileSentancesPart: List[String], wordList: List[String]): List[Option[String]] =
fileSentancesPart.map(sentancePart => {
if (isWordContained(wordList, sentancePart)) {
Some(sentancePart)
} else {
None
}
})
def isWordContained(wordList: List[String], sentancePart: String): Boolean = {
for (word <- wordList) {
if (sentancePart.contains(word)) {
return true;
}
}
false
}
lazy val lines = scala.io.Source.fromFile("c:\\data\\myfile.txt" , "latin1").getLines.toList.map(m => m.toUpperCase.split(" ")).flatten
val vw = validWords(lines, List("APPLE")) .flatten.size
println("size is "+vw)
}
The count is 79 as per the Scala code. But when I open the file with a text editor it finds 81 words with "Apple" contained. The search is case insensitive. Can spot where the bug is ? (I'm assuming the bug is with my code and not the text editor!)
I've wrote a couple of tests but the code seems to behave as expected in these simple use cases :
import scala.collection.mutable.Stack;
import org.scalatest.FlatSpec;
import org.scalatest._;
class ConvertTes extends FlatSpec {
"Valid words" should "be returned" in {
val fileWords = List("this" , "is" , "apple" , "applehere")
val validWords = List("apple")
lazy val lines = scala.io.Source.fromFile("c:\\data\\myfile.txt" , "latin1").getLines.toList.map(m => m.toUpperCase.split(" ")).flatten
val l : List[String] = validWords(fileWords, validWords).flatten
l.foreach(println)
}
"Entire line " should "be returned for matched word" in {
val fileWords = List("this" , "is" , "this apple is an" , "applehere")
val validWords = List("apple")
val l : List[String] = validWords(fileWords, validWords).flatten
l.foreach(println)
}
}
The HTML file being parsed (referred to as "c:\data\myfile.txt") in code above :
https://drive.google.com/file/d/0B1TIppVWd0LSVG9Edl9OYzh4Q1U/view?usp=sharing
Any suggestions on alternatives to code above welcome.
Think my issue is as per #Jack Leow comment. For code :
val fileWords = List("this", "is", "this appleisapple an", "applehere")
val validWords = List("apple")
val l: List[String] = validWords(fileWords, validWords).flatten
println("size : " + l.size)
size printed is 2, when it should be 3

Answer 1

I think you should do the following:
def validWords(
fileSentancesPart: List[String],
wordList: List[String]): List[Option[String]] =
fileSentancesPart /* add flatMap */ .flatMap(_.tails)
.map(sentancePart => {
if (isWordContained(wordList, sentancePart)) {
Some(sentancePart)
} else {
None
}
})
def isWordContained(
wordList: List[String],
sentancePart: String): Boolean = {
for (word <- wordList) {
//if (sentancePart.contains(word)) {
if (sentancePart.startsWith(word)) { // use startsWith
return true;
}
}
false
}

Answer 2

You could use regular expressions with a Source iterator:
val regex = "([Aa]pple)".r
val count = Source.fromFile("/test.txt").getLines.map(regex.findAllIn(_).length).sum