Pattern Matching for CSV processing - scala

Question

I have a csv file with three columns and I want to get the third column into an Iterator. I want to filter out the headers by using the trytoDouble method in combination of the Pattern Matching.
def trytoDouble(s: String): Option[Double] = {
try {
Some(s.toDouble)
} catch {
case e: Exception => None
}
}
val asdf = Source.fromFile("my.csv").getLines().map(_.split(",").map(_.trim).map(utils.trytoDouble(_))).map{
_ match {
case Array(a, b, Some(c: Double)) => c
}
}
results in
An exception or error caused a run to abort: [Lscala.Option;#2b4a2ec7 (of class [Lscala.Option;)
scala.MatchError: [Lscala.Option;#2b4a2ec7 (of class [Lscala.Option;)
what did I do wrong?

Answer 1

Try using an extractor like StringDouble below. If the unapply returns Some then it matches, if it returns None then it doesn't match.
object StringDouble {
def unapply(str: String): Option[Double] = Try(str.toDouble).toOption
}
val doubles =
Source.fromFile("my.csv").getLines().map { line =>
line.split(",").map(_.trim)
}.map {
case Array(_, _, StringDouble(d)) => d
}

Answer 2

It will always give scala.MatchError
scala> val url = "/home/knoldus/data/moviedataset.csv"
url: String = /home/knoldus/data/moviedataset.csv
scala> val asdf1 = Source.fromFile(url).getLines().map(_.split(",").map(_.trim).map(trytoDouble(_))).toList
asdf1: List[Array[Option[Double]]] = List(Array(Some(1.0), None, Some(1993.0), Some(3.9), Some(4568.0)), Array(Some(2.0), None, Some(1932.0), Some(3.5), Some(4388.0)), Array(Some(3.0), None, Some(1921.0), Some(3.2), Some(9062.0)), Array(Some(4.0), None, Some(1991.0), Some(2.8), Some(6150.0)), Array(Some(5.0), None, Some(1963.0), Some(2.8), Some(5126.0)), ....
As it will return non-empty iterator so, to see the result i have converted it toList.
If you notice return type is List[Array[Option[Double]]], and you are trying to match with Array of tuple3 but it always returns Array[Option[Double]].
Therefore, it will always throw error.

Answer 3

Try This:
val asdf = Array("1,2,3","1,b,c").map(_.split(",").map(_.trim).map(trytoDouble(_))).map{
_(2) match {
case x:Option[Double] => x
}
}

Answer 4

having only one match case is always breaking code.
See the REPL example,
scala> def trytoDouble(s: String): Option[Double] = {
| try {
| Some(s.toDouble)
| } catch {
| case e: Exception => None
| }
| }
when there's no Double in your CSV,
scala> List("a,b,c").map(_.split(",").map(_.trim).map(trytoDouble(_)))
res1: List[Array[Option[Double]]] = List(Array(None, None, None))
When you match above result (Array(None, None, None)) with Array(a, b, Some(c: Double)) always going to fail,
scala> List("a,b,c").map(_.split(",").map(_.trim).map(trytoDouble(_))).map { data =>
| data match {
| case Array(a, b, Some(c: Double)) => c
| }
| }
scala.MatchError: [Lscala.Option;#22604c7e (of class [Lscala.Option;)
at .$anonfun$res4$4(<console>:15)
at .$anonfun$res4$4$adapted(<console>:13)
at scala.collection.immutable.List.map(List.scala:283)
... 33 elided
And when there Double,
scala> List("a,b,100").map(_.split(",").map(_.trim).map(trytoDouble(_))).map { data => data match { case Array(a, b, Some(c: Double)) => c }}
res5: List[Double] = List(100.0)
You basically need to check for Some(c) or None.
BUT, if I understand you you are to extract third field as Double which can be done this way,
scala> List("a,b,100", "100, 200, p").map(_.split(",")).map { case Array(a, b, c) => trytoDouble(c)}.filter(_.nonEmpty)
res14: List[Option[Double]] = List(Some(100.0))

Answer 5

The other answers are great, but I just noticed that an alternative of minimal code change is to simply write
val asdf = Source.fromFile(fileName).getLines().map(_.split(",").map(_.trim).map( utils.trytoDouble(_))).flatMap{
_ match {
case Array(a, b, c) => c
}
}
or slightly more efficient (since we are only interested in the last column anyway):
val asdf = Source.fromFile(fileName).getLines().map(_.split(",")).flatMap{
_ match {
case Array(a, b, c) => trytoDouble(c.trim)
}
}
Important to notice is here that flatMap will remove None objects.