Formatting read-in CSV data - scala

Question

I'm trying to read in a CSV and store it in a matrix-like array of arrays object. A roadblock I am hitting is that the strings are read in with the surrouding quotes - that is to say, the string "price" is not just the word price, but """"price"""" in scala. Consequently, I want to remove those surrounding quotes. I also want to make sure that any numeric values are coerced to Double/Int, as they are read in as strings.
What I have now:
val rawObs = io.Source.fromFile(file).getLines() .map(_.split(",")).toArray
// An example element of the array is:
//scala> rawObs(2)
//res93: Array[String] = Array("3", 0, "2013-02-27", 1, 52, 52, 1, "1", "kg")
// Here, I make a function to remove surrounding strings and return a string, or if there are
// not surrounding strings, return a Double.
def fixRawObs(x: String) = {
// if it was actually meant to be a string:
if(x.startsWith(""""""")){
// delete any " quotes
var y = x.replaceAll(""""""", "")
} else { // this means that x needs to be coerced to Int or Double
var y = x.toDouble
}
y // return y
}
// but this won't compile, it returns <console>:14: error: not found: value y
// If it did compile, I'd want to do something like this:
rawObs.map(_.map(fixRawObs(_)))
// (although, is there a better way?)
So, basically, my first question is how to fix my fixRawObs function, and secondarily, is this even an okay way to do this or is there some nicer way to accomplish what I want? What I'm doing feels kind of hackish.
I'm super super new to Scala so it would be greatly appreciated if answers didn't assume much knowledge. Thank you!

Answer 1

There are a few problems with your code:
You are trying to store Strings and Doubles in an Array. Since the closest common super type of Strings and Doubles is Any, you will have an Array[Any]. With an Array[Any], you will need to cast the values inside as Strings or Doubles whenever you want to use them, and this is not desirable.
Your function fixRawObs() is not compiling because it is trying to return an inaccessible variable. "y" is declared inside of curly braces, which makes it inaccessible outside of the curly braces. "y" actually is not even necessary, because an if statement in Scala returns a value, just like a function can. You could do this:
def fixRawObs(x: String) = {
if(x.startsWith(""""""")) x.replaceAll(""""""", "")
else x.toDouble
}
The return type of this function is "Any", though, so you would still have to cast the return values to the proper types manually. Again, this is not a good approach.
I recommend creating a class, so that you can have a custom data structure that references your values using their proper types.
case class Row(
col1: String, col2: Double, col3: String, col4: Double,
col5: Double, col6: Double, col7: Double, col8: String, col9: String
)
It would be best if you rename the values with appropriate, descriptive names.
You can then create your row objects like this:
def stripQuotes(s: String): String = {
if(s.startsWith("\"") && s.endsWith("\"")) s.dropRight(1).dropLeft(1)
else s
}
val csv = io.Source.fromFile(file)
val rows = (for {
line <- file.getLines
s = line.split(",")
if(s.size == 9)
} yield {
new Row(
stripQuotes(s(0)),
s(1).toDouble,
stripQuotes(s(2)),
s(3).toDouble,
s(4).toDouble,
s(5).toDouble,
s(6).toDouble,
stripQuotes(s(7)),
stripQuotes(s(8))
)
}).toArray
csv.close()

Answer 2

You probably want to use a library that parses CSV files instead of trying to get through the edge cases by yourself. There are many options for Scala/Java (one two).
If you're practicing Scala, I'll explain why it won't compile. The issue is that you're trying to return y, which is defined in the scope of your loop and isn't available outside of it.
In Scala, the last statement of the function is the return value. So making your if statement the last one in the function, and returning the replaced/parsed value right away will do what you want.
def fixRawObs(x: String) = {
x.startsWith("\"") match {
case true =>
x.replaceAll("\"", "")
case false =>
x.toDouble
}
}
Note that the function will return an instance of Any - the superclass of all classes in Scala. This is because you are returning a String in one clause and a Double in another.
Knowing the specific format of your data (e.g. is a given field always a double or is it always a string), you can rewrite it to be more precise and support the actual types.

Answer 3

This simple library: product-collections does something similar to what you are trying to do with your array of arrays. It also has an intuitive csv reader.
Update: the library now handles case classes directly:
val csv = Array("3", 0, "2013-02-27", 1, 52, 52, 1, "1", "kg").mkString(",")
csv: String = 3,0,2013-02-27,1,52,52,1,1,kg
scala> case class Row(
| col1: String, col2: Double, col3: String, col4: Double,
| col5: Double, col6: Double, col7: Double, col8: String, col9: String
| )
defined class Row
scala> CsvParser(Row).parse(new java.io.StringReader(csv))
res33: Seq[Row] = List(Row(3,0.0,2013-02-27,1.0,52.0,52.0,1.0,1,kg))