Catching MalformedInputException during scala.io.Source.fromFile - scala

Question

I'm using the scala.io.Source.fromFile method to read a csv file. Sometimes the file will be encoded in a different encoding format. I'll allow the user to specify the file enconding but...if the user doesn't specify the proper encoding I'd like to catch the MalformedInputException and then my method will return a None (instead of Some[Iterator[String]]).
I'm using the onCodingException method of the Codec but it seems that is not get applied. See below my code:
def readFileAsIterator(fileName: String,
encoding: Option[String] = Some(defaultEncoding)): Option[Iterator[String]] = {
try {
val codecType = encoding.getOrElse(defaultEncoding)
implicit val codec = Codec(codecType)
codec.onCodingException {
case e: CharacterCodingException => {
throw (new MalformedInputException(2))
}
}
val fileLines = io.Source.fromFile(fileName)(codec).getLines()
Some(fileLines)
} catch {
case e: Exception => {
None
}
}
}
Someone has played around with this method and managed to make it work?

Answer 1

This
io.Source.fromFile(fileName)(codec).getLines()
returns Iterator[String] which is lazy. So exception happens on iterating, not immediately on iterator creation.
Think, in general case it is not possible to detect wrong encoding without parsing before, so you need either parse file first to understand if encoding is right and than return new created iterator (not one used for parsing!), or leave exception handling to caller code, which parses data.
Or kind of trade-off, e.g. read several first lines, if ok (no coding exceptions) create new iterator for caller, but understand that in some cases caller will get exception on later wrong encoding line.
Update
Response to your comment to me under another answer.
Check this:
def readFileAsIterator(fileName: String,
encoding: Option[String] = Some("IBM1098"),
touchIterator: Boolean = false): Option[Iterator[String]] = {
try {
val codecType = encoding.getOrElse("IBM1098")
implicit val codec = Codec(codecType)
codec.onCodingException {
case e: CharacterCodingException => {
throw new MalformedInputException(2)
}
case e: java.nio.charset.UnmappableCharacterException => {
throw new MalformedInputException(3)
}
}
if (!touchIterator) {
Some(scala.io.Source.fromFile(fileName)(codec).getLines())
} else {
val i = scala.io.Source.fromFile(fileName)(codec).getLines()
if (i.hasNext) {
Some(i)
} else {
None
}
}
} catch {
case e: Exception => {
log.info(s"Handled exception in func", e)
None
}
}
}
Two calls on file which cause exception (in my case it was UnmappableCharacterException) with touching iterator and without depending on additional argument.
Under the hood you have iterator as I said. It is lazy buffered iterator. So it is initialized on first call (in modified method I force to initialize it with hasNext).
I do not think that it reads whole file, just buffer part of it (so it is automated implementation of my "trade-off case").

Answer 2

There are two things which you should think about modifying here,
1 - Return a Try[Iterator[String]] instead of Option[Iterator[String]]
2 - encoding can be a String with a default value.
def readFileAsIterator(fileName: String, encoding: String = "UTF-8"): Try[Iterator[String]] = Try({
implicit val codec = Codec(encoding)
codec.onCodingException({
case e: CharacterCodingException =>
throw (new MalformedInputException(2))
})
io.Source.fromFile(fileName)(codec).getLines()
})

Answer 3

Had same error. I handled it using onMalformedInput() as shown below:
implicit val codec = Codec("UTF-8")
codec.onMalformedInput(CodingErrorAction.REPLACE)
codec.onUnmappableCharacter(CodingErrorAction.REPLACE)
for(line <- Source.fromFile("..").getLines()) {
...
}