Abort download if file too large in Scala - scala

Question

I am trying to download a file using the Scala APIs, but I would like it to abort if that file is too large (50MB).
I have managed to put together a very inefficient way that works for small files (< 10KB) but runs my CPU through the roof for large files:
var size = 0
val bytes = scala.io.Source.fromURL(url)(scala.io.Codec.ISO8859).toStream.map {
c =>
size = size + 1
if (size > (maxMbSize*1024*1024)) {
throw new Exception(s"File size is greater than the maximum allowed size of $maxMbSize MB")
}
c.toByte
}.toArray
I would like to be able to do this check more efficiently and also avoid the use of the var for the size. Is this possible?
Also I am using the play framework in case anyone knows of an API within that framework that may do what I am looking for.

Answer 1

You don't need to load the data into a byte array at all - you can generate the hash on-the-fly with a DigestInputStream using existing Java libraries. In this example I am loading the data from a String, but you can adapt to load from a URL. We use a tail-recursive function to eliminate the var, and return an Option so that we can indicate a over-size file by returning None.
import java.io._
import java.security._
import scala.annotation.tailrec
def calculateHash(algorithm: MessageDigest, in: String, limit: Int): Option[Array[Byte]] = {
val input = new ByteArrayInputStream(in.getBytes())
val dis = new DigestInputStream(input, algorithm)
#tailrec
def read(total: Int): Option[Array[Byte]] = {
if (total > limit) None
else {
val byte = dis.read()
if (byte == -1) Some(algorithm.digest())
else read(total + 1)
}
}
read(0)
}
Example usage:
val sha1 = MessageDigest.getInstance("SHA1")
calculateHash(sha1, "Hello", 5).get
//> res0: Array[Byte] = Array(-9, -1, -98, -117, 123, -78, -32, -101, 112, -109, 90, 93, 120, 94, 12, -59, -39, -48, -85, -16)
calculateHash(sha1, "Too long!!!", 5)
//> res1: Option[Array[Byte]] = None
You may also be able to get better performance by using the variant of DigestInputStream.read() that uses a buffer:
...
val buffer = new Array[Byte](1024)
#tailrec
def read(total: Int): Option[Array[Byte]] = {
if (total > limit) None
else {
val count = dis.read(buffer, 0, buffer.length)
if (count == -1) Some(algorithm.digest())
else read(total + count)
}
}
....

Answer 2

Since you are materializing all the data into memory, you don't really need to buffer it in small chunks (which is what the scala io library is doing). Also since you want the bytes anyway, you don't need to be decoding the bytes into Char's only to reverse the process.
To lose your size var, you can make use of the zipWithIndex function, which pairs each element with it's index. Do note that it starts at 0, so you need the + 1.
def readMyURL(url: String): Array[Byte] = {
val is = new java.net.URL(url).openStream()
val byteArray = Iterator.continually(is.read).zipWithIndex.takeWhile{
zb =>
if (zb._2 > (maxMbSize*1024*1024) + 1) {
throw new Exception(s"File size is greater than the maximum allowed size of $maxMbSize MB")
}
-1 != zb._1 // -1 is the end of stream
}.map(_._1.toByte).toArray
is.close()
byteArray
}
This is lazy, so the iterator will not traverse until you call toArray.
You may be able to get away with not closing that URL InputStream (it appears the scala io library does not do it).