Transform AWS S3 Object into ByteString issue - scala

Question

I'm working on a process of upload files from S3 to Facebook using Akka. According to Facebook API docs, files should be uploaded via small parts - chunks. Based on a file size, Facebook gives you an information about bytes offset which it expects to receive in a next request.
Firstly I make a GetObjectRequest to S3 via Java AWS SDK, in order to receive a chunk with a required bytes size:
val objChunkReq = new GetObjectRequest(get.s3ObjId.bucketName, get.s3ObjId.key)
objChunkReq.setRange(get.fbUploadSession.from, get.fbUploadSession.to)
Try(s3Client.getObject(objChunkReq)) match {
case Success(s3ObjChunk) => Right(S3ObjChunk(s3ObjChunk, get.fbUploadSession))
case Failure(ex) => Left(S3Exception(ex.getMessage))
}
Then in case if the S3 response is successful, I can work with the received chunk as with an InputStream in order to pass it then into Facebook HTTP request:
private def inputStreamToArrayByte(is: InputStream) = {
Try {
val reads: Int = is.read()
val byteStringBuilder = ByteString.newBuilder
while (is.read() != -1) {
byteStringBuilder.asOutputStream.write(reads)
is.read()
}
is.close()
byteStringBuilder.result()
}
}
The issue I faced is that size of s3ObjChunk from the first code snippet has twice bigger size in bytes than the resulting ByteString from the second one code snippet.
s3ObjChunk.getObjectMetadata.getContentLength == n
byteStringBuilder.result().length == n / 2
I have two assumptions:
a) I transform the InputStream into ByteString incorrectly
b) The ByteString compresses the InputStream
How to transform an S3 object InputStream into a ByteString correctly?

Answer 1

The issue with n vs n / 2 in the resulting output may be explained by a bug in the implementation.
is.read() is called twice in the loop, and none of its returns is indeed written into the output stream, but only the first one, stored in val reads.
The implementation should change to something like:
val byteStringBuilder = ByteString.newBuilder
val output = byteStringBuilder.asOutputStream
try {
var reads: Int = is.read() // note "var" instead of "val"
while (reads != -1) {
output.write(reads)
reads = is.read()
}
} finally {
is.close() // should it be here or closed by the caller?
// also close "output"
}
byteStringBuilder.result()
Or, another approach would be to use slightly more idiomatic stream reading with scala.io.Source, for example:
val byteStringBuilder = ByteString.newBuilder
val output = byteStringBuilder.asOutputStream
scala.io.Source.fromInputStream(is).foreach(output.write(_))
byteStringBuilder.result()