I have a method save that takes an Iteratee it and saves some data to it. Inside the method, the data is available as an enumerator producing byte-array chunks.
def save[E](consumer: Iteratee[Array[Byte], E]): Future[E] = {
val producer: Enumerator[Array[Byte]] = // ...
Iteratee.flatten(producer(consumer)).run
}
Wanted: Call save in order to have it write the data to a FileOutputStream.
I tried the following but am not sure whether this is the way to go:
def writeToStream(s: OutputStream) =
Iteratee.foreach((e: Array[Byte]) => s.write(e)).
mapDone(r => { s.close(); r })
save(writeToStream(new FileOutputStream(myFile)))
Question: Is this the way it's supposed to be done? I fear that this will not always close the stream (case of exceptions).
I am using the Play Framework Iteratee library from Play Framework 2.1 (which uses Scala futures).
The scaladocs for Iteratee say that it is in the responsibility of the "producer" not the iteratee to handle resources:
The Iteratee does not do any resource management (such as closing streams); the producer pushing stuff into the Iteratee has that responsibility.
You might be successful using the "onDoneEnumerating" method in Enumerator to clean up resources afterwards.
Scaladoc Iteratee
Related
I want to stream a file from s3 to actor to be parsed and enriched and to write the output to other file.
The number of parserActors should be limited e.g
application.conf
akka{
actor{
deployment {
HereClient/router1 {
router = round-robin-pool
nr-of-instances = 28
}
}
}
}
code
val writerActor = actorSystem.actorOf(WriterActor.props())
val parser = actorSystem.actorOf(FromConfig.props(ParsingActor.props(writerActor)), "router1")
however the actor that is writing to a file should be limited to 1 (singleton)
I tried doing something like
val reader: ParquetReader[GenericRecord] = AvroParquetReader.builder[GenericRecord](file).withConf(conf).build()
val source: Source[GenericRecord, NotUsed] = AvroParquetSource(reader)
source.map (record => record ! parser)
but I am not sure that the backpressure is handled correctly. any advice ?
Indeed your solution is disregarding backpressure.
The correct way to have a stream interact with an actor while maintaining backpressure is to use the ask pattern support of akka-stream (reference).
From my understanding of your example you have 2 separate actor interaction points:
send records to the parsing actors (via a router)
send parsed records to the singleton write actor
What I would do is something similar to the following:
val writerActor = actorSystem.actorOf(WriterActor.props())
val parserActor = actorSystem.actorOf(FromConfig.props(ParsingActor.props(writerActor)), "router1")
val reader: ParquetReader[GenericRecord] = AvroParquetReader.builder[GenericRecord](file).withConf(conf).build()
val source: Source[GenericRecord, NotUsed] = AvroParquetSource(reader)
source.ask[ParsedRecord](28)(parserActor)
.ask[WriteAck](writerActor)
.runWith(Sink.ignore)
The idea is that you send all the GenericRecord elements to the parserActor which will reply with a ParsedRecord. Here as an example we specify a parallelism of 28 since that's the number of instances you have configured, however as long as you use a value higher than the actual number of actor instances no actor should suffer from work starvation.
Once the parseActor replies with the parsing result (here represented by the ParsedRecord) we apply the same pattern to interact with the singleton writer actor. Note that here we don't specify the parallelism as we have a single instance so it doesn't make sense the send more than 1 message at a time (in reality this happens anyway due to buffering at async boundaries, but this is just a built-in optimization). In this case we expect that the writer actor replies with a WriteAck to inform us that the writing has been successful and we can send the next element.
Using this method you are maintaining backpressure throughout your whole stream.
I think you should be using one of the "async" operations
Perhaps this other q/a gives you some insperation Processing an akka stream asynchronously and writing to a file sink
Are there some code examples of using org.reactivestreams libraries to process large data streams using Java NIO (for high performance)? I'm aiming at distributed processing, so examples using Akka would be best, but I can figure that out.
It still seems to be the case that most (I hope not all) examples of reading files in scala resort to Source (non-binary) or direct Java NIO (and even things like Files.readAllBytes!)
Perhaps there is an activator template I've missed? (Akka Streams with Scala! is close addressing everything I need except the binary/NIO side)
Do not use scala.collection.immutable.Stream to consume files like this, the reason being that it performs memoization - that is, while yes it is lazy it will keep the entire stream buffered (memoized) in memory!
This is definitely not what you want when you think about "stream processing a file". The reason Scala's Stream works like this is because in a functional setting it makes complete sense - you can avoid calculating fibbonachi numbers again and again easily thanks to this for example, for more details see the ScalaDoc.
Akka Streams provides Reactive Streams implementations and provides a FileIO class that you could use here (it will properly back-pressure and pull the data out of the file only when needed and the rest of the stream is ready to consume it):
import java.io._
import akka.actor.ActorSystem
import akka.stream.scaladsl.{ Sink, Source }
object ExampleApp extends App {
implicit val sys = ActorSystem()
implicit val mat = FlowMaterializer()
FileIO.fromPath(Paths.get("/example/file.txt"))
.map(c ⇒ { print(c); c })
.runWith(Sink.onComplete(_ ⇒ { f.close(); sys.shutdown() } ))
}
Here are more docs about working with IO with Akka Streams
Note that this is for the current-as-of writing version of Akka, so the 2.5.x series.
Hope this helps!
We actually use akka streams to process binary files. It was a little tricky to get things going as there wasn't any documentation around this, but this is what we came up with:
val binFile = new File(filePath)
val inputStream = new BufferedInputStream(new FileInputStream(binFile))
val binStream = Stream.continually(inputStream.read).takeWhile(-1 != _).map(_.toByte)
val binSource = Source(binStream)
Once you have binSource, which is an akka Source[Byte] you can go ahead and start applying whatever stream transformations (map, flatMap, transform, etc...) you want to it. This functionality leverages the Source companion object's apply that takes an Iterable, passing in a scala Stream that should read in the data lazily and make it available to your transforms.
EDIT
As Konrad pointed out in the comments section, a Stream can be an issue with large files due to the fact that it performs memoization of the elements it encounters as it's lazily building out the stream. This can lead to out of memory situations if you are not careful. However, if you look at the docs for Stream there is a tip for avoiding memoization building up in memory:
One must be cautious of memoization; you can very quickly eat up large
amounts of memory if you're not careful. The reason for this is that
the memoization of the Stream creates a structure much like
scala.collection.immutable.List. So long as something is holding on to
the head, the head holds on to the tail, and so it continues
recursively. If, on the other hand, there is nothing holding on to the
head (e.g. we used def to define the Stream) then once it is no longer
being used directly, it disappears.
So taking that into account, you could modify my original example as follows:
val binFile = new File(filePath)
val inputStream = new BufferedInputStream(new FileInputStream(binFile))
val binSource = Source(() => binStream(inputStream).iterator)
def binStream(in:BufferedInputStream) = Stream.continually(in.read).takeWhile(-1 != _).map(_.toByte)
So the idea here is to build the Stream via a def and not assign to a valand then immediately get the iterator from it and use that to initialize the Akka Source. Setting things up this way should avoid the issues with momoization. I ran the old code against a big file and was able to produce an OutOfMemory situation by doing a foreach on the Source. When I switched it over to the new code I was able to avoid this issue.
I need to use to java.util.zip.ZipOutputStream to respond with a compressed file archive.
The data is several hundred megabytes uncompressed, so I would like to store as little of it as possible. It is coming from a serialization of SQL results.
I see examples of using an OutputStream to return a chunked result using Enumerator.outputStream:
http://greweb.me/2012/11/play-framework-enumerator-outputstream/
Play/Akka integration with Java OutputStreams
but those seem ill-advised when I read the documentation (emphasis mine)
Create an Enumerator of bytes with an OutputStream.
Not that calls to write will not block, so if the iteratee that is being fed to is slow to consume the input, the
OutputStream will not push back. This means it should not be used with large streams since there is a risk of
running out of memory.
Clearly, I can't use that. Or at least not without modification.
How can I create a response with an OutputStream (in this case, a gzipped archive) while being assured that only portions of it will be stored in memory?
I recognize the difference between InputStreams/OutputStreams and Play's Enumerator/Iteratee paradigm, so I expect there will be a specific way in which I need to generate my source data (serialization of SQL results) so that it doesn't outpace the rate of download. I don't know what it is.
In general you can't safely use any OutputStream with the Enumerator/Iteratee framework because OutputStream doesn't support non-blocking pushback. However, if you can control the writing to the OutputStream you can hack together something like:
val baos = new ByteArrayOutputStream
val zos = new ZipOutputStream(baos)
val enumerator = Enumerator.generateM {
Future.successful {
if (moreDateToWrite) {
// Write data into zos
val r = Some(baos.toByteArray)
baos.reset()
r
} else None
}
}
If all you need is compression, take a look at the Enumeratee instances provided in play.filters.gzip.Gzip and the play.filters.gzip.GzipFilter filter.
The only backpressure mechanism for OutputStream is blocking the thread. So one way or another, there will have to be a thread that is able to be blocked.
One way is to use piped streams.
import java.io.OutputStream
import java.io.PipedInputStream
import java.io.PipedOutputStream
import play.api.libs.iteratee.Enumerator
import scala.concurrent.ExecutorContext
def outputStream2(a: OutputStream => Unit, bufferSize: Int)
(implicit ec1: ExecutionContext, ec2: ExecutionContext) = {
val outputStream = new PipedOutputStream
Future(a(outputStream))(ec1)
val inputStream = new PipedInputStream(pipedOutputStream, bufferSize)
Enumerator.fromStream(inputStream)(ec2)
}
Since the operations are blocking, you must take care to prevent deadlock.
Either use two different thread pools, or used a cached (unbounded) thread pool.
I am learning Iteratee and related APIs for one of my requirements to stream live tweets. Using Play 2.1 and Scala 2.10. Is following the best way to use Iteratee which also produces result of saving tweet to MongoDB?
val wsStream = new Enumerator[Array[Byte]] {
def apply[A](iteratee: Iteratee[Array[Byte], A]) = {
WS.url("https://stream.twitter.com/1.1/statuses/filter.json?track=" + term)
.sign(OAuthCalculator(Twitter.KEY, tokens))
.get(_ => iteratee)
}
}
wsStream.apply(Iteratee.foreach(bytes => saveTweetToMongo(bytes)))
Note you you can apply multiple iteratees to the same enumerator. In order words you can create a streamingTweetIteratee and a saveTweetToMongoIteratee and apply both to the enumerator which provides tweets.
I often create a simple loggingIteratee which just funnels everything to STDOUT when I'm prototyping in the REPL. I apply both it and the iteratee I'm writing to the same enumerator.
I'm assuming you want to using WebSockets in order to stream tweets to a client? If you look at the chat demo that comes with Play! you'll get an idea of how to go about that.
I'm implementing long polling in Play 2.0 in potentially a distributed environment. The way I understand it is that when Play gets a request, it should suspend pending notification of an update then go to the db to fetch new data and repeat. I started looking at the chat example that Play 2.0 offers but it's in websocket. Furthermore it doesn't look like it's capable of being distributed. So I thought I will use Akka's event bus. I took the eventstream implementation and replicated my own with LookupClassification. However I'm stumped as to how I'm gonna get a message back (or for that matter, what should be the subscriber instead of ActorRef)?
EventStream implementation:
https://github.com/akka/akka/blob/master/akka-actor/src/main/scala/akka/event/EventStream.scala
I am not sure that is what you are looking for, but there is quite a simple solution in the comet-clock sample, that you can adapt to use AKKA actors. It uses an infinite iframe instead of long polling. I have used an adapted version for a more complex application doing multiple DB calls and long computation in AKKA actors and it works fine.
def enum = Action {
//get your actor
val myActorRef = Akka.system.actorOf(Props[TestActor])
//do some query to your DB here. Promise.timeout is to simulate a blocking call
def getDatabaseItem(id: Int): Promise[String] = { Promise.timeout("test", 10 milliseconds) }
//test iterator, you will want something smarter here
val items1 = 1 to 10 toIterator
// this is a very simple enumerator that takes ints from an existing iterator (for an http request parameters for instance) and do some computations
def myEnum(it: Iterator[Int]): Enumerator[String] = Enumerator.fromCallback[String] { () =>
if (!items1.hasNext)
Promise.pure[Option[String]](None) //we are done with our computations
else {
// get the next int, query the database and compose the promise with a further query to the AKKA actor
getDatabaseItem(items1.next).flatMap { dbValue =>
implicit val timeout = new Timeout(10 milliseconds)
val future = (myActorRef ? dbValue) mapTo manifest[String]
// here we convert the AKKA actor to the right Promise[Option] output
future.map(v => Some(v)).asPromise
}
}
}
// finally we stream the result to the infinite iframe.
// console.log is the javascript callback, you will want something more interesting.
Ok.stream(myEnum(items1) &> Comet(callback = "console.log"))
}
Note that this fromCallback doesn't allow you to combine enumerators with "andThen", there is in the trunk version of play2 a generateM method that might be more appropriate if you want to use combinations.
It's not long polling, but it works fine.
I stumbled on your question while looking for the same thing.
I found the streaming solution unsatisfying as they caused "spinner of death" in webkit browser (i.e. shows it is loading all the time)
Anyhow, didn't have any luck finding good examples but I managed to create my own proof-of-concept using promises:
https://github.com/kallebertell/longpoll