scala - Remove on Read - scala

Could anyone suggest an efficient and succinct way of realizing the following feature of "remove on read"? That is, when the attribute _events is read, it's cleared.
case class Event(nanos: Long)
case class History() {
private val _events = ArrayBuffer[Event]()
def add(event: Event): Unit = _events += event
def events: List[Event] = {
val builder = ArrayBuffer[Event]()
builder ++= _events
_events.clear()
builder.toList
}
}

Assuming you are not doing concurrency. You don't need to create a new ArrayBuffer. toList creates a new immutable List which will sustain when you clear. Just do
def events: List[Event] = {
val res = _events.toList
_events.clear()
res
}

You can use ConcurrentLinkedQueue of Java. The following solution supports concurrency, in a way that if you add objects, while retrieving the history, they will be added to the history, and to the result of the current call. However, they are read once in this solution. If you need to stop adding while retrieving the history, we can think about a different solution. Consider having the History class as follows:
case class History() {
private val _events = new ConcurrentLinkedQueue[Event]()
def add(event: Event): Unit = _events.add(event)
def events: List[Event] = {
if (_events.isEmpty) {
List()
} else {
_events.poll() +: events
}
}
}
Then, when running the following code:
def main(args: Array[String]): Unit = {
val h = History()
List(
Event(10),
Event(20),
Event(30)
).foreach(h.add)
println(s"Events are: ${h.events.mkString(", ")}")
println(s"Events are: ${h.events.mkString(", ")}")
}
The output is:
Events are: Event(10), Event(20), Event(30)
Events are:
Process finished with exit code 0

Related

How to access computed data from other scala file?

I need to access this data mapData from Calculation.scala. The way I call the function in Final.scala is as seen below.
When I see the output of fetch_data() or print it I see Future(< not completed >) and result is empty. I do not know how to wait until all data is downloaded and then access mapData? MAy I know how to do it? I am new to scala. In C++ I am aware of callbacks and handling is easy there. But in scala I a using Future, Await or OnComplete, but not clear how to do it.
Final.Scala
object finalComputation {
val calculationInfo = new Calculaton()
calclulationInfo.fetch_data()
val result = calculationInfo.getMapData()
def main(args: Array[String]): Unit = {
........
}
}
Calculation.scala
class Calculation {
var mapData = Map.empty[String, String]
def createMapData(metricItem: ActualMetrics) = {
mapData += (metricItem._1 -> metricItem._2)
}
def getMapData() = {
mapData
}
def fetch_data() = {
val totalData: Future[Done] =
querApi
.getData()
.map { data =>
(data)
}
}
}
Await.result(totalData, Duration.Inf).runForeach(unit => {
createMapData(parse.From(totalData))
})
}
}
Well, by starter don't mix concurrency with mutability.
Second, don't create imperative APIs that require some specific call order.
Third, Future is just a fancy wrapper over callbacks.
And fourth, don't initiate computations before the main
// file: Calculation.scala
class Calculation(queryApi: Api) {
def fetchData(): Future[Map[String, String]] =
querApi.getData().map { data =>
data.view.map { metric =>
val ActualMetrics(key, value) = parse(metric)
key -> value
}.toMap
}
}
// file: Main.scala
object Main {
def main(args: Array[String]): Unit = {
val calculation = new Calculation(...)
val dataF = calculation.fetchData()
val result = dataF.map { data =>
// Here you can process the fetched data
// It may be flatMap or foreach, instead of map;
// depending on what you want to do, check the Scaladoc
}
// I don't use future, but I think here you need to do a final await of result.
// In order to avoid the program to finish before the async computation.
}
}
I had to assume some types, but I hope this gives you a general idea of what to do.
My advice, pick any Scala course / book / tutorial and properly learn the language.

Reading from postgres using Akka Streams 2.4.2 and Slick 3.0

Trying out the newly minted Akka Streams. It seems to be working except for one small thing - there's no output.
I have the following table definition:
case class my_stream(id: Int, value: String)
class Streams(tag: Tag) extends Table[my_stream](tag, "my_stream") {
def id = column[Int]("id")
def value = column[String]("value")
def * = (id, value) <> (my_stream.tupled, my_stream.unapply)
}
And I'm trying to output the contents of the table to stdout like this:
def main(args: Array[String]) : Unit = {
implicit val system = ActorSystem("Subscriber")
implicit val materializer = ActorMaterializer()
val strm = TableQuery[Streams]
val db = Database.forConfig("pg-postgres")
try{
var src = Source.fromPublisher(db.stream(strm.result))
src.runForeach(r => println(s"${r.id},${r.value}"))(materializer)
} finally {
system.shutdown
db.close
}
}
I have verified that the query is being run by configuring debug logging. However, all I get is this:
08:59:24.099 [main] INFO com.zaxxer.hikari.HikariDataSource - pg-postgres - is starting.
08:59:24.428 [main] INFO com.zaxxer.hikari.pool.HikariPool - pg-postgres - is closing down.
The cause is that Akka Streams is asynchronous and runForeach returns a Future which will be completed once the stream completes, but that Future is not being handled and as such the system.shutdown and db.close executes immediately instead of after the stream completes.
Just in case it helps anyone searching this very same issue but in MySQL, take into account that you should enable the driver stream support "manually":
def enableStream(statement: java.sql.Statement): Unit = {
statement match {
case s: com.mysql.jdbc.StatementImpl => s.enableStreamingResults()
case _ =>
}
}
val publisher = sourceDb.stream(query.result.withStatementParameters(statementInit = enableStream))
Source: http://www.slideshare.net/kazukinegoro5/akka-streams-100-scalamatsuri
Ended up using #ViktorKlang answer and just wrapped the run with an Await.result. I also found an alternative answer in the docs which demonstrates using the reactive streams publisher and subscriber interfaces:
The stream method returns a DatabasePublisher[T] and Source.fromPublisher returns a Source[T, NotUsed]. This means you have to attach a subscriber instead of using runForEach - according to the release notes NotUsed is a replacement for Unit. Which means nothing gets passed to the Sink.
Since Slick implements the reactive streams interface and not the Akka Stream interfaces you need to use the fromPublisher and fromSubscriber integration point. That means you need to implement the org.reactivestreams.Subscriber[T] interface.
Here's a quick and dirty Subscriber[T] implementation which simply calls println:
class MyStreamWriter extends org.reactivestreams.Subscriber[my_stream] {
private var sub : Option[Subscription] = None;
override def onNext(t: my_stream): Unit = {
println(t.value)
if(sub.nonEmpty) sub.head.request(1)
}
override def onError(throwable: Throwable): Unit = {
println(throwable.getMessage)
}
override def onSubscribe(subscription: Subscription): Unit = {
sub = Some(subscription)
sub.head.request(1)
}
override def onComplete(): Unit = {
println("ALL DONE!")
}
}
You need to make sure you call the Subscription.request(Long) method in onSubscribe and then in onNext to ask for data or nothing will be sent or you won't get the full set of results.
And here's how you use it:
def main(args: Array[String]) : Unit = {
implicit val system = ActorSystem("Subscriber")
implicit val materializer = ActorMaterializer()
val strm = TableQuery[Streams]
val db = Database.forConfig("pg-postgres")
try{
val src = Source.fromPublisher(db.stream(strm.result))
val flow = src.to(Sink.fromSubscriber(new MyStreamWriter()))
flow.run()
} finally {
system.shutdown
db.close
}
}
I'm still trying to figure this out so I welcome any feedback. Thanks!

Asynchronous Iterable over remote data

There is some data that I have pulled from a remote API, for which I use a Future-style interface. The data is structured as a linked-list. A relevant example data container is shown below.
case class Data(information: Int) {
def hasNext: Boolean = ??? // Implemented
def next: Future[Data] = ??? // Implemented
}
Now I'm interested in adding some functionality to the data class, such as map, foreach, reduce, etc. To do so I want to implement some form of IterableLike such that it inherets these methods.
Given below is the trait Data may extend, such that it gets this property.
trait AsyncIterable[+T]
extends IterableLike[Future[T], AsyncIterable[T]]
{
def hasNext : Boolean
def next : Future[T]
// How to implement?
override def iterator: Iterator[Future[T]] = ???
override protected[this] def newBuilder: mutable.Builder[Future[T], AsyncIterable[T]] = ???
override def seq: TraversableOnce[Future[T]] = ???
}
It should be a non-blocking implementation, which when acted on, starts requesting the next data from the remote data source.
It is then possible to do cool stuff such as
case class Data(information: Int) extends AsyncIterable[Data]
val data = Data(1) // And more, of course
// Asynchronously print all the information.
data.foreach(data => println(data.information))
It is also acceptable for the interface to be different. But the result should in some way represent asynchronous iteration over the collection. Preferably in a way that is familiar to developers, as it will be part of an (open source) library.
In production I would use one of following:
Akka Streams
Reactive Extensions
For private tests I would implement something similar to following.
(Explanations are below)
I have modified a little bit your Data:
abstract class AsyncIterator[T] extends Iterator[Future[T]] {
def hasNext: Boolean
def next(): Future[T]
}
For it we can implement this Iterable:
class AsyncIterable[T](sourceIterator: AsyncIterator[T])
extends IterableLike[Future[T], AsyncIterable[T]]
{
private def stream(): Stream[Future[T]] =
if(sourceIterator.hasNext) {sourceIterator.next #:: stream()} else {Stream.empty}
val asStream = stream()
override def iterator = asStream.iterator
override def seq = asStream.seq
override protected[this] def newBuilder = throw new UnsupportedOperationException()
}
And if see it in action using following code:
object Example extends App {
val source = "Hello World!";
val iterator1 = new DelayedIterator[Char](100L, source.toCharArray)
new AsyncIterable(iterator1).foreach(_.foreach(print)) //prints 1 char per 100 ms
pause(2000L)
val iterator2 = new DelayedIterator[String](100L, source.toCharArray.map(_.toString))
new AsyncIterable(iterator2).reduceLeft((fl: Future[String], fr) =>
for(l <- fl; r <- fr) yield {println(s"$l+$r"); l + r}) //prints 1 line per 100 ms
pause(2000L)
def pause(duration: Long) = {println("->"); Thread.sleep(duration); println("\n<-")}
}
class DelayedIterator[T](delay: Long, data: Seq[T]) extends AsyncIterator[T] {
private val dataIterator = data.iterator
private var nextTime = System.currentTimeMillis() + delay
override def hasNext = dataIterator.hasNext
override def next = {
val thisTime = math.max(System.currentTimeMillis(), nextTime)
val thisValue = dataIterator.next()
nextTime = thisTime + delay
Future {
val now = System.currentTimeMillis()
if(thisTime > now) Thread.sleep(thisTime - now) //Your implementation will be better
thisValue
}
}
}
Explanation
AsyncIterable uses Stream because it's calculated lazily and it's simple.
Pros:
simplicity
multiple calls to iterator and seq methods return same iterable with all items.
Cons:
could lead to memory overflow because stream keeps all prevously obtained values.
first value is eagerly gotten during creation of AsyncIterable
DelayedIterator is very simplistic implementation of AsyncIterator, don't blame me for quick and dirty code here.
It's still strange for me to see synchronous hasNext and asynchronous next()
Using Twitter Spool I've implemented a working example.
To implement spool I modified the example in the documentation.
import com.twitter.concurrent.Spool
import com.twitter.util.{Await, Return, Promise}
import scala.concurrent.{ExecutionContext, Future}
trait AsyncIterable[+T <: AsyncIterable[T]] { self : T =>
def hasNext : Boolean
def next : Future[T]
def spool(implicit ec: ExecutionContext) : Spool[T] = {
def fill(currentPage: Future[T], rest: Promise[Spool[T]]) {
currentPage foreach { cPage =>
if(hasNext) {
val nextSpool = new Promise[Spool[T]]
rest() = Return(cPage *:: nextSpool)
fill(next, nextSpool)
} else {
val emptySpool = new Promise[Spool[T]]
emptySpool() = Return(Spool.empty[T])
rest() = Return(cPage *:: emptySpool)
}
}
}
val rest = new Promise[Spool[T]]
if(hasNext) {
fill(next, rest)
} else {
rest() = Return(Spool.empty[T])
}
self *:: rest
}
}
Data is the same as before, and now we can use it.
// Cool stuff
implicit val ec = scala.concurrent.ExecutionContext.global
val data = Data(1) // And others
// Print all the information asynchronously
val fut = data.spool.foreach(data => println(data.information))
Await.ready(fut)
It will trow an exception on the second element, because the implementation of next was not provided.

What are the implications of nesting an akka Actor class in an object to the members of the class?

I'm new to scala and akka and I came across code that looked something like this:
object Foo extends App {
class Bar extends Actor with ActorLogging{
val graph: TitanGraph = _
private def setGraph() = {
graph = TitanFactory.open(...)
}
setGraph()
...
}
def startBar() {
val barSystem = ActorSystem("BarSystem")
for(i <- 0 until numActors) {
val barActor = barSystem.actorOf(Props[Bar], name="BarActor" + i)
...
barActor ! Start
}
}
startBar
}
Does this have any effect on performance as compared to?:
object Foo extends App {
override def main(args: Array[String]): Unit = {
val barSystem = ActorSystem("BarSystem")
for(i <- 0 until numActors) {
val barActor = barSystem.actorOf(Props[Bar], name="BarActor" + i)
...
barActor ! Start
}
}
}
object Bar {
val graph: TitanGraph = _
private def setGraph() = {
graph = TitanFactory.open(...)
}
setGraph()
def props = Props[Bar]
...
}
class Bar extends Actor with ActorLogging{
...
}
In the first case, you're creating multiple instances of the graph but in the second case, I assume that you're using a single instance shared across workers?
Also, I read somewhere that it is good practice to keep the Actor's props factory in the companion object but not sure if the previous code breaks actor encapsulation or if it affects performance at all.
Whether you place an Actor inside an object or outside does not change anything besides the class file name. Concerning sharing a single Actor object instance, that is illegal and will not work, but luckily you cannot easily fall into this trap.

How to write tests for a Play Framework Filter?

I've written a Filter for my Play application:
object MyFilter {
def apply() = new MyFilter()
}
class MyFilter extends EssentialFilter {
def apply(next: EssentialAction) = new EssentialAction {
def apply(requestHeader: RequestHeader) = {
requestHeader.cookies.get("myCookie") match {
case Some(cookie) => {
requestHeader.session + ("importantValue", cookie.value)
next(requestHeader)
}
case None => {
val importantValue = ... // retrieve the value from somewhere
requestHeader.session + ("importantValue", importantValue)
next(requestHeader).map(_.withCookies(Cookie("importantValue", value)))
}
}
}
}
There're a lot of examples in the Play documentation about how to write test, but I've no idea how to unit test MyFilter. Can someone help me a little?
Thanks
Torben
The play framework source code shows test for filters.
for example:
val filter = SecurityHeadersFilter()
// Play.current is set at this point...
val rh = FakeRequest()
val action = Action(Ok("success"))
val result = filter(action)(rh).run()
header(X_FRAME_OPTIONS_HEADER, result) must beSome("DENY")
taken from here SecurityHeadersFilterSpec.scala
Not answering your question but requestHeader.session + ("importantValue", cookie.value) is never going to do anything. You're creating a new session object and don't do anything with it.