The use case is actually fairly typical. A lot of web services use authorization tokens that you retrieve at the start of a session and you need to send those back on subsequent requests.
I know I can do it like this:
lazy val myData = {
val request = ws.url("/some/url").withAuth(user, password, WSAuthScheme.BASIC).withHeaders("Accept" -> "application/json")
Await.result(request.get().map{x => x.json }, 120.seconds)
}
That just feels wrong as all the docs say never us Await.
Is there a Future/Promise Scala style way of handling this?
I've found .onComplete which allows me to run code upon the completion of a Promise however without using a (mutable) var I see no way of getting a value in that scope into a lazy val in a different scope. Even with a var there is a possible timing issue -- hence the evils of mutable variables :)
Any other way of doing this?
Unfortunately, there is no way to make this non-blocking - lazy vals are designed to be synchronous and to block any thread accessing them until they are completed with a value (internally a lazy val is represented as a simple synchronized block).
A Future/Promise Scala way would be to use a Future[T] or a Promise[T] instead of a val x: T, but that way implies a great deal of overhead with executionContexts and maps upon each use of the val, and more optimal resource utilization may not be worth the decreased readability in all cases, so it may be OK to leave the Await there if you extensively use the value in many parts of your application.
Related
I am using Scala concurrent module but I am not sure if it works asynchronously or synchronously ? I have a list of urls which are image urls and I want to make concurrent requests and get Array[Bytes]:
object Runner extends App {
def getBytesFromUrl(url:String) = {
requests.get(url).bytes
}
val urls = Seq(
"https://cdn.pixabay.com/photo/2014/02/27/16/10/flowers-276014__340.jpg",
"https://cdn.pixabay.com/photo/2014/02/27/16/10/flowers-276014__340.jpg",
"https://cdn.pixabay.com/photo/2014/02/27/16/10/flowers-276014__340.jpg"
)
// Would this make concurrent or sequential requests ?
val result = urls.map(url => getBytesFromUrl(url))
}
Would the above code make concurrent or sequential requests? And if it makes sequential requests then what's the right way to make concurrent requests?
I am using Scala concurrent module
You are not, at least not in the code shown.
Maybe you meant "request-scala library" in there?
but I am not sure if it works asynchronously or synchronously
For concurrency to exist you need asynchronously.
Now, again, if you were wondering about requests-scala then it is synchronously, that should be clear from the README and from the type signature.
Would the above code make concurrent or sequential requests?
Sequential, as the library explicitly states.
And if it makes sequential requests then what's the right way to make concurrent requests?
For one, you may consider using a different ecosystem since the author has been very clear that he doesn't think you need to be asynchronous at all to be efficient most of the time.
Another thing to consider is if you really need to make three HTTP calls concurrently.
Anyways, you can just use Future.traverse to do what you want.
object Runner {
def getBytesFromUrl(URL: String): Array[Byte] = {
requests.get(url).bytes
}
def getBytesFromUrls(urls: List[String]): Future[List[Array[Byte]]] =
Future.traverse(urls)(url => Future(getBytesFromUrl(url)))
}
You can then either compose that Future or Await it if you want to return to synchronous-land.
Personal disclaimer, if you actually have a lot of URLs and you need to do other asynchronous things with those bytes, like writing them to disk.
Then, I personally would recommend you to look to other ecosystems like typelevel, since those provide tools to write that kind of program in a more principled way.
I have some cats IO operations, and Future among then. Simplified:
IO(getValue())
.flatMap(v => IO.fromFuture(IO(blockingProcessValue(v)))(myBlockingPoolContextShift))
.map(moreProcessing)
So I have some value in IO, then I need to do some blocking operation using a library that returns Future, and then I need to do some more processing on the value returned from Future
Future runs on a dedicated thread pool - so far so good. The problem is after Future is completed. moreProcessing runs on the same thread the Future was running on.
Is there a way to get back to the thread getValue() was running on?
After the discussion in the chat, the conclusion is that the only thing OP needs is to create a ContextShift in the application entry point using the appropriate (compute) EC and then pass it down to the class containing this method.
// Entry point
val computeEC = ???
val cs = IO.contextShift(computeEC)
val myClass = new MyClass(cs, ...)
// Inside the method on MyClass
IO(getValue())
.flatMap(v => IO.fromFuture(IO(blockingProcessValue(v)))(myBlockingPoolContextShift))
.flatTap(_ => cs.shift)
.map(moreProcessing)
This Scastie showed an approach using Blocker and other techniques common in the Typelevel ecosystem but were not really suitable for OP's use case; anyways I find it useful for future readers who may have a similar problem.
I have basically list of unique ids, and for every id, i make a call to function which returns future.
Problem is number of futures in a single call is variable.
list.map(id -> futureCall)
There will be too much parallelism which can affect my system. I want to configure number of futures execution in parallel.
I want testable design so i can't do this
After searching alot, i found this
I didn't get it how to use it. I tried but it didn't work.
After that i have just imported it in my class where i am making call.
I have used same snippet and set default maxConcurrent to 4.
I replaced import global execution context with ThrottledExecutionContext
You have to wrap your ExecutionContext with ThrottledExecutionContext.
Here is a little sample:
object TestApp extends App {
implicit val ec = ThrottledExecutionContext(maxConcurrents = 10)(scala.concurrent.ExecutionContext.global)
def futureCall(id:Int) = Future {
println(s"executing $id")
Thread.sleep(500)
id
}
val list = 1 to 1000
val results = list.map(futureCall)
Await.result(Future.sequence(results), 100.seconds)
}
Alternatively you can also try a FixedThreadPool:
implicit val ec = ExecutionContext.fromExecutor(java.util.concurrent.Executors.newFixedThreadPool(10))
I am not sure what you are trying to do here. Default global ExecutionContext uses as many threads as you have CPU cores. So, that would be your parallelism. If that's still "too many" for you, you can control that number with a system property: "scala.concurrent.context.maxThreads", and set that to a lower number.
That will be the maximum number of futures that are executed in parallel at any given time. You should not need to throttle anything explicitly.
Alternatively, you can create your own executor, and give it a BlockingQueue with a limited capacity. That would block on the producer side (when a work item is being submitted), like your implementation does, but I would very strongly advice you from doing that as it is very dangerous and prone to deadlocks, and also much less efficient, that the default ForkJoinPool implementation.
Came across a problem I did not find an answer yet.
Running on playframework 2 with Scala.
Was required to write an Action method that performs multiple Future calls.
My question:
1) Is the attached code non-blocking and hence looking the way it should be ?
2) Is there a guarantee that both DAO results are caught at any given time ?
def index = Action.async {
val t2:Future[Tuple2[List[PlayerCol],List[CreatureCol]]] = for {
p <- PlayerDAO.findAll()
c <- CreatureDAO.findAlive()
}yield(p,c)
t2.map(t => Ok(views.html.index(t._1, t._2)))
}
Thanks for your feedback.
Is the attached code non-blocking and hence looking the way it should be ?
That depends on a few things. First, I'm going to assume that PlayerDAO.findAll() and CreatureDAO.findAlive() return Future[List[PlayerCol]] and Future[List[CreatureCol]] respectively. What matters most is what these functions are actually calling themselves. Are they making JDBC calls, or using an asynchronous DB driver?
If the answer is JDBC (or some other synchronous db driver), then you're still blocking, and there's no way to make it fully "non-blocking". Why? Because JDBC calls block their current thread, and wrapping them in a Future won't fix that. In this situation, the most you can do is have them block a different ExecutionContext than the one Play is using to handle requests. This is generally a good idea, because if you have several db requests running concurrently, they can block Play's internal thread pool used for handling HTTP requests, and suddenly your server will have to wait to handle other requests (even if they don't require database calls).
For more on different ExecutionContexts see the thread pools documentation and this answer.
If you're answer is an asynchronous database driver like reactive mongo (there's also scalike-jdbc, and maybe some others), then you're in good shape, and I probably made you read a little more than you had to. In that scenario your index controller function would be fully non-blocking.
Is there a guarantee that both DAO results are caught at any given time ?
I'm not quite sure what you mean by this. In your current code, you're actually making these calls in sequence. CreatureDAO.findAlive() isn't executed until PlayerDAO.findAll() has returned. Since they are not dependent on each other, it seems like this isn't intentional. To make them run in parallel, you should instantiate the Futures before mapping them in a for-comprehension:
def index = Action.async {
val players: Future[List[PlayerCol]] = PlayerDAO.findAll()
val creatures: Future[List[CreatureCol]] = CreatureDAO.findAlive()
val t2: Future[(List[PlayerCol], List[CreatureCol])] = for {
p <- players
c <- creatures
} yield (p, c)
t2.map(t => Ok(views.html.index(t._1, t._2)))
}
The only thing you can guarantee about having both results being completed is that yield isn't executed until the Futures have completed (or never, if they failed), and likewise the body of t2.map(...) isn't executed until t2 has been completed.
Further reading:
Are there any benefits in using non-async actions in Play Framework 2.2?
Understanding the Difference Between Non-Blocking Web Service Calls vs Non-Blocking JDBC
I have an actor which takes the result from another actor and applies some check on it.
class Actor1(actor2:Actor2) {
def receive = {
case SomeMessage =>
val r = actor2 ? NewMessage()
r.map(someTransform).pipeTo(sender)
}
}
now if I make an ask of Actor1, we now have 2 futures generated, which doesnt seem overly efficient. Is there a way to provide a foward with some kind of continuation, or some other approach I could use here?
case SomeMessage => actor2.forward(NewMessage, someTransform)
Futures are executed in an ExecutionContext, which are like thread pools. Creating a new future is not as expensive as creating a new thread, but it has its cost. The best way to work with futures is to create as much as needed and compose then in a way that things that can be computed in parallel are computed in parallel if the necessary resources are available. This way you will make the best use of your machine.
You mentioned that akka documentation discourages excessive use of futures. I don't know where you read this, but what I think it means is to prefer transforming futures rather than creating your own. This is exactly what you are doing by using map. Also, it may mean that if you create a future where it is not needed you are adding unnecessary overhead.
In your case you have a call that returns a future and you need to apply sometransform and return the result. Using map is the way to go.