I need to perform a single synchronous HTTP POST call: create an HTTP Post request with some data, connect to the server, send the request, receive the response, and close the connection. It is important to release all resources used to do this call.
Now I am doing it in Java with Apache Http Client. How can I do it with Scala dispatch library ?
Something like this should work (haven't tested it though)
import dispatch._, Defaults._
import scala.concurrent.Future
import scala.concurrent.duration._
def postSync(path: String, params: Map[String, Any] = Map.empty): Either[java.lang.Throwable, String] = {
val r = url(path).POST << params
val future = Http(r OK as.String).either
Await.result(future, 10.seconds)
}
(I'm using https://github.com/dispatch/reboot for this example)
You explicitly wait for the result of the future, which is either a String or an exception.
And use it like
postSync("http://api.example.com/a/resource", Map("param1" -> "foo") match {
case Right(res) => println(s"Success! Result was $res")
case Left(e) => println(s"Woops, something went wrong: ${e.getMessage}")
}
We all know that Scala's important feature is async. So there are lots of solutions to tell people how to do HTTP CALL ASYNC. For example, dispatch is the async http library. But here we care about how to implement sync http call by scala. I use a trick to do it.
Here is my solution:
val svc = url(#your_url)
val response = Http(svc OK dispatch.as.json4s.Json)
response onComplete {
// do_what_you_want
}
while(!response.isCompleted) {
Thread.sleep(1000)
}
// return_what_you_want
So here I use "isCompleted" to help me to wait until finish. So it is a trick to convert async to sync.
Related
I'm accessing data from a REST endpoint :
"https://api-public.sandbox.pro.coinbase.com/products/BTC-EUR/ticker"
To access the data once per second I use an infinite loop while(true) { to invoke a message send to the actor once per second which begins the process of invoking the REST request:
The actor to access the data is:
object ProductTickerRestActor {
case class StringData(data: String)
}
class ProductTickerRestActor extends Actor {
override def receive: PartialFunction[Any, Unit] = {
case ProductTickerRestActor.StringData(data) =>
try {
println("in ProductTickerRestActor")
val rData = scala.io.Source.fromURL("https://api-public.sandbox.pro.coinbase.com/products/BTC-EUR/ticker").mkString
println("rData : "+rData)
}
catch {
case e: Exception =>
println("Exception thrown in ProductTickerRestActor: " + e.getMessage)
}
case msg => println(s"I cannot understand ${msg.toString}")
}
}
I start the application using:
object ExchangeModelDataApplication {
def main(args: Array[String]): Unit = {
val actorSystem = ActorSystemConfig.getActorSystem
val priceDataActor = actorSystem.actorOf(Props[ProductTickerRestActor], "ProductTickerRestActor")
val throttler = Throttlers.getThrottler(priceDataActor)
while(true) {
throttler ! ProductTickerRestActor.StringData("test")
Thread.sleep(1000)
}
}
Throttler:
object Throttlers {
implicit val materializer = ActorMaterializer.create(ActorSystemConfig.getActorSystem)
def getThrottler(priceDataActor: ActorRef) = Source.actorRef(bufferSize = 1000, OverflowStrategy.dropNew)
.throttle(1, 1.second)
.to(Sink.actorRef(priceDataActor, NotUsed))
.run()
}
How to run the following code asynchronously instead of blocking using an infinite loop? :
throttler ! ProductTickerRestActor.StringData("test")
Thread.sleep(1000)
Also, the throttler, in this case, maybe redundant as I'm throttling requests within the loop regardless.
I would just use Akka Streams for this along with Akka HTTP. Using Akka 2.6.x, something along these lines would be sufficient for 1 request/second
import akka.actor.ActorSystem
import akka.http.scaladsl.Http
import akka.http.scaladsl.model._
import akka.stream.scaladsl._
import scala.concurrent.duration._
object HTTPRepeatedly {
implicit val system = ActorSystem()
import system.dispatcher
val sourceFromHttp: Source[String, NotUsed] =
Source.repeated("test") // Not sure what "test" is actually used for here...
.throttle(1, 1.second)
.map { str =>
HttpRequest(uri = "https://api-public.sandbox.pro.coinbase.com/products/BTC-EUR/ticker")
}.mapAsync(1) { req =>
Http().singleRequest(req)
}.mapAsync(1)(_.entity.toStrict(1.minute))
.map(_.data.decodeString(java.nio.charset.StandardCharsets.UTF_8))
}
Then you could, for instance (for simplicity, put this in a main within HTTPRepeatedly so the implicits are in scope etc.)
val done: Future[Done] =
sourceFromHttp
.take(10) // stop after 10 requests
.runWith(Sink.foreach { rData => println(s"rData: $rData") })
scala.concurrent.Await.result(done, 11.minute)
system.terminate()
Sending a request every second is not a good idea. If, for some reason, the request is delayed, you are going to get a lot of requests piled up. Instead, send the next request one second after the previous requests completes.
Because this code uses a synchronous GET request, you can just send the next request one second after mkString returns.
But using synchronous requests is not a good way to consume a RESTful API in Akka. It blocks the actor receive method until the request is complete, which can eventually block the whole ActorSystem.
Instead, use Akka Http and singleRequest to perform an asynchronous request.
Http().singleRequest(HttpRequest(uri = "https://api-public.sandbox.pro.coinbase.com/products/BTC-EUR/ticker"))
This returns a Future. Make the new request one second after the request completes (e.g. using onComplete on the Future).
Not only is this safer and more asynchronous, it also provides much more control over the REST API call than fromUrl
Disclaimer: I am new to sttp and Monix, and that is my attempt to learn more about these libraries. My goal is to fetch data (client-side) from a given API via HTTP GET requests -> parse JSON responses -> write this information to a database. My question pertains to the first part only. My objective is to run get requests in an asynchronous (hopefully fast) way while having a way to either avoid or handle rate limits.
Below is a snippet of what I have already tried, and seems to work for a single request:
package com.github.client
import io.circe.{Decoder, HCursor}
import sttp.client._
import sttp.client.circe._
import sttp.client.asynchttpclient.monix._
import monix.eval.Task
object SO extends App {
case class Bla(paging: Int)
implicit val dataDecoder: Decoder[Bla] = (hCursor: HCursor) => {
for {
next_page <- hCursor.downField("foo").downArray.downField("bar").as[Int]
} yield Bla(next_page)
}
val postTask = AsyncHttpClientMonixBackend().flatMap { implicit backend =>
val r = basicRequest
.get(uri"https://foo.bar.io/v1/baz")
.header("accept", "application/json")
.header("Authorization", "hushh!")
.response(asJson[Bla])
r.send() // How can I instead of operating on a single request, operate on multiple
.flatMap { response =>
Task(response.body)
}
.guarantee(backend.close())
}
import monix.execution.Scheduler.Implicits.global
postTask.runSyncUnsafe() match {
case Left(error) => println(s"Error when executing request: $error")
case Right(data) => println(data)
}
}
My questions:
How can I operate on several GET Requests (instead of a single request) via using Monix, while keeping the code asynchronous and composable
How can I avoid or handle hitting rate-limits imposed by the api server
On a side note, I am also flexible in terms of using another back-end if that will support the rate limiting objective.
You can use monix.reactive.Observable like this
Observable.repeatEval(postTask) // we generate infinite observable of the task
.throttle(1.second, 3) // set throttling
.mapParallelOrderedF(2)(_.runToFuture) // set execution parallelism and execute tasks
.subscribe() // start the pipline
while (true) {}
I am using the BlazeClientBuilder[IO].resource method to get Client[IO]. Now, I want to mock the client for unit testing but cannot figure out how to do so. Is there a good way of mocking this and how would I do that?
class ExternalCall(val resource: Resource[IO, Client[IO]], externalServiceUrl: Uri) {
def retrieveData: IO[Either[Throwable, String]] = {
for {
req <- IO(Request[IO](Method.GET, uri = externalServiceUrl))
response <- resource.use(client => {
client.fetch[String](req)(httpResponse => {
if (!httpResponse.status.isSuccess)
throw new Exception(httpResponse.status.reason)
else
httpResponse.as[String]
})
})
} yield Right(response)
}
}
Caller code
new ExternalCall(BlazeClientBuilder[IO](global).resource).retrieveData
It seems you only need to do something like
val resourceMock = mock[Resource[IO, Client[IO]]]
//stub whatever is necessary
val call = new ExternalCall(resourceMock).retrieveData
//do asserts and verifications as needed
EDIT:
You can see a fully working example below, but I'd like to stress that this is a good example of why it is a good practice to avoid mocking APIs that you don't own.
A better way to test this would be to place the http4s related code witin a class you own (YourHttpClient or whatever) and write an integration test for that class that checks that the http4s client does the right thing (you can use wiremock to simulate a real http server).
Then you can pass mocks of YourHttpClient to the components that depend on it, with the advantage that you control its API so it will be simpler and if http4s ever updates its API you only have one breaking class rather than having to fix tens or hundreds of mock interactions.
BTW, the example is written using mockito-scala as using the Java version of mockito would have yielded code much harder to read.
val resourceMock = mock[Resource[IO, Client[IO]]]
val clientMock = mock[Client[IO]]
val response: Response[IO] = Response(Status.Ok,
body = Stream("Mocked!!!").through(text.utf8Encode),
headers = Headers(`Content-Type`(MediaType.text.plain, Charset.`UTF-8`)))
clientMock.fetch[String](any[Request[IO]])(*) shouldAnswer { (_: Request[IO], f: Response[IO] => IO[String]) =>
f(response)
}
resourceMock.use[String](*)(*) shouldAnswer { (f: Client[IO] => IO[String]) =>
f(clientMock)
}
val data = new ExternalCall(resourceMock, Uri.unsafeFromString("http://www.example.com")).retrieveData
data.unsafeRunSync().right.value shouldBe "Mocked!!!"
You can easly mock Client using following snippet
import fs2.Stream
import org.http4s.Response
import org.http4s.client.Client
def httpClient(body: String): Client[IO] = Client.apply[IO] { _ =>
Resource.liftF(IO(Response[IO](body = Stream.emits(body.getBytes("UTF-8")))))
}
In order to have the client as resource you need to wrap it with IO and lift to Resource
Resource.liftF(IO(httpClient("body")))
I'm trying to convert this code, that uses the Play version 2.4 to the current version (2.6) and I'm having some issues because I'm still a noob in Scala.
def wsWeatherIntervals = WebSocket.using[String] {
request =>
val url = "http://api.openweathermap.org/data/2.5/weather?q=Amsterdam,nl"
val outEnumerator = Enumerator.repeatM[String]({
Thread.sleep(3000)
ws.url(url).get().map(r => s"${new java.util.Date()}\n ${r.body}")
})
(Iteratee.ignore[String], outEnumerator)
}
I followed this guide, but now I'm stuck on the stuff that I should return on the method.
This is the code that I'm trying to run using the version 2.6:
import play.api.mvc._
import scala.concurrent.Future
import akka.stream.scaladsl._
def wsWeatherIntervals = WebSocket.accept[String, Future[String]] { res =>
val url = "http://api.openweathermap.org/data/2.5/weather?q=Amsterdam,nl"
val source = Source.repeat({
Thread.sleep(3000)
ws.url(url).get().map(r => s"${new java.util.Date()}\n ${r.body}")
})
Flow.fromSinkAndSource(Sink.ignore, source)
}
But I'm getting this error when running the server, that points to the first line of the method:
could not find implicit value for parameter transformer: play.api.mvc.WebSocket.MessageFlowTransformer[String,scala.concurrent.Future[String]]
Note: I also tried to call WebSocket.apply instead of WebSocket.accept and I did some search about the differences between the two but didn't find anything useful. Can someone explain the difference between the two? Thanks.
The superficial error is that Play doesn't know how to turn a Future[String] into a Websocket message, for which you'd normally use an implicit transformer. However, in this case you don't want to return a Future[String] anyway but just a plain string which can be automatically marshalled (using the provided stringMessageFlowTransformer as it happens.) Here's something that should work:
def wsWeatherIntervals = WebSocket.accept[String, String] { res =>
val url = "http://api.openweathermap.org/data/2.5/weather?q=Amsterdam,nl"
def f = ws.url(url).get().map(r => s"${new java.util.Date()}\n ${r.body}")
val source = Source.unfoldAsync(f)(last => {
Thread.sleep(3000)
f.map(next => Some((last, next)))
})
Flow.fromSinkAndSource(Sink.ignore, source)
}
The unfoldAsync source lets us repeated run a function returning a future of the next element in the stream. (Since we want the stream to go on forever we return the value wrapped as Some.)
The Websocket.apply method is basically a more complicated version of accept which allows you to reject a websocket connection for some reason by returning a response, but if you need to do this it's better to use acceptOrResult, which handles transforming whatever your flow emits into websocket messages.
I am going to use Dispatch to write a simple HTTP client. I call dispatch.Http to get a future and call the future to get the response
val request = ...
val future = Http(request) // call the server asynchronously
val response = future() // wait for the response from the server
Now I wonder how I can wait with timeout. I would like the last API call to be:
// throw an exception if no response received within timeout
val response = future(timeout: Long)
Does it make sense ?
I understand that Dispatch return scala.concurrent.Future, which does not provide API with timeout. How would you suggest me implement it ?
You can create configured Http client:
val httpApplicationClient = Http.configure(
_.setRequestTimeoutInMs(3000)
)
val future = httpApplicationClient(request)
...
First, you can use Await:
import scala.concurrent.Await
import scala.concurrent.duration._
Await.result(future, 10 seconds) //you can specify timeout here
The problem is that it will throw an exception if the future did not manage to return in specified timeout.
If you need more flexibility here is the second approach:
val futureTimeout = Promise.timeout("Timed out" /* or throw new RuntimeException("timed out") */, 10 seconds)
Future.firstCompletedOf(Seq(futureResult, futureTimeout)).map {
case result: SomeType => //do something
case timedOut: SomeOtherType => //handle timeout
}