Webservice call in Play2 Scala - scala

I want to do something apparently simple: calling a webservice and saving the result in the database.
I'm inside an Akka Actor code and what I do is to call an object method:
object Service {
def run {
val response = WS.url("http://api.server.com/rest/")
.withAuth("test", "test", com.ning.http.client.Realm.AuthScheme.BASIC)
.get.value.get.get.body
}
}
How do I parse the body? I tried to print it on the console but I got NotSuchElement exception.
Any idea, thought? How do I parse arrays, attributes, elements of the XML ?
I'm in the play version 2.1.0

Things have changed a bit since the previous version. Play 2.1.0 depends on the scala.concurrent package instead of their own classes:
A Play Promise is now a Scala Future
A Play Redeemable is now a Scala Promise
I didn't have time to test it, but from the documentation I gathered it should be something like this:
import play.api.libs.ws.WS
import play.api.libs.concurrent.Execution.Implicits._
import scala.concurrent.Await
import scala.concurrent.duration._
import scala.language.postfixOps
object WebserviceCallParseXML {
val responseFuture = WS.url("http://api.server.com/rest/")
.withAuth("test", "test", com.ning.http.client.Realm.AuthScheme.BASIC)
.get()
val resultFuture = responseFuture map { response =>
response.status match {
case 200 => Some(response.xml)
case _ => None
}
}
val result = Await.result(resultFuture, 5 seconds)
println(if (result.isDefined) result.get else "No result found" )
}
The documentation about Future.value:
If the future is not completed the returned value will be None. If the future is completed the value will be Some(Success(t)) if it contains a valid result, or Some(Failure(error)) if it contains an exception.

Related

How to emit messages from a Sink and pass them to another function?

I am currently building a client-side WebSockets consumer using Akka-HTTP. Instead of trying to do the parsing in the Sink, I wanted to wrap the code in a function which emits the outcome from the Sink, and then use this output (from the function) later for further processing (more parsing...etc.).
I am currently able to print every message from the Sink; however, the return type of the function remains to be Unit. My objective is to Emit a String from the function, for each item that lands in the sink, and then use the returned string to do further parsing. I have the code I have so far (Note: it's mostly boiler plate).
import java.util.concurrent.atomic.AtomicInteger
import akka.Done
import akka.actor.ActorSystem
import akka.http.scaladsl.Http
import akka.http.scaladsl.model.StatusCodes
import akka.http.scaladsl.model.ws.{Message, TextMessage, WebSocketRequest, WebSocketUpgradeResponse}
import akka.http.scaladsl.settings.ClientConnectionSettings
import akka.stream.Materializer
import akka.stream.scaladsl.{Flow, Keep, Sink, Source}
import akka.util.ByteString
import scala.concurrent.ExecutionContext.Implicits.global
import scala.concurrent.Future
import scala.util.{Failure, Success, Try}
object client extends App {
def parseData(uri: String)(implicit system: ActorSystem, materializer: Materializer): Unit = {
val defaultSettings = ClientConnectionSettings(system)
val pingCounter = new AtomicInteger()
val customWebsocketSettings = defaultSettings.websocketSettings.withPeriodicKeepAliveData(
() => ByteString(s"debug-${pingCounter.incrementAndGet()}")
)
val customSettings = defaultSettings.withWebsocketSettings(customWebsocketSettings)
val outgoing = Source.maybe[Message]
val sink: Sink[Message, Future[Done]] = Sink.foreach[Message] {
case message: TextMessage.Strict => message.text // I Want to emit/stream this message as a String from the function (or get a handle on it from the outside)
case _ => println("Other")
}
val webSocketFlow: Flow[Message, Message, Future[WebSocketUpgradeResponse]] =
Http().webSocketClientFlow(WebSocketRequest(uri), settings = customSettings)
val (upgradeResponse, closed) =
outgoing
.viaMat(webSocketFlow)(Keep.right)
.toMat(sink)(Keep.both)
.run()
val connected = upgradeResponse.flatMap { upgrade =>
if (upgrade.response.status == StatusCodes.SwitchingProtocols) {
Future.successful(Done)
} else {
throw new RuntimeException(
s"Connection failed: ${upgrade.response.status}"
)
}
}
connected.onComplete {
case Success(value) => value
case Failure(exception) => throw exception
}
closed.onComplete { _ =>
println("Retrying...")
parseData(uri)
}
upgradeResponse.onComplete {
case Success(value) => println(value)
case Failure(exception) => throw exception
}
}
}
And in a seperate object, I would like to do the parsing, so something like:
import akka.actor.ActorSystem
import akka.stream.Materializer
import api.client.parseData
object Application extends App {
implicit val system: ActorSystem = ActorSystem()
implicit val materializer: Materializer = Materializer(system)
val uri = "ws://localhost:8080/foobar"
val res = parseData(uri) // I want to handle the function output here
// parse(res)
println(res)
Is there a way I can get a handle on the Sink from outside the function, or do I need to do any parsing in the Sink. I am mainly trying to not overcomplicate the Sink.
Update: I am also considering if adding another Flow element to the stream (which handles the parsing) is a better practice than getting values outside of the stream.
Adding a flow element seems to solve your problem while being totally idiomatic.
What you have to keep in mind is that the sinks semantic is meant to describe how to "terminate" the stream, so while it can describe very complex computations, it will always return a single value which is returned only once the stream ends.
Said differently, a sink does not return a value per stream element, it returns a value per whole stream.

Scala wait for async call to complete in loop

I am new to scala and I have to make an async call ( Elastic indexing using elastic4s ) while iterating through a for each loop. What is the best way to do that in scala.
val data = List("1","2","3","4")
data.foreach(element=>{
asyncCall(element)
})
How to do some action after all the asyncCall is completed. I don'nt want to do anything with the response. Just print success if everything success and print fail if any of the call fails. asyncCall returns Future[T].
You should use Future.sequence. It changes List[Future[T]] to Future[List[T]].
So in your example:
import scala.concurrent.Future
import scala.concurrent.ExecutionContext.Implicits.global
import scala.util.{Failure, Success}
val f = Future.sequence(data.map(asyncCall)) // Future[List[String]]
//you can now do something when future finishes:
f.onComplete{
case Success(s) => println(s"Evethings fine: $s")
case Failure(e) => println(s"Something went wrong: $e")
}

Scala - How to use a Timer without blocking on Futures with Await.result

I have an Rest API provided by akka-http. In some cases I need to get data from an external database (Apache HBase), and I would like the query to fail if the database takes too long to deliver the data.
One naïve way is to wrap the call inside a Future and then block it with an Await.result with the needed duration.
import scala.concurrent.duration._
import scala.concurrent.{Await, Future}
object AsyncTest1 extends App {
val future = Future {
getMyDataFromDB()
}
val myData = Await.result(future, 100.millis)
}
The seems to be inefficient as this implementation needs two threads. Is There an efficient way to do this ?
I have another use case where I want to send multiple queries in parallel and then aggregates the results, with the same delay limitation.
val future1 = Future {
getMyDataFromDB1()
}
val future2 = Future {
getMyDataFromDB2()
}
val foldedFuture = Future.fold(
Seq(future1, future2))(MyAggregatedData)(myAggregateFunction)
)
val myData = Await.result(foldedFuture, 100.millis)
Same question here, what is the most efficient way to implement this ?
Thanks for your help
One solution would be to use Akka's after function which will let you pass a duration, after which the future throws an exception or whatever you want.
Take a look here. It demonstrates how to implement this.
EDIT:
I guess I'll post the code here in case the link gets broken in future:
import scala.concurrent._
import scala.concurrent.duration._
import ExecutionContext.Implicits.global
import scala.util.{Failure, Success}
import akka.actor.ActorSystem
import akka.pattern.after
val system = ActorSystem("theSystem")
lazy val f = future { Thread.sleep(2000); true }
lazy val t = after(duration = 1 second, using = system.scheduler)(Future.failed(new TimeoutException("Future timed out!")))
val fWithTimeout = Future firstCompletedOf Seq(f, t)
fWithTimeout.onComplete {
case Success(x) => println(x)
case Failure(error) => println(error)
}

unable to recover in Future - scala

I expect that the following code will generate value 5 as I have written recover to handle the exception and return 5. But I always see the exception in IntelliJ. What is the mistake here?
import scala.concurrent._
import scala.concurrent.ExecutionContext.Implicits.global
import scala.util.{Failure, Success, Try}
import scala.concurrent.duration._
import scala.language.postfixOps
object ConcurrencyExample extends App {
val time = System.currentTimeMillis()
val futures1: Future[Int] = Future {
println("in Future.. sleeping")
Thread.sleep(10)
println("awakein future")
throw new Exception("future failed")
1
}
val result = futures1.map(x=>x+3)
result recover {
case e:Exception => 5
}
result onComplete{
case Success(v)=>println("result is "+v)
case Failure(e) =>println("failed result:"+e)
}
Await.result(result,100 millis)
}
result
in Future.. sleeping
awakein future
Exception in thread "main" LegacyException: future failed
It is not working because the future that would do the recovering is not the one you are calling Await.result() on.
From the Future's scaladoc, we learn that recover creates a new future that will handle any matching throwable that this future might contain. If there is no match, or if this future contains a valid result then the new future will contain the same.
Thus, it is this new future you are now interested in.
Try this:
val result = futures1.map(x=>x+3)
.recover {
case e:Exception => 5
}
Await.result(result,100 millis)

Use case for WebDriver (https://github.com/typesafehub/webdriver) to execute js

I'm working on a web scraper using this project (based on Scala, Spray, Akka and PhantomJS)
The problem is that I can't find a more specific example of how to use it, and the documentation is missing a lot of details
1- I would like to know how to give an specific URL so I can get data from it
2- How can I excecute, or pass a javascript file or function so that phantom can run and do some stuff(return specific data or whatever, from the site in point 1- )
Here is my Main.scala file: (Is almost the same as the one in the project)
package com.typesafe.webdriver.tester
import akka.actor.{ActorRef, ActorSystem}
import akka.pattern.ask
import com.typesafe.webdriver.{Session, PhantomJs, LocalBrowser}
import akka.util.Timeout
import scala.concurrent.duration._
import scala.concurrent.ExecutionContext.Implicits.global
import spray.json._
import spray.http._
object Main {
def main(args: Array[String]) {
implicit val system = ActorSystem("webdriver-system")
implicit val timeout = Timeout(5.seconds)
system.scheduler.scheduleOnce(7.seconds) {
system.shutdown()
System.exit(1)
}
val browser = system.actorOf(PhantomJs.props(system), "localBrowser")
browser ! LocalBrowser.Startup
for (
session <- (browser ? LocalBrowser.CreateSession).mapTo[ActorRef];
result <- (session ? Session.ExecuteNativeJs("return 5+5",JsArray(JsNumber(999)))).mapTo[JsNumber]
) yield {
println(result)
try {
system.shutdown()
System.exit(0)
} catch {
case _: Throwable =>
}
}
}
}
I would suggest you to use already created web scrappers in Scala.
For example ScalaWebDcraper which has nicely writted DSL and scrapping feature.
https://github.com/Rovak/ScalaWebscraper
It can be combined with Goose, which is a web article extractor. You can use it to fetch article data from the links you visit with the previous library.
https://github.com/jiminoc/goose
Also, checkout Metascrapper, a Scala Library for Scraping Page Metadata
https://beachape.com/blog/2013/09/05/introducing-metascraper-a-scala-library-for-scraping-page-metadata/
And check this question, lot's of valuable info inside.