Hello I'm writing scala code to pull the data from API.
Data is paginated, so I'm pulling a data sequentially.
Now, I'm looking a solution to pulling multiple page parallel and stuck to create WSClient programatically instead of Inject.
Anyone have a solution to create WSClient ?
I found a AhcWSClient(), but it required to implicitly import actor system.
When you cannot Inject one as suggested in the other answer, you can create a Standalone WS client using:
import akka.actor.ActorSystem
import akka.stream.ActorMaterializer
import play.api.libs.ws._
import play.api.libs.ws.ahc.StandaloneAhcWSClient
implicit val system = ActorSystem()
implicit val materializer = ActorMaterializer()
val ws = StandaloneAhcWSClient()
No need to reinvent the wheel here. And I'm not sure why you say you can't inject a WSClient. If you can inject a WSClient, then you could do something like this to run the requests in parallel:
class MyClient #Inject() (wsClient: WSClient)(implicit ec: ExecutionContext) {
def getSomething(urls: Vector[String]): Future[Something] = {
val futures = urls.par.map { url =>
wsClient.url(url).get()
}
Future.sequence(futures).map { responses =>
//process responses here. You might want to fold them together
}
}
}
Related
I am using Akka HTTP as a client to do a POST request and parse the answer. I am using Play JSON and I get the following compiler error:
could not find implicit value for parameter um: akka.http.scaladsl.unmarshalling.Unmarshaller[akka.http.javadsl.model.ResponseEntity,B]
[ERROR] Unmarshal(response.entity).to[B].recoverWith {
This is the dependency I added to use Play JSON instead of Spray:
"de.heikoseeberger" %% "akka-http-play-json"
My class definition is:
class HttpClient(implicit val system: ActorSystem, val materializer: Materializer) extends PlayJsonSupport {
and the method definition is:
private def parseResponse[B](response: HttpResponse)(implicit reads: Reads[B]): Future[B] = {
if (response.status().isSuccess) {
Unmarshal(response.entity).to[B].recoverWith {
....
In the imports I have:
import play.api.libs.json._
import scala.concurrent.ExecutionContext.Implicits.global
import de.heikoseeberger.akkahttpplayjson.PlayJsonSupport._
It seems to me that I have the required implicits in scope. The Marshal part has a similar logic (but with Writes instead of Reads) and compiles fine. What am I missing?
Check your other imports. Based on the error message, it appears that you're using akka.http.javadsl.model.HttpResponse instead of akka.http.scaladsl.model.HttpResponse; PlayJsonSupport only supports the Scala DSL:
private def parseResponse[B](response: HttpResponse)(implicit reads: Reads[B]): Future[B] = ???
// ^ this should be akka.http.scaladsl.model.HttpResponse
In other words, use
import akka.http.scaladsl.model._
instead of
import akka.http.javadsl.model._
I'm trying to create an API with Spray which listens to 2 prefixes. These 2 prefixes in turn listen to optional integers.
This is the setup that I am trying to achieve:
val itemRoute = {
pathPrefix("configs") {
<...>
}
pathPrefix("samples") {
<...>
}
}
This way, the API can listen to calls like http://www.example.com/samples/2
However, with said snippet, only one of the two prefixes are listened to.
I have tried different syntax styles, like putting a ~ inbetween twe pathPrefix blocks, and incorporating pathPrefixTest. Is this an issue with my syntax, and how can I achieve multiple pathPrefixes?
Use Akka http, Spray:
spray is no longer maintained and has been superseded by Akka HTTP.
Please check out the migration guide for help with the upgrade.
Commercial support is available from Lightbend.
Anyway, that example would work:
package test
import akka.http.scaladsl.server.Directives._
import akka.http.scaladsl.Http
import akka.actor.ActorSystem
import akka.stream.ActorMaterializer
object TesHttp {
val routes = pathPrefix("configs") {
complete {
"configs"
}
} ~
pathPrefix("samples") {
complete {
"samples"
}
}
def main(args: Array[String]) : Unit = {
implicit val system = ActorSystem()
implicit val mat = ActorMaterializer()
import system.dispatcher
println("Starting ..")
val binding = Http().bindAndHandle(routes, interface = "localhost", 9091)
}
}
Following this answer: https://stackoverflow.com/a/30806548/4496364
I use Play's ExecutionContext in my project.
Recently I needed to use Mockito to test some services in Play.
So, this is simplified version of it:
import scala.concurrent.{ Future, ExecutionContext }
import play.api.libs.concurrent.Execution.Implicits.defaultContext
case class Model(id: Int, name: String)
trait DAO {
def findAll(implicit ec: ExecutionContext): Future[List[Model]]
}
class Service(dao: DAO) {
def findAll: Future[List[Model]] = dao.findAll
}
Test:
import play.api.libs.concurrent.Execution.Implicits.defaultContext
// doesn't work when different ExecutionContext
// import scala.concurrent.ExecutionContext.Implicits.global
class FuturesTest extends PlaySpec with MockitoSugar with ScalaFutures {
"Service" should {
"return all future data" in {
val mockModel = Model(1, "name")
val mockDAO = mock[DAO]
when(mockDAO.findAll) thenReturn Future.successful(List(mockModel))
val service = new Service(mockDAO)
val futureData = service.findAll
whenReady(futureData) { data =>
data.map(_.name) must contain(mockModel.name)
}
}
}
}
Note the comment in test, i get a NullPointException when calling dao.findAll in the Service. At first I thought that Mockito can't handle Scala's Futures but I figured out that the ExecutionContext is the problem. Since I'm not a concurrency expert, can someone please explain why does this happen?
In case someone is looking, the answer was obvious...
import org.mockito.Matchers.any
..
mockDAO.findAll(any[ExecutionContext])
I wasn't familiar with how Mockito works, or with Scala implicits.
When you don't pass any[ExecutionContext] Scala will fill it with the implicit one from the test.
MyService.scala:33: could not find implicit value for parameter eh: spray.routing.ExceptionHandler
I have run into a "missing implicit" compilation error using Akka, in spray.io code that makes an http call to a separate back-end server, as part of responding to an http get. The code needs to import quite a lot of Spray and Akka libraries, so it's a bit hard figuring whether there may be some library conflicts causing this, and I'd rather figure how to logically trace this sort of problem for this and other cases.
The missing implicit is encountered on calling runRoute(myRoute)
Here's the code:
import spray.routing._
import akka.actor.Actor
import akka.actor.ActorSystem
import spray.http._
import MediaTypes._
import akka.io.IO
import spray.httpx.RequestBuilding._
import scala.concurrent.Future
import spray.can.Http
import spray.http._
import akka.util.Timeout
import HttpMethods._
import akka.pattern.ask
import akka.event.Logging
import scala.concurrent.duration._
// we don't implement our route structure directly in the service actor because
// we want to be able to test it independently, without having to spin up an actor
class MyServiceActor extends Actor with MyService with akka.actor.ActorLogging {
log.info("Starting")
// the HttpService trait defines only one abstract member, which
// connects the services environment to the enclosing actor or test
def actorRefFactory = context
// this actor only runs our route, but you could add
// other things here, like request stream processing
// or timeout handling
def receive = runRoute(myRoute)
}
// this trait defines our service behavior independently from the service actor
trait MyService extends HttpService {
implicit val system: ActorSystem = ActorSystem()
implicit val timeout: Timeout = Timeout(15.seconds)
import system.dispatcher // implicit execution context
//val logger = context.actorSelection("/user/logger")
val logger = actorRefFactory.actorSelection("../logger")
val myRoute =
{
def forward(): String = {
logger ! Log("forwarding to backend")
val response: Future[HttpResponse] =
(IO(Http) ? Get("http:3080//localhost/backend")).mapTo[HttpResponse]
"<html><body><h1>api response after backend processing</h1></body></html>"
}
path("") {
get {
respondWithMediaType(`text/html`) { // XML is marshalled to `text/xml` by default, so we simply override here
complete(forward)
}
}
}
}
}
I am wondering what's the best way to solve this, hopefully providing insight into how to solve similar problems with implicits being missing, as they are somehow inherently not straightforward to track down.
EDIT: when trying to directly pass implicits as in #christian's answer below, I get:
MyService.scala:35: ambiguous implicit values:
both value context in trait Actor of type => akka.actor.ActorContext
and value system in trait MyService of type => akka.actor.ActorSystem
match expected type akka.actor.ActorRefFactory
RoutingSettings.default, LoggingContext.fromActorRefFactory)
^
Not quite sure why being specific as in #christian's answer leaves room for ambiguity for the compiler...
I ran into the same "could not find implicit value for parameter eh: spray.routing.ExceptionHandler" error earlier today. I tried #Christian's approach but saw a few "implicit values for xxx" creeping up. After scouting the error message a little I found adding implicit val system = context.system to the actor that runRoute solved the problem.
runRoute expects a few implicits. You are missing an import:
import spray.routing.RejectionHandler.Default
Update:
I think we also did have some problems with runRoute because we are supplying the implicit parameters explicitly:
runRoute(route)(ExceptionHandler.default, RejectionHandler.Default, context,
RoutingSettings.default, LoggingContext.fromActorRefFactory)
Update2:
To fix the last error, remove the creation of the ActorSystem (in MyService you get the actor system from MyServiceActor - therefore you have to use a self type annotation). This compiles:
import akka.actor.Actor
import akka.io.IO
import spray.httpx.RequestBuilding._
import spray.http.MediaTypes._
import spray.routing.{RoutingSettings, RejectionHandler, ExceptionHandler, HttpService}
import spray.util.LoggingContext
import scala.concurrent.Future
import spray.can.Http
import spray.http._
import akka.util.Timeout
import HttpMethods._
import akka.pattern.ask
import akka.event.Logging
import scala.concurrent.duration._
// we don't implement our route structure directly in the service actor because
// we want to be able to test it independently, without having to spin up an actor
class MyServiceActor extends Actor with MyService with akka.actor.ActorLogging {
log.info("Starting")
// the HttpService trait defines only one abstract member, which
// connects the services environment to the enclosing actor or test
implicit def actorRefFactory = context
// this actor only runs our route, but you could add
// other things here, like request stream processing
// or timeout handling
def receive = runRoute(myRoute)(ExceptionHandler.default, RejectionHandler.Default, context,
RoutingSettings.default, LoggingContext.fromActorRefFactory)
}
// this trait defines our service behavior independently from the service actor
trait MyService extends HttpService { this: MyServiceActor =>
implicit val timeout: Timeout = Timeout(15.seconds)
implicit val system = context.system
//val logger = context.actorSelection("/user/logger")
val logger = actorRefFactory.actorSelection("../logger")
val myRoute =
{
def forward(): String = {
//logger ! Log("forwarding to backend")
val response: Future[HttpResponse] =
(IO(Http) ? Get("http:3080//localhost/backend")).mapTo[HttpResponse]
"<html><body><h1>api response after backend processing</h1></body></html>"
}
path("") {
get {
respondWithMediaType(`text/html`) { // XML is marshalled to `text/xml` by default, so we simply override here
complete(forward)
}
}
}
}
}
I'm working on a web scraper using this project (based on Scala, Spray, Akka and PhantomJS)
The problem is that I can't find a more specific example of how to use it, and the documentation is missing a lot of details
1- I would like to know how to give an specific URL so I can get data from it
2- How can I excecute, or pass a javascript file or function so that phantom can run and do some stuff(return specific data or whatever, from the site in point 1- )
Here is my Main.scala file: (Is almost the same as the one in the project)
package com.typesafe.webdriver.tester
import akka.actor.{ActorRef, ActorSystem}
import akka.pattern.ask
import com.typesafe.webdriver.{Session, PhantomJs, LocalBrowser}
import akka.util.Timeout
import scala.concurrent.duration._
import scala.concurrent.ExecutionContext.Implicits.global
import spray.json._
import spray.http._
object Main {
def main(args: Array[String]) {
implicit val system = ActorSystem("webdriver-system")
implicit val timeout = Timeout(5.seconds)
system.scheduler.scheduleOnce(7.seconds) {
system.shutdown()
System.exit(1)
}
val browser = system.actorOf(PhantomJs.props(system), "localBrowser")
browser ! LocalBrowser.Startup
for (
session <- (browser ? LocalBrowser.CreateSession).mapTo[ActorRef];
result <- (session ? Session.ExecuteNativeJs("return 5+5",JsArray(JsNumber(999)))).mapTo[JsNumber]
) yield {
println(result)
try {
system.shutdown()
System.exit(0)
} catch {
case _: Throwable =>
}
}
}
}
I would suggest you to use already created web scrappers in Scala.
For example ScalaWebDcraper which has nicely writted DSL and scrapping feature.
https://github.com/Rovak/ScalaWebscraper
It can be combined with Goose, which is a web article extractor. You can use it to fetch article data from the links you visit with the previous library.
https://github.com/jiminoc/goose
Also, checkout Metascrapper, a Scala Library for Scraping Page Metadata
https://beachape.com/blog/2013/09/05/introducing-metascraper-a-scala-library-for-scraping-page-metadata/
And check this question, lot's of valuable info inside.