How I can make a request with params from finagle client? - scala

I’m getting started with a Finagle server (twitter/finagle):
import com.twitter.finagle.{Http, Service}
import com.twitter.util.{Await, Future}
import java.net.InetSocketAddress
import org.jboss.netty.handler.codec.http._
object Server extends App {
val service = new Service[HttpRequest, HttpResponse] {
def apply(req: HttpRequest): Future[HttpResponse] =
Future.value(new DefaultHttpResponse(
req.getProtocolVersion, HttpResponseStatus.OK))
}
val server = Http.serve(":8080", service)
Await.ready(server)
}
Client (twitter/finagle):
import com.twitter.finagle.{Http, Service}
import com.twitter.util.{Await, Future}
import java.net.InetSocketAddress
import org.jboss.netty.handler.codec.http._
object Client extends App {
val client: Service[HttpRequest, HttpResponse] =
Http.newService("localhost:8080")
val request = new DefaultHttpRequest(
HttpVersion.HTTP_1_1, HttpMethod.GET, "/")
val response: Future[HttpResponse] = client(request)
response onSuccess { resp: HttpResponse =>
println("GET success: " + resp)
}
Await.ready(response)
}
How do I send data like Map("data_id" -> 5) from the client to the server? And where in the server do I receive it? Do I have to add a callback to the server?
I haven’t found it by searching. If you can give me a link with an example, that will be enough.

Finagle is a very thin library. That means that you'll have to handle most of the "magic" by yourself.
To make the request with parameters from the Client, I use these helper methods:
def buildUri(base: String, path: String, params: Map[String, String] = Map.empty): String = {
val p = if (params.isEmpty) ""
else params map { case (k,v) => urlEncode(k) + "=" + urlEncode(v) } mkString ("?", "&", "")
base + path + p
}
def urlEncode(url: String): String = URLEncoder.encode(url, "UTF-8")
And then I call it like this:
val url = buildUri(baseAddress, path, defaultParams ++ params)
val req = RequestBuilder().url(url).setHeader("Accept", "*/*").buildGet
client(req)
As for the server you have to do basically the same thing and parse the parameters by hand. Either using java.net.URI or even org.jboss.netty.handler.codec.http.QueryStringDecoder.
Of course you can also use URI and QueryStringEncoder to encode as well, instead of using my helper methods.
That said, if you want to do that on higher level, you can use one of these libraries above Finagle:
https://github.com/fwbrasil/zoot
http://finatra.info/ (this is for the server part only)

Related

Route akka-http request through a proxy

I am rewriting some application layer code in scala from using scalaj to akka-http
in order to reduce the number of third party dependencies in the project (we already use akka for other things in the same project.) The code simply wraps common types of request to an underlying general request provided by the library
Mostly it has been fine, but I am stuck on the problem of optionally adding a proxy to a request.
Requests should either be direct to the destination or via a proxy, determined by a parameter at runtime.
In my scalaj implementation, I have the following helper class and methods
object HttpUtils {
private def request(
host: Host,
method: HttpMethod,
params: Map[String, String],
postData: Option[String],
timeout: Duration,
headers: Seq[(String, String)],
proxy: Option[ProxyConfig]
): HttpResponse[String] = {
// most general request builder. Other methods in the object fill in parameters and wrap this in a Future
val baseRequest = Http(host.url)
val proxiedRequest = addProxy(proxy, baseRequest)
val fullRequest = addPostData(postData)(proxiedRequest)
.method(method.toString)
.params(params)
.headers(headers)
.option(HttpOptions.connTimeout(timeout.toMillis.toInt))
.option(HttpOptions.readTimeout(timeout.toMillis.toInt))
fullRequest.asString // scalaj for send off request and block until response
}
// Other methods ...
private def addProxy(proxy: Option[ProxyConfig], request: HttpRequest): HttpRequest =
proxy.fold(request)((p: ProxyConfig) => request.proxy(p.host, p.port))
}
case class ProxyConfig(host: String, port: Int)
Is there a way to build a similar construct with akka-http?
Akka HTTP does have proxy support that, as of version 10.0.9, is still unstable. Keeping in mind that the API could change, you could do something like the following to handle optional proxy settings:
import java.net.InetSocketAddress
import akka.actor.ActorSystem
import akka.stream.ActorMaterializer
import akka.http.scaladsl.{ClientTransport, Http}
implicit val system = ActorSystem()
implicit val materializer = ActorMaterializer()
case class ProxyConfig(host: String, port: Int)
val proxyConfig = Option(ProxyConfig("localhost", 8888))
val clientTransport =
proxyConfig.map(p => ClientTransport.httpsProxy(InetSocketAddress.createUnresolved(p.host, p.port)))
.getOrElse(ClientTransport.TCP)
val settings = ConnectionPoolSettings(system).withTransport(clientTransport)
Http().singleRequest(HttpRequest(uri = "https://google.com"), settings = settings)
In Akka Http 10.2.0, use bindflow for a Flow[HttpRequest, HttpResponse, NotUsed] defined by a RunnableGraph with Flowshape. Insided the RunnableGraph, an Http() outgoingConnection is used to connect to the remote proxy. Some example code:
import akka.actor.typed.ActorSystem
import akka.actor.typed.scaladsl.Behaviors
import akka.http.scaladsl.Http
import akka.http.scaladsl.model.{HttpRequest, HttpResponse}
import akka.stream._
import akka.stream.scaladsl.{Broadcast, Flow, GraphDSL, Merge}
import scala.concurrent.ExecutionContextExecutor
import scala.concurrent.duration.DurationInt
import scala.io.StdIn
import scala.util.{Failure, Success}
object Main {
def main(args: Array[String]) {
implicit val system: ActorSystem[Nothing] = ActorSystem(Behaviors.empty, "testproxy")
implicit val executionContext: ExecutionContextExecutor = system.executionContext
system.log.info("TestAkkaHttpProxy Main started...")
val remoteHost = "xxx.xxx.xxx.x"
val remotePort = 8000
val proxyHost = "0.0.0.0"
val proxyPort = 8080
val gateway = Flow.fromGraph(GraphDSL.create() { implicit b =>
import GraphDSL.Implicits._
// Broadcast for flow input
val broadcaster = b.add(Broadcast[HttpRequest](1))
// Merge for flow output
val responseMerge = b.add(Merge[HttpResponse](1))
// outgoing client for remote proxy
val remote = Http().outgoingConnection(remoteHost, remotePort)
// filter out header that creates Akka Http warning
val requestConvert = Flow[HttpRequest]
.map(req => { req.mapHeaders(headers => headers.filter(h => h.isNot("timeout-access")))
})
// connect graph
broadcaster.out(0) ~> requestConvert ~> remote ~> responseMerge
// expose ports
FlowShape(broadcaster.in, responseMerge.out)
})
// Akka Http server that binds to Flow (for remote proxy)
Http().newServerAt(proxyHost, proxyPort).bindFlow(gateway)
.onComplete({
case Success(binding) ⇒
println(s"Server is listening on 0.0.0.0:8080")
binding.addToCoordinatedShutdown(hardTerminationDeadline = 10.seconds)
case Failure(e) ⇒
println(s"Binding failed with ${e.getMessage}")
system.terminate()
})
system.log.info("Press RETURN to stop...")
StdIn.readLine()
system.terminate()
}
}

Akka Http Client Set Cookie on a HttpRequest

I am trying to make a GET request to a REST web service using Akka Http Client.
I am not able to figure out how do I set a cookie on the request before I make the GET.
I searched the web and I found ways to read the cookie on the server side. but I could not find anything which showed me how to set the cookie on the client side request.
Based on my own research I tried the following approach to set a cookie on http request
import akka.actor.ActorSystem
import akka.http.scaladsl.Http
import akka.http.scaladsl.model._
import akka.http.scaladsl.unmarshalling.Unmarshal
import akka.stream.scaladsl.{Sink, Source}
import akka.http.scaladsl.marshallers.sprayjson.SprayJsonSupport
import akka.http.scaladsl.model.headers.HttpCookie
import akka.stream.ActorMaterializer
import spray.json._
import scala.util.{Failure, Success}
case class Post(postId: Int, id: Int, name: String, email: String, body: String)
trait JsonSupport extends SprayJsonSupport with DefaultJsonProtocol {
implicit val postFormat = jsonFormat5(Post.apply)
}
object AkkaHttpClient extends JsonSupport{
def main(args: Array[String]) : Unit = {
val cookie = headers.`Set-Cookie`(HttpCookie(name="foo", value="bar"))
implicit val system = ActorSystem("my-Actor")
implicit val actorMaterializer = ActorMaterializer()
implicit val executionContext = system.dispatcher
val mycookie = HttpCookie(name="foo", value="bar")
val httpClient = Http().outgoingConnection(host = "jsonplaceholder.typicode.com")
val request = HttpRequest(uri = Uri("/comments"), headers = List(cookie))
val flow = Source.single(request)
.via(httpClient)
.mapAsync(1)(r => Unmarshal(r.entity).to[List[Post]])
.runWith(Sink.head)
flow.andThen {
case Success(list) => println(s"request succeded ${list.size}")
case Failure(_) => println("request failed")
}.andThen {
case _ => system.terminate()
}
}
}
But this gives an error
[WARN] [08/05/2016 10:50:11.134] [my-Actor-akka.actor.default-dispatcher-3] [akka.actor.ActorSystemImpl(my-Actor)]
HTTP header 'Set-Cookie: foo=bar' is not allowed in requests
The idiomatic way to construct any header for an akka-http client is by
using akka.http.scaladsl.model.headers.
In your case it would be
val cookieHeader = akka.http.scaladsl.model.headers.Cookie("name","value")
HttpRequest(uri = Uri("/comments"), headers = List(cookieHeader, ...))
The outgoing header must be 'Cookie' not 'Set-Cookie':
val cookie = HttpCookiePair("foo", "bar")
val headers: immutable.Seq[HttpHeader] = if (cookies.isEmpty) immutable.Seq.empty else immutable.Seq(Cookie(cookies))
val request = HttpRequest(uri = uri).withHeadersAndEntity(headers, HttpEntity(msg))

Why akka http is not emiting responses for first N requests?

I'm trying to use akka-http in order to make http requests to a single host (e.g. "akka.io"). The problem is that the created flow (Http().cachedHostConnectionPool) starts emitting responses only after N http requests are made, where N is equal to max-connections.
import scala.util.Failure
import scala.util.Success
import com.typesafe.config.ConfigFactory
import akka.actor.ActorSystem
import akka.http.scaladsl.Http
import akka.http.scaladsl.model.HttpRequest
import akka.http.scaladsl.model.Uri.apply
import akka.http.scaladsl.settings.ConnectionPoolSettings
import akka.stream.ActorMaterializer
import akka.stream.scaladsl.Sink
import akka.stream.scaladsl.Source
object ConnectionPoolExample extends App {
implicit val system = ActorSystem()
implicit val executor = system.dispatcher
implicit val materializer = ActorMaterializer()
val config = ConfigFactory.load()
val connectionPoolSettings = ConnectionPoolSettings(config).withMaxConnections(10)
lazy val poolClientFlow = Http().cachedHostConnectionPool[Unit]("akka.io", 80, connectionPoolSettings)
val fakeSource = Source.fromIterator[Unit] { () => Iterator.continually { Thread.sleep(1000); () } }
val requests = fakeSource.map { _ => println("Creating request"); HttpRequest(uri = "/") -> (()) }
val responses = requests.via(poolClientFlow)
responses.runForeach {
case (tryResponse, jsonData) =>
tryResponse match {
case Success(httpResponse) =>
httpResponse.entity.dataBytes.runWith(Sink.ignore)
println(s"status: ${httpResponse.status}")
case Failure(e) => {
println(e)
}
}
}
}
The output looks like this:
Creating request
Creating request
Creating request
Creating request
Creating request
Creating request
Creating request
Creating request
Creating request
Creating request
status: 200 OK
Creating request
status: 200 OK
Creating request
status: 200 OK
...
I am failing to find any configuration parameters which would allow emitting responses as soon as they are ready and not when the pool is out of free connections.
Thanks!
The reason is that you block the client from doing other work by calling Thread.sleep—that method is simply forbidden inside reactive programs. The proper and simpler approach is to use Source.tick.

How do I set a non-standard User-Agent using spray-client?

I'm building an application for a Telco, using Scala and Akka, and need to communicate with Account Information and Refill servers using the UCIP protocol.
UCIP is a simple protocol, built on XMLRPC; the only issue I'm having is that it requires clients to set the User-Agent header in the specific format User-Agent: <client name>/<protocol version>/<client version>, which spray parses as invalid.
I tried creating a custom User-Agent header, inheriting from spray.http.HttpHeader but it still doesn't work. Here's what I've got so far:
import akka.actor.ActorSystem
import akka.event.{Logging, LoggingAdapter}
import spray.client.pipelining._
import spray.http._
import spray.httpx._
case class `User-Agent`(value: String) extends HttpHeader {
def lowercaseName: String = "user-agent"
def name: String = "User-Agent"
def render[R <: Rendering](r: R): r.type = r ~~ s"User-Agent: $value"
}
class UcipClient(val url: String, val protocol: String, username: String, password: String) (implicit system: ActorSystem) {
val log = Logging.getLogger(system, this)
val logRequest: HttpRequest => HttpRequest = { r => log.debug(r.toString); r }
val logResponse: HttpResponse => HttpResponse = { r => log.debug(r.toString); r }
val pipeline = (
addHeader(`User-Agent`("USSD-UCIP/%s/1.0".format(protocol)))
~> addCredentials(BasicHttpCredentials(username, password))
~> logRequest
~> sendReceive
~> logResponse
)
def send(req: UcipRequest) = pipeline(Post(url, req.getRequest))
}
My requests keep returning "Sorry, Error occured: 403, Invalid protocol version Not defined", however, they return the correct response when I send the same details using curl.
What am I missing, and is this even possible with spray-client? I've spent a fair bit of time checking the internets (which led me towards the custom header route), but still haven't figured this out...would really appreciate any help :-)
Turns out I wasn't far from the answer. While examining the headers being sent over the wire, I noticed the User-Agent was being set twice: once by my code, and again by Spray (because it considered my header invalid).
Setting the spray.can.client.user-agent-header to the empty string "" removed the second header, and requests were successful. Here's the final version of the custom header:
import spray.http._
object CustomHttpHeaders {
case class `User-Agent`(val value: String) extends HttpHeader with Product with Serializable {
def lowercaseName: String = "user-agent"
def name: String = "User-Agent"
def render[R <: Rendering](r: R): r.type = r ~~ s"User-Agent: $value"
}
}
And the final UCIP client:
import akka.actor.ActorRefFactory
import com.typesafe.config.Config
import scala.concurrent.ExecutionContext.Implicits.global
import scala.xml.NodeSeq
import spray.client.pipelining._
import spray.http._
import spray.httpx._
class UcipFault(val code: Int, msg: String) extends RuntimeException(s"$code: $msg")
class AirException(val code: Int) extends RuntimeException(s"$code")
class UcipClient(config: Config, val url: String)(implicit context: ActorRefFactory) {
import CustomHttpHeaders._
val throwOnFailure: NodeSeq => NodeSeq = {
case f if (f \\ "fault").size != 0 =>
val faultData = (f \\ "fault" \\ "member" \ "value")
throw new UcipFault((faultData \\ "i4").text.toInt,
(faultData \\ "string").text)
case el =>
val responseCode = ((el \\ "member")
.filter { n => (n \\ "name").text == "responseCode" }
.map { n => (n \\ "i4").text.toInt }).head
if (responseCode == 0) el else throw new AirException(responseCode)
}
val pipeline = (
addHeader(`User-Agent`("USSD-UCIP/%s/1.0".format(config.getString("ucip.server-protocol"))))
~> addCredentials(BasicHttpCredentials(config.getString("ucip.server-username"), config.getString("ucip.server-password")))
~> sendReceive
~> unmarshal[NodeSeq]
~> throwOnFailure
)
def send(req: UcipRequest) = pipeline(Post(url, req.getRequest))
}

Stubbing SOAP requests in Scala

I use scalaxb to generate models and client part of the SOAP interface. For testing I use Betamax, which can also be used in Scala. However, scalaxb uses Netty as a transport, which ignores proxy settings set up by Betamax. How would you cope with this situation?
scalaxb uses cake pattern, so the service is built from 3 parts like in the following example:
import scala.concurrent.ExecutionContext.Implicits.global
import scala.concurrent._
import scala.concurrent.duration._
val service = (new stockquote.StockQuoteSoap12Bindings with
scalaxb.SoapClientsAsync with
scalaxb.DispatchHttpClientsAsync {}).service
val fresponse = service.getQuote(Some("GOOG"))
val response = Await.result(fresponse, 5 seconds)
println(response)
And tests:
import co.freeside.betamax.{TapeMode, Recorder}
import co.freeside.betamax.proxy.jetty.ProxyServer
import dispatch._
import org.scalatest.{Tag, FunSuite}
import scala.concurrent.duration._
import scala.concurrent.{Await, Future}
class StockquoteSpec extends FunSuite with Betamax {
testWithBetamax("stockquote", Some(TapeMode.READ_WRITE))("stockquote") {
val fresponse = service.getQuote(Some("GOOG"))
val response = Await.result(fresponse, 5 seconds)
println(response)
}
}
trait Betamax {
protected def test(testName: String, testTags: Tag*)(testFun: => Unit)
def testWithBetamax(tape: String, mode: Option[TapeMode] = None)(testName: String, testTags: Tag*)(testFun: => Unit) = {
test(testName, testTags: _*) {
val recorder = new Recorder
val proxyServer = new ProxyServer(recorder)
recorder.insertTape(tape)
recorder.getTape.setMode(mode.getOrElse(recorder.getDefaultMode()))
proxyServer.start()
try {
testFun
} finally {
recorder.ejectTape()
proxyServer.stop()
}
}
}
}
Versions:
net.databinder.dispatch 0.11.2
co.freeside.betamax 1.1.2
com.ning.async-http-client 1.8.10
io.netty.netty 3.9.2.Final
It is indeed possible to use proxy with Netty. Although Netty does not read system properties for proxy settings, the settings can be injected using ProxyServerSelector. It is created in build method of AsyncHttpClientConfig:
if (proxyServerSelector == null && useProxySelector) {
proxyServerSelector = ProxyUtils.getJdkDefaultProxyServerSelector();
}
if (proxyServerSelector == null && useProxyProperties) {
proxyServerSelector = ProxyUtils.createProxyServerSelector(System.getProperties());
}
if (proxyServerSelector == null) {
proxyServerSelector = ProxyServerSelector.NO_PROXY_SELECTOR;
}
The only obstacle is that scalaxb uses default config with useProxyProperties=false. You can override it with custom MyDispatchHttpClientsAsync that you can use when creating the service:
val service = (new stockquote.StockQuoteSoap12Bindings with
scalaxb.SoapClientsAsync with
MyDispatchHttpClientsAsync {}).service
And the source code of MyDispatchHttpClientsAsync (the key point is calling setUseProxyProperties(true)):
import com.ning.http.client.providers.netty.NettyAsyncHttpProvider
import com.ning.http.client.{AsyncHttpClientConfig, AsyncHttpClient}
import scalaxb.HttpClientsAsync
/**
* #author miso
*/
trait MyDispatchHttpClientsAsync extends HttpClientsAsync {
lazy val httpClient = new DispatchHttpClient {}
trait DispatchHttpClient extends HttpClient {
import dispatch._, Defaults._
// Keep it lazy. See https://github.com/eed3si9n/scalaxb/pull/279
lazy val http = new Http(new AsyncHttpClient(new NettyAsyncHttpProvider(new AsyncHttpClientConfig.Builder().setUseProxyProperties(true).build())))
// lazy val http = Http.configure(_.setUseProxyProperties(true)) // Maybe later. See https://github.com/eed3si9n/scalaxb/issues/312
def request(in: String, address: java.net.URI, headers: Map[String, String]): concurrent.Future[String] = {
val req = url(address.toString).setBodyEncoding("UTF-8") <:< headers << in
http(req > as.String)
}
}
}