How do I set a non-standard User-Agent using spray-client? - scala

I'm building an application for a Telco, using Scala and Akka, and need to communicate with Account Information and Refill servers using the UCIP protocol.
UCIP is a simple protocol, built on XMLRPC; the only issue I'm having is that it requires clients to set the User-Agent header in the specific format User-Agent: <client name>/<protocol version>/<client version>, which spray parses as invalid.
I tried creating a custom User-Agent header, inheriting from spray.http.HttpHeader but it still doesn't work. Here's what I've got so far:
import akka.actor.ActorSystem
import akka.event.{Logging, LoggingAdapter}
import spray.client.pipelining._
import spray.http._
import spray.httpx._
case class `User-Agent`(value: String) extends HttpHeader {
def lowercaseName: String = "user-agent"
def name: String = "User-Agent"
def render[R <: Rendering](r: R): r.type = r ~~ s"User-Agent: $value"
}
class UcipClient(val url: String, val protocol: String, username: String, password: String) (implicit system: ActorSystem) {
val log = Logging.getLogger(system, this)
val logRequest: HttpRequest => HttpRequest = { r => log.debug(r.toString); r }
val logResponse: HttpResponse => HttpResponse = { r => log.debug(r.toString); r }
val pipeline = (
addHeader(`User-Agent`("USSD-UCIP/%s/1.0".format(protocol)))
~> addCredentials(BasicHttpCredentials(username, password))
~> logRequest
~> sendReceive
~> logResponse
)
def send(req: UcipRequest) = pipeline(Post(url, req.getRequest))
}
My requests keep returning "Sorry, Error occured: 403, Invalid protocol version Not defined", however, they return the correct response when I send the same details using curl.
What am I missing, and is this even possible with spray-client? I've spent a fair bit of time checking the internets (which led me towards the custom header route), but still haven't figured this out...would really appreciate any help :-)

Turns out I wasn't far from the answer. While examining the headers being sent over the wire, I noticed the User-Agent was being set twice: once by my code, and again by Spray (because it considered my header invalid).
Setting the spray.can.client.user-agent-header to the empty string "" removed the second header, and requests were successful. Here's the final version of the custom header:
import spray.http._
object CustomHttpHeaders {
case class `User-Agent`(val value: String) extends HttpHeader with Product with Serializable {
def lowercaseName: String = "user-agent"
def name: String = "User-Agent"
def render[R <: Rendering](r: R): r.type = r ~~ s"User-Agent: $value"
}
}
And the final UCIP client:
import akka.actor.ActorRefFactory
import com.typesafe.config.Config
import scala.concurrent.ExecutionContext.Implicits.global
import scala.xml.NodeSeq
import spray.client.pipelining._
import spray.http._
import spray.httpx._
class UcipFault(val code: Int, msg: String) extends RuntimeException(s"$code: $msg")
class AirException(val code: Int) extends RuntimeException(s"$code")
class UcipClient(config: Config, val url: String)(implicit context: ActorRefFactory) {
import CustomHttpHeaders._
val throwOnFailure: NodeSeq => NodeSeq = {
case f if (f \\ "fault").size != 0 =>
val faultData = (f \\ "fault" \\ "member" \ "value")
throw new UcipFault((faultData \\ "i4").text.toInt,
(faultData \\ "string").text)
case el =>
val responseCode = ((el \\ "member")
.filter { n => (n \\ "name").text == "responseCode" }
.map { n => (n \\ "i4").text.toInt }).head
if (responseCode == 0) el else throw new AirException(responseCode)
}
val pipeline = (
addHeader(`User-Agent`("USSD-UCIP/%s/1.0".format(config.getString("ucip.server-protocol"))))
~> addCredentials(BasicHttpCredentials(config.getString("ucip.server-username"), config.getString("ucip.server-password")))
~> sendReceive
~> unmarshal[NodeSeq]
~> throwOnFailure
)
def send(req: UcipRequest) = pipeline(Post(url, req.getRequest))
}

Related

NNTP client using akka streams in scala

I'm trying to implement an NNTP client that streams a list of commands to the server and parsing the results back. I'm facing several problems :
the NNTP protocol doesn't have an "unique" delimiter that could be use to frame results. Some commands return multi-line responses. How to handle that with streams ?
how to "map" the command issued with the server response and wait the end of server response before sending the next command ? (Throttling is not relevant here)
how to stop the stream processing on disconnection ? (Actually, the program never returns)
Here is my current implementation :
import akka.stream._
import akka.stream.scaladsl._
import akka.{ NotUsed, Done }
import akka.actor.ActorSystem
import akka.util.ByteString
import scala.concurrent._
import scala.concurrent.duration._
import java.nio.file.Paths
import scala.io.StdIn
import scala.concurrent.ExecutionContext.Implicits.global
import scala.util.Success
import scala.util.Failure
object AutomatedClient extends App {
implicit val system = ActorSystem("NewsClientTest")
implicit val materializer = ActorMaterializer()
// MODEL //
final case class Command(query: String)
final case class CommandResult(
resultCode: Int,
resultStatus: String,
resultList: Option[List[String]])
final case class ParseException(message: String) extends RuntimeException
// COMMAND HANDLING FUN //
// out ->
val sendCommand: Command => ByteString = c => ByteString(c.query + "\r\n")
// in <-
val parseCommandResultStatus: String => (Int, String) = s =>
(s.take(3).toInt, s.drop(3).trim)
val parseCommandResultList: List[String] => List[String] = l =>
l.foldLeft(List().asInstanceOf[List[String]]){
case (acc, ".") => acc
case (acc, e) => e.trim :: acc
}.reverse
val parseCommandResult: ByteString => Future[CommandResult] = b => Future {
val resultLines = b.decodeString("UTF-8").split("\r\n")
resultLines.length match {
case 0 => throw new ParseException("empty result")
case 1 =>
val (code, text) = parseCommandResultStatus(resultLines.head)
new CommandResult(code, text, None)
case _ =>
val (code, text) = parseCommandResultStatus(resultLines.head)
new CommandResult(code, text, Some(parseCommandResultList(resultLines.tail.toList)))
}
}
// STREAMS //
// Flows
val outgoing: Flow[Command, ByteString, NotUsed] = Flow fromFunction sendCommand
val incoming: Flow[ByteString, Future[CommandResult], NotUsed] = Flow fromFunction parseCommandResult
val protocol = BidiFlow.fromFlows(incoming, outgoing)
// Sink
val print: Sink[Future[CommandResult], _] = Sink.foreach(f =>
f.onComplete {
case Success(r) => println(r)
case Failure(r) => println("error decoding command result")
})
// Source
val testSource: Source[Command, NotUsed] = Source(List(
new Command("help"),
new Command("list"),
new Command("quit")
))
val (host, port) = ("localhost", 1119)
Tcp()
.outgoingConnection(host, port)
.join(protocol)
.runWith(testSource, print)
}
And here is the result output :
CommandResult(200,news.localhost NNRP Service Ready - newsmaster#localhost (posting ok),None)
CommandResult(100,Legal Commands,Some(List(article [<messageid>|number], authinfo type value, body [<messageid>|number], date, group newsgroup, head [<messageid>|number], help, last, list [active wildmat|active.times|counts wildmat], list [overview.fmt|newsgroups wildmat], listgroup newsgroup, mode reader, next, post, stat [<messageid>|number], xhdr field [range], xover [range], xpat field range pattern, xfeature useragent <client identifier>, xfeature compress gzip [terminator], xzver [range], xzhdr field [range], quit, 480 Authentication Required*, 205 Goodbye)))
We can see that the second CommandResult contains the result of "list" command and "quit" command.

Akka flow for multiple http requests

In a project of mine I have an akka actor for sending post requests to my google fcm server. The actor takes a list of ids and should make as many requests as there are in the list. I print out the response from the server in runForeach(println(_)) but I only get one printout for a whole list of ids. Why does this happen?
class FCMActor(val key: String) extends Actor{
import fcm.FCMActor._
import akka.pattern.pipe
import context.dispatcher
private implicit def system: ActorSystem = ActorSystem()
final implicit val materializer: ActorMaterializer = ActorMaterializer(ActorMaterializerSettings(context.system))
def buildBody(id: Option[String]): String = {
Json.obj(
"to" -> id,
"priority" -> "high",
"data" -> Json.obj("message" -> "Firebase Clud Message"),
"time_to_live" -> 60
).toString()
}
def buildHttpRequest(body: String): HttpRequest = {
HttpRequest(method = HttpMethods.POST,
uri = s"/fcm/send",
entity = HttpEntity(MediaTypes.`application/json`, body),
headers = List(RawHeader("Authorization", s"key=$key")))
}
val connectionFlow: Flow[HttpRequest, HttpResponse, Future[Http.OutgoingConnection]] = {
Http().outgoingConnection("fcm.googleapis.com")
}
def send(ids: List[Option[String]]) = {
val httpRequests: List[HttpRequest] = ids.map(buildBody).map(buildHttpRequest)
println(httpRequests)
Source(httpRequests).via(connectionFlow).runForeach(println(_)) // << here I only get one println
}
override def receive: Receive = {
case SendToIds(ids: List[Option[String]]) =>
send(ids)
}
}
You are not consuming the response entity that the server sends you. To understand why this is important, check out the related docs page.
A quick code change to try and fix this is:
... .runForeach{ response =>
response.discardEntityBytes()
println(response)
}
Or, if you're actually interested in the entity, something along the lines of
... .runForeach{ _.entity.dataBytes
.runFold(ByteString.empty) { case (acc, b) => acc ++ b }
.map(println(_))
}

Akka Http Client Set Cookie on a HttpRequest

I am trying to make a GET request to a REST web service using Akka Http Client.
I am not able to figure out how do I set a cookie on the request before I make the GET.
I searched the web and I found ways to read the cookie on the server side. but I could not find anything which showed me how to set the cookie on the client side request.
Based on my own research I tried the following approach to set a cookie on http request
import akka.actor.ActorSystem
import akka.http.scaladsl.Http
import akka.http.scaladsl.model._
import akka.http.scaladsl.unmarshalling.Unmarshal
import akka.stream.scaladsl.{Sink, Source}
import akka.http.scaladsl.marshallers.sprayjson.SprayJsonSupport
import akka.http.scaladsl.model.headers.HttpCookie
import akka.stream.ActorMaterializer
import spray.json._
import scala.util.{Failure, Success}
case class Post(postId: Int, id: Int, name: String, email: String, body: String)
trait JsonSupport extends SprayJsonSupport with DefaultJsonProtocol {
implicit val postFormat = jsonFormat5(Post.apply)
}
object AkkaHttpClient extends JsonSupport{
def main(args: Array[String]) : Unit = {
val cookie = headers.`Set-Cookie`(HttpCookie(name="foo", value="bar"))
implicit val system = ActorSystem("my-Actor")
implicit val actorMaterializer = ActorMaterializer()
implicit val executionContext = system.dispatcher
val mycookie = HttpCookie(name="foo", value="bar")
val httpClient = Http().outgoingConnection(host = "jsonplaceholder.typicode.com")
val request = HttpRequest(uri = Uri("/comments"), headers = List(cookie))
val flow = Source.single(request)
.via(httpClient)
.mapAsync(1)(r => Unmarshal(r.entity).to[List[Post]])
.runWith(Sink.head)
flow.andThen {
case Success(list) => println(s"request succeded ${list.size}")
case Failure(_) => println("request failed")
}.andThen {
case _ => system.terminate()
}
}
}
But this gives an error
[WARN] [08/05/2016 10:50:11.134] [my-Actor-akka.actor.default-dispatcher-3] [akka.actor.ActorSystemImpl(my-Actor)]
HTTP header 'Set-Cookie: foo=bar' is not allowed in requests
The idiomatic way to construct any header for an akka-http client is by
using akka.http.scaladsl.model.headers.
In your case it would be
val cookieHeader = akka.http.scaladsl.model.headers.Cookie("name","value")
HttpRequest(uri = Uri("/comments"), headers = List(cookieHeader, ...))
The outgoing header must be 'Cookie' not 'Set-Cookie':
val cookie = HttpCookiePair("foo", "bar")
val headers: immutable.Seq[HttpHeader] = if (cookies.isEmpty) immutable.Seq.empty else immutable.Seq(Cookie(cookies))
val request = HttpRequest(uri = uri).withHeadersAndEntity(headers, HttpEntity(msg))

How to perform a simple json post with spray-json in spray?

I'm trying to perform a simple json post with spray. But it seems that i can get an http entity for a json object that can be Marshall.
here is my error:
[error]
...../IdeaProjects/PoolpartyConnector/src/main/scala/org/iadb/poolpartyconnector/thesaurusoperation/ThesaurusCacheService.scala:172:
could not find implicit value for evidence parameter of type
spray.httpx.marshalling.Marshaller[spray.json.JsValue]
[error] val request =
Post(s"$thesaurusapiEndpoint/$coreProjectId/suggestFreeConcept?",
suggestionJsonBody)
and the code that comes with it:
override def createSuggestedFreeConcept(suggestedPrefLabel: String, lang: String, scheme: String, b: Boolean): String = {
import system.dispatcher
import spray.json._
val pipeline = addCredentials(BasicHttpCredentials("superadmin", "poolparty")) ~> sendReceive
val label = LanguageLiteral(suggestedPrefLabel, lang)
val suggestion = SuggestFreeConcept(List(label), b, Some(List(scheme)), None, None,None, None)
val suggestionJsonBody = suggestion.toJson
val request = Post(s"$thesaurusapiEndpoint/$coreProjectId/suggestFreeConcept?", suggestionJsonBody)
val res = pipeline(request)
getSuggestedFromFutureHttpResponse(res) match {
case None => ""
case Some(e) => e
}
}
Please, does any one has an idea of what is going on with the implicit marshaller. I though spray Json would come with implicit marshaller.
I assume you already have a custom Json Protocol somewhere so that suggestion.toJson works correctly?
Try the following:
val body = HttpEntity(`application/json`, suggestionJsonBody.prettyPrint)
val request = Post(s"$thesaurusapiEndpoint/$coreProjectId/suggestFreeConcept?", body)
you could also use compactPrint rather than prettyPrint, in either case, it turns the Json into a string containing the json information.
Here is how i solved it:
override def createSuggestedFreeConcepts(suggestedPrefLabels: List[LanguageLiteral], scheme: String, checkDuplicates: Boolean): List[String] = {
import system.dispatcher
import spray.httpx.marshalling._
import spray.httpx.SprayJsonSupport._
val pipeline = addCredentials(BasicHttpCredentials("superadmin", "poolparty")) ~> sendReceive
suggestedPrefLabels map { suggestedPrefLabel =>
val suggestion = SuggestFreeConcept(List(suggestedPrefLabel), checkDuplicates, Some(List(Uri(scheme))), None, None, None, None)
val request = Post(s"$thesaurusapiEndpoint/$coreProjectId/suggestFreeConcept", marshal(suggestion))
val res = pipeline(request)
getSuggestedFromFutureHttpResponse(res) match {
case None => ""
case Some(e) => e
}
}
}
the key is:
import spray.httpx.marshalling._ import spray.httpx.SprayJsonSupport._
and
val request =
Post(s"$thesaurusapiEndpoint/$coreProjectId/suggestFreeConcept",
marshal(suggestion))
I marshall suggestion. The explanation is not super super straightforward. But by fetching around in the doc, it is explained.

How I can make a request with params from finagle client?

I’m getting started with a Finagle server (twitter/finagle):
import com.twitter.finagle.{Http, Service}
import com.twitter.util.{Await, Future}
import java.net.InetSocketAddress
import org.jboss.netty.handler.codec.http._
object Server extends App {
val service = new Service[HttpRequest, HttpResponse] {
def apply(req: HttpRequest): Future[HttpResponse] =
Future.value(new DefaultHttpResponse(
req.getProtocolVersion, HttpResponseStatus.OK))
}
val server = Http.serve(":8080", service)
Await.ready(server)
}
Client (twitter/finagle):
import com.twitter.finagle.{Http, Service}
import com.twitter.util.{Await, Future}
import java.net.InetSocketAddress
import org.jboss.netty.handler.codec.http._
object Client extends App {
val client: Service[HttpRequest, HttpResponse] =
Http.newService("localhost:8080")
val request = new DefaultHttpRequest(
HttpVersion.HTTP_1_1, HttpMethod.GET, "/")
val response: Future[HttpResponse] = client(request)
response onSuccess { resp: HttpResponse =>
println("GET success: " + resp)
}
Await.ready(response)
}
How do I send data like Map("data_id" -> 5) from the client to the server? And where in the server do I receive it? Do I have to add a callback to the server?
I haven’t found it by searching. If you can give me a link with an example, that will be enough.
Finagle is a very thin library. That means that you'll have to handle most of the "magic" by yourself.
To make the request with parameters from the Client, I use these helper methods:
def buildUri(base: String, path: String, params: Map[String, String] = Map.empty): String = {
val p = if (params.isEmpty) ""
else params map { case (k,v) => urlEncode(k) + "=" + urlEncode(v) } mkString ("?", "&", "")
base + path + p
}
def urlEncode(url: String): String = URLEncoder.encode(url, "UTF-8")
And then I call it like this:
val url = buildUri(baseAddress, path, defaultParams ++ params)
val req = RequestBuilder().url(url).setHeader("Accept", "*/*").buildGet
client(req)
As for the server you have to do basically the same thing and parse the parameters by hand. Either using java.net.URI or even org.jboss.netty.handler.codec.http.QueryStringDecoder.
Of course you can also use URI and QueryStringEncoder to encode as well, instead of using my helper methods.
That said, if you want to do that on higher level, you can use one of these libraries above Finagle:
https://github.com/fwbrasil/zoot
http://finatra.info/ (this is for the server part only)