How to perform a simple json post with spray-json in spray? - scala

I'm trying to perform a simple json post with spray. But it seems that i can get an http entity for a json object that can be Marshall.
here is my error:
[error]
...../IdeaProjects/PoolpartyConnector/src/main/scala/org/iadb/poolpartyconnector/thesaurusoperation/ThesaurusCacheService.scala:172:
could not find implicit value for evidence parameter of type
spray.httpx.marshalling.Marshaller[spray.json.JsValue]
[error] val request =
Post(s"$thesaurusapiEndpoint/$coreProjectId/suggestFreeConcept?",
suggestionJsonBody)
and the code that comes with it:
override def createSuggestedFreeConcept(suggestedPrefLabel: String, lang: String, scheme: String, b: Boolean): String = {
import system.dispatcher
import spray.json._
val pipeline = addCredentials(BasicHttpCredentials("superadmin", "poolparty")) ~> sendReceive
val label = LanguageLiteral(suggestedPrefLabel, lang)
val suggestion = SuggestFreeConcept(List(label), b, Some(List(scheme)), None, None,None, None)
val suggestionJsonBody = suggestion.toJson
val request = Post(s"$thesaurusapiEndpoint/$coreProjectId/suggestFreeConcept?", suggestionJsonBody)
val res = pipeline(request)
getSuggestedFromFutureHttpResponse(res) match {
case None => ""
case Some(e) => e
}
}
Please, does any one has an idea of what is going on with the implicit marshaller. I though spray Json would come with implicit marshaller.

I assume you already have a custom Json Protocol somewhere so that suggestion.toJson works correctly?
Try the following:
val body = HttpEntity(`application/json`, suggestionJsonBody.prettyPrint)
val request = Post(s"$thesaurusapiEndpoint/$coreProjectId/suggestFreeConcept?", body)
you could also use compactPrint rather than prettyPrint, in either case, it turns the Json into a string containing the json information.

Here is how i solved it:
override def createSuggestedFreeConcepts(suggestedPrefLabels: List[LanguageLiteral], scheme: String, checkDuplicates: Boolean): List[String] = {
import system.dispatcher
import spray.httpx.marshalling._
import spray.httpx.SprayJsonSupport._
val pipeline = addCredentials(BasicHttpCredentials("superadmin", "poolparty")) ~> sendReceive
suggestedPrefLabels map { suggestedPrefLabel =>
val suggestion = SuggestFreeConcept(List(suggestedPrefLabel), checkDuplicates, Some(List(Uri(scheme))), None, None, None, None)
val request = Post(s"$thesaurusapiEndpoint/$coreProjectId/suggestFreeConcept", marshal(suggestion))
val res = pipeline(request)
getSuggestedFromFutureHttpResponse(res) match {
case None => ""
case Some(e) => e
}
}
}
the key is:
import spray.httpx.marshalling._ import spray.httpx.SprayJsonSupport._
and
val request =
Post(s"$thesaurusapiEndpoint/$coreProjectId/suggestFreeConcept",
marshal(suggestion))
I marshall suggestion. The explanation is not super super straightforward. But by fetching around in the doc, it is explained.

Related

json4s, how to deserialize json with FullTypeHints w/o explicitly setting TypeHints

I do specify FullTypeHints before deserialization
def serialize(definition: Definition): String = {
val hints = definition.tasks.map(_.getClass).groupBy(_.getName).values.map(_.head).toList
implicit val formats = Serialization.formats(FullTypeHints(hints))
writePretty(definition)
}
It produces json with type hints, great!
{
"name": "My definition",
"tasks": [
{
"jsonClass": "com.soft.RootTask",
"name": "Root"
}
]
}
Deserialization doesn't work, it ignores "jsonClass" field with type hint
def deserialize(jsonString: String): Definition = {
implicit val formats = DefaultFormats.withTypeHintFieldName("jsonClass")
read[Definition](jsonString)
}
Why should I repeat typeHints using Serialization.formats(FullTypeHints(hints)) for deserialization if hints are in json string?
Can json4s infer them from json?
The deserialiser is not ignoring the type hint field name, it just does not have anything to map it with. This is where the hints come in. Thus, you have to declare and assign your hints list object once again and pass it to the DefaultFormats object either by using the withHints method or by overriding the value when creating a new instance of DefaultFormats. Here's an example using the latter approach.
val hints = definition.tasks.map(_.getClass).groupBy(_.getName).values.map(_.head).toList
implicit val formats: Formats = new DefaultFormats {
outer =>
override val typeHintFieldName = "jsonClass"
override val typeHints = hints
}
I did it this way since I have contract:
withTypeHintFieldName is known in advance
withTypeHintFieldName contains fully qualified class name and it's always case class
def deserialize(jsonString: String): Definition = {
import org.json4s._
import org.json4s.native.JsonMethods._
import org.json4s.JsonDSL._
val json = parse(jsonString)
val classNames: List[String] = (json \\ $$definitionTypes$$ \\ classOf[JString])
val hints: List[Class[_]] = classNames.map(clz => Try(Class.forName(clz)).getOrElse(throw new RuntimeException(s"Can't get class for $clz")))
implicit val formats = Serialization.formats(FullTypeHints(hints)).withTypeHintFieldName($$definitionTypes$$)
read[Definition](jsonString)

akka stream integrating akka-htpp web request call into stream

Getting started with Akka Streams I want to perform a simple computation. Extending the basic QuickStart https://doc.akka.io/docs/akka/2.5/stream/stream-quickstart.html with a call to a restful web api:
val source: Source[Int, NotUsed] = Source(1 to 100)
source.runForeach(println)
already works nicely to print the numbers. But when trying to create an Actor to perform the HTTP request (is this actually necessary?) according to https://doc.akka.io/docs/akka/2.5.5/scala/stream/stream-integrations.html
import akka.pattern.ask
implicit val askTimeout = Timeout(5.seconds)
val words: Source[String, NotUsed] =
Source(List("hello", "hi"))
words
.mapAsync(parallelism = 5)(elem => (ref ? elem).mapTo[String])
// continue processing of the replies from the actor
.map(_.toLowerCase)
.runWith(Sink.ignore)
I cannot get it to compile as the ? operator is not defined. As ar as I know this one would only be defined inside an actor.
I also do not understand yet where exactly inside mapAsync my custom actor needs to be called.
edit
https://blog.colinbreck.com/backoff-and-retry-error-handling-for-akka-streams/ contains at least parts of an example.
It looks like it is not mandatory to create an actor i.e.
implicit val system = ActorSystem()
implicit val ec = system.dispatcher
implicit val materializer = ActorMaterializer()
val source = Source(List("232::03::14062::19965186", "232::03::14062::19965189"))
.map(cellKey => {
val splits = cellKey.split("::")
val mcc = splits(0)
val mnc = splits(1)
val lac = splits(2)
val ci = splits(3)
CellKeySource(cellKey, mcc, mnc, lac, ci)
})
.limit(2)
.mapAsyncUnordered(2)(ck => getResponse(ck.cellKey, ck.mobileCountryCode, ck.mobileNetworkCode, ck.locationArea, ck.cellKey)("<<myToken>>"))
def getResponse(cellKey: String, mobileCountryCode:String, mobileNetworkCode:String, locationArea:String, cellId:String)(token:String): Future[String] = {
RestartSource.withBackoff(
minBackoff = 10.milliseconds,
maxBackoff = 30.seconds,
randomFactor = 0.2,
maxRestarts = 2
) { () =>
val responseFuture: Future[HttpResponse] =
Http().singleRequest(HttpRequest(uri = s"https://www.googleapis.com/geolocation/v1/geolocate?key=${token}", entity = ByteString(
// TODO use proper JSON objects
s"""
|{
| "cellTowers": [
| "mobileCountryCode": $mobileCountryCode,
| "mobileNetworkCode": $mobileNetworkCode,
| "locationAreaCode": $locationArea,
| "cellId": $cellId,
| ]
|}
""".stripMargin)))
Source.fromFuture(responseFuture)
.mapAsync(parallelism = 1) {
case HttpResponse(StatusCodes.OK, _, entity, _) =>
Unmarshal(entity).to[String]
case HttpResponse(statusCode, _, _, _) =>
throw WebRequestException(statusCode.toString() )
}
}
.runWith(Sink.head)
.recover {
case _ => throw StreamFailedAfterMaxRetriesException()
}
}
val done: Future[Done] = source.runForeach(println)
done.onComplete(_ ⇒ system.terminate())
is already the (partial) answer for the question i.e. how to integrate Akka-streams + akka-http. However, it does not work, i.e. only throws error 400s and never terminates.
i think you already found an api how to call akka-http client
regarding your first code snippet which doesn't work. i think there happened some misunderstanding of the example itself. you expected the code in the example to work after just copied. but the intension of the doc was to demonstrate just an example/concept, how you can delegate some long running task out of the stream flow and then consuming the result when it's ready. for this was used ask call to akka actor, because call to ask method returns a Future. probably the authors of the doc just omitted the definition of actor. you can try this one example:
import java.lang.System.exit
import akka.NotUsed
import akka.actor.{Actor, ActorRef, ActorSystem, Props}
import akka.pattern.ask
import akka.stream.ActorMaterializer
import akka.stream.scaladsl.{Sink, Source}
import akka.util.Timeout
import scala.concurrent.ExecutionContext.Implicits.global
import scala.concurrent.duration._
import scala.language.higherKinds
object App extends scala.App {
implicit val sys: ActorSystem = ActorSystem()
implicit val mat: ActorMaterializer = ActorMaterializer()
val ref: ActorRef = sys.actorOf(Props[Translator])
implicit val askTimeout: Timeout = Timeout(5.seconds)
val words: Source[String, NotUsed] = Source(List("hello", "hi"))
words
.mapAsync(parallelism = 5)(elem => (ref ? elem).mapTo[String])
.map(_.toLowerCase)
.runWith(Sink.foreach(println))
.onComplete(t => {
println(s"finished: $t")
exit(1)
})
}
class Translator extends Actor {
override def receive: Receive = {
case msg => sender() ! s"$msg!"
}
}
You must import ask pattern from akka.
import akka.pattern.ask
Edit: OK, sorry, I can see that you have already imported. What is ref in your code? ActorRef?

Convert HttpEntity.Chunked to Array[String]

I have the following problem.
I am querying a server for some data and getting it back as HttpEntity.Chunked.
The response String looks like this with up to 10.000.000 lines like this:
[{"name":"param1","value":122343,"time":45435345},
{"name":"param2","value":243,"time":4325435},
......]
Now I want to get the incoming data into and Array[String] where each String is a line from the response, because later on it should be imported into an apache spark dataframe.
Currently I am doing it likes this:
//For the http request
trait StartHttpRequest {
implicit val system: ActorSystem
implicit val materializer: ActorMaterializer
def httpRequest(data: String, path: String, targetPort: Int, host: String): Future[HttpResponse] = {
val connectionFlow: Flow[HttpRequest, HttpResponse, Future[OutgoingConnection]] = {
Http().outgoingConnection(host, port = targetPort)
}
val responseFuture: Future[HttpResponse] =
Source.single(RequestBuilding.Post(uri = path, entity = HttpEntity(ContentTypes.`application/json`, data)))
.via(connectionFlow)
.runWith(Sink.head)
responseFuture
}
}
//result of the request
val responseFuture: Future[HttpResponse] = httpRequest(.....)
//convert to string
responseFuture.flatMap { response =>
response.status match {
case StatusCodes.OK =>
Unmarshal(response.entity).to[String]
}
}
//and then something like this, but with even more stupid stuff
responseFuture.onSuccess { str:String =>
masterActor! str.split("""\},\{""")
}
My question is, what would be a better way to get the result into an array?
How can I unmarshall the response entity directly? Because .to[Array[String]] for example did not work. And because there are so many lines coming, could I do it with a stream, to be more efficent?
Answering your questions out of order:
How can I unmarshall the response entity directly?
There is an existing question & answer related to unmarshalling an Array of case classes.
what would be a better way to get the result into an array?
I would take advantage of the Chunked nature and use streams. This allows you to do string processing and json parsing concurrently.
First you need a container class and parser:
case class Data(name : String, value : Int, time : Long)
object MyJsonProtocol extends DefaultJsonProtocol {
implicit val dataFormat = jsonFormat3(Data)
}
Then you have to do some manipulations to get the json objects to look right:
//Drops the '[' and the ']' characters
val dropArrayMarkers =
Flow[ByteString].map(_.filterNot(b => b == '['.toByte || b == ']'.toByte))
val preppendBrace =
Flow[String].map(s => if(!s.startsWith("{")) "{" + s else s)
val appendBrace =
Flow[String].map(s => if(!s.endsWith("}")) s + "}" else s)
val parseJson =
Flow[String].map(_.parseJson.convertTo[Data])
Finally, combine these Flows to convert a Source of ByteString into a Source of Data objects:
def strSourceToDataSource(source : Source[ByteString,_]) : Source[Data, _] =
source.via(dropArrayMarkers)
.via(Framing.delimiter(ByteString("},{"), 256, true))
.map(_.utf8String)
.via(prependBrace)
.via(appendBrace)
.via(parseJson)
This source can then be drained into an Seq of Data objects:
val dataSeq : Future[Seq[Data]] =
responseFuture flatMap { response =>
response.status match {
case StatusCodes.OK =>
strSourceToDataSource(response.entity.dataBytes).runWith(Sink.seq)
}
}

akka HttpResponse read body as String scala

So I have a function with this signature (akka.http.model.HttpResponse):
def apply(query: Seq[(String, String)], accept: String): HttpResponse
I simply get a value in a test like:
val resp = TagAPI(Seq.empty[(String, String)], api.acceptHeader)
I want to check its body in a test something like:
resp.entity.asString == "tags"
My question is how I can get the response body as string?
import akka.http.scaladsl.unmarshalling.Unmarshal
implicit val system = ActorSystem("System")
implicit val materializer = ActorFlowMaterializer()
val responseAsString: Future[String] = Unmarshal(entity).to[String]
Since Akka Http is streams based, the entity is streaming as well. If you really need the entire string at once, you can convert the incoming request into a Strict one:
This is done by using the toStrict(timeout: FiniteDuration)(mat: Materializer) API to collect the request into a strict entity within a given time limit (this is important since you don't want to "try to collect the entity forever" in case the incoming request does actually never end):
import akka.stream.ActorFlowMaterializer
import akka.actor.ActorSystem
implicit val system = ActorSystem("Sys") // your actor system, only 1 per app
implicit val materializer = ActorFlowMaterializer() // you must provide a materializer
import system.dispatcher
import scala.concurrent.duration._
val timeout = 300.millis
val bs: Future[ByteString] = entity.toStrict(timeout).map { _.data }
val s: Future[String] = bs.map(_.utf8String) // if you indeed need a `String`
You can also try this one also.
responseObject.entity.dataBytes.runFold(ByteString(""))(_ ++ _).map(_.utf8String) map println
Unmarshaller.stringUnmarshaller(someHttpEntity)
works like a charm, implicit materializer needed as well
Here is simple directive that extracts string from request's body
def withString(): Directive1[String] = {
extractStrictEntity(3.seconds).flatMap { entity =>
provide(entity.data.utf8String)
}
}
Unfortunately in my case, Unmarshal to String didn't work properly complaining on: Unsupported Content-Type, supported: application/json. That would be more elegant solution, but I had to use another way. In my test I used Future extracted from entity of the response and Await (from scala.concurrent) to get the result from the Future:
Put("/post/item", requestEntity) ~> route ~> check {
val responseContent: Future[Option[String]] =
response.entity.dataBytes.map(_.utf8String).runWith(Sink.lastOption)
val content: Option[String] = Await.result(responseContent, 10.seconds)
content.get should be(errorMessage)
response.status should be(StatusCodes.InternalServerError)
}
If you need to go through all lines in a response, you can use runForeach of Source:
response.entity.dataBytes.map(_.utf8String).runForeach(data => println(data))
Here is my working example,
import akka.actor.ActorSystem
import akka.http.scaladsl.Http
import akka.http.scaladsl.model._
import akka.stream.ActorMaterializer
import akka.util.ByteString
import scala.concurrent.Future
import scala.util.{ Failure, Success }
def getDataAkkaHTTP:Unit = {
implicit val system = ActorSystem()
implicit val materializer = ActorMaterializer()
// needed for the future flatMap/onComplete in the end
implicit val executionContext = system.dispatcher
val url = "http://localhost:8080/"
val responseFuture: Future[HttpResponse] = Http().singleRequest(HttpRequest(uri = url))
responseFuture.onComplete {
case Success(res) => {
val HttpResponse(statusCodes, headers, entity, _) = res
println(entity)
entity.dataBytes.runFold(ByteString(""))(_ ++ _).foreach (body => println(body.utf8String))
system.terminate()
}
case Failure(_) => sys.error("something wrong")
}
}

How do I set a non-standard User-Agent using spray-client?

I'm building an application for a Telco, using Scala and Akka, and need to communicate with Account Information and Refill servers using the UCIP protocol.
UCIP is a simple protocol, built on XMLRPC; the only issue I'm having is that it requires clients to set the User-Agent header in the specific format User-Agent: <client name>/<protocol version>/<client version>, which spray parses as invalid.
I tried creating a custom User-Agent header, inheriting from spray.http.HttpHeader but it still doesn't work. Here's what I've got so far:
import akka.actor.ActorSystem
import akka.event.{Logging, LoggingAdapter}
import spray.client.pipelining._
import spray.http._
import spray.httpx._
case class `User-Agent`(value: String) extends HttpHeader {
def lowercaseName: String = "user-agent"
def name: String = "User-Agent"
def render[R <: Rendering](r: R): r.type = r ~~ s"User-Agent: $value"
}
class UcipClient(val url: String, val protocol: String, username: String, password: String) (implicit system: ActorSystem) {
val log = Logging.getLogger(system, this)
val logRequest: HttpRequest => HttpRequest = { r => log.debug(r.toString); r }
val logResponse: HttpResponse => HttpResponse = { r => log.debug(r.toString); r }
val pipeline = (
addHeader(`User-Agent`("USSD-UCIP/%s/1.0".format(protocol)))
~> addCredentials(BasicHttpCredentials(username, password))
~> logRequest
~> sendReceive
~> logResponse
)
def send(req: UcipRequest) = pipeline(Post(url, req.getRequest))
}
My requests keep returning "Sorry, Error occured: 403, Invalid protocol version Not defined", however, they return the correct response when I send the same details using curl.
What am I missing, and is this even possible with spray-client? I've spent a fair bit of time checking the internets (which led me towards the custom header route), but still haven't figured this out...would really appreciate any help :-)
Turns out I wasn't far from the answer. While examining the headers being sent over the wire, I noticed the User-Agent was being set twice: once by my code, and again by Spray (because it considered my header invalid).
Setting the spray.can.client.user-agent-header to the empty string "" removed the second header, and requests were successful. Here's the final version of the custom header:
import spray.http._
object CustomHttpHeaders {
case class `User-Agent`(val value: String) extends HttpHeader with Product with Serializable {
def lowercaseName: String = "user-agent"
def name: String = "User-Agent"
def render[R <: Rendering](r: R): r.type = r ~~ s"User-Agent: $value"
}
}
And the final UCIP client:
import akka.actor.ActorRefFactory
import com.typesafe.config.Config
import scala.concurrent.ExecutionContext.Implicits.global
import scala.xml.NodeSeq
import spray.client.pipelining._
import spray.http._
import spray.httpx._
class UcipFault(val code: Int, msg: String) extends RuntimeException(s"$code: $msg")
class AirException(val code: Int) extends RuntimeException(s"$code")
class UcipClient(config: Config, val url: String)(implicit context: ActorRefFactory) {
import CustomHttpHeaders._
val throwOnFailure: NodeSeq => NodeSeq = {
case f if (f \\ "fault").size != 0 =>
val faultData = (f \\ "fault" \\ "member" \ "value")
throw new UcipFault((faultData \\ "i4").text.toInt,
(faultData \\ "string").text)
case el =>
val responseCode = ((el \\ "member")
.filter { n => (n \\ "name").text == "responseCode" }
.map { n => (n \\ "i4").text.toInt }).head
if (responseCode == 0) el else throw new AirException(responseCode)
}
val pipeline = (
addHeader(`User-Agent`("USSD-UCIP/%s/1.0".format(config.getString("ucip.server-protocol"))))
~> addCredentials(BasicHttpCredentials(config.getString("ucip.server-username"), config.getString("ucip.server-password")))
~> sendReceive
~> unmarshal[NodeSeq]
~> throwOnFailure
)
def send(req: UcipRequest) = pipeline(Post(url, req.getRequest))
}