I am trying to create my gRPC payload by reading data from a csv file(it has guestID and category as columns).
I followed the example here https://github.com/phiSgr/gatling-grpc/blob/244ab372da6773102d79c65a7e4086f409d3fe94/src/test/scala/com/github/phisgr/example/GrpcExample.scala but I see a type mismatch error when I try the same. (it expects Seq[ContextKey] but here i am able to form Seq[Expression[ContextKey]])
val scn2: ScenarioBuilder = scenario("gRPC call - 50 users repeated 100 times")
.feed(csv("testtext.csv"))
.exec(
grpc("gRPC request with test message")
.rpc(RecommenderGrpc.METHOD_GET_RECOMMENDATIONS)
.payload(RequestContext.of(Map("test" -> "test"),
Seq(ContextKey.defaultInstance.updateExpr(
_.id :~ $("guestID"),
_.`type` :~ Type.GUEST
), ContextKey.defaultInstance.updateExpr(
_.id :~ $("category"),
_.`type` :~ Type.CATEGORY
)),
Seq())
)
)
(payload is a RequestContext object which takes in metadata, keys and items. metadata is a map, keys is a Seq of ContextKey and items is empty Seq. ContextKey contains string guestID or category and type).
How to use the variables in the feeder here?
Skip to the bottom for the solution.
Expression[T] is an alias for Session => Validation[T]. In plain English, that is a function that constructs the payload from the session with a possibility of failure.
You can consider an Expression[T], abstractly, "contains" a T.
Like how a Promise in JavaScript "contains" a future value. You cannot give a Promise of T to a function that expects a T. If one wants to transform or combine Promises, that code has to be turned inside out, and supplied as an argument to .then.1
aPromise + 1 // wrong
aPromise.then(a => a + 1)
This is the same reason why your code sample does not compile.
Users of Gatling are not necessarily familiar with Scala, or functional programming in general. It will be counterproductive to make them understand this "wrapping" stuff.2 So there are code that help you combine the Expressions.
For HTTP and other untyped stuff, the EL string is parsed and transformed in to an Expression under the hood.
Protobuf messages are strongly typed, the payload cannot be easily constructed using string interpolation. So the :~ operators on lenses are used to handle the plumbing so that you do not have to manually handle the Expression wrapping.
But now you have a function, RequestContext.of, that constructs the payload. The lens magic cannot help if you need that function. You have to write the Expression lambda yourself.3
.payload { session =>
for {
guestId <- session("guestId").validate[String]
category <- session("category").validate[String]
} yield RequestContext.of(
Map("test" -> "test"),
Seq(
ContextKey(id = guestID, `type` = Type.GUEST),
ContextKey(id = category, `type` = Type.CATEGORY)
),
Seq()
)
}
Needless to say this is very cumbersome, and people now use async-await with Promises.
An Expression is just the Either monad and the Reader monad stacked together, what's the problem?
I may be able to write a version with lenses if I know what RequestContext.of does.
Related
I am new to Scala and would like to learn the idiomatic way to solve common problems, as in pythonic for Python. My question regards reading JSON data with upickle, where the JSON value contains a string when present, and null when not present. I want to use a custom value to replace null. A simple example:
import upickle.default._
val jsonString = """[{"always": "foo", "sometimes": "bar"}, {"always": "baz", "sometimes": null}]"""
val jsonData = ujson.read(jsonString)
for (m <- jsonData.arr) {
println(m("always").str.length) // this will work
println(m("sometimes").str.length) // this will fail, Exception in thread "main" ujson.Value$InvalidData: Expected ujson.Str (data: null)
}
The issue is with the field "sometimes": when null, we cannot apply .str (or any other function mapping to a static type other than null). I am looking for something like m("sometimes").str("DEFAULT").length, where "DEFAULT" is the replacement for null.
Idea 1
Using pattern matching, the following works:
val sometimes = m("sometimes") match {
case s: ujson.Str => s.str
case _ => "DEFAULT"
}
println(sometimes.length)
Given Scala's concise syntax, this looks a bit complicated and will be repetitive when done for a number of values.
Idea 2
Answers to a related question mention creating a case class with default values. For my problem, the creation of a case class seems inflexible to me when different replacement values are needed depending depending on context.
Idea 3
Anwers to another question (not specific to upickle) discuss using Try().getOrElse(), i.e.:
import scala.util.Try
// ...
println(Try(m("sometimes").str).getOrElse("DEFAULT").length)
However, the discussion mentions that throwing an exception for a regular program path is expensive.
What are idiomatic, yet concise ways to solve this?
Idiomatic or scala way to do this by using scala's Option.
Fortunately, upickle Values offers them. Refer strOpt method in this source code.
Your problem in code is str methods in m("always").str and m("sometimes").str
With this code, you are prematurely assuming that all the values are strings. That's where the strOpt method comes. It either outputs a string if its value is a string or a None type if it not. And we can use getOrElse method coupled with it to decide what to throw if the value is None.
Following would be the optimum way to handle this.
val jsonString = """[{"always": "foo", "sometimes": "bar"}, {"always": "baz", "sometimes": null}]"""
for (m <- jsonData.arr) {
println(m("always").strOpt.getOrElse("").length)
println(m("sometimes").strOpt.getOrElse("").length)
}
Output:
3
3
3
0
Here if we get any value other than a string (null, float, int), the code will output it as an empty string. And its length will be calculated as 0.
Basically, this is similar to your "Idea1" approach but this is the scala way. Instead of "DEFAULT", I am throwing an empty string because you wouldn't want to have null values' length to be 7 (Length of string "DEFAULT").
As a personal project, I am writing yet another Scala library for DynamoDb. It contains many interesting aspect such as reading and writing from an AST (just as Json), handling HTTP request, streaming data…
In order to be able able to communicate with DynamoDb, one needs to be able to read from / to the DynamoDb format (the “AST”). I extracted this reading / writing from / to the AST in a minimalist library: dynamo-ast. It contains two main type classes: DynamoReads[_] and DynamoWrites[_] (deeply inspired from Play Json).
I successfully coded the reading part of the library ending with a very simple code such as :
trait DynamoRead[A] { self =>
def read(dynamoType: DynamoType): DynamoReadResult[A]
}
case class TinyImage(url: String, alt: String)
val dynamoReads: DynamoReads[TinyImage] = {
for {
url <- read[String].at(“url”)
alt <- read[String].at(“alt”)
} yield (url, alt) map (TinyImage.apply _).tupled
}
dynamoReads.reads(dynamoAst) //yield DynamoReadResult[TinyImage]
At that point, I thought I wrote the most complicated part of the library and the DynamoWrite[_] part would be a piece of cake. I am however stuck on writing the DynamoWrite part. I was a fool.
My goal is to provide a very similar “user experience” with the DynamoWrite[_] and keep it as simple as possible such as :
val dynamoWrites: DynamoWrites[TinyImage] = {
for {
url <- write[String].at(“url”)
alt <- write[String].at(“alt”)
} yield (url, alt) map (TinyImage.unapply _) //I am not sure what to yield here nor how to code it
}
dynamoWrites.write(TinyImage(“http://fake.url”, “The alt desc”)) //yield DynamoWriteResult[DynamoType]
Since this library is deeply inspired from Play Json library (because I like its simplicity) I had a look at the sources several times. I kind of dislike the way the writer part is coded because to me, it adds a lot of overhead (basically each time a field a written, a new JsObject is created with one field and the resulting JsObject for a complete class is the merge of all the JsObjects containing one field).
From my understanding, the DynamoReads part can be written with only one trait (DynamoRead[_]). The DynamoWrites part however requires at least two such as :
trait DynamoWrites[A] {
def write(a: A): DynamoWriteResult[DynamoType]
}
trait DynamoWritesPath[A] {
def write(path:String, a: A): DynamoWriteResult[(String, DynamoType)]
}
The DynamoWrites[_] is to write plain String, Int… and the DynamoWritesPath[_] is to write a tuple of (String, WhateverTypeHere) (to simulate a “field”).
So writing write[String].at(“url”) would yield a DynamoWritesPath[String]. Now I have several issues :
I have no clue how to write flatMap for my DynamoWritesPath[_]
what should yield a for comprehension to be able to obtain a DynamoWrite[TinyImage]
What I wrote so far (totally fuzzy and not compiling at all, looking for some help on this). Not committed at the moment (gist): https://gist.github.com/louis-forite/cad97cc0a47847b2e4177192d9dbc3ae
To sum up, I am looking for some guidance on how to write the DynamoWrites[_] part. My goal is to provide for the client the most straight forward way to code a DynamoWrites[_] for a given type. My non goal is to write the perfect library and keep it a zero dependency library.
Link to the library: https://github.com/louis-forite/dynamo-ast
A Reads is a covariant functor. That means it has map. It can also be seen as a Monad which means it has flatMap (although a monad is overkill unless you need the previous field in order to know how to process the next):
trait Reads[A] {
def map [B] (f: A => B): Reads[B]
def flatMap [B](f: A => Reads[B]): Reads[B] // not necessary, but available
}
The reason for this, is that to transform a Reads[Int] to a Reads[String], you need to first read the Int, then apply the Int => String function.
But a Writes is a contravariant functor. It has contramap where the direction of the types is reversed:
trait Writes[A] {
def contramap [B](f: B => A): Reads[B]
}
The type on the function is reversed because to transform a Writes[Int] to a Writes[String] you must receive the String from the caller, apply the transformation String => Int and then write the Int.
I don't think it makes sense to provide for-comprehension syntax (flatMap) for the Writes API.
// here it is clear that you're extracting a string value
url <- read[String].at(“url”)
// but what does this mean for the write method?
url <- write[String].at("url")
// what is `url`?
That's probably why play doesn't provide one either, and why they focus on their combinator syntax (using the and function, their version of applicative functor builder?).
For reference: http://blog.tmorris.net/posts/functors-and-things-using-scala/index.html
You can achieve a more consistent API by using something like the and method in play json:
(write[String]("url") and write[String]("alt"))(unlift(TinyImage.unapply))
(read[String]("url") and read[String]("alt"))(TinyImage.apply)
// unfortunately, the type ascription is necessary in this case
(write[String]("url") and write[String]("alt")) {(x: TinyImage) =>
(x.url, x.alt)
}
// transforming
val instantDynamoType: DynamoFormat[Instant] =
format[String].xmap(Instant.parse _)((_: Instant).toString)
You can still use for-comprehension for the reads, although it's a bit over-powered (sort of implies that fields must be processed in-sequence, while that's not technically necessary).
I'm interoperating with some Java code that uses iterator-like functionality, and presumes you will continue to test it's .next for null values. I want to put it into immutable Scala data structures to reason about it with functional programming.
Right now, I'm filling mutable data structures and then converting them to immutable data structures. I know there's a more functional way to do this.
How can I refactor the code below to populate the immutable data structures without using intermediate mutable collections?
Thanks
{
val sentences = MutableList[Seq[Token]]()
while(this.next() != null){
val sentence = MutableList[Token]()
var token = this.next()
while(token.next != null){
sentence += token
token = token.next
}
sentences += sentence.to[Seq]
}
sentences.to[Seq]
}
You might try to use the Iterator.iterate method in order to simulate a real iterator, and then use standard collection methods like takeWhile and toSeq. I'm not totally clear on the type of this and next, but something along these lines might work:
Iterator.iterate(this){ _.next }.takeWhile{ _ != null }.map { sentence =>
Iterator.iterate(sentence.next) { _.next }.takeWhile{ _ != null }.toSeq
}.toSeq
You can also extend Iterable by defining your own next and hasNext method in order to use these standard methods more easily. You might even define an implicit or explicit conversion from these Java types to this new Iterable type – this is the same pattern you see in JavaConversions and JavaConverters
I have a simple Spray scenario that doesn't work as I've expected.
I have a spray server that dispatches work to different modules, and then return their composed responses.
As I don't want to limit Module responses, I chose that the Module's method return ToResponseMarshallable, as I have Modules that need to return a plain String.
Module signature could be:
def moduleProcessing(): ToResponseMarshallable = randomString()
And my "complete" block look similar to this:
complete {
val response1 = moduleProcessing()
val response2 = moduleProcessing()
Seq(response1,response2)
}
In the previous example, I would expect to get:
[{"someRandomString"},{"anotherRandomString"}]
But I am getting:
[{},{}]
Of course it will propagate as expected, if I return a single response or if I change the signature of the moduleProcessing return type to any Marshallable Type.
Thanks in advance for your help!
I think it's strange that your moduleProcessing method returns directly a ToResponseMarshallable[T]. The spray way would be to have this method return a T and have an in scope Marshaller[T] for when you complete requests by providing a T.
Finally, I don't think Marshaller.of[Seq[T]] is a built in marshaller in spray.
This means you'll need to provide your own Seq marshaller. This could delegate the actual marshalling of each item in the seq to an implicit Marshaller[T] and concatenate the results. Many builtin spray marshallers require implicit Marshaller[T] values like this. Alternatively, if you want a simple Seq[T] to string marshaller, try something like the following:
implicit val CustomMarshaller = Marshaller.delegate[Seq[T], String](`text/html`)(_.mkString("{", ",", "}"))
I have something like this:
case HttpRequest(POST, Uri.Path("/userAction"), headers, entity: HttpEntity.NonEmpty, protocol) =>
val x = entity.as[spray.http.FormData].merge.asInstanceOf[spray.http.FormData].fields
sender ! HttpResponse(entity="Got it: " + x)
It seems a little bit inconvinient to unwrap the HttpEntity, is there a much more elegant way?
Furthermore: is it possible to get the data from inputs with multiple delacred variables? Is there a better way to access FormData, e.g. as Json or HashMap?
Is there an actual reason why you are implementing this with spray-can? In routing it's more easier and natural:
lazy val yourRoute: Route = {
(post & path("userAction")) {
entity(as[SomeType]) { data =>
// do what you want
}
}
}
In this version it all comes to the proper unmarshaller for your type (SomeType). But if you want to stick with spray-can, you can use spray-httpx unmarshaller. I don't have IDE near to me, but this can like this (actual code from my project with some Scalaz tricks):
case request # HttpRequest(...) =>
fromTryCatch(request.entity |> unmarshal[SomeType]) // returns Throwable \/ SomeType, like scala's Either type
Here it also comes to the right unmarshaller, e.g if you want to get your entity in Json format (i.e spray JsObject), then you need to emit a request with json payload, write entity(as[JsObject]) and provide an implicit unmarshaller from spray.json.SprayJsonSupport. I think that using this approach for FormData is a bit overhead, cause for simple Post payload you have formFields directive, and you can write:
lazy val yourRoute: Route = {
(post & path("userAction")) {
formFields(...) { fields => // just enumerate field names from your FormData
// do what you want
}
}
}
For Map approach if the same, just add spray.json.DefaultJsonProtocol into the scope, to add unmarshaller for maps.
Still using spray-routing DSL is far better then dealing with low-level can code. If you still want to use, then take a look at spray.httpx.unmarshalling package, it contains different types of unmarshallers, for entity, request and response.