As I read right from the Gatling documentation, foreach loop expects ChainBuilder as an executable expression.
foreach(sequence, elementName){chain} The chain is repeated for every
element in the sequence and the current element is stored in the
Session under the elementName key
However, if I want to use it inside resource method, it expects HttpRequestBuilders. Here's my usecase:
.exec(
http("Some name.")
.get("/api/call1?viewType=REGULAR")
.check(
jsonPath("$..reviewers[*]").ofType[Seq[Any]].saveAs("reviewers")
)
.resources(
foreach(session => session("reviewers").as[Seq[Any]], "reviewer") {
http("Request resource.")
.get("/api/users/${reviewer}/photo")
.headers(resourceRequestHeaders)
}
)
)
That's not possible as of Gatling 3.3 (and upcoming 3.4). Number of requests in the resources can only be static.
Related
I'm still pretty new to Gatling / Scala, so my apologies if I've misunderstood something obvious, but...
I have a scenario with a sequence of requests. One of them is along the lines of:
.exec (
http("Get IDs")
.post(<urlpath>)
.body(<body text>)
.headers(<headerinfo>)
.check(jsonPath("$[*].some.json.path").findAll.transform(_.map(_.replace("unwantedchars,""))).saveAs(myIds)
)
And that works fine, returning a vector of all matching json elements, with the unwanted chars removed. What I'm trying to do next is loop over the first 5 ids, and pass them into the next request. I've tried assorted variants of the following, but no amount of variation / googling has returned the actual solution:
.exec( session => {
val firstFive = session("myIds").as[Vector[String]].toArray.take(5)
for (inx <- 0 until 4){
exec(
http("get the item")
.get("/path/to/item/thing/" + firstFive(inx))
.headers(<etc etc>)
)
session
})
So I understand nested exec's aren't supported - but how can I create a block that combines the for-loop, and the actual HTTP requests?
Am I missing something obvious?
if you're looking to take 5, then you just need to put them in the session and then process them with a .foreach block.
so instead of taking the first five and trying to make exec calls (which, as you observed, won't work), save the list back to the session
.exec( session => {
val firstFive = session("myIds").as[Vector[String]].take(5)
session.set("firstFive", firstFive)
)
then use the DSL to loop through the collection
.foreach("${firstFive}", "id") {
exec(
http("get the item")
.get("/path/to/item/thing/${id}")
.headers(<etc etc>)
)
}
you could even add the selection of the first five to your transform step - so the whole thing could be...
.exec (
http("Get IDs")
.post(<urlpath>)
.body(<body text>)
.headers(<headerinfo>) .check(jsonPath("$[*].some.json.path").findAll.transform(_.map(_.replace("unwantedchars,"")).take(5).saveAs(myIds)
)
.foreach("${myIds}", "id") {
exec(
http("get the item")
.get("/path/to/item/thing/${id}")
.headers(<etc etc>)
)
}
So I have a scenario that works perfectly, its defined as follows:
val basicLoginScenario = createScenario(Config.testName, feeder.random,
setSessionParams(PARAM1, Config.param1),
setSessionParams(PARAM2, Config.param2),
setSessionParams(PARAM3, Config.param3),
setSessionParams(PARAM4, Config.param4),
exec(RestApi.userLogin))
exec(RestApi.transaction1))
exec(RestApi.transaction2)))
But when I surround it with exitBlockOnFail, I am getting the following error and it seems to happen before any HTTP request is sent or any request/response JSON is being parsed.
[GatlingSystem-akka.actor.default-dispatcher-4] ERROR io.gatling.http.action.HttpRequestAction - 'httpRequest-5' failed to execute: No attribute named 'cookie' is defined
This is the code with exitBlockOnFail:
val basicLoginScenario = createScenario(Config.testName, feeder.random,
exitBlockOnFail{
setSessionParams(PARAM1, Config.param1)
setSessionParams(PARAM2, Config.param2)
setSessionParams(PARAM3, Config.param3)
setSessionParams(PARAM4, Config.param4)
exec(RestApi.userLogin))
exec(RestApi.transaction1))
exec(RestApi.transaction2))
})
Note that the "cookie" parameter is being fetched from the userLogin transaction and is not used anywhere before it is fetched in this scenario, obviously not in setSessionParam which is:
def setSessionParams(key: String, value: Any) = {
exec(_.set(key, value))
}
Here is the userLogin transaction:
val userLogin = {
exec(http("Login")
.post("/login")
.body(ElFileBody("json/Login.json")).asJson
.check(jsonPath("$.result.cookie").saveAs("cookie")))
}
My feeder doesn't have a "cookie" parameter in it and the Login.json doesn't have a "cookie" parameter assigned in it, it only returns it. As I said at the beginning, the scenario works perfectly - the issue only occurs when i surround my transactions with exitBlockOnFail. Any idea what might cause it?
Your initial version works because 'exec' can take a varargs of execs whereas 'exitBlockOnFail' takes a chain. So when you supply several execs to 'exitBlockOnFail', only the last action is being executed.
so you can either wrap all the statements in an 'exec'
exitBlockOnFail{
exec(
setSessionParams(PARAM1, Config.param1),
...
exec(RestApi.transaction2)
)
}
or chain them
exitBlockOnFail{
setSessionParams(PARAM1, Config.param1)
.setSessionParams(PARAM1, Config.param1)
.setSessionParams(PARAM2, Config.param2)
.setSessionParams(PARAM3, Config.param3)
.setSessionParams(PARAM4, Config.param4)
.exec(RestApi.userLogin)
.exec(RestApi.transaction1)
.exec(RestApi.transaction2)
}
I have to query an API with pagination. But because the API is weird I have to query it until the response is empty (contains no items).
Actually it is done in Java with a do-while :
do {
objects = doQuery(from);
from += 200
} while ( !objects.isEmpty() )
But I would like to convert it in scala. My first idea is to use a stream with a step and takeWhile :
Stream.iterate(0)(_+200)
.map( from => doQuery(from) )
.takeWhile( objects -> !objects.isEmpty )
But another changes make doQuery returns a Future. Thus I cannot do the test in takeWhile and have no best idea on how to do it (maybe a recursive call).
Hopefully this new code will be into an Akka actor that will tell another for each object (no need to return anything)
This gives you Stream[Future[(Boolean, T)]], where the first time will be false as soon as the rest will be empty. Not sure how to do the takeWhile without blocking.
Stream.iterate(0)(_+200)
.scanLeft(Future((true, List[Int]())))({case (prev, from) =>
prev.flatMap({case (cont, _) =>
if(cont) {
doQuery(from).map(res => (res.isEmpty, res.toList))
} else {
Future((false, List()))
}
})
})
I would like to chain http request using akka-http-client as Stream. Each http request in a chain depends on a success/response of a previous requests and uses it to construct a new request. If a request is not successful, the Stream should return the response of the unsuccessful request.
How can I construct such a stream in akka-http?
which akka-http client level API should I use?
If you're making a web crawler, have a look at this post. This answer tackles a more simple case, such as downloading paginated resources, where the link to the next page is in a header of the current page response.
You can create a chained source - where one item leads to the next - using the Source.unfoldAsync method. This takes a function which takes an element S and returns Future[Option[(S, E)]] to determine if the stream should continue emitting elements of type E, passing the state to the next invocation.
In your case, this is kind of like:
taking an initial HttpRequest
producing a Future[HttpResponse]
if the response points to another URL, returning Some(request -> response), otherwise None
However, there's a wrinkle, which is that this will not emit a response from the stream if it doesn't contain a pointer to the next request.
To get around this, you can make the function passed to unfoldAsync return Future[Option[(Option[HttpRequest], HttpResponse)]]. This allows you to handle the following situations:
the current response is an error
the current response points to another request
the current response doesn't point to another request
What follows is some annotated code which outlines this approach, but first a preliminary:
When streaming HTTP requests to responses in Akka streams, you need to ensure that the response body is consumed otherwise bad things will happen (deadlocks and the like.) If you don't need the body you can ignore it, but here we use a function to convert the HttpEntity from a (potential) stream into a strict entity:
import scala.concurrent.duration._
def convertToStrict(r: HttpResponse): Future[HttpResponse] =
r.entity.toStrict(10.minutes).map(e => r.withEntity(e))
Next, a couple of functions to create an Option[HttpRequest] from an HttpResponse. This example uses a scheme like Github's pagination links, where the Links header contains, e.g: <https://api.github.com/...> rel="next":
def nextUri(r: HttpResponse): Seq[Uri] = for {
linkHeader <- r.header[Link].toSeq
value <- linkHeader.values
params <- value.params if params.key == "rel" && params.value() == "next"
} yield value.uri
def getNextRequest(r: HttpResponse): Option[HttpRequest] =
nextUri(r).headOption.map(next => HttpRequest(HttpMethods.GET, next))
Next, the real function we'll pass to unfoldAsync. It uses the Akka HTTP Http().singleRequest() API to take an HttpRequest and produce a Future[HttpResponse]:
def chainRequests(reqOption: Option[HttpRequest]): Future[Option[(Option[HttpRequest], HttpResponse)]] =
reqOption match {
case Some(req) => Http().singleRequest(req).flatMap { response =>
// handle the error case. Here we just return the errored response
// with no next item.
if (response.status.isFailure()) Future.successful(Some(None -> response))
// Otherwise, convert the response to a strict response by
// taking up the body and looking for a next request.
else convertToStrict(response).map { strictResponse =>
getNextRequest(strictResponse) match {
// If we have no next request, return Some containing an
// empty state, but the current value
case None => Some(None -> strictResponse)
// Otherwise, pass on the request...
case next => Some(next -> strictResponse)
}
}
}
// Finally, there's no next request, end the stream by
// returning none as the state.
case None => Future.successful(None)
}
Note that if we get an errored response, the stream will not continue since we return None in the next state.
You can invoke this to get a stream of HttpResponse objects like so:
val initialRequest = HttpRequest(HttpMethods.GET, "http://www.my-url.com")
Source.unfoldAsync[Option[HttpRequest], HttpResponse](
Some(initialRequest)(chainRequests)
As for returning the value of the last (or errored) response, you simply need to use Sink.last, since the stream will end either when it completes successfully or on the first errored response. For example:
def getStatus: Future[StatusCode] = Source.unfoldAsync[Option[HttpRequest], HttpResponse](
Some(initialRequest))(chainRequests)
.map(_.status)
.runWith(Sink.last)
I've come across the following code in a Gatling scenario (modified for brevity/privacy):
val scn = scenario("X")
.repeat(numberOfLoops, "loopName") {
exec((session : Session) => {
val loopCounter = session.getTypedAttribute[Int]("loopName")
session.setAttribute("xmlInput", createXml(loopCounter))
})
.exec(
http("X")
.post("/rest/url")
.headers(headers)
.body("${xmlInput}"))
)
}
It's naming the loop in the repeat block, getting that out of the session and using it to create a unique input XML. It then sticks that XML back into the session and extracts it again when posting it.
I would like to do away with the need to name the loop iterator and accessing the session.
Ideally I'd like to use a Stream to generate the XML.
But Gatling controls the looping and I can't recurse. Do I need to compromise, or can I use Gatling in a functional way (without vars or accessing the session)?
As I see it, neither numberOfLoops nor createXml seem to depend on anything user related that would have been stored in the session, so the loop could be resolved at build time, not at runtime.
import com.excilys.ebi.gatling.core.structure.ChainBuilder
def addXmlPost(chain: ChainBuilder, i: Int) =
chain.exec(
http("X")
.post("/rest/url")
.headers(headers)
.body(createXml(i))
)
def addXmlPostLoop(chain: ChainBuilder): ChainBuilder =
(0 until numberOfLoops).foldLeft(chain)(addXmlPost)
Cheers,
Stéphane
PS: The preferred way to ask something about Gatling is our Google Group: https://groups.google.com/forum/#!forum/gatling