I would like to uses jmeter for api functional testing, the jmeter dashboard reporting is not ideal for functional testing.
I have attempted to integrate extent 2.41.2 reporting with groovy script that validates responses (http and expected response code).
I have attempted to use the idea given in Using extentreports for jmeter test results
However that has failed. I used a js2322 assertion to check for valid responses, but then I get errors whenever attempt to run.
I'm not sure whether it should be setup as post processor step instead of an assertion.
Has anyone got any ideas on how this can be achieved?
You can assert result by using prev which is SampleResult:
prev - (SampleResult) - gives access to the previous SampleResult (if any)
Here's example of checking token exists in response and if not return relevant assertion:
import org.apache.jmeter.assertions.AssertionResult;
boolean assertToken = prev.getResponseDataAsString().contains("token");
prev.setSuccessful(assertToken);
if (!assertToken) {
AssertionResult assertionResult = new AssertionResult("Assertion expected to contain token")
assertionResult.setFailureMessage("Assertion failure message: Test failed: text expected to contain /token/");
assertionResult.setFailure(true);
prev.addAssertionResult(assertionResult);
}
Related
Our project passed “Turn on” execution and query in the test
suite of the google action.
But “Turn off” execution and query failed and I couldn’t find in our part.
After “Turn off” execution, I responded “false” for “on” attribute but
error report was shown - actual state: {"on":true,"online":true}.
Below is the exchanged data and the error report.
Google action request for execution:
{"inputs":[{"context":{"locale_country":"US","locale_language":"en"},"intent":"action.devices.EXECUTE","payload":{"commands":[{"devices":[{"id":"eg_cam_87901"}],"execution":[{"command":"action.devices.commands.OnOff","params":{"on":false}}]}]}}],"requestId":"960820806239759768"}
Our Response for execution:
{"requestId":"960820806239759768","payload":{"commands":[{"ids":["eg_cam_87901"],"status":"SUCCESS","state":{"online":false,"on":false,"currentToggleSettings":null}}]}}
Google action error report
Turn off the bedroom
AssertionError: Expected state to include: {"on":false}, actual state: {"on":true,"online":true}: expected false to be true
Google action request for query:
{"inputs":[{"intent":"action.devices.QUERY","payload":{"devices":[{"id":"eg_cam_87901"}]}}],"requestId":"4409204964431192716"}
Our Response for query:
{"requestId":"4409204964431192716","payload":{"devices":{"eg_cam_87901":{"online":false,"status":"SUCCESS","on":false,"currentToggleSettings":null}}}}
Google action error report
Query after 'Turn off the bedroom'
Error: the string "Error from HA: deviceOffline" was thrown, throw an Error :) at new f (js/app_compiled.js?f77b3b67-f1c5-4a38-8a20-fe5eafbc2c40:18143:10) at Runner.fail (js/app_compiled.js?f77b3b67-f1c5-4a38-8a20-fe5eafbc2c40:4491:11) at js/app_compiled.js?f77b3b67-f1c5-4a38-8a20-fe5eafbc2c40:4802:18 at done (js/app_compiled.js?f77b3b67-f1c5-4a38-8a20-fe5eafbc2c40:4189:5) at js/app_compiled.js?f77b3b67-f1c5-4a38-8a20-fe5eafbc2c40:4236:11 at w.Ic (js/app_compiled.js?f77b3b67-f1c5-4a38-8a20-fe5eafbc2c40:18137:24) at r.run (js/app_compiled.js?f77b3b67-f1c5-4a38-8a20-fe5eafbc2c40:18129:169) at js/app_compiled.js?f77b3b67-f1c5-4a38-8a20-fe5eafbc2c40:18171:310 at w.Jl (js/app_compiled.js?f77b3b67-f1c5-4a38-8a20-fe5eafbc2c40:18137:454) at r.Ms (js/app_compiled.js?f77b3b67-f1c5-4a38-8a20-fe5eafbc2c40:18130:301) at h (js/app_compiled.js?f77b3b67-f1c5-4a38-8a20-fe5eafbc2c40:18126:375) at y.Jl [as Ic] (js/app_compiled.js?f77b3b67-f1c5-4a38-8a20-fe5eafbc2c40:18139:326) at q (js/app_compiled.js?f77b3b67-f1c5-4a38-8a20-fe5eafbc2c40:18167:89) at e (js/app_compiled.js?f77b3b67-f1c5-4a38-8a20-fe5eafbc2c40:18158:235) at XMLHttpRequest.w (js/app_compiled.js?f77b3b67-f1c5-4a38-8a20-fe5eafbc2c40:18167:241)
You've got a couple issues here.
First, you need to verify that your sync response includes the appropriate device attributes - your execution/query responses both return NULL for your currentToggleSettings instead of the string of the availableToggles in your sync response. If you are just using OnOff for your device, you don't actually even need to implement any of the Toggle trait features. Again, verify what is in your initial sync response.
Second, you're returning online: false as well as on:false, which is why you're getting the following error:
Error: the string "Error from HA: deviceOffline" was thrown
online:false should only be used when you intend to return the TTS response of "the device is not available right now" when a user issues a command/query, and when the device cannot be controlled. This should also include status:ERRROR in your response. You can read more about handling errors and exceptions in the docs.
Returning online:true, on:false or online:true, on:true when responding to an execute/query request should fix the deviceOffline error.
I'm trying to validate whether error messages returned by API are proper or not.
So, I stored all local error message strings in HashMap errorMessage
.doIf(errorMessages.size()>1) {
exec(session => {
assert(ResponseJSON.contains(errorMessages.get("errorMessage1")))
for ((k,v)<- errorMessages){
assert(ResponseJSON.contains(v))
}
}
I could see the error on console as
hook-3' crashed with 'java.lang.AssertionError: assertion failed', forwarding to the next one
But, the Gatling scenarios are not failing here, what is I'm missing ?
Try using an exitHereIfFailed to exit the scenario.
I need to customise the error that shows up when someone did not send the CSRF code with the POST request.
So that no one will know what happened with the error and they will not even try to hack in to the CSRF mechanism.
Hope this is clear
For now Sails.js CSRF hook uses res.forbidden() function to handle wrong CSRF token.
It uses it with this message:
return res.forbidden("CSRF mismatch");
So you could rewrite this response by placing a new file with name forbidden.js into /api/responses
Actually you cound copy this one: https://github.com/balderdashy/sails/blob/master/lib/hooks/responses/defaults/forbidden.js
And add condition to check data before production mode check:
...
else sails.log.verbose('Sending 403 ("Forbidden") response');
if (data == 'CSRF mismatch') {
//Return another response for example:
return res.jsonx(500, {/* data here */});
}
// Only include errors in response if application environment
// is not set to 'production'. In production, we shouldn't
// send back any identifying information about errors.
if (sails.config.environment === 'production') {
...
Anyway as long as you will use development mode for sails. You will see all errors when getting 500 or any other error from sails. But in production mode all error messages will be hidden. And your users wouldn't get any error details. Except of error code.
So in production mode without any changes you will get only HTTP 403 code.
A Gatling scenario with an exec chain. After a request, returned data is saved. Later it's processed and depending on the processing result, it should either fail or pass the test.
This seems like the simplest possible scenario, yet I can't find any reliable info how to fail a test from within an exec block. assert breaks the scenario and seemingly Gatling (as in: the exception throw doesn't just fail the test).
Example:
// The scenario consists of a single test with two exec creating the execChain
val scn = scenario("MyAwesomeScenario").exec(reportableTest(
// Send the request
exec(http("127.0.0.1/Request").get(requestUrl).check(status.is(200)).check(bodyString.saveAs("MyData")
// Process the data
.exec(session => {
assert(processData(session.attributes("MyData")) == true, "Invalid data");
})
))
Above the scenario somewhere along the line "guardian failed, shutting down system".
Now this seems a useful, often-used thing to do - I'm possibly missing something simple. How to do it?
You have to abide by Gatling APIs.
With checks, you don't "fail" the test, but the request. If you're looking for failing the whole test, you should have a look at the Assertions API and the Jenkins plugin.
You can only perform a Check at the request site, not later. One of the very good reasons is that if you store the bodyString in the Sessions like you're doing, you'll end using a lot of memory and maybe crashing (still referenced, so not garbage collectable). You have to perform your processData in the check, typically in the transform optional step.
were you looking for something like
.exec(http("getRequest")
.get("/request/123")
.headers(headers)
.check(status.is(200))
.check(jsonPath("$.request_id").is("123")))
Since the edit queue is already full.
This is already resolved in the new version of Gatling. Release 3.4.0
They added
exitHereIf
exitHereIf("${myBoolean}")
exitHereIf(session => true)
Make the user exit the scenario from this point if the condition holds. Condition parameter is an Expression[Boolean].
I implemented something using exitHereIfFailed that sounds like exactly what you were trying to accomplish. I normally use this after a virtual user attempts to sign in.
exitHereIfFailed is used this way
val scn = scenario("MyAwesomeScenario")
.exec(http("Get data from endpoint 1")
.get(request1Url)
.check(status.is(200))
.check(bodyString.saveAs("MyData"))
.check(processData(session.attributes("MyData")).is(true)))
.exitHereIfFailed // If we weren't able to get the data, don't continue
.exec(http("Send the data to endpoint 2")
.post(request2Url)
.body(StringBody("${MyData}"))
This scenario will abort gracefully at exitHereIfFailed if any of the checks prior to exitHereIfFailed have failed.
When failing an authentication strategy in Warden, a message can be passed to the fail! method. How can I access this message in my default failure application action such that I can display it in a flash message? I tried request.env['warden'].message, but after looking at the code for Warden::Proxy this variable is only set for winning strategies. Thoughts?
When using the fail or fail! methods, you can obtain the failure message from the environment warden object.
Example:
def unauthenticated
flash[:alert] = env["warden"].message unless env["warden"].message.blank?
end
http://www.rubydoc.info/github/hassox/warden/Warden/Proxy#message-instance_method
Perhaps a different way of failing strategies (from recent documentation), is to use throw(:warden, :foo => 'bar'). Anything you throw with will be available in your failure application through request.env['warden.options'], including a string of the desired url as well. HTH