We are currently in the process of identifying a suitable Automation Framework using JMeter for our RESTful APIs. A typical POST Request in our suite would be something as shown below :
URL : https://host123.com/createuser
Message body(JSON) :
{ "UserName" ,"Password","FirstName","LastName","PhoneNumber" }
There is an equivalent message body for XML as well
One framework we are interested in as shown :
The JSON/XML Repository would contain all the XML/JSON message bodies of every unique API Endpoint(We have close to 350 such unique API URLs).
The Test case repository would contain all the relevant tests containing data to be passed into the JSON/XMLs. One such example is shown below :
JMeter would run these tests and export the response to a file which would be parsed and presented graphically by another reporting plugin/utility.
Could you please let me know if the above data driven framework is suited for Automating RESTful Services ? Also if Jmeter is the ideal tool for performing these tests.
Not sure regarding the "ideal" but JMeter is definitely capable in helping you automating your scenario.
Some references:
CSV Data Set Config or JDBC Request Sampler (depending in what format you have your data) - to read from Test Case Repository
Beanshell Pre Processor or Sampler combined with i.e. GSON to dynamically construct JSON or XML test structures
XPath Assertion - to validate XML response from server
JSON Path Extractor and Assertion (available through JMeter Plugins
Hope this helps
D.
JMeter is a perfect solution for this.
If you want to automate running JMeter and graphing here are some solutions using Jenkins and the CLI:
https://blog.codecentric.de/en/2014/01/automating-jmeter-tests-maven-jenkins/
Need Step by Step Guide to execute the Jmeter Scripts in Jenkins (with Hudson build) over Ubuntu
Another option is a paid solution with http://BlazeMeter.com which is basically JMeter as a service. They have an API and a Jenkins Plugin as well. A lot simpler but not free.
Lastly, also look at the JMeter plugin project which has some great JMeter extras
http://jmeter-plugins.org/
I would use Staf/Stax to call XML test cases, run JMeter and collect the results.
Here is a very good article about this.
Related
Is it possible to use WSDL in rest assured, I'm looking for open source API automation testing tool. My services are in WSDL, I can use SOAPUI which will extract wsdl into separate end points. but Is there any way can we extract end in WSDL manually or else how I can use this with Karate or Rest Assured.
As the author of Karate, let me give you my point of view.
You don't need to worry about WSDL. All you need is a sample SOAP envelope as XML (plain-text) and you will be easily able to derive all your tests, complex scenarios and edge cases with that. This is what many teams are doing today, and you can refer this detailed set of examples to get a sense of the possibilities: xml.feature
Even if you have some complexities like encryption and signed-headers etc, you can easily plug them into your tests using Java-interop, look at this example in the documentation: https://github.com/intuit/karate#http-basic-authentication-example
I have a SOAP endpoint and will be having more than 1000 request messages which have different values for the request parameters but same operation of SOAP Message. I want to execute them in a sequence if the previous request that got triggered was 200 OK?
Is there any way to do this without JAVA program? Is there any client that will help me?
I assume you already have some sort of loop in your test case that reads your variable properties from a file or perhaps Excel and feeds them into your SOAP request. Ready API/soapUI Pro gives you this functionality, but for open source soapUI you'll have to write your own Groovy test steps.
Then, you can use a soapUI Compliance, Status and Standards assertion to check you've received a valid or invalid HTTP status code and react accordingly.
Is there any way to do this without JAVA program? Is there any client
that will help me?
After re-reading the question, it seems to me you're not yet using SoapUI, though it has been tagged as a SoapUI question. It happens quite a lot on here where people are askign general SOAP questions, but tag SoapUI. BTW, Craig's answer should be accepted if you are using SoapUI.
In terms of options, you have lots....
Code. You can use Python, C#, Java, Javascript, etc. etc. to create a program that will call your endpoint. Any programming language will have the libraries to call web services. So, if you do know a language, you could use that.
SoapUI. There is a free version, which will allow you to call web services. In your question, you want to call the same service over and over with different parameters. In testing speak, this is a data-driven test. These can be achieved in the free SoapUI, but it is a fiddle. However, the full-licensed version offers data-driven tests out of the box. I use these all the time. Very easy to set-up. If you use SoapUI, then Craig's answer about using Assertions would stop the test if you got a status code other than a 200.
Postman. this is another free tool, which I have used a little. I haven't tried data-driven tests, but I'm sure the docs will tell you if they're supported. If you try Postman, then you ought to look at Danny Dainton's excellent tutorial on GitHub
JMeter. Another free tool. This is primarily used for performance and load testing, but would still meet your needs.
I found many similar questions related to this, but not the particular answer I am looking for. Actually my requirement is little different, so I end up posting the following issue.
I want to automate Rest APIs, and I got two options for the same case.
The first one is Rest Assured and second one is Play Framework.
For example, to test this RestAPI:
http://servername:9000/dbs/all/list/m1/p1/sch1
(↑ This gives xml response)
So, I have written a code in Java with Rest assured, and is working fine. I integrate this with Maven project so that can be integrated with Jenkins.
Sample code:
import com.jayway.restassured
public class TestNGSimpleTest2 {
#Test
public void testApi() {
expect().
statusCode(200).
body("Status", equalTo("Su22ccess")).
when().
get("http://localhost:9000/dbs/all/list/m1/p1/sch1");
}
So my first question is:
Is the rest assured is the best tool to use?
Does Play framework is better?
I found many other tool like Jmeter, RightAPI etc. to test RestAPI. But I dont think this is automable. Am I right?
For automating REST API testing, as a starting point I recommend using Postman and newman.
Postman provides an excellent UI for building requests, and newman is its command-line counterpart. After you create a set of requests and corresponding tests within the Postman UI, you can run the entire collection from Jenkins via newman, preventing a deployment if tests fail.
The RestAssured code you have posted will work just fine for basic cases. It's not necessarily the "right tool" if you want to:
continuously add new test case and don't have many resources
propogate alerts with well-formed error messages (especially to places like Slack or GitHub)
reduce false-positives
re-use the same tests for monitoring
Building these features takes time and resources which, depending on the size of your team may or may not be a good call.
Some of the commercial solutions you posted can solve some of these problems for you.
Assertible is a codeless solution that supports the workflow you described directly: https://assertible.com/blog/automated-api-testing-with-jenkins
We can integrate Jenkins and JMeter for automating RestAPI testing.
The reason for that is,
In Jenkins we can schedule our tests/builds in whichever way(every minute/hour/day/month.....) or based on the commits etc..
we can bundle n number of APIs together in JMeter and execute in a single test.(Maintaining is easy)
There is a jenkins plugin "Performance" which can be used to check the response time for each API call, which compares the response time with respect to the previous response times.
JMeter has an in-build threading feature, which helps execute tests much faster than any single threaded tests.
Screenshots:
Steps
We can prepare our APIs in JMeter
Configure the tests in non GUI Mode in Jenkins.
Install and Configure the Performance plugin in Jenkins.
I'd like to use Gatling for REST performance and scalability web service testing. I'm currently using JMeter for this as I wasn't aware of gatling when I started this project. Gatling would integrate better and would be better for the project a number of reasons.
I'd like to ask one main question:
Obviously, there's a lot of overhead in configuring Gatling with the correct web service information. I've already done this in JMeter and I'm not keen to do it again. For one of the sub-projects, we have a WADL but we have no such thing for the other. Is it possible, out of the box, to import:
a. JMeter test plans and
b. WADL documents
into Gatling?
I've looked through the docs but unfortunately I can't find anything that references these.
No, Gatling has neither.
Building a jmx converter is something we might investigate in 2013, as you're not the first one to ask for it. At this point, I'm a bit skeptical, as the logic and the configuration of the 2 JMeter and Gatling are quite different, so the features and the way to use them don't map 1:1.
The easiest way to work with REST APIs is to use the recorder, so you'd dump request bodies as template and then inject data into them. See http://gatling.io/docs/2.1.6/http/http_request.html#request-body
If you work with JSON, you can use our JsonPath (or standard regex) checks in order to make assertions on the response body, or even capture data. See http://gatling.io/docs/2.1.6/http/http_check.html#defining-the-check-type
Using HttpSampler with Raw Post body And last 2.8 version is the right way to test Webservices.
Is it the way you are doing it ?
Upcoming 2.9 has new performance improvements related to memory and cpu consumed by Post Processors.
Regarding a, I don't think so.
I'm currently developping a REST Web service with Spring MVC.
And I am struggling to find the best way to do integration tests on my WS.
First solution : using rest-assured
Advantage : fluent api, really easy to use with its cool DSL
Drawback : when i perform POST or PUT requests on my WS, the state of my database is modified, and next tests are corrupted.
Second solution : unit test the controllers and perform integration tests at the service level separately
Advantage : i can control the state of my database, using Spring Test Framework and perform rollback after each test
Disadvantage : i do not perform end-to-end integration tests anymore.
Question : how can i use rest-assured to do integration tests without modifying the state of my database ?
Thanks a lot.
Why don't you delete the rest assured doubles and redirects before every test and set them up fresh for the test?
RestClient.delete "#{RestAssured::Server.address}/redirects/all"
RestClient.delete "#{RestAssured::Server.address}/doubles/all"
Or alternatively you can use different doubles for the GET and POST/PUT calls to the rest assured and use the redirects in between these calls.
I am not sure, your request makes sense as you state it.
RestAssured is just a framework to support you with testing. You can also write unit tests, that do the equivalent of PUT and DELETE (basically the internal implementations), which then modify the database state.
Or you can only issue HEAD and GET requests with RestAssured and not modify the database state by this.
Both of the options will only test parts of the code path if you leave any updates out, so your issue is orthogonal to the selection of RestAssured or hand written unit tests.
Of course you can mock your backend away, but either the mocks are trivial and you don't gain any insight. Or they are complex and you will need separate tests to assure that the mock objects to what you think they are doing.
In order to perform integration tests on a REST Spring MVC Web Service, the SpringSource team has provided a new library called spring-test-mvc, which is now integrate to spring-test.
http://blog.springsource.org/2012/11/12/spring-framework-3-2-rc1-spring-mvc-test-framework/
For my special purpose, it is more adapted than Rest-Assured.