We have 2 fairly large automation projects going, both using BDD. One is in Lettuce for a desktop app, the other is for a website using JBehave (we are just getting started with the web project).
We have tried using Thucydides for reporting for our JBehave project, and started implementing tests using that. However, we ran into Allure and it looks a lot better and lets us use standard JBehave framework without relying on someone's code that has its own unknown-to-us issues. Luckily, we found Allure early enough.
2 questions:
1) We spent 2 days trying to make Allure work with JBehave, but the only example on GitHub isn't working well (all scenarios are reported together without breakdown by individual stories or scenarios). Also, JBehave doesn't have an #AfterStep decorator and it's a requirement for us to save screenshots after each step, successful or not. Thucydides for all its faults took care of that. Does Allure have something similar? If not, then at least is there a working example of how to make it report stories and scenarios correctly when run from JBehave?
2) I haven't tried yet, but doesn't look like there is an adapter for Lettuce (Python). Can someone recommend a way to produce Allure reports from Lettuce?
Thanks a lot!!
Allure doesn't support JBehave and Lettuce yet. But you can implement such adapters by yourself.
First step you need to read the following section https://github.com/allure-framework/allure-core/wiki#development in documentation. Then if you are ready to contribute you should follow the following instructions:
JBehave
We already have Java adaptor. So all you need is add allure-java-adaptor-api module as dependency and then implement JBehave listener.
Lettuce
There are the same. You can use allure-python bindings and all you need is implement Lettuce handlers. Python team are going to move bindings (aka allure-python-adaptor-api) to separate module, you can force it by comment in https://github.com/allure-framework/allure-python/issues/63
So, if you have any questions/suggestions you can also use our gitter chat room (https://gitter.im/allure-framework/allure-core) or our mailing list (allure#yandex-team.ru)
Hope it helps.
To achieve integration between JBehave and Allure you can create your own implementation of org.jbehave.core.reporters.StoryReporter. From the methods in this interface you can fire the Allure events that correspondent to the JBehave abstractions. In our implementation we fire TestSuite*Events from e.g. StoryReporter#beforeStory() and TestCase*Events from e.g.StoryReporter#afterScenario().
The caveat is that for some JBehave events you have to fire multiple Allure events. For example for a failed jbehave step we fire the following allure events:
public class AllureStoryReporter implements StoryReporter {
...
#Override
public void notPerformed(final String step) {
getLifecycle().fire(new StepStartedEvent(step).withTitle(step));
getLifecycle().fire(new StepCanceledEvent());
getLifecycle().fire(new StepFinishedEvent());
}
}
Of course the created reporter needs to be registered to be used during reporting by JBehave.
This results in comprehensive Allure reports.
At least for JBehave, the support was added since then
Related
I am working on protractor for testing an angularjs application. I am also able to fetch the report but I want to mention some more points and details about the test execution. For example Its model name, Test case name, Severity, Priority, Where the test failed if it gets fails etc. Where should I add all this points so that I can be able to fetch a complete detailed report.Currently I am able to get the report I have attached here.
Please help me in getting the solution as I am new to protractor. Thanks a lot in advance.
Jasmine framework does specs reporting, not Protractor in e2e testing. You can either leverage some of the popular ones listed below or need to create your own using custom reporter.
https://www.npmjs.com/package/jasmine-spec-reporter
https://www.npmjs.com/package/protractor-html-reporter
https://www.npmjs.com/package/protractor-beautiful-reporter
http://jasmine.github.io/2.1/custom_reporter.html
Or you can try with allure plugin here http://allure.qatools.ru/
I also advice to use allure report. It is easy to setup and has a good documentation. Just want to mention that there is Allure 2 is ready. Take a look at Git Hub and integration for JS
I dont know why, but JBehave does not take in consideration failures in the given stories. If there is a failure in a givenstory, it will not perform the rest of the steps of that story, but it will execute the rest of the given stories. Here is a example:
GivenStories: stories/web/pmv/Story1.story,
stories/web/pmv/Story2.story,
stories/web/pmv/Story3.story,
stories/web/pmv/Story4.story,
stories/web/pmv/Story5.story
When the user do something
Then something happens
For instance, if the Story2.story fails, I was expecting that the rest of the given stories and the last 2 steps were not executed. But they are.
Anyone knows why is that?
How can I fail all stories if one single step or story fails?
I noticed as well that the reports statistics just reflect the last given story and the following steps. Is this correct? Why?
I have the following configuration:
configuredEmbedder().embedderControls()
.doGenerateViewAfterStories(true)
.doIgnoreFailureInStories(false)
.doIgnoreFailureInView(false)
.useThreads(2)
.useStoryTimeoutInSecs(60);
MostUsefulConfiguration:
.useStoryControls(
new StoryControls()
.doDryRun(false)
.doSkipScenariosAfterFailure(true)
.doResetStateBeforeScenario(false))
When i added the config doResetStateBeforeScenario(false), the following steps after the failure, even the ones inside the givestories were not performed. But in yet, the statistics show no error, because it was not last given story or the steps on the main story. In the end the maven build had no errors, in yet there failures in the test.
Any thoughts?
OK. After some searching around I managed to find that this issue was fixed in JBehave 3.8.
JIRA link: http://jira.codehaus.org/browse/JBEHAVE-841
I updated to latest jbehave version and this works fine.
So I'm trying out this new TDD thing (about time haha). Anyways I have two unit testing files currently one for application and one for logic. The application logic test was autogenerated by xcode and when I go to manage schemes I can see the (void)testExample but I can't see the other tests I have created in my logic file, nor are they being run. Attaching picture.
Well it would help to read the docs a bit more carefully: http://developer.apple.com/library/ios/#documentation/DeveloperTools/Conceptual/UnitTesting/03-Writing_Test_Case_Methods/writing_tests.html#//apple_ref/doc/uid/TP40002143-CH4-SW1
You just have to prepend your methods with 'test'.
i.e.
-(void)testmynewcase {
}
I was googling a lot in order to find a solution for my problems with UI Automation. I found a post that nice summarizes the issues:
There's no way to run tests from the command line.(...)
There's no way to set up or reset state. (...)
Part of the previous problem is that UI Automation has no concept of discrete tests. (...)
There's no way to programmatically retrieve the results of the test run. (...)
source: https://content.pivotal.io/blog/iphone-ui-automation-tests-a-decent-start
Problem no. 3 can be solved with jasmine (https://github.com/pivotal/jasmine-iphone)
How about other problems? Have there been any improvements introduced since that post (July 20, 2010)?
And one more problem: is it true that the only existing method for selecting a particular UI element is adding an accessibility label in the application source code?
While UI Automation has improved since that post was made, the improvements that I've seen have all been related to reliability rather than new functionality.
He brings up good points about some of the issues with using UI Automation for more serious testing. If you read the comments later on, there's a significant amount of discussion about ways to address these issues.
The topic of running tests from the command line is discussed in this question, where a potential solution is hinted at in the Apple Developer Forums. I've not tried this myself.
You can export the results of a test after it is run, which you could parse offline.
Finally, in regards to your last question, you can address UI elements without assigning them an accessibility label. Many common UIKit controls are accessible by default, so you can already target them by name. Otherwise, you can pick out views from their location in the display hierarchy, like in the following example:
var tableView = mainWindow.tableViews()[0];
As always, if there's something missing from the UI Automation tool that is important to you, file an enhancement request so that it might find its way into the next version of the SDK.
Have you tried IMAT? https://code.intuit.com/sf/sfmain/do/viewProject/projects.ginsu . It uses the native javascript sdk that Apple provides and can be triggered via command line or Instruments.
In response to each of your questions:
There's no way to run tests from the command line.(...)
Apple now provides this. With IMAT, you can kick off tests via command line or via Instruments. Before Apple provided the command line interface, we were using AppleScript to bring up Instruments and then kick off the tests - nasty.
There's no way to set up or reset state. (...)
Check out this state diagram: https://code.intuit.com/sf/wiki/do/viewPage/projects.ginsu/wiki/RecoveringFromTestFailures
Part of the previous problem is that UI Automation has no concept of discrete tests. (...)
Agreed. Both IMAT and tuneup.js (https://github.com/alexvollmer/tuneup_js#readme) allow for this.
There's no way to programmatically retrieve the results of the test run. (...)
Reading the trailing plist file is not trivial. IMAT provides a jUnit like report after a test run by reading the plist file and this is picked up by my CI Tool (Teamcity, Jenkins, CruiseControl)
Check out http://lemonjar.com/blog/?p=69
It talks about how to run UIA from the command line
Try to check the element hierarchy, the table can be placed over a UIScrollView.
var tableV = mainWindowTarget.scrollViews()[0].tableViews()[0].scrollToElementWithName("Name of element inside the cell");
the above script will work even the element is in 12th cell(but the name should be exactly the same as mentioned inside the cell)
I would like to be able to run a set of unit tests by linking to them in my application (e.g. I want to be able to click on a link and have it run a set of jUnit tests). The problem is that GWT and jUnit don't seem to be designed for this capability -- only at build time can you run the tests it seems.
I would like to be able to include my test code in my application and, from onModuleLoad for example, run a set of tests.
I tried to just instantiate a test object:
StockWatcherTest tester = new StockWatcherTest();
tester.testSimple();
but I get:
No source code is available for type com.google.StockWatcher.client.StockWatcherTest;
even though I include the module specifically.
Would anyone know a way to do this? I just want to be able to display the test results within the browser.
If you are trying to test UI elements in GWT using JUnit, unfortunately you may not do so. JUnit testing is limited to RPC and non-UI client-side testing. See this thread for a great discussion on what you can and cannot do with GWT jUnit testing.
If you are not trying to test UI elements, but are instead trying to inject your RPC code or client-side logic with test values (hence why you want to be able to click on a link and run a set of JUnit tests), then you should follow the following guide from testearly.com: Testing GWT with JUnit. In short, you should make sure that the method you are testing does not include any UI elements and if the method you are testing is asynchronous in nature, you must add a timer.
In 2.0, HTMLUnit was added. You may wish to use this instead of firing up a browser each time you wish to test.