Is it possible to use the Code Coverage devtool (from upcoming Chrome release) by selenium+maven automation?
E.g. like JaCoCo for java projects?
I.e. I imagine approximately such scenario for our web-product:
There is a test suite with several test cases.
The Chrome Code Coverage collection is enabled somehow from outside
Then test suite is executed (by maven):
The each test case uses Selenium to run Chrome browser, executes testing scenario and close the webdriver (browser is closed).
Then, after the suite is finished the code coverage data (CSS/JS related) could be collected and could be viewed as a report later.
Related
Please tell me if it's possible to associate Azure DevOps test cases with JUnit automated tests from eclipse. If yes, is there any step by step document I can follow? Please help
As I know it's not supported scenario.
You can find the supported tests from this document:
Coded UI test, Selenium tests, and unit tests written using Version 1 of the MSTest framework can be associated with a test case.
Tests that use MSTest v2, NUnit, and xUnit frameworks can be associated with a test case workitem when using Visual Studio 15.9 Preview 2 or later. However, these tests cannot be run using Microsoft Test Manager and XAML builds.
Tests that use the .NET core framework can be associated with a test case workitem when using Visual Studio 15.9 Preview 2 or later. To run the .NET core tests the appropriate target framework must be specified in a runsettings file. However, these tests cannot be run using Microsoft Test Manager and XAML builds.
Tests that use other test frameworks such as Chutzpah (for JavaScript tests such as Mocha or QUnit), or Jest cannot be associated with a test case.
Associating generic tests may work, but running these tests is not supported.
So we recommend using Visual Studio to associate the automated tests. And for now the Junit framework is not supported in such scenario.
I want to link my manual test cases in Azure test plan to robot framework automation test cases( python based test case using selenium library).
Performed below steps :
Created a test project containing my robot framework automated tests.
Checked test project into an Azure DevOps repository.
Created a build pipeline for the project, ensuring that it contains the automated test.
When I executed build pipeline, it appears in Visual Studio - Solution explorer but not in Test Explorer.
enter image description here
My question is how can I get robot framework test cases to appear in VS - Test Explorer so that I can associate it to manual test cases in Azure test plan.
Can you please tell me the steps that I need to do in VS to get Test Scenarios.robot in Test Explorer ?
The software I worked on has both unit tests and system tests. System tests can take minutes to run, they take input values and we validate the results against expected output. There are hundreds of system tests. The software must be built (done this) and tested on both windows and Linux.
How can I automate testing with VSTS? I'd like to avoid doing this at build stage, because it would slow the builds down. I can't see how to automate this in the Test stage. Do I need additional extensions to do this? Everything seems so geared up for web development, e.g. selenium tests, how do we run automated tests for good old binary programs?
I would suggest using Release Management to deploy your application to a test environment and then run your tests as a part of your Release Definition. You can then choose to run tests in parallel to make sure that your system tests don't take days to run.
On a side note, having so many system tests is a code smell. I would suggest looking into building as many fast running unit tests as possible and only using system tests when absolutely necessary.
I have 3 sets of related questions:
1.Are both TestNg and Jenkins competitive tools in the industry. Do they have the same function?
2. Are both used primarily to run Selenium Test Cases?
3. Can both be used to run test cases on IDE's like Eclipse?
1.Are both TestNg and Jenkins competitive tools in the industry. Do they have the same function?
No they are not. Jenkins is an Orchestrator tool that is predominantly used for Continuous Integration (As and when developer checks-in code, it must be compiled, unit tests should be run on it and if the tests pass, it should be deployed to some place) and Continuous Delivery (All of Continuous Integration + run functional tests on the deployed code and if the functional tests pass, promote the code to the next environment and eventually ship to production in an automated fashion). Jenkins cannot do this by itself. So it makes use of build tools such as Ant/Gradle/Maven to help it compile the code and run tests on top of it.
TestNG is a test framework, that lets developers write unit tests and also lets test engineers build their functional automation using it. Its pretty much like JUnit, except that it lets a test engineer visualise a test just as he/she would do in the real world. So they both have totally different functions.
Are both used primarily to run Selenium Test Cases?
Jenkins is agnostic to what it orchestrates. So it doesn't care whether you are using it to build selenium tests or production code. To Jenkins a build is a build. TestNG on the other hand can be used to run Selenium Tests as well. TestNG merely cares about running tests. What goes inside the test is left to the engineer who is building those tests.
Can both be used to run test cases on IDE's like Eclipse?
Jenkins is usually started and left running as a standalone server and one usually interacts with it via a Web UI. So it has got nothing to do with an IDE. TestNG can be run via the below 3 mechanisms :
Via IDE
Via command line
Via build tools.
I have a question that's very similar to what's discussed here:
Integration Test of REST APIs with Code Coverage
I deployed a war file that exposes the REST APIs to a web server and I'm using TestNG to write test cases for the REST APIs. I'm not unit testing - I'm only end-to-end / integration testing. Currently, I'm running test cases from eclipse in my machine.
My goal is to get coverage reports on the TestNG test cases.
Since the tests are local to my machine and the REST API is deployed in another server, EclEmma doesn't provide any meaningful data when I run the tests cases in my machine.
Is there a way to point EclEmma to the web server instead of my local machine and get the code coverage report?
Would it be better/possible to include the tests in the war file and run the tests from the web server? That should allow me to get the meaningful code coverage report, right?
The easiest way forward in cases like this is normally to start the web server inside of your IDE and run tests with coverage measuring in there. Even better to start the web server from within the tests - then a build tool like maven can also do code coverage reporting.