In my Protractor conf.js file, I'd like to re-use the same spec files multiple times; however, it's seems to not be possible.
Some background:
We are reading test cases from a JSON file, launching reports, then testing grid results and various DOM elements.
All reports have the same format. The primary differences lie in the report titles, data columns, actual data results, etc.
So in my conf.js file, ideally I'd like to re-use the same spec files multiple times - but my understanding is that I cannot do this.
For example, my spec array:
specs: [
'spec/report1-spec.js',
'spec/report-grid-details-spec.js',
'spec/report2-spec.js',
'spec/report-grid-details-spec.js',
'spec/report3-spec.js',
'spec/report-grid-details-spec.js',
]
I've read this post (http://ramt.in/how-to-run-identical-jasmine-specs-multiple-times-with-protractor/ ) where you can move your spec files into a node module, but 1) I don't want to move all specs files there, and 2) it doesn't work anyway when I move even one spec file into a module export file.
If I can't do it, then I'll just move my report-grid-details-spec.js code into a common page object file and call it whenever it's needed.
Just wondering if anyone out there has found a solution to this need to re-use spec files multiple times in one conf.js configuration.
Thank you,
Bob
If I can't do it, then I'll just move my report-grid-details-spec.js code into a common page object file and call it whenever it's needed.
This would probably be the easiest way to approach the problem. Though, I like the idea of putting specs into modules - it is a plus to reusability overall.
The thing is, jasmine does not allow executing the same test in a single test run. And, from what I understand, there is no easy way to change the behavior.
One of the possible workarounds is to completely restart protractor and, hence, recreate the jasmine testing environment so that the next report-grid-details-spec.js would run in a new jasmine environment - this is something that protractor-flake project uses to retry the failing tests (it basically restarts protractor through command-line passing the failing specs as a comma-separated list to the specs argument, source).
Related
We use NUnit 3 to run deployment smoke tests with parameters read from VCS. It works like this:
Powershell reads the parameters from VCS and compose NUnit 3 console command line.
NUnit3 console runs the tests.
Some parameters are passwords.
The problem is that the end test result XML lists all the test parameters, including the passwords.
Is it possible to instruct NUni3 somehow to avoid including the test parameters in the test result XML?
There is no feature like that because the assumption is you would not pass in anything that needs to be secure.
Why is that? Imagine there were an argument like --secret:password=XXX which worked like a parameter but was not displayed in the results. In that case, your password would still be in clear in your script, for anyone to read. Also, it would be available to any test, which could do what it wanted with it, like write it somewhere.
A better approach is to use some sort of encryption, so that you are only passing in a key, which is not usable except by an account or program that knows how to decrypt it. There are various approaches to doing this,depending on how you are running tests. I believe you will find that VCS has a way of encrypting passwords that you may be able to use.
In any case, without such a "secret" option, the only way you could avoid publishing the password would be to create your own output format by writing an engine result writer extension. Your extension code would receive the entire nunit 3 output document and you could modify it to remove the passwords before saving the file.
I have a really complicated system which use multiple languages and frameworks (Java Python Scala Bash). In each module I need to retrieve configuration values which are similar and change frequently. Currently I'm maintaining multiple conf files which holds lots of duplicates.
I wonder if there is out of the box RestAPI which can retrieve variables by demand from remote location.
All I manage to find by now are ways to load the entire file from remote source which is half a solution from me:
YAML.parse(open('https://link_to_file/file.yaml'))
My goal, which I fail to find a lead to it, is to make a direct call.
MyRemoteAPI.get("level1.level2.x")
P.S
YAML is not mandatory solution for me, I'm Open for suggestions.
I don't know about an out-of-the-box API, but it's fairly trivial to build. Make a service that will read the YAML file and traverse to the appropriate key. e.g. using a dynamic language like Ruby (+Rails), you could do something like
def value
config = YAML.load_file '/local/path/to/config.yaml'
render plain: config.dig(params[:key].split('.'))
end
dig essentially traverses a structure and safely returns nil if a key isn't found, so this returns the value at the "leaf" of the requested path.
You might also want to cache the structure in memory to prevent constantly reading from the file, e.g. could do something like ##config ||= YAML.parse(open('https://link_to_file/file.yaml')) or config = Rails.cache.fetch('config', expire_in: 1.hour) { ... }. And/or cache the API's HTTP response.
I am in a process of implementing Page object Model, I have one query regarding it, please see below:
I have created page files which is having locators and methods for the page, I have spec file in which I am doing the assertions by calling these methods. My question is that for one page I have over 100 test cases, now should I create single assertion file for single tests or should I create 100 assertion file for 100 test.
Please let me know what is the best way to manage it.
Regards,
Manan
I think it makes the most sense to group tests into files by functionality. It's hard to run only some tests from a file, so split out any groups of tests you think you might want to run independently. Are some of them suitable for a quick smoke test suite? Maybe those should be in a separate file.
You shouldn't need to create a new file for neither every assertion nor test case. I am confused by your question because in my understanding, the assertion is part of the test case, and test+assertion are part of the same function (assertion being the end goal of the test).
Regarding the Page Object Model: The important part of the pattern is ensuring the separation of page/DOM detail from test flow (i.e. tests should possess no knowledge of the DOM, but instead rely on page objects to act on actual pages).
I've got a Perl-based test suite with 10,000+ tests that I would like to make run faster. I've tested using the -j flag to prove, and I have found that most-but-not-all of my tests are ready to run in parallel.
While I can work on making the remaining tests to be "parallel friendly", I expect there always be some tests which are not. What's a good way to manage this? I would like for it to be easy to run the whole set of tests efficiently, and make it easy to mark tests as "not-parallel-ready" if I need to.
Here are some options I see:
prove could be patched to support some tests as not-parallel-ready
Jenkins is being used to manage the test suite runs. I could split off the non-parallel tests into their own run. In other words, give up and use two test runs.
Perhaps there is a way to merge two TAP result streams together that I have yet to recover.
I'm not too concerned with how I will manage the list of exceptions. Either I can keep a list in a file as part of the test harness infrastructure, or I could put something in each test header that would mark it as such, and our test harness could determine the list of exceptions dynamically.
( The test suite is partially based on Test::Class, and I'll also be looking at Test::Class::Load to speed it up as well. )
I found a solution. It's in the documentation for aggregate_tests() for TAP::Harness. It includes a code sample for how I could write my own harness for this purpose:
...This is useful, for example, in the case where some tests should
run in parallel but others are unsuitable for parallel execution.
my $formatter = TAP::Formatter::Console->new;
my $ser_harness = TAP::Harness->new( { formatter => $formatter } );
my $par_harness = TAP::Harness->new(
{ formatter => $formatter,
jobs => 9
}
);
my $aggregator = TAP::Parser::Aggregator->new;
$aggregator->start();
$ser_harness->aggregate_tests( $aggregator, #ser_tests );
$par_harness->aggregate_tests( $aggregator, #par_tests );
$aggregator->stop();
$formatter->summary($aggregator);
From there it looks like I could:
Sub-class App::Prove and override _runtests(), which is where the new functionality above could be merged in.
Fork prove so that it calls My::App::Prove instead of App::Prove.
Now that I better understand how the pieces fit together I can see how I might create a patch for prove that would add an option like --exclude-from-parallel FILE, which would allow you to specify a file, which contains a list of test files to be excluded from parallel testing.
UPDATE 2012-08-16: I have a patch for prove now, and have submitted it for review. You can view and comment on the Pull Request. No summary is produced after the run output. It's not clear why.
I've now found the best solution so far to this problem. In turns out that prove has had undocumented support for marking some tests to be run in sequence instead of parallel since 2008. It's backed by a rather fancy "rules" system in TAP::Parser::Scheduler that allows for complex specifications of ordering arrangements for parallel and sequential test runs.
Here's the basic current recipe for prove:
# All tests are allowed to run in parallel, except those starting with "p"
--rules='seq=t/p*.t' --rules='par=**'
I have a new pull request that adds documentation for this feature, and have started a discussion about possibly offering a simpler syntax for basic exceptions as well. See the pull request for details.
I found another solution which advertised this feature, but I could only get trivial cases to work. It's to use Test::Steering. It allows me to do this:
include_tests( { jobs => 4 }, #parallel_tests );
include_tests( #serial_tests );
With this solution, be aware:
Before it actually works, I currently have to patch the code to fix a basic bug with it that has remained unpatched for multiple years.
Additional code is needed to handle generating the list of of parallel and serial tests to run.
I didn't actually get a combined summary for my real-world test... both sections emitted their own summary reports, so it didn't really work. Maybe I missed something, or maybe it's broken.
Test::Parallel also provides an easier way to run some tests in parallel
have a look at the sample from https://metacpan.org/pod/Test::Parallel
Another option: use a rules file for TAP::Harness.
You can build custom rules in a YAML file (called testrules.yml by default). I needed something similar to what you describe, which I was able to do with a testrules.yml file that looked like this:
---
seq:
# tests that are not parallel-ready (will run in isolation)
- seq:
- t/test1.t
- t/test2.t
# tests that can run in parallel
- par:
# wildcard for everything else
- **
In my case, I was using this with code that directly called App::Prove, rather than command-line prove. But I think it would work with prove too?
I have a list of several thousand NUnit tests that I want to run (generated automatically by another tool). (This is a subset of all of the tests, and changes frequently)
I'd like to be able to run these via NUnit-Console.exe. Unfortunately the /run option only takes a direct list of files which in my case would not fit on a single command line. I'd like it to pickup the list from a filename.
I appreciate that I could use categories, but the list I want to run changes frequently and so I'd prefer not to have to start changing source code.
Does anyone know if there is a clean way to get NUnit to run my specified tests?
(I could break it down into a series of smaller calls to NUnit-console with a full command line, but that's not very elegant)
(If it's not possible, maybe I should add it as an NUnit feature request.)
Had a reply from Charlie Poole (from NUnit development team), that this is not currently possible but has been added as a feature request for NUnit 2.6
I see what you're saying, but like you say you can run a single fixture from the command line.
nunit-console /fixture:namespace.fixture tests.dll
How about generating all the tests in the same fixture? Or place them all in the same assembly?
nunit-console tests.dll
As mentioned in the nunitLink, we need to mention the scenario/test case name. It simple but it has bit of a trick in it. Directly mentioning the test case name will not serve the purpose and you will end up with the 0 testcases executed. We need to write the exact path for the same.
I don't know how it works for other languages but using c# I have found a solution. Whenever we create a feature file corresponding feature.cs file get's created in Visual Studio. Click on the featureFileName.feature.cs and look for namespace and keep it aside(Part 1)
namespace MMBank.Test.Features
Scroll a bit down you will get the class name. Note that as well and keep it aside(Part 2)
public partial class HistoricalTransactionFeature
Keep scrolling down, you will see the code which nunit understands for execution basically.
[NUnit.Framework.TestAttribute()]
[NUnit.Framework.DescriptionAttribute("TC_1_A B C D")]
[NUnit.Framework.CategoryAttribute("MM_Bank")]
Below the code you can see the function/method name which will most likely be TC_1_ABCD(certain parameters)
public virtual void TC_1_ABCD(string username, string password, string visit)
You will be having multiple such methods based on no. of scenarios you have in your feature file. Note the method(test case) which you want to execute and keep it aside(Part 3)
Now collate all the parts with dots. Finally you will land up with something like this,
MMBank.Test.Features.HistoricalTransactionFeature.TC_1_ABCD
This is it. Similarly you can create the test case names from multiple feature files and stack them up in text file. Every test case name should be in different line. For command you can browse through above nunit link for execution using command prompt.