SpecFlow + WatiN + NUnit + TeamCity on Windows Server 2008 R2 - nunit

I just recently ran into some trouble setting up a fully automated nightly build / testing / deployment environment. To explain the setup in short. We use Mercurial as source control system and TeamCity as build server. It currently only has one build agent on a separate machine to keep things simple.
The idea is as follows: each night a TeamCity build runs. Pulls latest from Mercurial repository, runs a build, runs unit tests, deploys to test server, runs acceptance tests (on build agent) against freshly deployed test server and if all of that passes deploys the lot to a staging server. I've got everything working except for acceptance tests. These are done with SpecFlow and WatiN. Everyhting runs nicely on development machines (Windows 7 32 bit). However when I try to run the acceptance tests (NUnit in TeamCity) the runner reports fail beacuse of step [Browser].ContainsText(browserText). All steps up to that ran correctly and if I replicate the steps in the same browser on build agent (got full admin access) I can see the text being present. So to my best knowledge the test should pass.
I found this stackoverflow thread that got me thinking I may be doing somwthing wrong with how I keep the browser instance around. To explain: I have a set of common steps such as "when I click X button" or "when I enter text X into field Y" and the one with an issue "then then I am shown X". Because I'm making as much as possible of tests reusable I've also generalized a #requires_browser scenario requirement which is then defined like this:
[BeforeScenario("requires_browser")]
public void RequiresBrowserBeforeFeature()
{
Settings.MakeNewIeInstanceVisible = false;
ScenarioContext.Current.Set<Browser>(new IE(true));
}
[AfterScenario("requires_browser")]
public void RequiresBrowserAfterFeature()
{
ScenarioContext.Current.Get<Browser>().Close();
ScenarioContext.Current.Get<Browser>().Dispose();
}
and then the common step:
[Then("I am shown \"(.*)\"")]
public void ThenIAmShown(string text)
{
Assert.IsTrue(ScenarioContext.Current.Get<Browser>().ContainsText(text));
}
From this I can define other common steps that enter text into text boxes, click buttons and so on. All this works perfectly on development environment, running in visual studio 2010 [as administrator].
Am I wrong in thinking that ScenarioContext should be preserved until the test is finished?
What I've tried so far:
Configuring build agent service to run as local system (SYSTEM). This causes the issue as described - the ThenIAmShown(string text) method will always fail. It has interaction with local desktop enabled.
Configured build agent service to run as network administrator (full admin rights). This setup won't even runs tests. The exception I get:
TearDown method failed. WatiN.Core.Exceptions.BrowserNotFoundException : Could not find an IE window matching constraint: Timeout while waiting to attach to newly created instance of IE.. Search expired after '30' seconds.
TearDown : System.Collections.Generic.KeyNotFoundException : The given key was not present in the dictionary.
at WatiN.Core.IE.CreateIEPartiallyInitializedInNewProcess()
at WatiN.Core.IE.CreateNewIEAndGoToUri(Uri uri, IDialogHandler logonDialogHandler, Boolean createInNewProcess)
...
Already added assemblies Interop.SHDocVw.dll and Microsoft.mshtml.dll to bin directory (Copy To Output in project references).
Running NUnit UI from the build agent. This runs the unit tests as expected. Leads me to believe there might be an issue with security. Runnint NUnit 2.0.5 on Windows Server 2008 R2 Enterprise.
Anyone had this issue?

Related

How to Build a definition and publish test results for a Java project with maven, Junit and selenium on Visual Studios Team Services VSTS

I have an automation script that uses maven POM.xml to import all the dependencies needed from selenium and junit. The main test uses selenium to open a browser, verify some information, close the browser and the test ends.
When run as Junit it works fine: run as Junit test
When run as Maven Test it works fine as well: run as maven test
In both scenarios, the program opens the browser and navigates through the website as it should do for an automated test.
Now I need to integrate it to VSTS so I can visualize the overall pass/fail test on the VSTS dashboard but I'm not familiarized with this tool too much yet.
So far this is what I have managed to do:
Deploy an agent on my WindowsPC (I want to execute and deploy the project on an Azure VM or another azure instance later on) NOTE: this is the same pc I'm successfully running the program using eclipse as shown in the screenshoots above. https://learn.microsoft.com/en-us/vsts/build-release/actions/agents/v2-windows?view=vsts
Create a build definition on VSTS but when I queue the definition the build fails: build definition and the build fail.
I don't know why it can't find mt config.txt file since it is located on the same hosted agent in that same directory. I'll appreciate if someone is capable of guiding me through this process so I can run the program from the VSTS and visualize the overall tests that fail and pass on the VSTS dashboard.
UPDATE: I moved the config.txt file to the public directory and the build was successful(I still need to fix this issue because I do not want my work in a public folder).
Now the problem I have is that even though the build is successful and it looks like it is running my "3 tests", When I look at my pc, nothing is happening. it should open chrome and take a screenshot, then open Firefox and take another screenshot and finally open internet explorer and take another screenshot and save each test on different folders but it is only generating folders for chrome and internet explorer (but still those folders does not have the screenshot I'm asking, maybe because the browser is not being open on the computer.)
Here is the log: https://drive.google.com/open?id=1S_MhAUmzj8i9phPQiqS06s0_1cCRrbF0
test output report generated on my computer
test output on vsts
Look at the error message. The error message tells you precisely what the problem is: java.io.FileNotFoundException: Y:\Automation Team\CopaQA\Architecture\local\config.txt (The system cannot find the path specified)
You need to not rely on hard-coded paths.
You say you registered a build agent against your VSTS account... but did you change the agent queue for your build? If the agent queue is "Hosted", you're using Microsoft's hosted agent.
I don't know why it can't find mt config.txt file since it is located on the same hosted agent in that same directory.
It turns out that Java.IO. can't read files located on a shared network drive, I solved this by using the UNC path to that file (//"computername"/"directory"/"file.txt")
Now the problem I have is that even though the build is successful and
it looks like it is running my "3 tests", When I look at my pc,
nothing is happening.
It took me a little reading to realize that to perform UI tests my agent needs to be set up in INTERACTIVE MODE. it can be done following this guide: https://learn.microsoft.com/en-us/vsts/build-release/actions/agents/v2-windows?view=vsts

TeamCity nunit tests different output

I have tests that have to be run on CI server. The thing is that when I am running the tests in nunit console, they are all passing, but on teamcity server I keep receiving a bunch of errors, connected with sqllocaldb instance. I guess this is due to the fact that running teamcity from 'local system' privileges gives such an output, but I don't know how to bypass this thing, even using 'Run as' plugin.
Maybe you can suggest an option/configuration/workaround to me to get the right results?
Created new windows profile, called "TeamCity", configured everything and made service to be run under this profile, now everything works fine.

How to run scheduled Coded UI Tests on Virtual Machine without having a RDP connection

Situation in short:
Virtual Machine with Visual Studio 2013 installed. PowerShell script
runs on the VM to execute Get Latest, Build and Execute Coded UI
Tests. Windows Scheduled Task to execute PowerShell nightly.
auto-logon is enabled (or I'm doing something wrong?)
yes, I've read post Is it possible to run Coded UI tests without having to connect via remote desktop?
I've seen posts about TCM. Does this help and how can I use it in my
situation?
I made some tests in Microsoft Test Manager and I also executed and recorded them.
I've loaded these tests in a test project (and changed the script providing categories and custom checks).
I then categorized these (as Development or Acceptance).
I executed out using a PowerShell script on a VM (with Visual Studio 2013 installed) with following actions:
Get Latest
Build
Run latest build with a selected set aka category using mstest.exe
So far everything is going perfectly. All the tests pass.
However, when I create a Scheduled Task on my VM run the PowerShell script everything fails because of a missing session.
Do I have the VM (I have no knowledge of Virtual Machines) then unlock or something?
Side-Note:
I also tried to fix this with a Test Agent and Controller, but once I had installed these, all other users of TFS lost their rights, so I prefer not to do this again.
I would be very grateful if you know something that can solve this.
I spent hours on Google finding a solution for this issue, but no solution helped me.
Do I need to provide more information?
The problem you're seeming to have is that the testagent is not setup correctly. You need an active desktop session for Coded UI to be able to run (it needs it to perform all the actions such as clicks).
Microsoft has some nice info about setting up your test agent here.
But to tackle your exact problem of the test failing because of a missing session I'd suggest the following:
Run AutoLogOn.exe from the sysinternals suite (can be found at http://live.sysinternals.com/). It will automatically log in with specified user when the machine starts, and keeps the desktop session active.
Alright...I'm making progress.
I've installed test agent and controller. It's all running fine.
Next I've opened Lab Center on my own MTM to create a new environment.
Test Controller is found, but I receive the message in this post "Microsoft Test Manager cannot install test agent on these machines" when creating new Lab Center environment
File and Printer sharing exception is enabled. I don't get the other message.
I don't understand what is wrong.
Maybe I'm completely on the wrong track and it's not necessary to use the Lab Center.
Then the only remaining issue is the non-active desktop issue.

Coypu/SpecFlow acceptance tests hang when run via Jenkins job

I'm running acceptance tests on a project via SpecFlow, NUnit and Coypu (for browser automation, using the WatiN driver). The running of the tests are invoked via a powershell/psake script.
If I run these tests on my local box, they run fine. However, we have a build server on which a Jenkins job will automatically run these tests, and when run via this Jenkins job they don't execute -- they just hang.
Looking in task manager I can see there's two instances of iexplore.exe that are created when the Jenkins job runs. However after a certain point they just hang - no changes in memory usage or CPU.
nunit-agent-x86.exe and nunit-console-x86.exe are also running but mostly hung, just nunit-agent-x86.exe going up very slowly in memory.
If I kill one of the iexplore.exe processes things continue, but the SpecFlow specs all subsequently fail.
At the point of killing iexplore.exe, the following exception is in the log:
Unhandled Exception: System.Runtime.InteropServices.COMException: The remote procedure call failed. (Exception from HRESULT: 0x800706BE)
If I invoke the psake script manually when logged in to the server, the specs run OK.
This issue began to occur when I tried to use basic DI for the BrowserSession as in the gist here: https://gist.github.com/2301407
Before that I was sharing the BrowserSession via a static property of an NUnit [SetupFixture] class. Things were working mostly OK that way, except a small issue with a test involving a modal dialog not working correctly, but I wasn't sure I was doing it right so wanted to do the technique in the gist
I'm a bit lost as to what's causing the hang. Any ideas what it is or tips to track it down?
UPDATE: After switching to Firefox as the browser, and Selenium as the driver, the problem has gone away...
I got a couple of responses on the Coypu list. I've yet to test them out as everything is fine with Firefox at the moment, but in case they are of assistance to anyone else...
On Thursday, 2 August 2012 16:38:30 UTC+1, Adiel wrote:
[...]i believe that watin needs nunit to run in STA (single-threaded) which could possibly be related to your problem.
In other words, perhaps you made your tests thread safe with the static singleton browser session, but now via specflow's IOC you are
getting multiple instances due to the way nunit is running.
On Thursday, 2 August 2012 16:41:11 UTC+1, Matt Ellis wrote:
This sounds like it's Internet Explorer's protected mode getting in the way. IE runs different zones, such as Internet and Intranet
(and about:blank) in different processes, and IIRC WatiN doesn't
handle that very well. If you can disable protected mode on your
server, you should be fine.

How to run Selenium RC + PHPUnit + NetBeans remotely?

I have Selenium Server, PHPUnit and NetBeans up and running on a machine that I want to be my dedicated testing box. How can I set it up to were I modify test cases (I already figured that part out) and tell the test machine to run the test remotely?
I'd use a continuous integration server like Jenkins. Usually CI servers are used to build an application on every commit to a repository, but it's just as easy to manually start a "build" that just consists of running all your tests (and recording the results, and running code coverage if you want, etc).
I found Jenkins to be really easy to set up (I followed a nice tutorial at http://blog.jepamedia.org/2009/10/28/continuous-integration-for-php-with-hudson) - the only extra work I had to do besides creating a build script was to make sure that Selenium RC is running on the test machine, and it sounds like you've already done that.
To make it even easier, if you set up Jenkins (or any other CI server, I'm sure) to build on a commit to your repository, then you don't even have to log onto the test machine to edit the tests - anybody can commit test changes, the CI server will run the tests, and everybody can see the results. Not quite as important if you're developing solo, but still a handy trick.
We can run the test cases from remote server and case will get execute into local machine. We have to follow the below steps,
Install phpunit and necessary packages in server
edit the test case and change host as local ip address ( use static ip address )
run the selenium RC in local server
run the test case in server
Test case will get execute into local machine.