We are running a setup on CodeShip Pro which generally works fine but every third/forth build fails as some components (normally DB server/tables and or seeded data) already exists in the environment.
Doing a manual "Reset Cache" and restart the build after 20-30 seconds solves the issue and the pipeline runs all tests just fine.
There is a CodeShip command to clear the cache as cs clear-cache but I have not been able to implement that in any way using the Pro setup.
As I understand the documentation everything runs on "Services" which in turn are Docker containers. Using the "Steps" I am able to "control" and run commands against (in) the containers, but not execute anything against the "host" (=the CodeShip service).
Anyone that come up with a way to have CodeShip to clear the cache (=run cs clear-cache) as the first "serial" step of each build?
Related
I'm currently trying to integrate automatic test execution on the deployment process using Azure Devops but it's causing me an issue where some tests don't run properly.
The application I'm testing has a log-in page and I've set up a couple of tests to see if it opened properly. These ones run smoothly, but the ones that require going through the login process do not.
I've set up some snapshots amidst the process and the result was that I was getting an error on the last step of logging in. I've checked the application logs and the result I got was that no credentials were sent through the request which I found odd since they were displayed on the snapshots.
All the data being inserted was correct, I've checked the endpoints and settings on the agent that was running them and everything seemed fine.
I tried then to run the automated tests through powershell command directly on the same agent and all of them passed with no issue whatsoever.
I've tried changing my task to a powershell command but it still gives me an error.
I have no idea where my issue could be or where I should be even able to start looking for the problem, as clearly it seems that the problem is on the azure devops task but everything seems fine with it, I've tinkered around with the settings a bit but none of them seemed to have any impact on the results I was getting.
I'm using version 2 of the vstest task and am targetting the test assembly that was deployed to the agent. Any idea on what I could be missing or pointers on how to find the solution?
These ones run smoothly, but the ones that require going through the
login process do not
Based on this description, it seems that it will open a new window automatically by the task.
If I'm right, and your test will open a window to login with your private-hosted agent. At this time, the mode of your private-hosted agent must be in interactive which it can allow auto-logon enabled.
Just check this official document for more detail information.
I'm trying to incorporate a test suite that runs nightly on an Azure VM.
As of now, I have a Build process using TFS2015 that publishes my test files, starts the VM, and copies the files.
I'm trying to then use the "PowerShell on Target Machine" task to execute a script that launches a batch file. The reason it'll be executing a batch file is because I can't have the build process wait until that script is finished (it takes around 3 hours for the tasks in the batch file to complete).
My initial logic was to have the powershell script create a task using schtasks. This part works and the task itself is created on the virtual machine, however, it never runs at the scheduled time.
The other issue is that if I manually create these tasks, the task is executed, but everything is executed in the background. I need everything to be executed in the foreground.
I'm aware that this is by design since you should not be able to run foreground processes/applications remotely since it isn't "your session".
So the question will remain, are there any work arounds to this?
I'm trying to do launch selenium webservers and then execute protractor automation tests on the virtual machine. So one batch file starts the selenium server and the second launches protractor with a defined suite. If these are ran in the background (essentially headless) all my tests break.
Any insight would be helpful, or if I need to expand on my question or provide further details please let me know. Thanks.
I'm aware that this doesn't answer your specific question, but have you looked at moving your Selenium tests into VSTS? They're officially supported in the build/release pipelines and other people are taking this approach with protractor here and elsewhere. People are also making it work with TFS.
Situation in short:
Virtual Machine with Visual Studio 2013 installed. PowerShell script
runs on the VM to execute Get Latest, Build and Execute Coded UI
Tests. Windows Scheduled Task to execute PowerShell nightly.
auto-logon is enabled (or I'm doing something wrong?)
yes, I've read post Is it possible to run Coded UI tests without having to connect via remote desktop?
I've seen posts about TCM. Does this help and how can I use it in my
situation?
I made some tests in Microsoft Test Manager and I also executed and recorded them.
I've loaded these tests in a test project (and changed the script providing categories and custom checks).
I then categorized these (as Development or Acceptance).
I executed out using a PowerShell script on a VM (with Visual Studio 2013 installed) with following actions:
Get Latest
Build
Run latest build with a selected set aka category using mstest.exe
So far everything is going perfectly. All the tests pass.
However, when I create a Scheduled Task on my VM run the PowerShell script everything fails because of a missing session.
Do I have the VM (I have no knowledge of Virtual Machines) then unlock or something?
Side-Note:
I also tried to fix this with a Test Agent and Controller, but once I had installed these, all other users of TFS lost their rights, so I prefer not to do this again.
I would be very grateful if you know something that can solve this.
I spent hours on Google finding a solution for this issue, but no solution helped me.
Do I need to provide more information?
The problem you're seeming to have is that the testagent is not setup correctly. You need an active desktop session for Coded UI to be able to run (it needs it to perform all the actions such as clicks).
Microsoft has some nice info about setting up your test agent here.
But to tackle your exact problem of the test failing because of a missing session I'd suggest the following:
Run AutoLogOn.exe from the sysinternals suite (can be found at http://live.sysinternals.com/). It will automatically log in with specified user when the machine starts, and keeps the desktop session active.
Alright...I'm making progress.
I've installed test agent and controller. It's all running fine.
Next I've opened Lab Center on my own MTM to create a new environment.
Test Controller is found, but I receive the message in this post "Microsoft Test Manager cannot install test agent on these machines" when creating new Lab Center environment
File and Printer sharing exception is enabled. I don't get the other message.
I don't understand what is wrong.
Maybe I'm completely on the wrong track and it's not necessary to use the Lab Center.
Then the only remaining issue is the non-active desktop issue.
I have Selenium Server, PHPUnit and NetBeans up and running on a machine that I want to be my dedicated testing box. How can I set it up to were I modify test cases (I already figured that part out) and tell the test machine to run the test remotely?
I'd use a continuous integration server like Jenkins. Usually CI servers are used to build an application on every commit to a repository, but it's just as easy to manually start a "build" that just consists of running all your tests (and recording the results, and running code coverage if you want, etc).
I found Jenkins to be really easy to set up (I followed a nice tutorial at http://blog.jepamedia.org/2009/10/28/continuous-integration-for-php-with-hudson) - the only extra work I had to do besides creating a build script was to make sure that Selenium RC is running on the test machine, and it sounds like you've already done that.
To make it even easier, if you set up Jenkins (or any other CI server, I'm sure) to build on a commit to your repository, then you don't even have to log onto the test machine to edit the tests - anybody can commit test changes, the CI server will run the tests, and everybody can see the results. Not quite as important if you're developing solo, but still a handy trick.
We can run the test cases from remote server and case will get execute into local machine. We have to follow the below steps,
Install phpunit and necessary packages in server
edit the test case and change host as local ip address ( use static ip address )
run the selenium RC in local server
run the test case in server
Test case will get execute into local machine.
I just recently ran into some trouble setting up a fully automated nightly build / testing / deployment environment. To explain the setup in short. We use Mercurial as source control system and TeamCity as build server. It currently only has one build agent on a separate machine to keep things simple.
The idea is as follows: each night a TeamCity build runs. Pulls latest from Mercurial repository, runs a build, runs unit tests, deploys to test server, runs acceptance tests (on build agent) against freshly deployed test server and if all of that passes deploys the lot to a staging server. I've got everything working except for acceptance tests. These are done with SpecFlow and WatiN. Everyhting runs nicely on development machines (Windows 7 32 bit). However when I try to run the acceptance tests (NUnit in TeamCity) the runner reports fail beacuse of step [Browser].ContainsText(browserText). All steps up to that ran correctly and if I replicate the steps in the same browser on build agent (got full admin access) I can see the text being present. So to my best knowledge the test should pass.
I found this stackoverflow thread that got me thinking I may be doing somwthing wrong with how I keep the browser instance around. To explain: I have a set of common steps such as "when I click X button" or "when I enter text X into field Y" and the one with an issue "then then I am shown X". Because I'm making as much as possible of tests reusable I've also generalized a #requires_browser scenario requirement which is then defined like this:
[BeforeScenario("requires_browser")]
public void RequiresBrowserBeforeFeature()
{
Settings.MakeNewIeInstanceVisible = false;
ScenarioContext.Current.Set<Browser>(new IE(true));
}
[AfterScenario("requires_browser")]
public void RequiresBrowserAfterFeature()
{
ScenarioContext.Current.Get<Browser>().Close();
ScenarioContext.Current.Get<Browser>().Dispose();
}
and then the common step:
[Then("I am shown \"(.*)\"")]
public void ThenIAmShown(string text)
{
Assert.IsTrue(ScenarioContext.Current.Get<Browser>().ContainsText(text));
}
From this I can define other common steps that enter text into text boxes, click buttons and so on. All this works perfectly on development environment, running in visual studio 2010 [as administrator].
Am I wrong in thinking that ScenarioContext should be preserved until the test is finished?
What I've tried so far:
Configuring build agent service to run as local system (SYSTEM). This causes the issue as described - the ThenIAmShown(string text) method will always fail. It has interaction with local desktop enabled.
Configured build agent service to run as network administrator (full admin rights). This setup won't even runs tests. The exception I get:
TearDown method failed. WatiN.Core.Exceptions.BrowserNotFoundException : Could not find an IE window matching constraint: Timeout while waiting to attach to newly created instance of IE.. Search expired after '30' seconds.
TearDown : System.Collections.Generic.KeyNotFoundException : The given key was not present in the dictionary.
at WatiN.Core.IE.CreateIEPartiallyInitializedInNewProcess()
at WatiN.Core.IE.CreateNewIEAndGoToUri(Uri uri, IDialogHandler logonDialogHandler, Boolean createInNewProcess)
...
Already added assemblies Interop.SHDocVw.dll and Microsoft.mshtml.dll to bin directory (Copy To Output in project references).
Running NUnit UI from the build agent. This runs the unit tests as expected. Leads me to believe there might be an issue with security. Runnint NUnit 2.0.5 on Windows Server 2008 R2 Enterprise.
Anyone had this issue?