I'm trying to integrate Robot Framework (an acceptance testing framework) with TeamCity. In order to do this it needs to send messages to the console output which TeamCity will then read and return realtime test progress/results. I'm doing this by calling the command line to run the tests with a simple exec task. Everything seemed to be working other than I was only getting the results at the end of the run and not on the fly.
After a bit of struggling with NAnt I swapped to using MSBuild and everything worked first time.
I have what I need now, but for completeness I'd like to find out why I couldn't get it working with NAnt. As far as I can tell the issue is that NAnt is prefixing all console output with [exec]. Is it possible to suppress this?
I don't think the console output is configurable.
NAnt is open source: you could fork your own copy and/or submit a feature patch.
Related
I am adding a PowerShell script to run in my build to get the Version Number for the build. When I use the builder(Box with ...) to get the file, it takes me to my TFS Project and I work my way down to the file. When the build definition runs it fails at that step, because it cant find my script.
What it has done is prepended the local Servers work directory to the front of the Path it had me choose.
I feel this is a Bug or how am I suppose to get the most current copy of a script in TFS when the Build Definition runs.
Tried with a simple powershell script on my side, but couldn't reproduce your issue, the script is working.
Only add one step in build, and map to $/teamproject in Repositity, then select the script:
To narrow down the issue, you could create a new pipeline with only one task-Powershell, check what will happen.
If you still get error, please share detail logs with system.debug=true enable.
Also take a look at this tutorial about how to use powershell fetch/change build number, which may helps-- Use a PowerShell script to customize your build pipeline
Update
According to your error info, you are lacking of the definition.
Please make sure you have specified the value in options--build number format
$(BuildDefinitionName)_$(Year:yyyy).$(Month).$(DayOfMonth)$(Rev:.r)
Then it should run without any problem:
I'm currently moving one of our projects to DNX (.NET Core now) and I was forced to update to nunit3. Because of other considerations, we run compile the test project as a console app with its own entry point, basically self-hosting the NUnit runner.
I now need to report the results to TeamCity via the XML Reporter, which doesn't seem to parse Nunit3 TestResults.xml files.
Any advice on how to work around this?
The NUnit 3 console has the option to produce results formatted in the NUnit 2 style.
Use the option:
--result=[filename];format=nunit2
Docs: https://github.com/nunit/nunit/wiki/Console-Command-Line
To add to the answer above:
NUnitLite inherits the --result CLI parameter which seems to do the trick.
Another option, which I went for in the end is using the --teamcity CLI parameter:
dotnetbuild --project:<path to project directory> -- --teamcity
which will integrate with TC's service messages. This will also do real-time updates.
Is it possible to run NCover automatically from code instead of running NCover manually or via command line?
Here is the scenario, I have written a few tests, I execute all the tests and after the tests are completed, NCover should run automatically for that particular test project and store the coverage report as an XML in a location.
Is this possible to do? Kindly help.
Running NCover from the command line was the only option with NC3. When we updated NC4 the default works like this --> you create a project, the NCover service watches for a process to start that meets the match rules defined in the project, and then collects coverage on it.
This doc may be of some help: http://www.ncover.com/support/docs/desktop/user-guide/coverage_scenarios/how_do_i_collect_data_from_nunit
If you have more questions, please reach out to us at support#ncover.com.
In our msbuild script we can run unit tests without using the vsmdi file:
We would like to enforce tests to be run on check in. The "Testing policy" seems to require a .vsmdi file. Is there a way to do this without one?
The out-of-the-box testing policy simply checks for the presence of a TestResults folder that has been created since the last checkin, indicating that MSTest has been run. If it finds one it then scans the test results for a reference to the .vsmdi file that is mentioned.
You could write a custom policy that does much the same thing but instead of looking for a .vsmdi file in the test results look for something else as an indicator that tests you want run have actually been run (and that they pass).
Nunit works quite well with CruiseControl.NET, but there is one thing that irritates me a lot.
If there is a test that causes Nunit to crash, I would only get little information about the crash because the XML report of Nunit doesn't get a chance to be created and be merged into the CruiseControl report.
I need a way to report the progress even when Nunit crashes during the execution.
I have been tried to force each test to output some information to the console to resolve this problem. I have thought about using SetUp method, but I haven't found any good way to get the name of the current running test.
I think a better answer would be to create an NUnit Add-in that implements EventListener interface to capture the TestStarted event to output the progress to the console or a file.
The EventListener interface is documented on NUnit website: http://nunit.org/index.php?p=eventListeners&r=2.5
In addition, we can make the Dashboard report better even when NUnit crashes during its execution. We can use the following procedure to ensure that the DashBoard always shows something about the tests.
Run tests with the EventListener which outputs the progress to a separate file
After running tests, use another program to check the file
If the file does not contain a specific "end line", generate a special XML report based on the file and merge it into the CruiseControl log
If getting the name of the current running test is what you're after you could grab it with the following:
using System.Diagnostics;
...
[Test]
public void SomeTestThatWillCrash()
{
StackFrame sf = new StackFrame();
Console.WriteLine("Now running method: " + sf.GetMethod().Name);
...
}
CruiseControl.net recommends that you use NUnit through your builder (i.e. NAnt/MSBuild). See here: http://confluence.public.thoughtworks.org/display/CCNET/NUnit+Task. As they describe - it will allow you to run these tests locally first - which should give you an exception that you can clear up.
That being said - are your developers running these unit tests prior to checking in code? That could ease this issue. If its an integration issue - I would suggest grabbing the latest code base and running the tests locally to see what is out of sorts.
I don't know if NUnit is able to create the results file even when it crashes. Even if it did - you could run into problems if that file is not well formed due to the crash.
You could use #jpoh's approach but do it in the TestSetup method which would require you do it per-fixture. If really needed, you could write a base class that all your test fixtures inherit from that implement this method.
Another solution is to use MSBuild to run NUnit and use the task in the MSBuildCommunityTasks library. This allows you to continue on error and also get the error code back from NUnit. You won't get what method caused the problem, but might help some. Here is my MSBuild target:
<Target Name="UnitTest"
DependsOnTargets="BuildIt">
<NUnit Assemblies="#(TestAssemblies)"
ToolPath="$(NUnitx86Path)"
WorkingDirectory="%(TestAssemblies.RootDir)%(TestAssemblies.Directory)"
OutputXmlFile="#(TestAssemblies->'%(FullPath).$(NUnitFile)')"
Condition="'#(TestAssemblies)' != ''"
ExcludeCategory="$(ExcludeNUnitCategories)"
ContinueOnError="true">
<Output TaskParameter="ExitCode" ItemName="NUnitExitCodes"/>
</NUnit>
<!-- Copy the test results for the CCNet build before a possible build failure (see next step) -->
<CallTarget Targets="CopyTestResults" Condition="'#(TestAssemblies)' != ''"/>
<Error Text="Test error(s) occured" Code="%(NUnitExitCodes.Identity)" Condition=" '%(NUnitExitCodes.Identity)' != '0' And '#(TestAssemblies)' != ''"/>
</Target>
This probably won't fit your needs as is, but is something to try out and play with.
That said, I would agree with #rifferte that it sounds like you need to debug the problem locally and not rely on CC.NET to handle the reporting.