I've got a build that completed successfully, but the number of completed tests is about 50% lower than I was expecting. On closer inspection, it seems that the Mono instance running NUnit died with SIGABRT. (!!) Yet TeamCity still considers this a successful test run, since none of the tests returned a failure condition.
How do I force TeamCity to consider this a failed build? (TeamCity 8.x)
The same problem occurs with TeamCity 9 if an NUnit [OneTimeSetUp] function fails. No tests run, NUnit returns with exit code 1, but TeamCity considers it a success.
If there's anything in the build log relating to your SIGABRT, you can add a Failure Condition, e.g.:
Fail build if its build log ...
contains text matching regexp ...
"SIGABRT" or in my case "Process exited with code [^0]"
Related
I see unreliable behaviour in the dotnet command when executed from a powershell script.
Executing in a powershell script:
dotnet build "$slnPath"
ignores any compilation errors and continues executing the script. I have to check $lastexitcode to see, if there are any errors.
On the other hand, the command:
dotnet test "$slnPath"
immediately terminates execution of the powershell script, if there are any failed unit tests.
Is that normal behavior? Do I need to write different error handling depending on the arguments of the dotnet command?
The dotnet test command launches the test runner console application specified for a project. The test runner executes the tests defined for a unit test framework (for example, MSTest, NUnit, or xUnit) and reports the success or failure of each test. If all tests are successful, the test runner returns 0 as an exit code; otherwise if any test fails, it returns 1.
But dotnet build building requires the project.assets.json file, which lists the dependencies of your application. The file is created when dotnet restore is executed. Without the assets file in place, the tooling can't resolve reference assemblies, which results in errors.
You can read what each command do(and may be found some -key to control they) at this adress tool_description
I'm running a TeamCity 8.1.2 build server with 3 agents on AWS cloud.
I'm experiencing unusually long delays following completion of an NUnit build step. I have just over 30,000 unit tests running, which take around 3 mins to complete. However the build log shows the step takes 11 mins to complete (at least, sometimes 15 mins), most of which seems to be due to a delay at the end of the step, internal to TeamCity.
EDIT This delay has increased over time, the build used to complete within 5-6 mins.
Here's an excerpt from the build log of a fairly quick run. Note the 7 min delay between 'Process exited' and '##teamcity':
[22:39:17]Step 6/7: Run Unit Tests (NUnit) (11m:26s)
[22:39:17][Step 6/7] ##teamcity[buildStatisticValue key='buildStageDuration:buildStepRUNNER_20' value='0.0']
[22:39:17][Step 6/7] Starting: D:\BuildAgent\plugins\dotnetPlugin\bin\JetBrains.BuildServer.NUnitLauncher.exe #TeamCityImplicit
[22:39:17][Step 6/7] in directory: D:\builds\X.Web\Build
[22:39:30][Step 6/7] Start TeamCity NUnit Test Runner
[22:39:30][Step 6/7] Running NUnit-2.6.3 tests under .NET Framework v4.0 x64
[22:39:39][Step 6/7] X.Tests.dll (2m:39s)
[22:42:59][Step 6/7] X.ServiceInterface.Tests.dll (1s)
[22:43:04][Step 6/7] X.Data.Tests.dll (24s)
[22:43:30][Step 6/7] Process exited with code 0
[22:50:44][Step 6/7] ##teamcity[buildStatisticValue key='buildStageDuration:buildStepRUNNER_20' value='686467.0']
[22:50:44]Step 7/7: Pack Feed NuGet Packages (NuGet Pack) (6s)
Any ideas? I'll drop a support ticket to jetbrains, but I thought I'd throw this out to the community in case someone's seen it before. Thanks in advance for any help.
TeamCity agent needs to report tests back to the server. I would say that the amount of tests reported causes the delay between step finish and build finish
I am using sonar-runner to run tests and code coverage over my C# code with the help of gallio plugin. The tests are running fine, but I am not able to see any code coverage on the sonar web UI.
My Sonar settings are as follows:
sonar-project.properties
mentioning only relevant bits
sonar.gallio.coverage.tool = NCover
sonar.NCover.installDirectory = C:/Program Files/NCover
sonar.donet.visualstudio.testProjectPattern = .Test
sonar.dotnet.buildConfigurations = "Release|x86"
Any idea what coule be missing??
sonar.projectKey=XXX:XXX
sonar.projectVersion=trunk
sonar.projectName=XXX
sources=.
sonar.language=cs
sonar.dotnet.visualstudio.solution.file=Project.sln
sonar.dotnet.excludeGeneratedCode=false
sonar.dotnet.4.0.sdk.directory=C:/WIndows/Microsoft.NET/Framework/v4.0.30319
sonar.dotnet.version=4.0
# Gallio
sonar.gallio.mode=
sonar.gallio.coverage.tool=NCover
sonar.gallio.runner=IsolatedAppDomain
sonar.NCover.installDirectory=c:/Program Files/NCover
sonar.gallio.installDirectory=C:/Program Files/Gallio
sonar.dotnet.test.assemblies=$(SolutionDir)/../**/bin/**/*.Tests.Unit.dll
# FXCop
sonar.fxcop.mode=
#StyleCop
sonar.stylecop.mode=
#NDeps
sonar.ndeps.mode=skip
sonar-runner.properties
You said
sonar.dotnet.buildConfigurations = "Release|x86"
If that's true, your build likely isn't generating .pdb files, which are needed to figure out the mapping between the binaries and your source files.
Does it work if you try it with a Debug build?
I was seeing this same behavior with NCover in Sonar. I found that Sonar was generating invalid arguments for Gallio's NCover runner.
Try piping the output from Sonar's runner into a text file so that you can examine the arguments more easily (on the command line, you can just type sonar-runner > output.txt to do this).
You will likely see a line like this in your output:
INFO .u.c.CommandExecutor - Executing command: C:\Program Files\Gallio\bin\Gallio.Echo.exe /r:Local /report-directory:E:\Reports\.sonar /report-name-format:gallio-report /report-type:Xml E:\Projects\UnitTests\bin\Release\UnitTests.dll /runner-property:NCoverCoverageFile=E:\Reports\.sonar\coverage-report.xml /runner-property:NCoverArguments=//ias MyFirstAssembly;MySecondtAssembly;MyThirdAssembly
If you attempt to execute this manually via Gallio on the command line, you will get an error:
Cannot find file 'MyFirstAssembly;MySecondtAssembly;MyThirdAssembly'
If you edit this list manually down to a single entry such as MyFirstAssembly*, everything will work as expected.
This seems to indicate that Sonar is generating invalid command line arguments for Gallio. As much as I love NCover, the easiest solution was to use OpenCover instead.
I've followed this tutorial to setup unit testing on my app when I got a little stuck.
At bullet point 8 in that tutorial it shows this image, which is what I should be expecting when I build: alt text http://developer.apple.com/library/ios/documentation/Xcode/Conceptual/iphone_development/art/logic_test_failure_text_editor.jpg
However this isn't what I get when I build. I get this error message: Command /bin/sh failed with exit code 1 as well as the error message the unit test has created. Then, when I expand on the first error I get this:
PhaseScriptExecution "Run Script" "build/3D Pool.build/Debug-iphonesimulator/LogicTests.build/Script-1A6BA6AE10F28F40008AC2A8.sh"
cd "/Users/james/Desktop/FYP/3D Pool"
setenv ACTION build
setenv ALTERNATE_GROUP staff
...
setenv XCODE_VERSION_MAJOR 0300
setenv XCODE_VERSION_MINOR 0320
setenv YACC /Developer/usr/bin/yacc
/bin/sh -c "\"/Users/james/Desktop/FYP/3D Pool/build/3D Pool.build/Debug-iphonesimulator/LogicTests.build/Script-1A6BA6AE10F28F40008AC2A8.sh\""
/Developer/Tools/RunPlatformUnitTests.include:412: note: Started tests for architectures 'i386'
/Developer/Tools/RunPlatformUnitTests.include:419: note: Running tests for architecture 'i386' (GC OFF)
objc[12589]: GC: forcing GC OFF because OBJC_DISABLE_GC is set
Test Suite '/Users/james/Desktop/FYP/3D Pool/build/Debug-iphonesimulator/LogicTests.octest(Tests)' started at 2010-01-04 21:05:06 +0000
Test Suite 'LogicTests' started at 2010-01-04 21:05:06 +0000
Test Case '-[LogicTests testFail]' started.
/Users/james/Desktop/FYP/3D Pool/LogicTests.m:17: error: -[LogicTests testFail] : Must fail to succeed.
Test Case '-[LogicTests testFail]' failed (0.000 seconds).
Test Suite 'LogicTests' finished at 2010-01-04 21:05:06 +0000.
Executed 1 test, with 1 failure (0 unexpected) in 0.000 (0.000) seconds
Test Suite '/Users/james/Desktop/FYP/3D Pool/build/Debug-iphonesimulator/LogicTests.octest(Tests)' finished at 2010-01-04 21:05:06 +0000.
Executed 1 test, with 1 failure (0 unexpected) in 0.000 (0.002) seconds
/Developer/Tools/RunPlatformUnitTests.include:448: error: Failed tests for architecture 'i386' (GC OFF)
/Developer/Tools/RunPlatformUnitTests.include:462: note: Completed tests for architectures 'i386'
Command /bin/sh failed with exit code 1
Now this is very odd as it is running the tests (and succeeding as you can see my STFail firing) because if I add a different test which passes I get no errors, so the tests are working fine. But why am I getting this extra build fail?
It may also be of note that when downloading solutions/templates which should work out the box, I get the same error. I'm guessing I've set something up wrong here but I've just followed a tutorial 100% correctly!
EDIT: According to this blog, this post and a few other websites, I'm not the only one getting this problem. It has been like this since the release of xCode 3.2, assuming the apple dev center documents and tutorials etc are pre-3.2 as well.
However some say its a known issue whereas others seem to think this was intentional. I for one would like both the extended console and in code messages, and I certainly do not like the "Command /bin/sh..." error and really think they would have documented such an update. Hopefully it will be fixed soon anyway.
UPDATE:
Here's confirmation it's something changed since the release of xCode 3.2.1.
This image:
alt text http://ing0.co.uk/info/pics/unittest-xcode-3.2.1.png is from my test build using 3.2.1. This one is from an older version (3.1.4):
alt text http://ing0.co.uk/info/pics/unittest-xcode-3.1.4.png. (The project for both was unchanged).
Double click 'Run script' under 'Targets'/'Your script target name' then type
# Run the unit tests in this test bundle.
"${SYSTEM_DEVELOPER_DIR}/Tools/RunUnitTests" 1> /tmp/RunUnitTests.out
Basically you need to append
1> /tmp/RunUnitTests.out
to what is already there
Have you tried to set the build configuration to "Debug" for the test case ?
i did that, as shown at http://nothing2fancy.com/2009/08/04/failed-tests-for-architecture-i386-gc-off/ and it worked
try to check this issue SenTestCase in Xcode 3.2 and XCBuildLogCommandInvocationSection Errors
the workaround in that question work for me.
Menu: Project --> New Build Phase --> New RunScript Build Phase --> Paste the following into the empty text area:
"${SYSTEM_DEVELOPER_DIR}/Tools/RunUnitTests" 1> /tmp/RunUnitTests.out
Now try to build and you should get an error on failed unit tests.
I don't know it a right way or not, but it's work for me.
In General Tab of info Test Case, I
add my main application executable
target as a direct dependency.
And no error for ‘i386′ (GC OFF)”.
Good luck.
When you create your firt test case classes, make sure the Target is set to your UnitTest target, not your application target.
If you forget to do this when you create the class, you can "Get Info" on the existing class, and set the target there....
OK, never played with unit tests in XCode (of course I liked JUnit in good old java times). So it is just a trial: What compiler are you using, GCC or LLVM? Maybe GCC is the better choice to be on the safe side.
Do you have the same behaviour when "Build Active Architecture Only" in Project settings is enabled? This switch removes duplicate error messages when building for armv6 and armv7, maybe it's related.
Please have a look at apple's latest sample. When you download the code there is a read me file in which its clearly written:
———————————————————————————————————————————————————————————————————————————————
Running Logic Tests
To run the logic tests:
1. Choose Project > Set Active Target > CalculatorTests.
2. Choose Project > Set Active SDK > Simulator.
3. Choose Build > Build. Xcode runs the test cases implemented in
the CalculatorTests.m file.
4. Choose Build > Build Results to open the Build Results window, containing
the tests results. You may have to click the Show Transcript button (the
third button on the bottom-left corner of the build results list) to view
the build transcript.
The logic-tests results look similar to this:
PhaseScriptExecution <project_directory>/build/iPhoneUnitTests.build/Debug-iphonesimulator/CalculatorTests.build/Script-17AA84010F99894F00167681.sh
cd <project_directory>
/bin/sh -c <project_directory>/build/iPhoneUnitTests.build/Debug-iphonesimulator/CalculatorTests.build/Script-17AA84010F99894F00167681.sh
/Developer/Tools/RunPlatformUnitTests.include:364: note: Started tests for architectures 'i386'
/Developer/Tools/RunPlatformUnitTests.include:371: note: Running tests for architecture 'i386' (GC OFF)
objc[1222]: GC: forcing GC OFF because OBJC_DISABLE_GC is set
objc[1222]: GC: forcing GC OFF because OBJC_DISABLE_GC is set
Test Suite '<project_directory>/build/Debug-iphonesimulator/CalculatorTests.octest(Tests)' started at 2009-05-19 16:55:28 -0700
Test Suite 'CalculatorTests' started at 2009-05-19 16:55:28 -0700
<time> otest[1222:80f] -[CalculatorTests testAddition] setUp
<time> otest[1222:80f] -[CalculatorTests testAddition] start
<time> otest[1222:80f] -[CalculatorTests testAddition] end
<time> otest[1222:80f] -[CalculatorTests testAddition] tearDown
Test Case '-[CalculatorTests testAddition]' passed (0.007 seconds).
<time> otest[1222:80f] -[CalculatorTests testDivision] setUp
<time> otest[1222:80f] -[CalculatorTests testDivision] start
<time> otest[1222:80f] -[CalculatorTests testDivision] end
<time> otest[1222:80f] -[CalculatorTests testDivision] tearDown
Test Case '-[CalculatorTests testDivision]' passed (0.003 seconds).
<time> otest[1222:80f] -[CalculatorTests testInputException] setUp
<time> otest[1222:80f] -[CalculatorTests testInputException] start
<time> otest[1222:80f] -[CalculatorTests testInputException] end
<time> otest[1222:80f] -[CalculatorTests testInputException] tearDown
...
Test Case '-[CalculatorTests testSubtractionNegativeResult]' passed (0.002 seconds).
Test Suite 'CalculatorTests' finished at 2009-05-19 16:55:28 -0700.
Executed 6 tests, with 0 failures (0 unexpected) in 0.021 (0.022) seconds
Test Suite '<project_directory>/build/Debug-iphonesimulator/CalculatorTests.octest(Tests)' finished at 2009-05-19 16:55:28 -0700.
Executed 6 tests, with 0 failures (0 unexpected) in 0.021 (0.024) seconds
/Developer/Tools/RunPlatformUnitTests.include:388: note: Passed tests for architecture 'i386' (GC OFF)
/Developer/Tools/RunPlatformUnitTests.include:399: note: Completed tests for architectures 'i386'
Remember that logic tests are executed as part of the build process to provide
you with build errors for failed unit tests. Logic unit-test bundles are not
intented to run in iPhone Simulator or a device.
———————————————————————————————————————————————————————————————————————————————
Hence there is some change in the sentesting kit. I don't think it would be worth for filing a bug.
That is the expected behavior. The build will fail if one of the tests fail and the build will succeed if all the tests pass.
I am new to XCode and testing, and have followed the steps outlined in the iPhone Development Guide under "Unit Testing Applications".
I added a unit-test bundle to my project, named it "LogicTests", set it as the active target, then added a sample "LogicTests" unit-test class under a new group called "Tests". I wrote the simple one-line test from the doc (STFail(#"Must fail to succeed");), then tried to build my project.
I keep getting the errors: Command /bin/sh failed with exit code 1 and /Developer/Tools/RunPlatformUnitTests.include:448: error: Failed tests for architecture 'i386' (GC OFF).
The second error looks like the test is being run and failing properly, but the first error I am completely clueless about... What could the problem be here?
Thanks,
-Greta
Well this behavior is totally normal. If the unit test failed it will produce a return code of 1. Test the opposite with STTrue(1,#"some message") and it should not return 1 and finish sucessfully.
The test thinks your compiling for Intel not ARM architecture. Unfortunately, I can't remember off the top of my head how to set up the target properly.