TeamCity NUnit build step ends with long delay? - nunit

I'm running a TeamCity 8.1.2 build server with 3 agents on AWS cloud.
I'm experiencing unusually long delays following completion of an NUnit build step. I have just over 30,000 unit tests running, which take around 3 mins to complete. However the build log shows the step takes 11 mins to complete (at least, sometimes 15 mins), most of which seems to be due to a delay at the end of the step, internal to TeamCity.
EDIT This delay has increased over time, the build used to complete within 5-6 mins.
Here's an excerpt from the build log of a fairly quick run. Note the 7 min delay between 'Process exited' and '##teamcity':
[22:39:17]Step 6/7: Run Unit Tests (NUnit) (11m:26s)
[22:39:17][Step 6/7] ##teamcity[buildStatisticValue key='buildStageDuration:buildStepRUNNER_20' value='0.0']
[22:39:17][Step 6/7] Starting: D:\BuildAgent\plugins\dotnetPlugin\bin\JetBrains.BuildServer.NUnitLauncher.exe #TeamCityImplicit
[22:39:17][Step 6/7] in directory: D:\builds\X.Web\Build
[22:39:30][Step 6/7] Start TeamCity NUnit Test Runner
[22:39:30][Step 6/7] Running NUnit-2.6.3 tests under .NET Framework v4.0 x64
[22:39:39][Step 6/7] X.Tests.dll (2m:39s)
[22:42:59][Step 6/7] X.ServiceInterface.Tests.dll (1s)
[22:43:04][Step 6/7] X.Data.Tests.dll (24s)
[22:43:30][Step 6/7] Process exited with code 0
[22:50:44][Step 6/7] ##teamcity[buildStatisticValue key='buildStageDuration:buildStepRUNNER_20' value='686467.0']
[22:50:44]Step 7/7: Pack Feed NuGet Packages (NuGet Pack) (6s)
Any ideas? I'll drop a support ticket to jetbrains, but I thought I'd throw this out to the community in case someone's seen it before. Thanks in advance for any help.

TeamCity agent needs to report tests back to the server. I would say that the amount of tests reported causes the delay between step finish and build finish

Related

Is it normal if i have more than 200 objectboxes declared in a project?

Im having a project where the requirement is to manage 200++ models persisted in objectbox. I've proceed with the plan, but i find it uncertain whenever i run "flutter packages pub run build-runner build" and it tooks around 15 - 20 minutes to finish. Is this normal??
I've created 200++ models with complex attributes and declared as Box for each of them, I'm not sure is this too overwhelm?
I expect the "flutter packages pub run build-runner build" process takes less than 3 minutes to complete.

How to stop powershell script after failed dotnet command?

I see unreliable behaviour in the dotnet command when executed from a powershell script.
Executing in a powershell script:
dotnet build "$slnPath"
ignores any compilation errors and continues executing the script. I have to check $lastexitcode to see, if there are any errors.
On the other hand, the command:
dotnet test "$slnPath"
immediately terminates execution of the powershell script, if there are any failed unit tests.
Is that normal behavior? Do I need to write different error handling depending on the arguments of the dotnet command?
The dotnet test command launches the test runner console application specified for a project. The test runner executes the tests defined for a unit test framework (for example, MSTest, NUnit, or xUnit) and reports the success or failure of each test. If all tests are successful, the test runner returns 0 as an exit code; otherwise if any test fails, it returns 1.
But dotnet build building requires the project.assets.json file, which lists the dependencies of your application. The file is created when dotnet restore is executed. Without the assets file in place, the tooling can't resolve reference assemblies, which results in errors.
You can read what each command do(and may be found some -key to control they) at this adress tool_description

TeamCity seems to ignore exit code from NUnit

I've got a build that completed successfully, but the number of completed tests is about 50% lower than I was expecting. On closer inspection, it seems that the Mono instance running NUnit died with SIGABRT. (!!) Yet TeamCity still considers this a successful test run, since none of the tests returned a failure condition.
How do I force TeamCity to consider this a failed build? (TeamCity 8.x)
The same problem occurs with TeamCity 9 if an NUnit [OneTimeSetUp] function fails. No tests run, NUnit returns with exit code 1, but TeamCity considers it a success.
If there's anything in the build log relating to your SIGABRT, you can add a Failure Condition, e.g.:
Fail build if its build log ...
contains text matching regexp ...
"SIGABRT" or in my case "Process exited with code [^0]"

Teamcity not DLL's for some NUnit Test projects

I get this error when running my Moq tests through Teamcity 5
Test(s) failed.
System.IO.FileNotFoundException :
Could not load file or assembly 'Moq,
Version=3.1.416.3, Culture=neutral,
PublicKeyToken=69f491c39445e920' or
one of its dependencies. The system
cannot find the file specified. at
MyCode.Tests.SomeHandlerTests.Setup()
The tests run fine on my local; they just fail on the build server.
I made sure the assemblies are in the Bin (looking at them now over RDP just be double sure).
So the issue was to do with the Test DLL search path under the nunit settings
It was:
..\Tests\**\*Test*.dll
But is now:
..\Tests\*\bin\Debug\*Test*.dll
And things work nicely
UPDATE
http://confluence.jetbrains.com/display/TCD8/NUnit
You can use this pattern
**\*.dll
as long as you add this pattern in the "Do not run tests from" field
**\obj\**\*.dll
I had a similar issue, but found that I had different version's of Moq between my 2 Test projects.
The issue that I had was that the correct version was not available.
Just do
Update-Package Moq
From the Package Manager command line

Unit testing in Xcode doesn't work like in Apple's example [duplicate]

I've followed this tutorial to setup unit testing on my app when I got a little stuck.
At bullet point 8 in that tutorial it shows this image, which is what I should be expecting when I build: alt text http://developer.apple.com/library/ios/documentation/Xcode/Conceptual/iphone_development/art/logic_test_failure_text_editor.jpg
However this isn't what I get when I build. I get this error message: Command /bin/sh failed with exit code 1 as well as the error message the unit test has created. Then, when I expand on the first error I get this:
PhaseScriptExecution "Run Script" "build/3D Pool.build/Debug-iphonesimulator/LogicTests.build/Script-1A6BA6AE10F28F40008AC2A8.sh"
cd "/Users/james/Desktop/FYP/3D Pool"
setenv ACTION build
setenv ALTERNATE_GROUP staff
...
setenv XCODE_VERSION_MAJOR 0300
setenv XCODE_VERSION_MINOR 0320
setenv YACC /Developer/usr/bin/yacc
/bin/sh -c "\"/Users/james/Desktop/FYP/3D Pool/build/3D Pool.build/Debug-iphonesimulator/LogicTests.build/Script-1A6BA6AE10F28F40008AC2A8.sh\""
/Developer/Tools/RunPlatformUnitTests.include:412: note: Started tests for architectures 'i386'
/Developer/Tools/RunPlatformUnitTests.include:419: note: Running tests for architecture 'i386' (GC OFF)
objc[12589]: GC: forcing GC OFF because OBJC_DISABLE_GC is set
Test Suite '/Users/james/Desktop/FYP/3D Pool/build/Debug-iphonesimulator/LogicTests.octest(Tests)' started at 2010-01-04 21:05:06 +0000
Test Suite 'LogicTests' started at 2010-01-04 21:05:06 +0000
Test Case '-[LogicTests testFail]' started.
/Users/james/Desktop/FYP/3D Pool/LogicTests.m:17: error: -[LogicTests testFail] : Must fail to succeed.
Test Case '-[LogicTests testFail]' failed (0.000 seconds).
Test Suite 'LogicTests' finished at 2010-01-04 21:05:06 +0000.
Executed 1 test, with 1 failure (0 unexpected) in 0.000 (0.000) seconds
Test Suite '/Users/james/Desktop/FYP/3D Pool/build/Debug-iphonesimulator/LogicTests.octest(Tests)' finished at 2010-01-04 21:05:06 +0000.
Executed 1 test, with 1 failure (0 unexpected) in 0.000 (0.002) seconds
/Developer/Tools/RunPlatformUnitTests.include:448: error: Failed tests for architecture 'i386' (GC OFF)
/Developer/Tools/RunPlatformUnitTests.include:462: note: Completed tests for architectures 'i386'
Command /bin/sh failed with exit code 1
Now this is very odd as it is running the tests (and succeeding as you can see my STFail firing) because if I add a different test which passes I get no errors, so the tests are working fine. But why am I getting this extra build fail?
It may also be of note that when downloading solutions/templates which should work out the box, I get the same error. I'm guessing I've set something up wrong here but I've just followed a tutorial 100% correctly!
EDIT: According to this blog, this post and a few other websites, I'm not the only one getting this problem. It has been like this since the release of xCode 3.2, assuming the apple dev center documents and tutorials etc are pre-3.2 as well.
However some say its a known issue whereas others seem to think this was intentional. I for one would like both the extended console and in code messages, and I certainly do not like the "Command /bin/sh..." error and really think they would have documented such an update. Hopefully it will be fixed soon anyway.
UPDATE:
Here's confirmation it's something changed since the release of xCode 3.2.1.
This image:
alt text http://ing0.co.uk/info/pics/unittest-xcode-3.2.1.png is from my test build using 3.2.1. This one is from an older version (3.1.4):
alt text http://ing0.co.uk/info/pics/unittest-xcode-3.1.4.png. (The project for both was unchanged).
Double click 'Run script' under 'Targets'/'Your script target name' then type
# Run the unit tests in this test bundle.
"${SYSTEM_DEVELOPER_DIR}/Tools/RunUnitTests" 1> /tmp/RunUnitTests.out
Basically you need to append
1> /tmp/RunUnitTests.out
to what is already there
Have you tried to set the build configuration to "Debug" for the test case ?
i did that, as shown at http://nothing2fancy.com/2009/08/04/failed-tests-for-architecture-i386-gc-off/ and it worked
try to check this issue SenTestCase in Xcode 3.2 and XCBuildLogCommandInvocationSection Errors
the workaround in that question work for me.
Menu: Project --> New Build Phase --> New RunScript Build Phase --> Paste the following into the empty text area:
"${SYSTEM_DEVELOPER_DIR}/Tools/RunUnitTests" 1> /tmp/RunUnitTests.out
Now try to build and you should get an error on failed unit tests.
I don't know it a right way or not, but it's work for me.
In General Tab of info Test Case, I
add my main application executable
target as a direct dependency.
And no error for ‘i386′ (GC OFF)”.
Good luck.
When you create your firt test case classes, make sure the Target is set to your UnitTest target, not your application target.
If you forget to do this when you create the class, you can "Get Info" on the existing class, and set the target there....
OK, never played with unit tests in XCode (of course I liked JUnit in good old java times). So it is just a trial: What compiler are you using, GCC or LLVM? Maybe GCC is the better choice to be on the safe side.
Do you have the same behaviour when "Build Active Architecture Only" in Project settings is enabled? This switch removes duplicate error messages when building for armv6 and armv7, maybe it's related.
Please have a look at apple's latest sample. When you download the code there is a read me file in which its clearly written:
———————————————————————————————————————————————————————————————————————————————
Running Logic Tests
To run the logic tests:
1. Choose Project > Set Active Target > CalculatorTests.
2. Choose Project > Set Active SDK > Simulator.
3. Choose Build > Build. Xcode runs the test cases implemented in
the CalculatorTests.m file.
4. Choose Build > Build Results to open the Build Results window, containing
the tests results. You may have to click the Show Transcript button (the
third button on the bottom-left corner of the build results list) to view
the build transcript.
The logic-tests results look similar to this:
PhaseScriptExecution <project_directory>/build/iPhoneUnitTests.build/Debug-iphonesimulator/CalculatorTests.build/Script-17AA84010F99894F00167681.sh
cd <project_directory>
/bin/sh -c <project_directory>/build/iPhoneUnitTests.build/Debug-iphonesimulator/CalculatorTests.build/Script-17AA84010F99894F00167681.sh
/Developer/Tools/RunPlatformUnitTests.include:364: note: Started tests for architectures 'i386'
/Developer/Tools/RunPlatformUnitTests.include:371: note: Running tests for architecture 'i386' (GC OFF)
objc[1222]: GC: forcing GC OFF because OBJC_DISABLE_GC is set
objc[1222]: GC: forcing GC OFF because OBJC_DISABLE_GC is set
Test Suite '<project_directory>/build/Debug-iphonesimulator/CalculatorTests.octest(Tests)' started at 2009-05-19 16:55:28 -0700
Test Suite 'CalculatorTests' started at 2009-05-19 16:55:28 -0700
<time> otest[1222:80f] -[CalculatorTests testAddition] setUp
<time> otest[1222:80f] -[CalculatorTests testAddition] start
<time> otest[1222:80f] -[CalculatorTests testAddition] end
<time> otest[1222:80f] -[CalculatorTests testAddition] tearDown
Test Case '-[CalculatorTests testAddition]' passed (0.007 seconds).
<time> otest[1222:80f] -[CalculatorTests testDivision] setUp
<time> otest[1222:80f] -[CalculatorTests testDivision] start
<time> otest[1222:80f] -[CalculatorTests testDivision] end
<time> otest[1222:80f] -[CalculatorTests testDivision] tearDown
Test Case '-[CalculatorTests testDivision]' passed (0.003 seconds).
<time> otest[1222:80f] -[CalculatorTests testInputException] setUp
<time> otest[1222:80f] -[CalculatorTests testInputException] start
<time> otest[1222:80f] -[CalculatorTests testInputException] end
<time> otest[1222:80f] -[CalculatorTests testInputException] tearDown
...
Test Case '-[CalculatorTests testSubtractionNegativeResult]' passed (0.002 seconds).
Test Suite 'CalculatorTests' finished at 2009-05-19 16:55:28 -0700.
Executed 6 tests, with 0 failures (0 unexpected) in 0.021 (0.022) seconds
Test Suite '<project_directory>/build/Debug-iphonesimulator/CalculatorTests.octest(Tests)' finished at 2009-05-19 16:55:28 -0700.
Executed 6 tests, with 0 failures (0 unexpected) in 0.021 (0.024) seconds
/Developer/Tools/RunPlatformUnitTests.include:388: note: Passed tests for architecture 'i386' (GC OFF)
/Developer/Tools/RunPlatformUnitTests.include:399: note: Completed tests for architectures 'i386'
Remember that logic tests are executed as part of the build process to provide
you with build errors for failed unit tests. Logic unit-test bundles are not
intented to run in iPhone Simulator or a device.
———————————————————————————————————————————————————————————————————————————————
Hence there is some change in the sentesting kit. I don't think it would be worth for filing a bug.
That is the expected behavior. The build will fail if one of the tests fail and the build will succeed if all the tests pass.