I've managed to set up unit tests for my library in Xcode 4. I've performed builds with tests that I know will pass and fail (i.e. STAssertTrue(YES) and STAssertTrue(NO) ) just to make sure it's working. I'm using the default apple SenTest libraries following this document.
However, when my tests are running I'm getting this error in the build log :
An internal error occurred when handling command output: -[IDEActivityLogSectionRecorder endMarker]: unrecognized selector sent to instance 0x20310b580
To be clear, it's not affecting the running of the tests at all, just the output into the build window. All the tests run each time so I can tell a pass / fail by looking to see if the build succeeds or fails.
However, when my tests fail I can't find out which one fails because the output seems to stop when it gets to that error.
Does anyone have experience with unit testing / Xcode 4 / this error?
I just posted this on another thread, but I'm going the opposite direction for Xcode 4.
Please see my blog post exploring the topic, leave a comment if you think I'm wrong.
I realise it doesn't directly answer your question, but forget SenTestingKit and use GHUnit. It'll take you about 10 minutes to figure out (much more straightforward than OCUnit) and will save you a lot of headaches. IMHO, Apple should be shipping it with Xcode instead of OCUnit.
GHUnit can run your tests in a true application environment (with a GUI), or on the command line. It literally just drops into your existing project as a separate target.
https://github.com/gabriel/gh-unit
Related
I've got some conditions where I need my frameworks to report a fatalError or a preconditionFailure and crash. I have set up some unit tests around these conditions to make sure they hit correctly.
It works really well if the unit tests are run without a Test Host application, but the moment a Test Host application is involved, any fatalError or preconditionFailure hits an automatic XCode break with the message Thread 1: Fatal error: <CUSTOM ERROR HERE>. and stops the execution of the rest of the tests until I press the Continue program execution button manually.
This is obviously an issue because I can't just let my tests run in the background and I need to be actively hitting the Continue button every time one of these tests hits which is quite annoying.
Is there any way to disable this XCode break?
Looks like this is fixed in Nimble 7.1.3, a library which I use to test these particular cases. I also found this specific issue in the Nimble library issue 478 which has a comment on a workaround you can use to disable the debug executable when running you tests. Hopefully this helps anyone else stuck with the same problem I had.
I'm following this guide about logic unit testing:
https://developer.apple.com/library/ios/#documentation/DeveloperTools/Conceptual/UnitTesting/02-Setting_Up_Unit_Tests_in_a_Project/setting_up.html
I've added tests to my existing project just as in instruction, but when i try to run them in simulator, I'm getting this error:
Stack trace:
Also, I've tried to create completely new project and repeat procedure, but the problem remains. Application unit testing runs as expected.
Found a solution, Xcode 4.3 set the LLDB as a default debugger in all of your schemes, I changed it to GDB (Product->Edit Scheme->Test->Debugger) and the problem disappears. Why it causes the crash is another interesting question.
Ok
SO recently I've being having a lot of trouble with an application working in debug mode but not working in distribution mode.
Is it possible to use xcode debugging tools such as break points and variable tracing with an adhoc distribution build of an app?
If it's not how would one usually go about debugging such a thing?
Assuming that your crash logs aren't giving you any clues (you'll need to hook up to the device to get them) there are lots of things you can do.
But I'd start by looking at the crash logs ... the clues will be there and don't forget Apple make distribution crash logs available to you through iTunesConnect.
1) Copious logging is one thing. Lots of developers use a switch so that in debug, logs go to the console but for other builds they are dropped. Consider a different option where you log to a file instead. You could then push the log files to a remote server for debugging. It's a bit of a slog to set this all up, but once you've done it, you'll wonder how you ever lived without it.
2) Another option is to use Flurry and log events when you detect that things have gone wrong. This can cover more controlled problems when things aren't as expected rather than random crashes. This can be a useful feature for released apps provided your terms and conditions are clear about what data you are logging and why.
3) Make sure you do a clean build, I'm sure you've already done this, but sometimes it clears these issues.
4) Are you using external libraries / modules? I've come across issues with older versions of TT where the arm6/7 build settings were wrong and this was causing issues for distribution builds. Basically check through the build settings for each profile and make sure it is what you expect.
5) Suspect a race condition. In distribution mode (often because the logging is turned off) you will find that your application runs a little bit faster. That can reveal timing issue bugs in badly written code.
So yes ... there is a lot you can do ... you just can't attach the debugger ;-)
Not possible to debug an application in distribution mode.
The build configuration difference between Distribution, Debug, and Release is really whatever parameters you have set for that in XCode. If your Distribution config is giving you problems and the release isn't, the easiest way to fix it is to go back through Apple's steps on copying the Release config and making the changes to make it a Distribution config, like you did originally.
The alternative is to go through every line of the configurations for Release and Distribution and find what's different. The other way is a LOT faster. :)
NOTE: Unit testing is a lot easier to setup nowadays. This tutorial is not really relevant for Xcode version 5 and above.
It took me quite some time but I finally managed to make it work for my project.
To create the "logic" tests I followed Apple guidelines on creating logic tests.
This works fine once you understand that the logic tests are run during build.
To be able to debug those tests it is required to create a custom executable that will call those tests. The article by Sean Miceli on the Grokking Cocoa blog provides all the information to do this. Following it however did not yield immediate success and needed some tweaking.
I will go over the main steps presented in Sean's tutorial providing some "for dummies" outline which took me some time to figure out:
Setup a target that contains the unit tests but DOES NOT run them
Setup the otest executable to run the tests
Setup the otest environment variables so that otest can find your unit tests
The following was performed with XCode 3.2.5
Note for XCode 4
In XCode 4 it is possible to debug your unit tests DIRECTLY. Just write your test, add it to your target as one of the tests and set a breakpoint in it. That's all. More will come.
Step 1 - Setting up the target
Duplicate your unit tests target located under your project Targets. This will also create a duplicate of your unit tests product (.octest file). In the figure below "LogicTest" is the original target.
Rename both the unit tests target and the unit tests product (.octest file) to the same name. In the figure below "LogicTestsDebug" is the duplicate target.
Delete the RunScript phase of the new target
The name of both can be anything but I would avoid spaces.
Step 2 - Setting up otest
The most important point here is to get the correct otest, i.e. the one for your current iOS and not the default Mac version. This is well described in Sean's tutorial. Here are a few more details which helped me setting things right:
Go Project->New Custom Executable. This will pop open a window prompting you to enter an Executable Name and an Executable Path.
Type anything you wish for the name.
Copy paste the path to your iOS otest executable. In my case this was /Developer/Platforms/iPhoneSimulator.platform/Developer/SDKs/iPhoneSimulator4.2.sdk/Developer/usr/bin/otest
Press enter. This will bring you to the configuration page of your executable.
The only thing to change at this point is to select "Path Type: Relative to current SDK". Do not type in the path, this was done at step 3.
Step 3 - Setting up the otest arguments and environment variables
The otest arguments are straightforward to setup... But this proved to be my biggest problem. I initially had named my logic test target "LogicTests Debug". With this name and "LogicTests Debug.octest" (with quotes) as argument to otest I kept having otest terminating with exit code 1 and NEVER stopping into my code...
The solution: no space in your target name!
The arguments to otest are:
-SenTest Self (or All or a test name - type man otest in terminal to get the list)
{LogicTestsDebug}.octest - Where {LogicTestsDebug} needs to be replaced by your logic test bundle name.
Here is the list of environment variables for copy/pasting:
DYLD_ROOT_PATH: $SDKROOT
DYLD_FRAMEWORK_PATH: "${BUILD_PRODUCTS_DIR}: ${SDK_ROOT}:${DYLD_FRAMEWORK_PATH}"
IPHONE_SIMULATOR_ROOT: $SDKROOT
CFFIXED_USER_HOME: "${HOME}/Library/Application Support/iPhone Simulator/User"
DYLD_LIBRARY_PATH: ${BUILD_PRODUCTS_DIR}:${DYLD_LIBRARY_PATH}
DYLD_NEW_LOCAL_SHARED_REGIONS: YES
DYLD_NO_FIX_PREBINDING: YES
Note that I also tried the DYLD_FORCE_FLAT_NAMESPACE but this simply made otest crash.
Step 4 - Running your otest executable
To run your otest executable and start debugging your tests you need to:
Set your active target to your unit test target (LogicTestsDebug in my case)
Set your active executable to your otest executable
You can build and run your executable and debug your tests with breakpoints.
As a side note if you are having problems running your otest executable it can be related to:
Faulty path. I had lots of problem initially because I was pointing to the mac otest. I kept crashing on launch with termination code 6.
Faulty arguments. Until I removed the space from bundle (.octest) name I kept having otest crash with exit code 1.
Wrong path in environment variables. Sean tutorial has lots of follow-up questions giving some insight on what other people tried. The set I have now seems to work so I suggest you start with this.
You may get some message in the console which might lead you to think something is wrong with your environment variables. You may notice a message regarding CFPreferences. This message is not preventing the tests from running properly so don't focus on it f you have problems running otest.
Last once everything is working you will be able to stop at breakpoints in your tests.
One last thing...
I've read on many blogs that the main limitation of the integrated XCode SenTestKit is that tests cannot be run while building the application. Well as it turns out this is in fact quite easy to manage. You simply need to add your Logic tests bundle as a dependency to your application project. This will make sure your logic tests bundle is built, i.e. all tests are run, before your application is built.
To do this you can drag and drop your logic test bundle onto your application target.
This post is intended as a "How-to" more than a real question. Therefore this answer is just meant to allow me to mark the "How-to" as "answered". This will probably be flagged by the community as irregular. I'm up for suggestions on where to post future "How-to" articles.
One final note though on this topic.
For those who still wonder whether writing unit tests is worth it I would definitely say Yes!
I am currently writing an application with CoreData and retrieval of data from a web service (xml parsing). The complete model can be tested and debugged without having to:
run the actual application on the simulator or device. Not having to use the device to run the tests is a huge gain of time. It's the difference between 2 minutes and 5 seconds per run.
without the need to create views or controllers when testing the model. The complete development and testing can focus on the model only in the first iteration. Once the model is cleared for integration the rest of the development can follow.
To debug the xml parsing I can simply use "hard-coded" files which I completely control.
The crux is of course to write the tests as you implement features in the code. It really is a time saver down the line in terms of debugging of the complete application.
Voilà, I'll leave it at that.
I was able to run the test case in debugger in the following simple steps:
Product > Build For > Testing
Put a break point in part of the test you want to debug
Product > Test
This is on Xcode 6.0.1 and seems much more convenient than the long procedure described above.
After upgrading to the 4.1 iOS SDK my unit test bundles always return with the following two errors and one warning:
An internal error occurred when handling command output: -[XCBuildLogCommandInvocationSection setTestsPassedString:]: unrecognized selector sent to instance
An internal error occurred when handling command output: -[XCBuildLogCommandInvocationSectionRecorder endmarker:]: unrecognized selector sent to instance
Run unit tests for architecture 'i386' (GC OFF) did not finish
I get these errors despite it reporting that all X out of X tests passed. I read about a similar bug in an older version of XCode that also occurred immediately after it was released (something about a bug in a regular expression for time) that could be mitigated by setting your time zone to Pacific Time, but I can't say for sure it's a regression to there specifically.
Is anyone else having this problem?
There is a fix available on the apple dev forums,
https://devforums.apple.com/thread/68687
it's a date parsing problem. The fix is quick, it requires including one .m added to your test bundle. The fix is available above or from:
http://gist.github.com/586296
It does seem to be a regression with some part of the reporting code not respecting timezone issues. The output is tagged as ending before it began, so gets very confused and chokes.
One work-around, that I found somewhere on Google, is to change the Run Script stage of the target.
Change
"${SYSTEM_DEVELOPER_DIR}/Tools/RunUnitTests"
to
"${SYSTEM_DEVELOPER_DIR}/Tools/RunUnitTests" 1> /tmp/RunUnitTests.out
It’s working round the issue, rather than solving it, but does work.
This is a confirmed bug in Xcode 3.2.4/4.1 SDK. It has been reported to Apple.
Upgrading to Xcode 3.2.5/4.2 beta SDK solves the problem. I don't know if access to beta SDK:s is open, or if you need to be a registered developer, though.
I'm getting the same issue too.
Oddly enough, I can get the unit tests to complete properly by running them through the debugger as per http://www.grokkingcocoa.com/how_to_debug_iphone_unit_te.html
So it looks like something wrong with the XCode interface to otest...
I really don't want to set my timezone to PST which this post seems to be indicating as I'm in the UK...