I found examples on how to debug your unit test in Cocoa or the ADC page here.
But I can't get the debugging to work for an iPhone app target. I can get the tests up and running and they are run during the build, but what I need is to debug the tests for some of the more complex failures.
You might consider moving your tests to GHUnit, where they run in a normal application target, so debugging is straightforward.
This can be done by setting up a separate Executable for the project that uses the otest tool to run the unit tests, after setting a bunch of relevant environment variables for the executable. I have used this method to successfully debug SenTestKit logic unit tests.
I found the following links helpful:
http://www.grokkingcocoa.com/how_to_debug_iphone_unit_te.html (also contains help to fix common errors encountered setting up the project).
http://cocoawithlove.com/2009/12/sample-iphone-application-with-complete.html (covers both logic tests and application tests)
http://developer.apple.com/mac/library/documentation/Darwin/Reference/ManPages/man1/otest.1.html (Man Page for otest XCode tool)
The NSLog messages show up in Console.app
Should give you a starting point.
In Xcode 4, you can set breakpoints in your unit tests.
Create a new project with "include unit tests" checked.
Put a breakpoint in the failing unit test.
Press Command-U to test.
If you do Build & Go instead of just build, then you can set breakpoints in your unit tests and debug them traditionally. This is if you are using the google toolbox for iphone unit testing; i don't know how you are doing it and if the process is different.
Related
Maybe I have never seen this but shouldn't Unit tests show a TEST_TARGET_NAME
in build settings? I just added a Unit test target to an app that has Obj-C code mixed with Swift and I can't get the Unit test to run correctly. It crashes on run time. I have been looking at the build setting from unit tests to the UI test target that is already there and a few things come to mind. First, the non-existent test target name from build settings under User-Defined is not there and
Bundle Loader
has stuff inputted in it on the Unit test but not the UITest target and for some reason the 'Product Bundle Identifier' on the Unit test is different than on the UI test target. Also, 'Weak References in Manual Retain Release' is set to YES on UI but NO on Unit. Test Host is populated on Unit but not on UITest. Again the UI test target seems to run just fine but it doesn't reference any classes from the app. It literally just runs the UIApplication and does a run through the app. The Unit Test I get one main crash on run-time dealing with a
Thread 1: implicit Objective-C entrypoint +[<class name> UI] is deprecated and will be removed in Swift 4.
I will say I picked up this project and trying to clean a lot of it up but I feel like the previous developers just started upgrading to the latest and greatest versions of Xcode and never really migrating cleanly. Can anyone provide any input on this?
I have a number of logic unit-tests (where my project files have a target membership of the App and AppTests). I want to add a call to xcodebuild test-without-building to my build system so that my unit-tests run for each build.
However, the tests cannot run on the release build (because release doesn't build for testing).
Is my only choice to build both the release version and the debug version during my build, so that I can use the debug version only to perform the tests? That's very different and very much worse to the other test frameworks I've used (GTest, Catch). Why can't the tests stand on their own?
The test-without-building command is not actually meant to "run the tests without rebuilding the app", but rather it's supposed to be used in tandem with the build-for-testing command.
Please refer to the "Advanced Testing and Continuous Integration" WWDC 2016 session for more info.
The gist is: use build-for-testing to build an .xctestrun file, which is then used by test-without-building to run the tests. This is particularly useful to run big suites across different machines, although I have never done it myself.
Now that we have established that you can't use test-without-building on its own, the only option to run the test from the command line and if they pass build a Release version is to use xcodebuild test, which is going to build the app.
As for the why does it need to be in Debug, I don't have a precise answer. In iOS land, at least in the teams I worked with, the difference between Debug and Release builds is always only on things like the options passed to the compiler in terms of optimization, the architectures to build on, and the type of code signing.
This means that the code that runs in Debug vs Release is exactly the same, and since we already established that you'll need to build the app twice, one to run the tests, one to generate the releasable version, running the tests in Debug seems acceptable.
I'm playing around with Microsoft Test Manager 2013 (though it appears it is just MTM2012) to try and get a better understanding of test cases and test suites as I want to use this at work. So I was hoping that I could run a test suite on a build which gets included in this test suite. That is what I WANT to do, but it could very well be wrong. So maybe a better scope of what I'm doing at work might lend to a better answer.
My company makes tablet PC's. I write programs for those tablets. For sake of argument lets just say there are 5 tablets, that run a similar array of OS's. Tablet1,2,3 and 4 can run WinXP, WinXP embedded, Win7, and Win7 Embeded, and Tablet5 can run Win7, Win7 Embedded, and Win8 embedded. Lets say i'm making a Display test program. Naturally this display test will run differently on each tablet, but the program it self is supposed to be able to handle that along with not being able to worry about OS. So I wrote out a very simple test. Open Program, try to open again, verify only 1 instance, check display, close program.
I figured it would be good to make a Test Suite called "Complete Display Program Test" and put 5 sub test suites to that for each tablet. Then moved the 5 test cases to a single test suite. I configured all test cases to only have the correct tablet/OS configuration. Queued a build and waited for it to finish. I then attached that build to the main test suite. I then clicked on run a test for tablet 1 but I didn't see the build attached to the test runner. I've looked around a little bit to see why or how and haven't found anything. Question is is how do I do that? Or if you are scratching your head and wondering why in the world I am doing it this way then by all means suggest another way. This is the second time I have ever looked into MTM so I might not be doing it right.
Thank you for your time.
When running manual tests from MTM you will not see the build you are using in Test Runner.
But if you complete the test and set the test outcome you will be able to check which build you've ran the test against.
Just double-click on the test or select "View Results" to display test results:
This column is not visible by default. You will have to right-click on the column row and select the column "Buld number" to be displayed.
You will also be able to see the build number in "Analyse Test Runs" area:
The things are slightly different if you are running automated test.
Consider following approach:
Automate your Test Cases
See How to: Associate an Automated Test with a Test Case for details.
Create a Build Definition building your application under test AND assemblies containing your tests.
I strongly recommend build application you want to test and test assemblies using in the same Build Definition. (You will see why a little bit later).
Run this build definition and deploy the latest version of the application to the environment where you want run the tests.
This is very important to understand: if you run automated tests the tests assemblies only would be deployed automatically to the environment.
It's your job to deploy the right version of the application you are going to test.
Now you can run tests from MTM.
You can do it the way described by #AndrewClear in the comment to this answer: "choose "Run with Options" when you're beginning a test run" and select the latest build.
Now test assemblies containing tests which are using to automate Test Cases will be deployed automatically to the test environment and the tests will be executed.
That is the point you should recognize why is it so important to build application and tests with a single Build Definition: since the build number you've just selected when starting the tests will be stored along with the test results on TFS you will later know what version of you application you were testing (assuming you deployed the right version, of course).
You could go a little bit further if you want even more automation (This is the way I'm currently running automated tests)
Use Deploy-Build-Test template (this is a good place to start reading about Setting Up Automated Build-Deploy-Test Workflows).
Using this approach you will be able to automate deployment of the application you want to test.
When trying to build the unit tests created using the default XCode Unit Test bundle target, it looks like it's stuck on the "Run custom shell script 'Run Script'" phase.
I also notice a high cpu usage on process "otest" to the point where the fans kick in within seconds.
The only useful message I see when expanding the line is
/Developer/Tools/RunPlatformUnitTests.include:419: note: Running tests for architecture 'i386' (GC OFF)
Couldn't open shared capabilities memory GSCapabilities (No such file or directory)
The only option I have at that time is to stop the build.
Have to say I was running unit tests perfectly fine up to this moment but can't say for sure what I did to cause that.
That's on XCode 3.2.4
After updating to 3.2.5 now the run script does fail with an error
Test rig '/Developer/Platforms/iPhoneSimulator.platform/Developer/SDKs/iPhoneSimulator4.2.sdk/Developer/usr/bin/otest' exited abnormally with code 138 (it may have crashed).
Guess they problem is related?
Did find some answers on SO about how exception handling now works differently when using NSInvocation (which otest seems to use) but not really a solution to this.
I had this happen to me. I made it go away by scrapping my old testing target profile, creating a new one, and pointing all my tests to it. I was too frustrated to compare the profiles line by line to figure out what had changed.
This looks like an infinite loop to me. Try adding some NSLog statements and/or debugging your tests with gdb (by adding otest as a custom executable).
This happened to me after updating to Xcode 9 and using script for updating localizable strings file, a minor bug caused the script to never finish. After updating BartyCrouch, everything worked normally.
https://github.com/Flinesoft/BartyCrouch/issues/66
I am building a logic test suite using Xcode 3.1.4. I am able so far to build the test target and see the results inside de *.m test files as compiler errors (if they fail). The problem is that I can't see any information in the Debugger Console and I can't debug since the debugger does not work either.
1.How can I do to see TestCases ouput on the debugger console?
2.Are there any configuration issues regarding Unit Testing and Debugging test cases?
I would be very greateful if anyone could help me.
Thanks!!
You can setup Google Toolbox for Mac. With it, you will be able to step through your code both on simulator and device.
http://www.grokkingcocoa.com/how_to_debug_iphone_unit_te.html worked great for me, once I unchecked the DYLD_FRAMEWORK_PATH environment variable.