We are facing the issue that Xcode detects runtime tests.
Because of certain states of the tests these tests will always fail.
It's not possible to get rid of them, by cleaning the Project nor deleting the Derived Data.
Some Test classes are inherited by others.
Specs:
Xcode 8.3.3
Swift 3.1
Edit
Add more specific details
Subclass:
TestClassA: XCTestCase
func testA1()
func testA2()
TestClassB: TestClassA
func testB1()
Now the Problem is that when I run all tests for TestClassB Xcode will run testB1 and then testA1 and testA2. Both marked with rt and they will fail.
How can i achieve that this is not happening by not loosing the subclass structure.
Screenshot
I assume you're overriding the setup and teardown do assure the app will be in the proper state for every test case, the problem here would be that
the rT test will appear since the subclasses are inheriting the tests from their parent.
To avoid this while maintaining the subclassing structure, keep the setup and teardown functions in the classes you have but move the tests to a subclass of the setup class.
For example:
Subclass:
TestClassA: XCTestCase // This one will have the setup and teardown for A
TestClassATests: TestClassA
func testA1()
func testA2()
TestClassB: TestClassA // This one will have the setup and teardown for B, and you may reuse A setup
TestClassBTests: TestClassB
func testB1()
func testB2()
Related
I am testing a framework.
Every test case set static variable and it looks like XCTest shares static area of a framework.
As a result, running tests at the same time makes test failure from the second test.
Note that running tests individually makes all test succeed.
I am testing with Unit test not UI test, but should I reset App at the stage of class tearDown()?
If so, please tell me how to do it, because
XCUIApplication.terminate() fails in the subclass of XCTestCase.
I am using Xcode 11 Swift 5.1
I believe what you're looking for are the instance methods, not the type methods.
Furthermore, consider setUp over tearDown:
Here is the description for the instance method setUp from Apple's documentation:
Before each test begins, XCTest calls setUpWithError(), followed by setUp(). Override this method to reset state for each test method. If state preparation might throw errors, override setUpWithError().
https://developer.apple.com/documentation/xctest/xctest/1500341-setup
Example usage:
override func setUp() {
// reset state for each test
}
For comparison, here is the description for the type method setUp from Apple's documentation:
The setUp() class method is called exactly once for a test case, before its first test method is called. Override this method to customize the initial state for all tests in the test case.
https://developer.apple.com/documentation/xctest/xctestcase/1496262-setup
Example usage:
override class func setUp() {
// customize the initial state for all tests
}
I don't know what the purpose of your static variables are, but it sounds like they are responsible for changing behavior in your app. I recommend using mock objects and/or data. I would consider looking into writing "testable code." It will help you keep your code clean and also make it easier for you to write unit tests. It is also object-oriented.
Let's say I have this test code
class MyTests: XCTestCase {
// I want to call this func first when I run class test.
func testLoginUser() {
... do some test
... expectation.fulfiil() // jump to the next testGetUserProfile func.
}
// I want to run this func after testLoginUser fulfill expectation
func testGetUserProfile() {
}
}
Questions:
As I understood that all functions which started from prefix test will be called automatically by pressing this button on the image below with undefined queue of calls. So testGetUserProfile func can be called before actually testLoginUser fulfilled am I right?
If so then does it mean that I need to rename testGetUserProfile to getUserProfile function and call it from testLoginUser after expectation fulfilled?
Is there a way to program a queue of test func calls one by one after each of them fulfilled?
The order of test method execution is defined by their names, by default Xcode runs them in alphabetical order.
Xcode 11 has introduced Test Plans, where you can configure the execution order to be either "random" or "alphabetical".
I have a unit test suite where every test succeeds when it is executed individually.
However, if I execute the whole suite, one test hangs when it should init a singleton. It hangs only if it is executed after a certain other unit test - if I change the order, the whole test suit executes successfully.
If I pause the execution of the hanging unit test, the stack trace is the following:
The execution hangs at the statement static let shared = StoreManager():
class StoreManager: NSObject, CalledByDataStoreInStoreManager {
static let shared = StoreManager() // Instantiate the singleton
// …
}
The other unit test that it executed before and that causes the test to hang does not use the StoreManager singleton.
My question is:
What could be the reason that the 1st test lets the initialisation of the singleton in the 2nd test fail, although this singleton is not used in the 1st test?
So the obvious answer is that there's a side effect occurring in your 1st test that is affecting your second test. Exactly what that side effect is, though, is dependent on two things:
What is being tested in the first test
What occurs in the initializer for StoreManager
From your description, it sounds like the first test is somehow affecting what's then used in the StoreManager init, which is unaffected when that test is not run before your 2nd test.
Solved: The side effect that the answer of BHendricks pointed me to, was the following:
Since my app uses also the watch, the watch session is activated in the app delegate, as recommended by Apple. This was done by calling a function in a WatchSessionManager instance, and the initialisation of this instance tried to init a singleton that is also tried to be init by the Storemanager.
This created an init cycle deadlock.
Since I use a mock app delegate for unit tests, I activate now the watch connectivity session there without initialising the unnecessary initialisation of the WatchSessionManager instance, and the deadlock is avoided.
I am developing an iOS application and am trying to integrate Typhoon into the testing. I am currently trying to mock out a dependency in a view controller that comes from the storyboard, so with in my assembly:
public dynamic var systemComponents: SystemComponents!
public dynamic func storyboard() -> AnyObject {
return TyphoonDefinition.withClass(TyphoonStoryboard.self) {
(definition) in
definition.useInitializer("storyboardWithName:factory:bundle:") {
(initializer) in
initializer.injectParameterWith("Main")
initializer.injectParameterWith(self)
initializer.injectParameterWith(NSBundle.mainBundle())
}
}
}
I want to create a CameraModeViewController (the class I am unit testing) with its dependency upon a system-camera-functions-providing protocol mocked out. The dependency is dynamic var cameraProvider: CameraAPIProvider?. I think I correctly created a replacement collaborating assembly to replace systemComponents; MockSystemComponents is a subclass of SystemComponents that overrides functions. This is where I inject the mock:
let assembly = ApplicationAssembly().activateWithCollaboratingAssemblies([
MockSystemComponents(camera: true)
])
let storyboard = assembly.storyboard()
subject = storyboard.instantiateViewControllerWithIdentifier("Camera-Mode") as! CameraModeViewController
The next line of code in the tests is let _ = subject.view, which I learned is a trick to call viewDidLoad and get all the storyboard-linked IBOutlets, one of which is required for this test.
However, I am getting very mysterious result: sometimes but not always, all the tests fail because in the viewDidLoad I make a call to the dependency (cameraProvider), and I get an "unrecognized message sent to class" error. The error seems to indicate that at the time the message is sent (which is a correct instance method in protocol CameraAPIProvider) the field is currently a CLASS and not an instance: it interprets the message as +[MockSystemCamera cameraStreamLayer] as reported in the error message.
~~~BUT~~~
Here's the kicker: if I add a breakpoint between the calls to assembly.storyboard() and subject.view, the tests always pass. Everything is set up correctly, and the message is correctly sent to an instance without this "class method" bogus interpretation. Therefore, I have to wonder if Typhoon does some kind of asynchronous procedure in the injection that I have to wait for? Possibly only when dealing with storyboard-delivered view controllers? And if so, is there any way to make sure it blocks?
After digging around in Typhoon's source for a while, I get the impression that in the TyphoonDefinition(Instance Builder) initializeInstanceWithArgs:factory: method there is an __block id instance that is temporarily a Class type, and then is replaced with an instance of that type; and possibly this can be called asynchronously without blocking, so the injected member is left as a Class type?
UPDATE: Adding the code for MockSystemComponents(camera:). Note that SystemComponents inherits from TyphoonAssembly.
#objc
public class MockSystemComponents: SystemComponents {
var cameraAvailable: NSNumber
init(camera: NSNumber) {
self.cameraAvailable = camera
super.init()
}
public override func systemCameraProvider() -> AnyObject {
return TyphoonDefinition.withClass(MockSystemCamera.self) {
(definition) in
definition.useInitializer("initWithAvailable:") {
(initializer) in
initializer.injectParameterWith(self.cameraAvailable)
}
}
}
}
UPDATE #2: I tried replacing the constructor injection in the MockSystemComponents.systemCameraProvider() with a property injection. Different issue, but I suspect it's equivalent in cause: now, the property that is injected (declared optional) is still nil some of the time when I go to unwrap it (but not always -- probably about 4/5 of test runs fail, about the same as before).
UPDATE #3: have tried using the following code block, using factory construction according to this answer (note that setting factory directly didn't work as that OP did, but I think I correctly used the feature added in response to Jasper's issue). The results are the same as when using property injection like Update #2 above), so no dice there.
This issue was in fact arising even before the call to the instantiation. In fact, the problem was assemblies aren't generally intended to be stateful. There are a few ways to get around this, but the one I used -- having a member variable and an initializer method -- is NOT recommended. The problem with doing this is that in the activateWithCollaboratingAssemblies method, all the instance methods of the assembly are enumerated for definitions, and initializers will actually get called on the collaborating assembly. Consequently, even if you create your assembly with an initializer, it may get called again with a bogus value.
Note that the reason there appeared to be async behavior is actually that there is nondeterministic order in which definitions are assembled (property of storing them in an NSDictionary). This means that if activateWithCollaboratingAssemblies happens to enumerate methods which depend on state first, they'll work fine; but if the initializer is enumerated first, and the state is destroyed, definitions that are created after will be borked.
I'm testing some code that uses StructureMap for Inversion of Control and problems have come up when I use different concrete classes for the same interface.
For example:
[Test]
public void Test1()
{
ObjectFactory.Inject<IFoo>(new TestFoo());
...
}
[Test]
public void Test2()
{
ObjectFactory.Initialize(
x => x.ForRequestedType<IFoo>().TheDefaultIsConcreteType<RealFoo>()
);
// ObjectFactory.Inject<IFoo>(new RealFoo()) doesn't work either.
...
}
Test2 works fine if it runs by itself, using a RealFoo. But if Test1 runs first, Test2 ends up using a TestFoo instead of RealFoo. Aren't NUnit tests supposed to be isolated? How can I reset StructureMap?
Oddly enough, Test2 fails if I don't include the Initialize expression. But if I do include it, it gets ignored...
If you must use ObjectFactory in your tests, in your SetUp or TearDown, make a call to ObjectFactory.ResetAll().
Even better, try to migrate your code away from depending on ObjectFactory. Any class that needs to pull stuff out of the container (other than the startup method) can take in an IContainer, which will automatically be populated by StructureMap (assuming the class itself is retrieved from the container). You can reference the IContainer wrapped by ObjectFactory through its Container property. You can also avoid using ObjectFactory completely and just create an instance of a Container that you manage yourself (it can be configured in the same way as ObjectFactory).
Yes, NUnit tests are supposed to be isolated and it is your responsibility to make sure they are isolated. The solution would be to reset ObjectFactory in the TearDown method of your test fixture. You can use ObjectFactory.EjectAllInstancesOf() for example.
Of course it doesn't reset between tests. ObjectFactory is a static wrapper around an InstanceManager; it is static through an AppDomain and as tests run in the same AppDomain this is why it is not reset. You need to TearDown the ObjectFactory between tests or configure a new Container for each test (i.e., get away from using the static ObjectFactory).
Incidentally, this is the main reason for avoiding global state and singletons: they are not friendly to testing.
From the Google guide to Writing Testable Code:
Global State: Global state is bad from theoretical, maintainability, and understandability point of view, but is tolerable at run-time as long as you have one instance of your application. However, each test is a small instantiation of your application in contrast to one instance of application in production. The global state persists from one test to the next and creates mass confusion. Tests run in isolation but not together. Worse yet, tests fail together but problems can not be reproduced in isolation. Order of the tests matters. The APIs are not clear about the order of initialization and object instantiation, and so on. I hope that by now most developers agree that global state should be treated like GOTO.