automatic app testing in xcode - iphone

Currently I'm testing an app that I've written with ios 4. I've spent a good bit of time looking for memory leaks and that side of things seems o.k.
I have a problem where the app seems to "randomly" quit after going from one view to another. Sometimes this will happen after a minute or 2 and sometimes it won't happen at all. Because of this it's really very difficult to determine where the flaw in my application is and under what circumstances because my observations of whats going on is subjective. Also it can get very annoying very quickly sitting there hitting buttons until something happens and then trying to remember the sequence of your actions.
What would be a good method to hunt down the cause of this seemingly randomly occurring problem?
For example,
I remember speaking with QA engineer before and he spoke of tools that he used to automate a user using an application. He'd leave it running over night and then in the morning he could examine what combination of user actions and under what circumstances a problem occurred.
Are there such tools available for testing iphone applications? Ideally what I'd like to see is a report of
action a
action b
action c
= everything ok.
action a
action c
action b
= there was a problem.

There's also fonemonkey: http://www.gorillalogic.com/fonemonkey
As for random crashes... I am tester, not developer, and random crashes are usually one of the topics I examine more thoroughly on every iphone app. My favorite scenarios are: leave the phone for 2 min, for 10 min, and switch apps (on iOS 4). In all cases it's better to use a real phone (and the older the better), not a simulator (some crashes never happen on simulator). Commonly nested views are affected (e.g. get to some view, let phone sleep, click "Back" button to return to previous view - crash). Usually developers say that incorrect retain / release, autorelease while another object holds pointer, and other memory issues are to blame. Also Device Logs (which will have crash report) usually can pinpoint problem pretty accurately.

Hey.
Instruments by Apple. Apple reference.
Instruments has its limitations and drawbacks, but just try it. It may be confusing at the begging so try to follow tutorial, or some other.

Check out running the app with zombies enabled. Also, instruments in Xcode is very helpful.

Related

Recognise an out of memory crash (IOS)

Our apps are live on the app store.
I wish to recognise crashes of out of memory that some users are getting.
I understand there is no way to 100% recognise an out of memory crash.
Is there any way to recognise these crashes(with a pretty large probability) by doing some logic in the applicationDidReceiveMemoryWarning? (I am not talking about finding it in xcode during development time, i am talking about code that will recognise the out of memory crash from actual users and will log something to file)
While I was looking for any service or library that give me OOM tracking, I could only find this article from Facebook engineering:
https://code.facebook.com/posts/1146930688654547/reducing-fooms-in-the-facebook-ios-app/
The idea is to deduce the reason why the app needs to be launch, checking different aspects (like if the app was at background, if there is an app/OS update,...).
Discarding all the other possible reason that can force the previous app exit, you can know if the reason is a background out of memory or a foreground out of memory.
It would be nice to have a library that implements the Facebook article procedure. But nowadays I couldn't find any, probably there is some reason that make this difficult or may be impossible to add it as an sdk.
If anyone knows any service, please share it with everyone with a comment or a new answer.
Edit:
I have discovered this github (https://github.com/jflinter/JRFMemoryNoodler) with an implementation of the Facebook post procedure. I haven't tried yet, but we will deploy it in our apps to try it.
Look out for the applicationWillTerminate message in your app delegate. This is called if you app is terminated by the system (due to e.g. low memory), but not if the user leaves the app in the usual way by pressing the home key. Note: if your app is in the background and memory runs out, your app gets killed without any messages being sent to it.
YMMV, especially with older versions of iOS, and it's worth researching to ensure that the above is accurate.
The images at this blog post are quite informative (although slightly dated).
For more info, see How to know whether app is terminated by user or iOS (after 10min background)
Firstly Analyse your application by clicking on the Product at the top menu bar of your Xcode and click on Analyse section it will show you the number of leaks on in the application and can take you to the place where leaks occurred. This is how you can find the memory leak and rectify it.
Secondly it above does not worked then see to the view controller where crash occurred and check whether you have left any object to release.
Hope this might help you to resolve your problem.

Xcode 4 question about object library and NSLog

Two questions. All of a sudden, not sure why, maybe I hit the incorrect hot key, but my Object Library for .xibs in IB now shows NS items used in Mac development and not my typical iOS elements. How do I bring that back?
Second question, if I run an app in the simulator and it says in the console:
Received memory warning.Level = 1
What should I start with in troubleshooting? This is on an old 3G iPhone. Thanks!
regarding your second question: you should start troubleshooting your app for an excess of memory allocation and, if you can, look for memory leaks
maybe you want to look at this topic: iPhone OS Memory Warnings. What Do The Different Levels Mean?
For the first question: restart XCode.
For the second question: If the warning is infrequent and your app is not crashing, don't worry about it. Regardless or warnings or not you should be doing memory testing for leaks and also make sure your app responds to didReceiveMemoryWarning notifications. These warnings are normal and vary greatly depending on what other processes are running.

iPhone app crashing on some devices -- looking for advice

Hey everyone, I'm in a bit of a pickle and am looking for some advice.
I have an app that's been released to the app store..nothing special just a first time sort of thing. The app runs perfectly fine on my device(s) but a couple buddies always get a crash.
So here's the details:
It crashes by sometimes stalling and other times by just exiting to the main screen.
I've tried to replicate the crash (or any crash) using
the same iPhone generation/model
on an ipod touch as well
the same OS
different OS
using debug/release/distribution/downloading from app store builds
since it's an app that connects to a google maps service, I've tried with both
a wifi connection and the att network.
The only thing that MAY be different is any other apps on their phones that are not on mine.
The fact that I cannot replicate with exactly the same setup leads me to believe it's a memory issue, maybe uninitialized variables or incorrect cleanup at some point (i'm from C++, so I know this sort of thing can vary from machine to machine). I'm somewhat new to obj C and may very well be missing something there. I'm just curious to hear what others have to say, does this sound like a memory issue with the info I've provided? Any other ideas to test/suggestions come to mind to try and reproduce the crash?
Thanks!
One possibility is that it's a backgrounding issue. You don't say whether your app tries to use those features or whether your friends experience the crash in that context, so it's hard to say.
I would provision your friends with a beta that logs verbosely if you can't pin it down.

How do YOU handle crashes in your iPhone apps?

If been looking around the web and can't seem to find any good solutions to sending allowing your user to submit bug reports from your iPhone app.
How do you handle crashes and exceptions?
Do you send the error user-data to a server,
grab a log file from somewhere and attach,
or do you ignore it and pretend it never happened?
Anybody got any experience with this?
Update
I am aware of how to prepare you software testing it with Static Analysis, Leak Detection, User Testing etc.
But errors might still happen when a user (mis)using my software. Always assume your user is trying to break your software.
What I want to figure out is how (I, or rather the app) can provide me with useful data when/if errors happen. As they do even in top quality products – like my own ofc. :)
I'm looking anyone that has experience with allowing the use to send error reports, stack traces, logs etc. to see how they handle the problem.
Some people use built-in analytics like Flurry which will post exception data to Flurry's website which you can review later.
Also, Apple has a "crash log" reporting area on iTunes Connect, but I'm not sure if it works since I've yet to see something come through and I kinda doubt each and every person has run my apps flawlessly. Not saying I write poor code (hopefully), but not every device is created equal either so I have to imagine it has crashed at least ONCE. There's always the option of logging and sending to a server later though.
Most of the time though, if you give users your e-mail address within the app itself (like on an instructions or about screen), they will e-mail me about any issues. That's a little bit nicer since it gives you a chance to correct the issue before they hit the review forms on iTunes.
You can also try BugSense. It's free, realtime, error reporting for iOS
PS: I am one of the founders.
Crashes most of the time comes with problems of memory management. To test memory leaks and find in your code on to what areas you were leaking a memory. Use the Instruments if your using XCode.
In your XCode go to Run -> Run with performance tool -> Leaks.
You can check everything here when it comes to memory allocations.
Note: To avoid crashes, make sure you released objects correctly and check your scheduled timers as well.
Regards,
ZaldzBugz

Should we required to check iPhone is jailbraked

Some of our application is already in AppStore...
But suddenly one thing comes into my mind, that I want to clear before submitting my next application.
The thing is : As a programmer's point of view, should we require to handle if iPhone Device is jailbreaked ? If yes, then how we can tackle with this ?
Thanks in advance....
On a general note, jail-breaking the device is an issue between the user, Apple and potentially the carrier. You are not a side in this relationship, and the user has no contractual obligations to you with regards to their device.
You could choose to attempt detecting jail-broken devices in an attempt to prevent piracy of your app. However:
If the device is jail-broken, there's nothing you can do to reliably verify it's not jail-broken, since none of the OS APIs (including networking) is guaranteed to function as you expect. Your code could be running in a non-jail-broken simulation on top of jail-broken device.
Of course, you could check by attempting to do one of the things you currently know Apple actively prevents apps from doing. However, there's no guarantee that Apple is not going to allow that particular action in future. And, there's the chance that your app might get rejected because you are attempting to do something prohibited by Apple.
There is no official criteria from Apple on what constitutes a jail-broken device and what does not. And even if there was, you are not guaranteed to be notified in a timely manner (or at all) by Apple if they decide to change any such criteria. But even assuming you do get notified somehow, you can't update your app quick enough to avoid falsely detected jail-broken devices, thus potentially denying access to your app to legitimate users.
If you would like to cut off a large group of users, then sure, go ahead and require it.
Unless your application specifically requires it, there should be no reason to force users to have a jailbroken iPhone or a non-jailbroken iPhone.
If you program is legitimate (no private API calls etc), then you should not concern yourself with JB. You don't need to handle anything differently if the users phone has been JB'd. If it has, and your software doesn't run (say memory issues because they are using backgrounder to run 2 other things) then that's their problem not yours. Make your code well behaved, not leak memory, dump cache's etc with memory warnings, and you should be fine.
As you asked for the "programmer's point of view", I'd say: make sure your app runs on as many devices as possible.
Meaning: as long as you app is safe to run on an iPhone whether it's JB or not, I wouldn't care.
One thing I have found, at least early on (not seen it for a while) is that most reports I got of strange behaviour with my app (vConqr) turned out to be from people with jailbroken phones.
That's not to say I think that's good reason to block them. But if you do any sort of custom crash reporting, or other diagnostics it could be useful to log the fact to save time on troubleshooting.
Do a search on the Internet. You'll find several articles that shows some ways you can detect a pirated app. I make no claim on their effectiveness, but I do use some of these in my own apps. These techniques do not try to detect if a phone is jailbroken; they focus on detecting if your app has been tampered with.