Not so long ago iPhone development was quite simple, only a few OS versions and even less devices.
Now however, there are 2 major OS versions and 5 different devices to consider.
As a company about to release several applications testing has been become more and more of an issue.
What are the best ways to test all combinations, do I need to acquire every generation of iPhone and iPod Touch? Are there any gotchas with specific hardware/OS combinations I might need to code around?
I guess my question is, "What's the minimum amount of testing required to cover all the bases?"
In my experience, you won't have much compatibility trouble between iPod/iPhone. There are other gotchas to be aware of:
The devices run at different speeds. iPhone v1 and iPhone 3G run at 412MHz; iPod Touch runs at 532MHz, and the new 3GS runs at 600MHz. This can have a big impact on performance and even functionality if you're getting fancy.
There's a huge performance difference between EDGE/3G/WiFi networks; often the differences are counter-intuitive. EDGE can often have better latency (time to first byte) than 3G, while 3G has 10x better bandwidth. You'll want to test your app under all three conditions.
Are you using Core Location? iPhone v1 and iPod Touch do not have GPS.
Are you using the camera? The iPod Touch does not have a camera.
Is your app compatible with jailbroken phones? A lot of people have done it, and if your app crashes on them, they will blame you, not the Dev Team hackers, and this will be reflected in your App Store ratings. Note especially that background apps can use up memory that you might have thought would be available exclusively for your app. Leave yourself some overhead.
So, what do you need to buy? If you're a serious developer, yes, you should have all 5 devices available. But do you need to test every build on all 5 devices? Does every one of your developers need 5 devices each? No.
One developer can probably test everything that matters with one iPhone 3G and an iPod Touch. Toss in a 3GS and your coverage is probably nearly perfect. (Note that development on 3GS is much nicer just because the CPU is faster, so your apps deploy more quickly.)
At work we have one device per developer, but they're a mix of 3G and iPod devices (and, today, one 3GS).
I don't have much experience with the iPhone itself, but in general this might be a good application of pairwise testing. In practice, you can get 90% coverage with a small fraction of the testing of an exhaustive test pass. Then later if you find certain configurations have gotchas you can add them to your set of configurations and still not need to do an exhaustive pass.
I am not sure if its a best practice.
But I have heard many people using ibetatest for exactly what you are looking for. Lots of enthusiastic beta testers out there.
You shouldn't have any issues on 95% of your code running on different versions, and something like ibetatest should catch the remaining 5%.
Get all the combinations is the short answer.
I started out with just the iPod, but it's 100MHz
faster than the previous iPhone model. My app's
performance sensitive, so I'm gonna have to get
an old iPhone. I can hardly ask my beta testers
(volunteers?) to run the GL performance tool.
I'd suggest a couple things:
you probably will need a range of devices for your own internal testing. As has already been pointed out, there are various differences between each generation of iPod Touch/iPhone
you may want to look into using crowd-sourcing to supplement your internal testing. This potentially allows you to have a bigger audience to test your apps and any cost of using crowd-sourcing is potentially offset by the fact that you can spend less on devices and potentially less on internal testing staff and at the same time, potentially producing a better product. One crowd-sourced test group not mentioned above is uTest.com
Related
I am wondering how good of an idea it is to start creating iphone applications without actually having an iphone?
I found that there are simulators for iphone...
Are they good enough or is it likely that I might encounter some problems down the road when creating an app?
-I don't have an idea of an app yet.
-I don't have a real iphone. There ARE some handsets in my job's office, though, but I don't want to use them too much.
Thanks! And if you think that it is an okay idea to create apps and test them on simulators, which ones would you recommend?
Bad idea.
You can get a lot of work done but you really, really need an actual device to do your final testing.
Remember that it's a simulator and not an emulator. There are significant differences in performance. Lots of things work fine on your Mac but poorly on an real device. There are, perhaps surprisingly, a number of situations where the reverse is true (i.e., faster on the iPhone). You get better at figuring out these differences after a while, but without working on a real device you'll never know.
If cost is the main factor, you don't actually need an iPhone; you could work with an iPod touch instead.
xCode has an Emulator in itself, so why not? All you need is a Mac-PC
check http://developer.apple.com/library/ios/#documentation/Xcode/Conceptual/ios_development_workflow/25-Using_iOS_Simulator/ios_simulator_application.html#//apple_ref/doc/uid/TP40007959-CH9-SW1
Do you have a mac?
iOS SDK has a really good emulator where you can try your apps, and i think if works on it, it should work on a real iPhone.
Consider that the simulator is good enough, to create your app, but it all depends on what you use. For example if you use the accelerometer or the Bluetooth simulator is not suitable to your case. Overall, in many respects is very limited, but to begin with can go just fine.
To use the devices of which you speak, you must have a valid license from the developer and associate devices to your license, otherwise you will not use them (though not with the jailbreak)
the iPhone simulator does not support everything, which is supported by a real device. Some things are impossible to test on a simulator, including but not limited to features/API associated with: calendars, camera, gyroscope, accelerometer, music, ...
The iPhone simulator that comes with xCode is fine for the development of most applications. In my experience the area where it falls down is when you are dealing with stuff that runs in real time such as audio and graphics. The processor on the computer is obviously faster and if you are only testing on the simulator you might not realise that what you are trying to achieve is outside the capabilities of the device.
This could have impact on your frame rate (as you will get a higher frame rate on the computer) and in depth audio with lots of DSP faces the same issues.
If you are developing Line Of Business apps the simulator is probably all you need. Of course you need to test on a device before release, but most of the bug testing and debugging is possible in the simulated environment.
Now more than one year passed since a retina display device appeared. Does anyone know some numbers how many users still suffer under the low-resolution devices? How long do they use their iPhones or iPods until they realize that a retina display is so much better?
I'm sick of having to create every graphic two times, trashing the binary with all these low-resolution files. I wonder if anyone has stopped supporting low resolution hardware without a big loss.
And: If I wanted to stop supporting low resolution devices, what kind of settings in the info.plist must I make? And what would happen if a low-res device still installs my retina-only app?
Why do people still use IE 6? or why do some people still have Windows XP? Simply b/c they can't afford new ones. Money is to blame. There is no simple answer as to when should you stop supporting certain technology. In the technology world you should usually wait for a big company (like Facebook) to stop supporting certain devices. Only those big sharks can force people to switch.
If you want your users to be happy and want your software to be great you should always support all available devices/versions if possible. Little work for you is a big benefit for your users.
In the end you don't have to do anything you don't want to. But your software will resemble it.
You can at least wait for Apple to stop providing iOS updates to low-resolution devices (i.e. iPhone 3G S) since it is capable of running iOS 5
Maybe its not possible right now as it isnt yet supported here.
But releasing your app with a iOS deployment target of iOS 4.2 may reduce the number of non-retina display users to a great extent.
And the main reason for users not switching to new devices is explained perfectly in the previous answer. :)
For the sake of coding itself, I know that I don't need to buy iPhone as there's pretty good emulator.
However, as I will develop iPhone apps for clients (will not have direct contacts to clients) via freelancers sites, do you think that I might get rejected (not chosen) by the contractor because I don't have iPhone at home?
Do contractors accept this way of working:
I develop the app, test it in the emulator and send it to them
They test it in iPhone and send me the list of the bugs
I fix the bugs and send them the app back
They find new bugs and...
Yes, or at least an iPod Touch.
To clarify:
Yes. You really need one. Debugging the kind of errors that cause it not to open on the device at all, for example, can be mind-numbingly tedious if you don't have the device handy.
For most purposes, of course, an iPod Touch should do just fine, but the crux of the matter is that the testers can only test what they see; only the developer can actually test crucial stuff, most of the time.
So to repeat. Yes, you'll need a device. A thousand times yes.
Your client may have something of an issue paying money for software that was never tested on actual hardware. No matter how good an emulator is, you should always try the software on the real machine your program will be running on. The emulator will simulate the way the API responds etc, but you could be blindsided by things such as interference from other running applications, subtle timing bugs, interaction between different versions of the firmware or hardware, etc.
In short, I don't think there is a legal reason you have to test on a real iPhone, but from a Q/C point of view, I think there is no question you need the real hardware to run it on.
Paying customers generally dislike being treated as a beta tester.
I don't think that you won't get the clients - but I do think its a terrible idea not to have a device to test on.
There are many things that won't work properly in the simulator. For instance you can't simulate a camera function, you can't simulate GPS (properly - it always sets you at the apple HQ), you can't simulate sound recording, or test with a real contact address book or a real set up. You can't test whether there is an internet connection or if there are any iphone specific bugs.
On the other side of the coin there are loads of things that will work in the iphone simulator that won't work on the device itself. For instance NSXML and such won't work on an iphone, but WILL work in the simulator.
If you can get hold of one of the new ipod touches they do pretty much most things you will need, and you don't have to get into a data plan or anything. I would suggest AT LEAST getting one of those. You can't make apps if you can't test them properly.
Other things:
In app purchases - #Stephen Darlington
Whenever working for clients, the clients do not pay you for having an iphone or not... or being able to test it on a actual iPhone. The clients pay you for the product you deliver. They expect it to work on the device.
My recommendation is to get yourself an iPhone 3, 3gs and 4 if you want the best results. But, if money is a object here... try developing minor projects which are reliable in the simulator. AND ask friends/family which do have iPhones to test it for you on their device. Its better to ask mates to do this, then ask the client, this way you have a better comunication with your client, your client has more faith in you and... well lets face it, it is the developers responsibility to deliver quality code. Right?
There are some tests you cannot perform in the emulator.
And I am not sure contractors will like this ping-pong test approach (somebody will get tired after couple passes).
You can get an old second-gen iPod touch at a very good price since there are many people who would like to get rid of it. And Apple advices to test apps on older hardware to achieve the best performance. So you'd better get something 'hard' to play with.
Another aspect is performance. The simulator (running on a powerful Mac) will be much faster than a device. This was a huge difference with the original first iPhone.
As an alternative to the iPod: look for a cheap original iPhone on eBay or so. But remember that this will not run iOS4.
I've tested my app thoroughly on the simulator, but I don't have an iphone/ipad/ipod touch on which to test the app. Are there likely to be bugs that dont expose themselves until I test it on the device?
if i had a macbook, id take my code along with me and meet up with a friend or a stranger to test the app, but im working with a mac mini :(
Thanks for the input.
The one major concern is performance. Devices, especially older ones, have orders of magnitude worse performance than Mac mini, both in terms of CPU and memory. It is possible that your app is crazy slow on a device, yet runs fine in simulator.
Other areas to think about:
network connectivity, performance in poor/no network conditions (good way to test in simulator: yank out ethernet/turn off airport... but some reachability code works differently in device and simulator I think, and you cannot simulate mobile-only [no wifi] situation)
As Rengers said, you should get your friends' device IDs and then create a provisioning profile so they can run the test version of your app which you can e.g just zip up and email to them.
You will really want to pick up a device, an iPod Touch isn't that expensive, and you can even get iPhone 3GSs for less than $100 now at some retailers (contract required). The reason for this is that there have been many cases of the simulator doing one thing while a device does another, in addition to the resource differences (both in terms of available cpu time and memory)
Yes, there are bugs that will only show on the device. It depends on the complexity of your app. For a fairly simple application, chances a bug will show up on the device only will be a little lower.
However, I strongly recommend to test on the device. Bugs aside, you can't get a good measure of performance on the simulator. A real devices has much tighter ram and cpu constraints.
If you have a developers account, afaik you can distribute apps to testers. Perhaps someone else with an 'iDevice' is willing to test for you?
I know there are emulators, but is this good enough? If someone is serious about iPhone development, do they absolutely need an iPhone?
Just my personal opinion: if you're serious it means that you're committed to quality of your product. If you're committed to quality there is no way to deliver a product without actually launching it on the target platform :)
As noted in other posts you'll have tough time testing the multi-touch screen and other aspects of the hardware on your emulator.
Don't forget that most types of iPhone apps also work on the iPod Touch, which is a one time cost and no monthly fees. Even network apps work if the iPod Touch is connected to WiFi.
During development of my first iPhone app, I wrote code that worked fine on the iPhone Simulator, but which did not work on the device. So I would say "Yes, you definitely need to test on an actual device."
The simulator is not an emulator. It is not running the actual iPhone OS; it is running a set of Mac OS X libraries that are very similar, but not identical, to iPhone OS. The simulator is great for debugging and saving time during the code-and-test cycle, so you will use it a lot more than the device, but a device is indispensible.
You really do need to touch-and-feel your app on a real device. A UI that works great while pointing and clicking with a mouse might be terrible when used with thumbs and fingers. If there is any text entry, you need to feel how painful it is to type using the onscreen keyboard, to determine whether it makes sense to provide alternative data-entry methods.
There are also significant performance differences between the simulator and actual devices. You need to test with the oldest (slowest) device you want to support to verify it is not too slow, doesn't run out of memory, etc.
As others have suggested, an iPod Touch is also sufficient, so the cost of a device isn't huge. Also, try to find beta testers with a variety of different models.
Necessary: How the app handles in your hands is critical to something like the iPhone. you cannot tell how it will feel to use when plastered straight in front of you in the emulator on a big screen.
If you cannot hold it you won't be getting the true user experience.
If you need to learn Obj-C, go with the emulator for a while until you learn the ropes and save the expense for later. But yes, eventually you will need an iPhone for final testing. How long you can wait will depend on the features that your app uses, If all you are doing is button presses, you can wait a long time. If you are dragging, using location services, etc., you'll need a device earlier in the development cycle.
Are you trying to convince yourself or your boss? ;-)
I'd say you need one. Emulation of such a new device can only go wrong. Plus don't forget the tactile aspects.
The iPod touch is a reasonable substitute provided you are not using:
GPS, BlueTouch or Camera - the iPod touch doesn't have these
Cellular network - although the iPod touch has WiFi, the latency of a cellular network is way way higher than that of a wifi network. If you are doing anything like designing a custom protocol for your application, you will want to check real-world performance - and if you do this too late in the development cycle, you will be in for an unpleasant surprise.
Whether you develop on the iPod touch or on the iPhone, you absolutely must have a device. This is not optional! The simulator is good, but it is not perfect, and there is no substitute for having a device which correctly indicates performance, screen resolution, brightness, form factor and all the other factors that you will need to consider in your application.
If you buy an iPod touch, you will probably end up getting an iPhone too. I'd just go straight for the iPhone. That way you can use it as your main phone, and get a real feel for how the platform behaves and what an application needs to do to make it great.
Kind-of "yes".
Just download iPhone SDK (it's easy and free) and check out the emulator that is in there. You'll see whether that suits your needs or not. The emulator is not indicative of real hardware performance, there's no touch input, some quirks might be different, some things can not work, etc.
The iPhone Simulator makes it easy to test your applications using the power and convenience of your desktop or laptop computer. Although, your development computer may not simulate complicated touch events, such as multifinger touches, the Simulator lets you perform pinches. To perform a pinch, hold Option while tapping on the Simulator screen.
I'd say it depends on the kind of application you are developing. For a successful iPhone app, one which is properly integrated on the system, you are going to need to be able to test your tactile interface. That's hardly accomplished with the Emulator.
So, my answer is Yes, you do need an iPhone to develop iPhone apps. Fortunately, if you cannot afford one, an iPod Touch (200 bucks) is a very competent replacement. The underlying hardware is pretty much the same.
Necessary. If you plan to develop a successful product it needs to be one the end users (not just the developers) find easy to use.
The best way to do that would be to load your app on an iPhone then take it to various people and ask them to use it while you watch them to see if they experience any issues.
Users can get mighty creative in trying to do things a developer never intended - just ask any support tech.
Unless you're app is going to sell for less then $500 total it's a relatively small investment to build a quality app.
If you are serious about development, an iPhone (or iPod touch) is a must. However, the official SDK comes with a very complete "iPhone simulator". This will allow you getting a feel for Objective C and the entire development workflow. The SDK requires Leopard.
You don't need a Mac for this. You can use OSX86 on your PC, either installed on and booted from disk or through VmWare.
It works. In fact, you can even synch the iPhone through Leopard running in vmWare.
Now, testing on a real iPhone is a necessity because of performance, memory usage etc. Also you need it for the entire authentification procedure, getting the keys etc. (if you want to sell your stuff on the Appstore), testing this really requires an iPhone.
If you buy an iPod touch, you will
probably end up getting an iPhone too.
I'd just go straight for the iPhone.
That way you can use it as your main
phone, and get a real feel for how the
platform behaves and what an
application needs to do to make it
great.
I absolutely agree with this.
If you are seriously developing an iPhone application - for fun or for profit - you will have to run it on a real iPhone to test out compatibility and usability at some point. Since you going to have to get one at some point, you may as well get one now. Don't go for half measures. An iPod Touch may be [significantly] cheaper to start with, but will be money wasted when you go and get your iPhone. (Of course, if you are planning an app that runs on the iPhone as well as the iPod Touch, then you MUST test it on both. You cannot assume that if it is good on one it must be good on the other).
Also, by having an iPhone from day one, you can familiarize yourself with its user interface, its norms and the common metaphors the apps use. That will heavily feed into your own application design process, and make sure that your app looks, feels, and works like a first class iPhone citizen.
From experience developing on other mobile platforms, once you get to a certain point, it really is best to have a physical device to test on. If this is something that you would also be using yourself, if it much easier to get some real world type of testing by using the application out and about.
I also think it helps one to understand the platform better by having the device or devices you are targeting with your app,
if you are going to develop native apps for the iphone, I would say get an iphone or ipod touch to target. emulators are good, but eventually you will need to target the real thing. if you are developing web specific content there are lots of things you can do without it (there are some great dev videos free from apples dev site which will only cost you a sign up) but eventually I would think you would still want to test with the real deal
Get a cheap used iPod touch, develop, get money, buy an iPhone 5.
I'm a nokia dev now, I'm thinking of going to iPhone, Actually I have the Mac to work, just the device itself ;)
I've tried iPhoney and compared to my iPhone (Mark 1) it's not the same, it's close - but not close enough to rely on if the interface is of importance to you.
You absolutely need the real device. The performance difference between the simulator and the actual iPhone/iPod Touch hardware is huge. Code that will run nice and fast in the simulator can easily turn out to be too slow to be usable on the real thing. Also the API provided by the simulator is not 100% identical to the real thing, so code that works fine in the sim, may not work on the device. The only way to know for sure is to test often on the actual device.
As others have mentioned, the iPod touch works well as a development device. So if you don't need any of the features of the iPhone, it's a good, cheaper, alternative.