What is the best and most efficient way to benchmark an iOS application? We are mainly looking to get response times for the application to communicate with our API and complete the processing of the returned data.
If you're looking for API response times, you can simply add the two lines of code that measure the times within your app (log at request start, log at request end.)
You can also look into using Apple's Instruments toolset to measure device CPU performance and leaks.
For the quickset-and-dirtiest method of measuring your performance, just NSLog at the start of the request, end of the request/start of processing, and end of processing. That will give you an idea of whether your device or the server are causing a hold-up (something I assume you're looking for). Xcode will timestamp the outputs and you can analyze them after a few trials.
Also, if you run your app in the simulator, it will not give a good representation of phone speed, as it just runs at your computer's speed, but it will give you the option of using the new Network Link Conditioner in Lion to simulate slow and sketchy network connections, so you can see how the network performance would feel out in the field - just make sure to bear in mind you won't have the comfort of the extra processing power.
Flurry provides decent, free analytics and support timed events. Take a look: http://www.flurry.com/product/analytics/index.html
This is good if you want to collect data from other folks running your app.
Related
What I want to do:
I want to have my iPhone to frequently (formally defined later) upload my GPS location to a central server. I want to do this in the most battery efficient way.
Research I'm aware of:
Apple Documentation:
https://developer.apple.com/library/ios/#documentation/CoreLocation/Reference/CLLocation_Class/CLLocation/CLLocation.html#//apple_ref/doc/uid/TP40007126
Stack Overflow Links:
Response 1
Response: proof of existence; some other tool can do it
How to reduce iPhone battery consumption while using GPS
Response 2
Response: track only cell tower changes
iPhone GPS - Battery Draining Extremely Fast
iPhone running periodical process in the background - battery optimized way
Question
My question is a bit vague in that my definition of "frequently" is really dependent on what the battery life can tolerate. For example, if the battery can take updates of every 5 minutes, I'd like to do it every five minutes; if the battery could do this every 10 seconds, I'd like to do it every 10 seconds.
I really want to understand
the different ways (change on significant location, timer, background?) continuously uploading GPS locations can be implemented
advantages / disadvantages
approximately how long the battery life can last in each case
This seems like a fairly generic and common problem. Does anyone know of either:
an in depth analysis of the various methods
or if there is a single "optimal" way to do this?
[Moderators: feel free to mark this comment wiki. I'd love to just get lots of different answers + cost benefit analysis of them.]
Have a look at the Apple Documentation Here.
The significant-change location service offers a low-power location service for devices with cellular radios. This service is available only in iOS 4.0 and later and can also wake up an application that is suspended or not running.
It all depends on the use case. If the user will stay at a certain location and all you want to do is to track if he is leaving the country, tracking the significant location changing will be most suitable. However, this will not be precise enough for navigation apps.
One advantage of choosing the "Apple-Algorithm" is that they'll optimize it for you, if battery issues occur. (-:
Are there any tricks/methods to optimize the battery usage that you app uses?
I've an app that can play streaming audio in the background quiet well. It does the basics like put the app in the background after a little time of no interfacing with the screen by the user.
Are there any other tricks to be done to stop it eating the battery like a fat man at an all you can eat buffet!
Thanks,
-Code
You can decrease the usage of the internet and in the case that you are using the Location librairies don't use them except if you need them in which case don't use cutting eye accuracy. The most valid trick for this is no GPS no Geolocation.
I hope this helps!
Reading in data over a network connection is one of the most power hungry operations on the device. Depending on your protocol, you maybe able to optimize the streaming by buffering larger chunks at a time (if possible). Obviously if this is realtime streaming of a live feed that is not an option.
Review Apple's guidelines here: Performance, make sure you scroll down to the Reduce Power Consumption section. Basically, to reduce power consumption, you should do as little as possible. If you are turning on frameworks like CoreLocation or using the accelerometers, you should disable those as often as possible. Try releasing as many resources as you possibly can when in the background. Less memory means less overhead for the system to keep track of as well.
I'm currently testing the latest iOS4 Feature to put my location aware app in the background. Well, it does work! But on the other hand it's quite hart to handle the immense power usage.
The app consumed about 50% battery power in the last four hours. It read the entire official documentation by Apple on this topic but I'm still not sure which parts of my application are still running and which functionality is suspended (beside the UI Drawing, which should be clear).
I don't use any real boilerplate code but extended libraries like ASIHTTPRequest to talk to my webservice. Tests with a friends car did you show that the Network Connectivity and and the Location Services is still running when I'm using i.e.
[locationManager startMonitoringSignificantLocationChanges];
Apples Documentation on the different application states
Background: The application is in the background and executing code
[...]
Should I write a "bare metal" functions to receive and send this location data? Should I remove all other objects for the time the application resides in the background to reduce the memory footprint? It seems there isn't any best practice yet.
Any ideas? Maybe you guys can provide me with some of your insights. Thanks.
Edit:
There's a new Instruments tool called Energy Diagnostics Instruments to record any power usage (for iPhone 3GS and later) with an attached device. Also there's another service on the device in the Settings App -> Developer -> Power Usage. It's great to test your power usage in field. The created logs can be pulled later in instruments.
Reference: WWDC 2010 Session 309 - Advanced Performance Analysis with Instruments
Sounds like your app is transmitting location data over the cellular network. Turning on the cellular radio is one of the most rapid causes of power drain, especially if the user has a weak signal connection to the cell tower.
You might want to save and package up a bunch of location data, and send the data in a quick burst as seldom as possible (twice per day, when the user stops moving for 30 minutes, only after the user gets to one of their favorite restaurants, etc.) Turning on the radio less than half as often could get you close to doubling the battery life (unless the user is doing something else with the device as well).
I have tried Instruments and Shark to profile an iPhone app, but both use a data sampling approach, by regularly taking screenshots of thread stacks.
I would prefer to have a full coverage profiling tool, which would record every single function call and the time spent in functions and their subroutines. This seems intuitively better than getting samples. Something like AQtime on Windows would be great.
So my questions are:
Can sample-based profiling be trusted and be as useful as function-call-based profiling?
Can Instruments or Shark do this type of profiling?
Are there other tools available, which would be closer to what I want?
I've found that the time profiling used by Shark is very accurate for determining what your bottlenecks are in your code. You can adjust the sampling interval to be more fine-grained by showing the Mini Config editor using Config | Show Mini Config Editor and lowering the sample time.
Instruments in Xcode 3.2 also now has a nice Time Profiler instrument, although that's Mac-only. I've found that Instruments works well for profiling, but it can drop samples if the system is under heavy load. Generally, I start with Instruments, given how easy it is to use, then move on to Shark if I need a more detailed view of what's going on.
If you really want to do function-call-based profiling, I'd look at DTrace. I've written a couple of articles about tuning Cocoa applications using DTrace here and here. The latter one even shows an example of tuning the startup time of an iPhone application using a custom DTrace script.
Unfortunately, DTrace currently does not run on the iPhone itself, but you can still gather a lot of interesting information using it by running your application in the simulator. While the exact timing information will be nowhere near what it is on the device, knowing exactly what methods are executed how many times and in what order can give some clues as to where to optimize. I use DTrace to provide a different perspective on information gathered by Shark and Instruments, and to answer specific questions about my application.
Yes, stack sampling can be counterintuitive, and yes it can be trusted. Look at this link, and the first comment.
It's not about asking "How long did this take?"
It's about asking (in a doubtful voice) "Was that nanosecond necessary?"
Our lovely app that downloads mp3s from our server into a local file on the phone then plays from that file was rejected for using too much bandwidth.
I understand the rejection (we are downloading rather than streaming) and don't quibble with their decision... our first priority was quality of user experience.
I am just wondering... what do I do now?
There are no hard and fast rules... Apple just says, "Must not in Apple's reasonable judgment excessively use or unduly burden network capacity or bandwidth".
Anyone have data on what Apple considers reasonable data transfer rates?
Should I fill up the buffer file in short spurts? Should stream the file at a constant rate (and how would I limit the transfer rate from within the app?)
Any and all suggestions are welcome.
Thanks
I have talked with Apple Developer support, and just FYI. You are only allowed to stream 1 MB per minute over the Cellular network. Support suggest that you test your app in the following way:
"The basic measurement methodology is to turn off all background updating (particularly Mail's automatic mail downloads and Calendar updates), reset the transfer statistics in "Settings:General:Usage:", and then launch your application. Let it run for a fixed amount of time (five minutes is reasonable), and then exit your application. Once you've finished the test, the numbers listed under "Cellular Network Data" in "Settings:General:Usage:" are what you should focus on reducing.
Using what I just described, I'd suggest 4.8 MB every 5 minutes as the guideline you use to ensure your application stays within our bandwidth requirements."
Hope that helps at least a little.
Have you considered HTTP Live Streaming? It's built into OS 3.0.
Basically, you split your media into small (say, 10 second) snippets and put it on a standard web-server. Then you create little text 'meta-descriptor' files in EXTM3U format that point out where the bits are. The interesting thing is that you can create multiple versions of each snippet at different bit-rates. So if your bandwidth is good, the iPhone player dynamically chooses the higher bit-rates but when it's low it automatically switches to the lower bit-rate version of the snippet. It does this on-the-fly to adapt to changing conditions.
So if you split your MP3 into multiple 10-second bits at say, 3 different bit-rates then when the user is connected through WiFi they get the high-quality stuff but if they're on 3G or EDGE they get progressively lower-quality (and smaller sized) content.
If this violates your downloadable media concept, then perhaps you can use the same trick and keep multiple size files for each connection type. Then if you're on WiFi (or get a fast turnaround on a heartbeat ping to the server) download the big file vs. medium or small size ones.
Here's a decent step-by-step on segmenting content. They focus on video but it should work with audio content as well.
I would suggest throttling the bandwidth on the network/http request when connected to cellular, and having no throttle on wifi.
A combo of using Reachability to detect the network status, and implementing your http request using CFNetwork with a sleep until the next second whenever you have already downloaded as much as allowed per second should get you there.
You should check out this project, either to use, or just to see an example of how to code this well:
http://allseeing-i.com/ASIHTTPRequest
The ASIHTTPRequest class implements this, using either the old or 2.0 version of Reachability, and also provides a class level throttle, so even if you have multiple concurrent downloads and uploads, so long as all of them go through ASIHTTPRequest, they will be properly taken together and throttled properly.
A bit more on how to use this here:
http://allseeing-i.com/ASIHTTPRequest/How-to-use#bandwidth_throttling
Lots of good stuff in this (and no, I am not the author of any of it).
On a more serious note, what is it trying to do? There are many markets for iPhone outside of the US which don't have good download rates for phones, nor are the download cap sizes good, so having an app do lots of downloading isn't a good thing.
Could you possibly drop the data rate on your mp3 or some such? Make it optional to do the downloading with a warning that it will use your download?
YOu could not download the mp3 unless you are on wireless, and inform them of that. If you the mp3 are too important, then just tell them it only works on wireless, or include a few mp3 on the device. But 30min of mp3 is ridiculous, that is about what, 30meg.... think about it, 30 meg is just too much.