I am writing an app that caches streaming content from the web on the iPhone. Right now, I'm saving data to disk as it arrives (in chunk sizes ranging from 1KB to about 60KB), but application response is somewhat sluggish (better than I was expecting, but still pretty bad).
My question is: does anyone have a rule of thumb for how frequent and large writes to the device memory should be to maximize performance?
I realize this seems application-specific, and I intend to do performance tuning for my scenario, but this applies generally to any app on the iPhone downloading a lot of data because there is probably a sweet spot (given sufficient incoming data availability) for write frequency/size.
These are the resources I've already read related to the issue, but no one addresses the specific issue of how much data to accumulate before dumping:
Best way to download large files from web to iPhone for writing to disk
The Joy in Discovering You are an Idiot
One year later, I finally got around to writing a test harness to test chunking performance of streaming downloads.
Here's the set-up: Use an iPhone 4 to download a large file over a Wi-Fi connection* with an asynchronous NSURLConnection. Periodically flush downloaded data to disk (atomically), whenever the amount of data downloaded exceeds a threshold.
And the results: It doesn't make a difference. The performance difference between using 32kB and 512kB chunks (and several sizes in-between) is smaller than the variance between runs using the same chunking size. The file download time, as expected, is comprised almost entirely of time spent waiting on the network.
*Average throughput was approximately 8Mbps.
Related
does anyone know of best practices or common strategies in backend design for serving dynamic images and videos to client applications?
Background: I'm currently building an application that allows users to upload their own images and videos. I'm not really sure about how to serve these media files back to the client in the most efficient way. Do I store the files on the same VPS that my application server is running on? Do I need to save the files in different qualities / densities to better adjust for the clients' screen resolution? (I'll have mostly mobile clients)
I tried googling these questions but apparently I'm asking the wrong questions :-)
I would really appreciate maybe a reference or professional vocabulary on these topics.
Thanks in advance.
1) You need to split web server and application server.
First of all do not try to stream media files from your backend unless you can offload low-level stuff to OS - most likely you will do it wrong.
Use proxy server as an web server to serve such files.
nginx will do.
Also you need to have backup of your media files the same way as you do backup of your database.
Storing static huge media files along with application server is wrong move - it will not scale at all.
You can add cron task to move files to some CDN server - when your move is complete you replace URL in database to match new location.
So by using nginx you will save precious CPU and RAM while file is getting moved to external server.
And CDN will help you to dedicate bandwidth and CPU/RAM resources to application server.
2) Regarding image resolution and downsampling:
Screens of modern handsets have the same or even better resolution compared to typical office workstation.
Link speeds have much bigger impact on UX.
If client has smartphone with huge screen but with slow link you still have to deliver image or video as fast as possible even if quality of media will not be match the resolution of handset.
It makes sense to downsample images on demand and store result on disk for nginx/CDN to serve it again.
In case of videos it makes sense to make "bad" version with big compression(quality loss) for the cases of slow link - device will downsample it itself during playback.
And you can keep client statistics (screen sizes/downlink speeds) and generate optimized versions of such video file later when you see that it is "popular".
FYI: Several years ago some social meda giant dropped idea to prepare all possible versions of the same media file in favour of FPGA on-the-fly resampler.
I do not remember the name of the company and URL to the article. It was probably instagram.
Some cloud providers have offers with FPGA or CUDA on board to do heavy lifting.
So in some cases you could exchange storage for heave horsepower to do conversion on the fly.
I am looking into making a system for work where you can download huge video files, (Im talking 4k full length videos which have a file size of sometimes 500GB) and I'm looking into the best way of doing this.
Would it simply need a file manager to split the download? or could I use bittorrent?
any suggestions?
Bittorrent can be used to to 1:1 transfers and has the benefit of hash-verifying the contents and being able to resume the transfer when it has been interrupted. But that can also achieved with other tools such as rsync.
Bittorrent's strength is making the transfer between many nodes scalable and being able to work in a decentralized manner.
I was discussing the requirements of an app and I am a little lost with a few things.
1)
I will be getting data back from a web server via JSON. The user wants the data cached for a month so that there is no unnecessary data usage. My concern is that the data base on the webserver is about 300mb in size. If the user is very active this could mean there could be a lot of data cached on the phone. Is there a limit to the amount of storage you can use? Would 50mb of cached data be realistic to store?
2)
If i was to store all this data, what would be the best way to handle it? Since i would need to be able to search and manage the data from the device. Is this a realistic request from the client, or is the iphone really not suited to this kind of operation. Would it be best to not cache the data at all and just pull it down with JSON as required.
Many Thanks,
-Code
You can use CoreData framework with SQLite for storage.
According to the accepted answer to this question, there is no specific limit to how much data you can store other than the device's flash storage size.
Is there a limit to the amount of storage you can use? Would 50mb of cached data be realistic to store?
No, there is no limit except the capacity of the iPhone's flash memory. 50 MB is not that much IMO, considering there are apps that are hundreds of megabytes large.
I'm having issues with audio files on the iPhone web-app. Seems as each time an audio file is played, it's loaded first then played, even if repeating the same audio on a page that hasn't refreshed (done via javascript). From what I've research manifest files would be great but they are for offline application. I'm now researching HTML5 databases.
Does anyone know if HTML5 databases can store audio files such as mp3? The end result it then to pull the mp3 from the database. It might still have to load the file each time from the database but I'm hoping it's quicker than retrieving it from a server.
Thank you.
I think what you are after is possible, however you have a significant hurdle in that the implementation of HTML5 databases on most browsers is limited to 5mb as per w3c recommendations:
A mostly arbitrary limit of five
megabytes per origin is recommended.
Having said that the way its implemented in iPhone Safari is that databases can grow until they reach 5MB in size at which point the browser will ask the user if they wish to allow for the extra size, asking again at 10, 50, 100 and 500MB (see section "Estimated Database Size" in this post by html5doctor).
There is no limit on the number of databases you can build per domain in safari, however according to this post by Cantina Consulting you can have a total of 50MB across all databases in a single domain.
Given these parameters, a possible work-around for this implementation is to split your mp3 blobs across multiple databases, creating a new database each time your reach 4.9MB, however even if you follow this design it may not be ideal as you will still experience the following:
50MB is not a lot of audio files, a typical 5/6min song is about 5MB at 128Khz, so that only gives you space for about 1CD (60 min) of mp3 songs, after this you will need user cooperation to use additional database space.
You will still have significant security issues trying to play the mp3 blobs from the javascript runtime, it may be possible to bypass these tricking flash into thinking they are mp3 stream but I'm not sure how you'd go about it.
Feel free to have a play around with this iPhone HTML5 SQL Client I put together, you may want to use something similar for experimenting with your local mp3 Database.
My problem is that my application size is very high,
is there any idea to reduce size of application?
if i make application without content and content is uploaded my server then how i sync the application with content put on my server?
i want to know that once user download application after that when he use application then we stream the content and save his document folder.
once user stream then never required for streaming.
is it possible????
Thanks,
Reducing the size of your application depends on the TYPE of contents of your application. I highly doubt that the application code is the cause, and since you did not mention what they are I am assuming they are some kind of resource.
If your resources are images, try to use image compression programs. Or convert them to smaller sized images or optimize the images.
If your resources are documents / text files / files that have a high compression ratio when zipped. Then you can try to zip your resources and access them inside the compressed file (this will mean additional coding, and probably slower in performance).
These are just examples.
It is not advisable to stream large contents because it uses the network bandwidth which, depending on the user's plan, can cause a big spike in phone bills.
Yes it is possible that you can download your content and can save to application's document folder, when user runs your application for the first time. Thought it may affect the first impression to your user as it will take time to download.