want to open a large text file and then search the content of it.
I loaded the file with stringWithContentsOfFile into a NSString.
Every thing works on a 30mb file. But I am concerned what happens if I load a 200mb file, which I want to do.
Is the complete NSString in the memory? if so it wouldn't work on iPhone. is there a solution for such large files on the iPhone?
A good way to read a large file would be to buffer small chunks of it at a time.
Not sure of the exact API methods you could use to do this, but it is fairly standard practice for audio, video, etc to read a small amount of the file into memory, process this, and remove it from memory as you continue through the file.
Since the limit isn't documented the way to check would be actually profiling it on target. People may be able to give their limits here but that is totally relative. Also a good memory management scheme and design would help you avoid problems of running out of memory.
Related
I've been working with video_player package from Flutter. I'm mostly testing videos taken from the url. My code is very similar to the ones from the examples.
Everywhere I read about it, I can see that this library does not support caching of the videos. But what this exactly means? What exactly is happening behind the scenes and how the behaviour would change if the caching would actually be implemented? How this is different from buffering? Are the video files simply downloaded to our device?
If yes, then were those files are kept?
One additional question is, how can I check the network consumption caused by using such connection? I've tried using Dev Tools but the network tab is always empty.
One last thing is, is it possible to pre-initialize next videos, so when we would like to switch between them, they are already partially pre-loaded?
U can use a package that helps you manage caching futter_cache_manager
https://pub.dev/packages/flutter_cache_manager
U can use this in combination with video_player. However, u would have to download the whole file first to be then be able to retrieve it for video_player to consume it.
An idea would be to stream the video and also download a copy locally. This however would consume more data than just downloading and caching the video first, then playing it locally.
As for how to check for network consumption, i am not sure.
Starting with the theory:
1.Cache is a high-speed storage area while a buffer is a normal storage area on ram for temporary storage.
2.Cache is made from static ram which is faster than the slower dynamic ram used for a buffer.
3.The buffer is mostly used for input/output processes while the cache is used during reading and writing processes from the disk.
4.Cache can also be a section of the disk while a buffer is only a section of the ram.
5.A buffer can be used in keyboards to edit typing mistakes while the cache cannot.
When it comes to video buffering just look at youtube app. You can see the buffer being made when grey line grows bigger before the red line. Mostly information stored this way cannot be accessible at all as Android uses combination of RAM allocation for both caching and buffering as it sees fit for current active process.
Technically you could try pre-loading different videos by starting and pausing all of them at once but I cannot imagine how much tampering with system memory control it would take, even youtube doesn't work like that.
I am using a custom UITableViewCell with 3 UIImages in a UITableView with 50-100 rows. Its similar to a UITableViewCell the Facebook iPhone app uses for its news feed view.
The application has 4 similar UITableViews which may be open at the same time via a UITabConroller.
The images are lazy loaded, there is a cache on disk so that no images are loaded twice from the server and there is also a NSMutableDictionary for images allowing in-memory reuse of the same image eg: a users profile picture appears multiple times
This setup is extremely fast but takes a lot of memory even after using the NSMutableDictionary for image reuse.
I tried a variation without the NSMutableDictionary where images are either loaded from the server or pulled from the disk cache every time cellForRowAtIndexPath is called. This setup is extremely memory efficient but causes a noticeable lag in the UITableView scrolling.
A mid-way approach is to free the NSMutableDictionary for images when a low memory warning is received.
Will really appreciate recommendations to optimize memory usage and speed in this scenario and or an insight into how the Facebook iphone app or three20 execute this conceptually.
I have an app that is very similar to yours in many respects (uses Three20, has several tabs across the bottom, each tab can have a table, each cell can have one or two images); and the approach I'm taking is the one you mentioned near the end of your post:
A mid-way approach is to free the NSMutableDictionary for images when a low memory warning is received.
Personally, I quite like iOS's approach to memory management, of warning me when memory is getting tight. The Mac/PC approach of "just use all the memory you want, we'll swap it out to disk if memory gets tight" has the disadvantage that even though the OS is the only one who really knows how much pressure there is on memory, it isn't telling you. I think what every polite app would really like to say (if apps could talk) is, "I'd be happy to use as much memory as you'll give me, but I don't want to be a bother, I don't want to slow down any other apps, so if you could please give me a hint as to how much memory I can use without causing problems, I would appreciate it."
Well that's what iOS's memory warnings give you, in my opinion. So, just keep as many images cached in memory as you want; and when you get a memory warning, empty the in-memory cache. To me it's really the best of both worlds.
Also, you should definitely take a look at Three20's TTURLCache, although I can't tell you a lot about it because I haven't dug into it very much. What I do know is:
If you retrieve your messages via TTURLImageResponse, it will automatically cache them in TTURLCache's image cache.
You can also store and load your own images (and other data) in the TTURLCache.
Three20 seems to take an approach similar to what I am talking about. Take a look at this code from Three20Network/Sources/TTURLCache.m (the NO argument means don't remove from disk, only remove from memory):
- (void)didReceiveMemoryWarning:(void*)object {
// Empty the memory cache when memory is low
[self removeAll:NO];
}
In addition, that class also allows you to set a maximum size for the in-memory cache, but by default there is no maximum size.
you would purge images when they go offscreen, then read the images from the locale cache on demand from a secondary worker thread when needed. since one can zip through tables, add support for read cancellation (esp. for requests which come off the server). NSOperation is a good api for this.
if you know your table's small, then you could opt to avoid purging in such cases.
also, rescaling the image to the size you'll display it as is often a good idea (depending on how far you want to take an optimization). assuming the source is larger than the displayed size: this will reduce memory requirements, drawing speed, disk space, and disk read times.
you can also read three20's sources to see what they have done.
I have an iPhone application that will save number of images in it.
I used SQLite in-order to save the images into the application.
There were lot of memory issues after i saved more than 20 images.
Do any one know how many images users can save in their app database?.
if it depends on iphone memory, how can we get that max limit?.
One more thing:
I have removed the database and used the file system to store the images into application.
but same problem replicated.
Can any one suggest me on this.
I owe a lot for your great help.
Thanks in advance.
I believe storing the image in the file system is a much better idea, could you provide us with more information, like image size, and also some of your saving code?
When you say 'memory issue', do you mean leaks causing out of memory exceptions or are you sure your database has filled the device's disk completely. To know how much SQLite can store read the discussion on this question.
On the other hand, if you are having out of memory exceptions (didReceiveMemoryWarning) you need to tune your code. Specially, when working with many images, just avoiding the use of 'imageNamed' factory method does the job. This is because it creates an autorelease object which remains longer in the memory. Instead create UIImages using the 'initWithContentsOfFile' to create the image and release it immediately after it is used. If you still face the memory issue, you probably have some leaks and need to post some code for people to answer more correctly.
Apple just released some sample code on lazy loading images in a UITableView a week ago. I checked it out and implemented it into my own UITableView (which is a drawRect one for fast scrolling), to see if there was a difference from what I was already doing.
After implementing I am not sure what is best; the new code or what I already had. I am not seeing much of a speed improvement on my 3GS.
"Sandbox" method: Load images lazily, then save to local tmp folder in the sandbox. Each time the cell is displayed it looks for whether an image with that filename is already located in the sandbox folder. If it is, it retrieves the image and displays it, if not it continues with the download, saves it locally and then displays it. The benefit with this is that the images won't be blank the second time you open the app. They will already be downloaded and ready for displaying.
Caching method: This also loads the images lazily, however, now I include a UIImage on each object in the array that's displayed in the tableview. Instead of saving the image locally, I now download the image and put it into the array for the object. Now, instead of checking for the filename every single time, it jut check whether the UIImage != nil and uses the cached image (or downloads if nil).
A small difference is also that the caching code resizes the image before caching it to the exact size of what is displayed in the cell, whereas the image used in the sandbox code example is actually a bit larger than what it needs to display, which means it has to resize on the fly when scrolling as well. I read months ago that this could be a bit expensive to do, and I am also not sure whether it makes much of a difference in terms of then using a cached image instead of the sandbox-stored image and therefore more CPU intensive anyway (compared to what you save from caching with the caching code above).
I guess my question would be whether I should even bother with the caching code? Again, the new code won't immediately load images on a new launch, whereas the old code actually does because it's already in the sandbox. Since I am not reusing images, I have a lot of images to load (from the sandbox or cache) so I am not noticing a huge difference in speed. In fact, on my 3GS it's almost impossible to tell, in my opinion. The scrolling is not silky smooth, and I assume this is due to the large amount of images that I cannot reuse (different image for each cell). I am also wondering whether the sandbox method would get slower once there's 1000+ images in the folder, for example, eventually having it look through many more images than just 100 or so.
I hope I am making sense. I wanted to be pretty thorough with the details, and I am happy to give more details if needed.
Thanks!
If you have code that already works, and there's not a pressing problem, then don't change it.
If your scrolling actually is too slow, then perhaps you could use a mixture of ideas, and try to get the UIImage, and if it's not there, load it from the sandbox, and if it's not there, then download it.
The only good way to tell if there is any discernible difference in performance is to use profiling tools like Instruments (for measuring things like display framerate for the two techniques) or Shark (to determine hotspots in your code). There could be small differences in your exact implementation that could potentially cause significant differences between any general answer we could give and the actual performance you see in your application.
The thing that primarily concerns me with the "sandbox" method is not performance but disk space usage. Users won't appreciate you filling up their iPhone or iPod Touch with unnecessary files, especially if all the images aren't consistently used or if the set of used images changes often. Without knowing more about your application its impossible to guess how often these cached images would be loaded.
If you're testing locally on your own device, you might be on Wifi network. My recommendation would be to turn Wifi off for part of your testing to see how the two approaches perform when you have to fetch all the images over the cellular network. I would also recommend trying to find an older device (iPhone 3G or worse) because the 3GS does in fact hide potential performance issues that could be annoying for users on older devices.
I have personally used the LazyTableImages technique in my apps many times (provided it hasn't changed drastically between WWDC09 and the recent 'release') and find it to be just what I need. Caching images on disk wouldn't be an option in my case, however, and you shouldn't take my anecdote too strongly into account - profile your own code and use the results it shows.
Edit: The obvious answer is that accessing an in-memory cache is going to be faster than accessing the filesystem, but of course the final word on that is left up to profiling. If the images are already in memory, they don't need to be read from flash and parsed by UIImage. The traditional tradeoff comes into play here though - in-memory caching vs. disk space.
While it may be faster for you to store your images in-memory, you need to be very sure that you correctly handle memory warnings in your application (as you should be doing anyway!). Otherwise long period of use will lead to many, many images in your in-memory cache and trigger memory warnings and if your application is not built to handle these, at best your application will be killed by the OS due to lack of memory resources.
There are pros and cons in both approaches that you present - I suggest using elements of both in your app.
It's better to keep your images in memory and save them later (perhaps when your app quits). If you have a lot of images, it might be faster to use Core Data to save them, than as regular files.
It's also better to avoid doing any resizing on the fly, i.e. in your tableView:cellForRowAtIndexPath: or tableView:willDisplayCell:forRowAtIndexPath: methods or in any method that has to do with drawing your cells' content view. If you can, ask the image provider (content management?) to supply images at the size that your table view displays.
I have a question and I am very open to suggestions (even very odd ones!)
I am writing an iPhone app, which does a request (URL with parameters) to a server. As a response, the iPhone receives XML. All is well.
Right now, I am looking to improve my application's speed by measuring the time it takes to perform certain tasks. I've found that, of all the tasks performed (downloading, XML Parsing, sending the request, handeling the request, parsing objects from XML), downloading the actual XML takes the longest.
Now, my XML files are very, very, very easy and very, very, very small. I use them primarily to read RSS-like data and show them in a UITableView.
My app works very well, and there is nothing that feels really slow, but there is one application in the App Store right now which does something very similar to my application, but is way faster and feels more 'snappy', if you know what I mean. It also has the great feature to load the headlines one by one from the RSS-feed.
Currently I'm experimenting with gzip compression of my data, but the compression only makes my data half the size and it doesn't seem to do any real good for performance. The main thing is, that the data has to be downloaded, before it gets parsed. It would be very cool to have a 'stream' of data, which is parsed as it comes in. That way, I can do two jobs almost simultaneous and load headlines one by one (making user interactivity more attractive).
Anyone has an idea of how to improve my performance? Either by great compression tips or entirely different ways to communicate with the server.. All is welcome!
UPDATE: putting the latency and responsiveness of the server aside; how could I get a source of XML to be 'streamed' to my iPhone (downloaded byte for byte) and at the same time get parsed? Right now it is a linear process of downloading -> parsing -> showing, but it could become semi-parallel by downloading & parsing at the same time (and show each item when it is done downloading, instead of loading it all at the same time in a UITableView)
Assuming you're using NSXMLParser's initWithContentsOfURL:, that's probably part of the problem. It seems like that downloads the entire contents of the URL, then hands it off to the parser to parse all at once.
Despite the fact that the NSXMLParser is an event-driven parser, it doesn't seem to support streaming data to the parser in an incremental manner. You could, of course, replace NSXMLParser with some other parsing library that handles incremental data in a more sensible way.
An alternative would be to use NSURLConnection, and create a new NSXMLParser and re-parse the data each time some data comes in, in the connection:didReceiveData: method of your NSURLConnection's delegate. You''d have to write some extra code to ignore the extra events from re-parsing the beginning of the file more than once.
This seems like it'd be more work than just grabbing some other library and adapting it, but maybe not, depending on how you're handling the downstream creation of your table data.
If NSMutableArray is the underlying data structure of your UITableView you should try
using initWithContentsOfURL. The format on the server needs to be in apples "plist" xml which is easy to generate. I'm guessing that if cocoa already has resources acquired for
processing xml it would be quicker to use them instead of creating your own xml parser instance.
How are you parsing the XML data. The need to parse XML as you receive it is the whole reason pull parsing was invented, and they have the NSXMLParser event driven parser that operates on a stream of data...
That also means compression of the data is counterproductive.
For very small amounts of XML, the issue of speed is probably about latency of connection.
Thus, other big factor could be the DNS lookup. You could do that yourself once beforehand and cache the IP, perhaps rechecking only on failure to connect to your server...
Since your xml files are very small, (just 8k or so, I imagine?) I don't think you'd see a big performance boost trying to parse them as they come in. You can use an asynchronous NSURLConnection to do that, but I can't see it helping with a small file very much. Have you considered the time it takes to generate the XML file on the server? Are you using PHP to create the XML, or accessing a static file? For testing purposes, it might be interesting to access static files and see how that compares.
I ran into a problem like this a while ago, and it turned out the mySQL database on my server was being slow, and it really had nothing to do with my app.
Also, use Instruments to look at the amount of memory that is allocated while you're downloading and parsing an XML document. Some XML parsers load the entire XML document into memory and then allow you to index randomly into the document's elements, while others provide step by step parsing only. The step-by-step method is much better for limited devices like the iPhone and it's probably what you're using, but a quick look at memory consumption should tell you.