iPad application
I have a large text file which I want to divide into several pieces and process them in a loop.
I used the following code:
NSArray * contentArray = [stringFromFile componentsSeparatedByString:#" "];
for(...){
contentRange = NSMakeRange(lastPosition , newLength);
NSArray * subContentArray = [contentArray subarrayWithRange:contentRange];
NSString *subContent = [subContentArray componentsJoinedByString:#" "];
....
//Process sub content here...
}
After running, I receive malloc error, code 12
Observing the Activity monitor, the memory and the VM size increases and until system memory is exhausted and the application crashes.
Is there any way to prevent this?
One way to work around this issue will be to use a custom NSAutoreleasePool to constantly clear temporarily allocated memory. Like so:
NSAutoreleasePool *pool = [[NSAutoreleasePool alloc] init];
NSUInteger i=0;
for (...) {
... // your method contents
if (++i % 10 == 0) {
// you might want to play with the frequency of the releases, depending on the size of your loop
[pool release];
pool = [[NSAutoreleasePool alloc] init];
}
}
[pool release];
The autorelease pool thing will take care of the memory problem for the most part, but if the file is any size it might be slow.
It seems like scanning the original string in a loop getting substrings directly from the string would be faster. Also use at least 1/2 the memory, since you duplicate the string in the components array.
The code would use rangeOfString on the original string, until a substring of appropriate length was found, then process the substring, and onto the next one.
iPhone 3G phones only have about 5 - 15 MB of memory total for your application. You are on an iPad, though I see which does give you about 4x that amount.
Related
I have A LOT of rows that need to be inserted into a database. I download about 50 .txt files that are tab delimited with the information I need. I parse the data and insert it into the database in the same method. When I had it as separate methods, the simulator on my 27" iMac would crash because it ran out of memory. The method looks like this:
- (BOOL)insertDataWithFileName:(NSString *)fileName {
NSLog(#"%s %#", __FUNCTION__, fileName);
if (_db == nil) {
return NO;
}
// Get the data from the file
NSData *data = [[NSData alloc] initWithContentsOfFile:[[self applicationDocumentsDirectory] stringByAppendingPathComponent:fileName]];
NSString *responseString = [[NSString alloc] initWithData:data encoding:NSUTF8StringEncoding];
// Parse each line by newlines first
NSArray *lines = [responseString componentsSeparatedByString:#"\n"];
NSString *ID = [[fileName componentsSeparatedByString:#"."] objectAtIndex:0];
// Set up the database insert statement
NSString *tableName = #"Hardcodedtablename"; // remove after prototype
BOOL oldshouldcachestatements = _db.shouldCacheStatements;
[_db setShouldCacheStatements:YES];
[_db beginTransaction];
NSString *insertQuery = [NSString stringWithFormat:#"INSERT INTO %# values(null, ?, ?, ?, ?, ?, ?, ?);", tableName];
BOOL success;
// Parse each line by tabs
for (NSString *line in lines) {
NSAutoreleasePool *pool = [[NSAutoreleasePool alloc] init];
NSArray *fields = [line componentsSeparatedByString:#"\t"];
// Need to check since some of the .txt files have an empty newline at the EOF.
if ([fields count] == NUM_VARIANT_FIELDS) {
NSNumber *loc = [NSNumber numberWithInteger:[[fields objectAtIndex:0] integerValue]];
NSString *carType = [fields objectAtIndex:1];
NSString *model = [fields objectAtIndex:2];
NSString *AA = [fields objectAtIndex:7];
NSString *BB = [fields objectAtIndex:8];
NSNumber *freq = [[fields objectAtIndex:11] isEqualToString:#""] ? [NSNumber numberWithDouble:-1.0] : [NSNumber numberWithDouble:[[fields objectAtIndex:11] doubleValue]];
NSArray *argArray = [[NSArray alloc] initWithObjects:ID, loc, carType, model, AA, BB, freq, nil];
success = [_db executeUpdate:insertQuery withArgumentsInArray:argArray];
[argArray release];
}
[pool drain];
}
[_patientAnnotatedDatabase commit];
[_patientAnnotatedDatabase setShouldCacheStatements:oldshouldcachestatements];
return success;
}
As you can see from my for loop, I actually don't need all the files in the array after it gets separated by the \t. I set up an autorelease pool to reduce the memory footprint in the for loop. The largest file has 270,000 rows in it. There are about 50 of these files. So after 2 files on the device, it crashes. No error message though.
I was wondering what my options were at this point. Can the iPad actually handle that amount of data? Can I make optimizations on this method?
Or would I have to create a database on the server side and download the database to the device instead? If I did download the database from the server to the device, does FMDB support an ability to just transfer those tables from one DB to another? I wasn't able to find anything like that while I was googling. Thanks!
It really depends on what you are using the data for. If you are using it to search or for some purpose where you will really need ALL of it, you are basically using this much memory:
270,000 rows * 25 chars/row * 50 files * 1 byte/char = 321 Mb of RAM
I'm guessing about the 25 chars/row part, but when you are using over 100 Mb of memory, you are using a lot of memory. However, the iPad does have about 1GB of RAM for you to use, so you would seem to be within the amount of available memory. But, because of the NSArrays and SQLite FMDB stuff you are using to store it (as well as other NSObjects), the data with encapsulation could amount to over 1GB, when added to other processes running on the iPad.
So, all of this memory might be your problem. You might be better of saving all of this data on a webserver that can handle it, and call whatever functions you would like through said server. It probably would be much faster, and it would allow you to make adjustments to the data easily, if it needed to be updated. It also would allow your app to be used on devices like the iPhone that do not have enough RAM.
If sheer amount of data is your problem, then your solution is the one above. However, I still believe that the iPad is capable of handling 300+ Mb of RAM, so your problem could lie in the code. Make sure you are accessing all of your data correctly, especially when you are separating the data into arrays from the files, because the smallest "out of bounds" array error could cause your program to fail.
Have you also tried running your code with the memory profiler enabled. You can enable this app in the new Xcode by holding down the "Run" round button until you can select an option that says "Profiler." This app will show you how much memory your app is using, so you can make sure its under 1GB.
Unfortunately, I had NSZombieEnabled set on in my scheme to track down an earlier bug. With it turned off, it seems to be now not crashing at this moment.
Once again I'm hunting memory leaks and other crazy mistakes in my code. :)
I have a cache with frequently used files (images, data records etc. with a TTL of about
one week and a size limited cache (100MB)). There are sometimes more then 15000 files in a
directory. On application exit the cache writes an control file with the current cache
size along with other useful information. If the applications crashes for some reason
(sh.. happens sometimes) I have in such case to calculate the size of all files on application start to make sure I know the cache size.
My app crashes at this point because of low memory and I have no clue why.
Memory leak detector does not show any leaks at all. I do not see any too. What's wrong
with the code below? Is there any other fast way to calculate the total size of all files
within a directory on iPhone? Maybe without to enumerate the whole contents of the directory?
The code is executed on the main thread.
NSUInteger result = 0;
NSAutoreleasePool *pool = [[NSAutoreleasePool alloc] init];
NSDirectoryEnumerator *dirEnum = [[[NSFileManager defaultManager] enumeratorAtPath:path] retain];
int i = 0;
while ([dirEnum nextObject]) {
NSDictionary *attributes = [dirEnum fileAttributes];
NSNumber* fileSize = [attributes objectForKey:NSFileSize];
result += [fileSize unsignedIntValue];
if (++i % 500 == 0) { // I tried lower values too
[pool drain];
}
}
[dirEnum release];
dirEnum = nil;
[pool release];
pool = nil;
Thanks,
MacTouch
Draining the pool "releases" it, it doesn't just empty it. Think of autorelease pools as stacks, so you have popped yours, meaning that all these new objects are going into the main autorelease pool and not being cleaned up until it gets popped. Instead, move the creation of your autorelease pool to inside the loop. You can do something like
NSAutoreleasePool* pool = [[NSAutoreleasePool alloc] init];
int i = 0;
while( shouldloop ) {
// do stuff
if( ++i%500 == 0 ) {
[pool drain];
pool = [[NSAutoreleasePool alloc] init];
}
}
[pool drain];
I have a requirement to create some NSDecimalNumber objects objects as part of my application (as I require the precision of calculation they offer) but I note that in calculations they return NSDecimalNumber objects which are, presumably, autoreleased.
My question is really whether this is potentially problematic in an iPhone application where I may carry out lots of calculations.
The question is not just relating to NSDecimalNumber specifically but to the sometimes unavoidable creation of autoreleased objects in the course of developing an iPhone application.
Any detailed answers on this point would be gratefully received.
Remember, you can create your own NSAutoreleasePool objects.
For example:
for (int i = 0; i < 1000; ++i) {
NSAutoreleasePool * p = [[NSAutoreleasePool alloc] init];
for (int j = 0; j < 1000; ++j) {
NSString * s = [NSString stringWithFormat:#"%d,%d", i, j];
NSLog(#"%#", s);
}
[p release];
}
If you do that, you'll never have more than 1000 of those strings in memory at a time.
Yes, creating lots of autoreleased instances on the iPhone can create memory problems, particularly in a tight loop, which is why I tend to avoid them when I can. You can create your own autorelease pools to manage this, but they also add some performance overhead and additional code that you have to keep track of.
It is for this reason that when I do high-precision calculations, I tend to use the NSDecimal C struct instead of NSDecimalNumbers. In fact, I performed some benchmarks on this and found a significant performance increase when going with the C struct (copied from my answer here):
NSDecimal
Additions per second: 3355476.75
Subtractions per second: 3866671.27
Multiplications per second: 3458770.51
Divisions per second: 276242.32
NSDecimalNumber
Additions per second: 676901.32
Subtractions per second: 671474.6
Multiplications per second: 720310.63
Divisions per second: 190249.33
As you can see, there is almost a fivefold increase in calculation speed between the NSDecimal paths and NSDecimalNumber ones. The biggest difference between the NSDecimal and NSDecimalNumber calculations was the memory allocation of the NSDecimalNumber instances. Therefore, you should avoid allocating temporary autoreleased instances wherever you can.
If you are worried about handling too many autoreleased objects you can create your own autorelease pool (see memory management):
for (count = 0; count < limit; count++)
{
NSAutoreleasePool *loopPool = [[NSAutoreleasePool alloc] init];
NSString *fileContents;
NSString *fileName;
fileName = [args objectAtIndex:count];
fileContents = [[[NSString alloc] initWithContentsOfFile:fileName] autorelease];
// this is equivalent to using stringWithContentsOfFile:
/* Process the file, creating and autoreleasing more objects. */
[loopPool release];
}
Every object that is given a chunk of memory will have to relinquish it. The question is when.
I try to alloc/init/release when I can, so that objects only hang around as long as they are needed.
If I use autorelease I have less control over when the object is released. If the application tries to access a released object then it can crash. So tighter memory management is a good thing, I think.
So long as you retain autoreleased objects returned from a method, you should be okay. (Unless you're asking about something else, in which case I apologize in advance.)
I have a simple example of what I don't understand about memory management on the iPhone:
- (IBAction)AssignAndReleaseOne :(id)sender {
for (int i=0;i<10;i++) {
someString = [[NSString alloc] initWithString:#"String Assigned"];
}
[someString release];
}
- (IBAction)AssignAndReleaseTen :(id)sender {
for (int i=0;i<10;i++) {
someString = [[NSString alloc] initWithString:#"String Assigned"];
[someString release];
}
}
I would expect to get a memory leak in the first method because I alloc 10 times (or is it 11 :) with only one release, but Instruments doesn't report any errors?
Am I or is Instruments correct?
Thanks Chris.
You won't get a leak, surprisingly enough. See :
Memory issue of NSString
You should be getting the memory leak you expect.
Instruments' leak detection algorithm is expensive to run, so it is only executed after a specified amount of time (I think it's defaulted to 10 seconds). You may have to let the application run for a while before Instruments picks up the leak.
I'm using Instruments to try to determine if there are places in my application that I could be more efficient with use of memory. I've taken the time to get somewhat familiar with Instruments but I'm generally a newbie with hunting memory management issues having come from a Java background. I seem to be using about 1.82mb by calls to this method:
+ (NSString *)stringFromDateWithFormat:(NSDate *)date withFormat:(NSString *)format
{
NSDateFormatter *dateFormatter;
NSString *result;
if (nil == date || nil == format)
return nil;
result = nil;
if (nil != (dateFormatter = [[NSDateFormatter allocWithZone:[self zone]] init])) {
[dateFormatter setDateFormat:format];
if (nil != (result = [dateFormatter stringFromDate:date])) {
[dateFormatter release];
return result;
}
[dateFormatter release];
}
return nil;
}
As I'm releasing the date formatter I'm wondering if the NSString result is my issue. It seems to me that the stringFromDate library call would return an autoreleased object so there's nothing I can do to 'manually' manage it. A bit unsure of how to optimize this method.
Is this method getting called a lot of times in a loop? Autoreleased objects only get released when the NSAutoreleasePool they're in gets released. As I understand it, the default autorelease pool is created and release every event loop. It's possible you're creating too many autoreleased objects in the course of a single event loop. The solution is to create your own NSAutoreleasePool in an appropriate place, and release it to clear up autoreleased objects. An extreme example that illustrates the point:
int i;
NSAutoreasePool* pool = nil;
for (i = 0; i < 1000000; ++i) {
/* Create a new pool every 10000 iterations */
if ((i % 10000) == 0) {
if (pool) [pool release];
pool = [[NSAutoreleasePool alloc] init];
}
[someObj someMethodThatCreatesAutoreleasedObjects];
}
[pool release];
In that example, the current pool is released every 10,000 iterations and a new one is created. You can read more about memory management in the Memory Management Programming Guide section on autorelease pools.
You need to return an autoreleased object anyway, so there's really nothing you should be doing about the result string. I don't see any mistakes related to memory, but your code is definitely more verbose than needed. Keep in mind that in Objective-C, if you call a method on nil, you get back nil (or 0 for an integer, but not for floating point values). You can take out all those if statements and the two return paths, and your code will still work the same way. Also, I would just use alloc instead of allocWithZone.
I am not a 100% with this. I am also just learning Mac/Iphone development. But you can use the auto release pool to help with memory management. It is used to get around release problems.
Here is an article with lots on memory management. Check out the left sided menu.