My question is about core data and memory usage. I have used core data before, but this time the amount of data is higher and this made me realise that there was much more to know. I have seen that there are several other similar posts and I got interesting information from them, but after applying it my apps still crashes. I have been dealing with this issue for a week now. Somebody please help.
Basically I have three subsequent similar loops of 64, 15, and 17 iterations respectively. They work fine on simulator. Tested on a couple of iPads they get memory warnings and they crash at the same iteration (number 34 of the first loop). Tested on iPad 2 it will crash at number 14 of the second loop. Instruments shows a memory usage of about 1.5 MB both live and overall. There are leaks for a few KB.
The loops perform the following (code below)
Execute a fetch with core data
For every record take a parameter stored as a row property attribute (String)
Call a procedure which takes that parameter and which returns data (about hundreds of KB)
Store these data in another property attribute (Transformable) of the same row
Pretty common task isn't it?
Now, since I got into memory issues, I tried to use the all the known (by me) tools at my disposal, which are:
release owned objects asap
create autorelease pools and drain them asap for not owned objects
save context asap
turn objects into faults asap
After applying all these techniques I got an exciting result: the app crashes at the exactly same point as before.
Here it is the code.
- (void) myMainProcedure {
[self performLoop1];
[self performLoop2]; // Similar to loop1
[self performLoop3]; // Similar to loop1
}
- (void) performLoop1 {
NSError * error = nil;
NSAutoreleasePool * myOuterPool;
NSAutoreleasePool * myInnerPool;
NSManagedObjectContext * applicationContext = [[[UIApplication sharedApplication] delegate] managedObjectContext];
[applicationContext setUndoManager:nil];
NSEntityDescription * myEntityDescription = [NSEntityDescription entityForName:#"EntityName"
inManagedObjectContext:applicationContext];
NSFetchRequest * myFetchRequest = [[NSFetchRequest alloc] init];
[myFetchRequest setEntity:myEntityDescription];
NSString * column = #"columnName";
NSPredicate * aWhereClause = [NSPredicate predicateWithFormat:
#"(%K = %#)", column, [NSNumber numberWithInt:0]];
[myFetchRequest setPredicate: aWhereClause];
myOuterPool = [[NSAutoreleasePool alloc] init];
NSArray * myRowsArray = [applicationContext executeFetchRequest:myFetchRequest
error:&error];
NSMutableArray * myRowsMutableArray = [[NSMutableArray alloc] initWithCapacity:0];
[myRowsMutableArray addObjectsFromArray: myRowsArray];
[myOuterPool drain];
[myFetchRequest release];
EntityName * myEntityRow;
int totalNumberOfRows = [myRowsMutableArray count];
myOuterPool = [[NSAutoreleasePool alloc] init];
for (int i = 0; i < totalNumberOfRows; i++) {
myInnerPool = [[NSAutoreleasePool alloc] init];
myEntityRow = [myRowsMutableArray objectAtIndex:0];
NSString * storedSmallAttribute = myEntityRow.smallAttribute;
UIImageView * largeData = [self myMethodUsingParameter: smallAttribute];
myEntityRow.largeAttribute = largeData;
[myRowsMutableArray removeObjectAtIndex:0];
[applicationContext save:&error];
[applicationContext refreshObject:myEntityRow mergeChanges:NO];
[myInnerPool drain];
[largeData release];
}
[myOuterPool drain];
[myRowsMutableArray release];
}
- (UIImageView *) myMethodUsingParameter : (NSString *) link {
UIImageView * toBeReturned = nil;
NSURL *pdfURL = [NSURL fileURLWithPath:link];
CGPDFDocumentRef pdf = CGPDFDocumentCreateWithURL((CFURLRef)pdfURL);
CGPDFPageRef page = CGPDFDocumentGetPage(pdf, 1);
CGRect pageRect = CGPDFPageGetBoxRect(page, kCGPDFMediaBox);
UIGraphicsBeginImageContext(pageRect.size);
CGContextRef context = UIGraphicsGetCurrentContext();
CGContextSetRGBFillColor(context, 1.0,1.0,1.0,1.0);
CGContextFillRect(context,pageRect);
CGContextSaveGState(context);
CGContextTranslateCTM(context, 0.0, pageRect.size.height);
CGContextScaleCTM(context, 1, - 1);
CGContextSetInterpolationQuality(context, kCGInterpolationHigh);
CGContextSetRenderingIntent(context, kCGRenderingIntentDefault);
CGContextDrawPDFPage(context, page);
CGContextRestoreGState(context);
UIImage *imageToBeReturned = UIGraphicsGetImageFromCurrentImageContext();
UIGraphicsEndImageContext();
CFRelease(pdf);
toBeReturned = [[UIImageView alloc] initWithImage:imageToBeReturned];
UIGraphicsEndImageContext();
return toBeReturned;
}
Please note that
The mutable array was introduced as a (apparently useless) strategy to
have objects released sooner
Pools have been added ad part of the same strategy
The statement about interpolation quality was the only one to improve
the situation (say, to move the crash
a little bit forward)
Retain count for managed objects within the cycle ranges from 6 to 10
(?) I know that rc is not a valuable
information but still, I made a test
and I found out that I could send
multiple release messages to managed
objects before forcing the app to
crash for this. But the point is that
I am not supposed to release an
object I don't own, am I? ....
The entity upon which the request is set has got also some
bidirectional relationships with
other entities, but still, is this
relevant?
Thank you.
I think you should get a fresh look at your problem.
There are many ways to deal with Core Data that are much simpler than your approach.
Creating class files for your Core Data entities may help you.
Also, when you are saving your files, you should seriously evaluate each objects necessity and whether it can be done a better way.
In your case, I would offer two suggestions:
For each PDF URL,
a.Assign a unique identifier.
b.Save this unique identifier in your
core data store.
c.Add url to a queue for a
background process that creates
your PDF (background process will
allow the user to keep working while
the PDFs are being generated. You
could update your delegate on
progress or create a temporary image
that is replaced when the PDF is
created.)
d.Save the image in your app
directory (or photo library) using
the unique identifier as a name.
e.When needed, load the image from
disk into a UIImageView or
whatever appropriate.
For each PDF URL,
a.Draw the PDF.
b.Get UIImage representation
c.Convert to PNG NSData
(UIImagePNGRepresentation(image))
d.Save NSData in CoreData.
e.Load NSData and convert to UIImage
when needed.
Related
I have A LOT of rows that need to be inserted into a database. I download about 50 .txt files that are tab delimited with the information I need. I parse the data and insert it into the database in the same method. When I had it as separate methods, the simulator on my 27" iMac would crash because it ran out of memory. The method looks like this:
- (BOOL)insertDataWithFileName:(NSString *)fileName {
NSLog(#"%s %#", __FUNCTION__, fileName);
if (_db == nil) {
return NO;
}
// Get the data from the file
NSData *data = [[NSData alloc] initWithContentsOfFile:[[self applicationDocumentsDirectory] stringByAppendingPathComponent:fileName]];
NSString *responseString = [[NSString alloc] initWithData:data encoding:NSUTF8StringEncoding];
// Parse each line by newlines first
NSArray *lines = [responseString componentsSeparatedByString:#"\n"];
NSString *ID = [[fileName componentsSeparatedByString:#"."] objectAtIndex:0];
// Set up the database insert statement
NSString *tableName = #"Hardcodedtablename"; // remove after prototype
BOOL oldshouldcachestatements = _db.shouldCacheStatements;
[_db setShouldCacheStatements:YES];
[_db beginTransaction];
NSString *insertQuery = [NSString stringWithFormat:#"INSERT INTO %# values(null, ?, ?, ?, ?, ?, ?, ?);", tableName];
BOOL success;
// Parse each line by tabs
for (NSString *line in lines) {
NSAutoreleasePool *pool = [[NSAutoreleasePool alloc] init];
NSArray *fields = [line componentsSeparatedByString:#"\t"];
// Need to check since some of the .txt files have an empty newline at the EOF.
if ([fields count] == NUM_VARIANT_FIELDS) {
NSNumber *loc = [NSNumber numberWithInteger:[[fields objectAtIndex:0] integerValue]];
NSString *carType = [fields objectAtIndex:1];
NSString *model = [fields objectAtIndex:2];
NSString *AA = [fields objectAtIndex:7];
NSString *BB = [fields objectAtIndex:8];
NSNumber *freq = [[fields objectAtIndex:11] isEqualToString:#""] ? [NSNumber numberWithDouble:-1.0] : [NSNumber numberWithDouble:[[fields objectAtIndex:11] doubleValue]];
NSArray *argArray = [[NSArray alloc] initWithObjects:ID, loc, carType, model, AA, BB, freq, nil];
success = [_db executeUpdate:insertQuery withArgumentsInArray:argArray];
[argArray release];
}
[pool drain];
}
[_patientAnnotatedDatabase commit];
[_patientAnnotatedDatabase setShouldCacheStatements:oldshouldcachestatements];
return success;
}
As you can see from my for loop, I actually don't need all the files in the array after it gets separated by the \t. I set up an autorelease pool to reduce the memory footprint in the for loop. The largest file has 270,000 rows in it. There are about 50 of these files. So after 2 files on the device, it crashes. No error message though.
I was wondering what my options were at this point. Can the iPad actually handle that amount of data? Can I make optimizations on this method?
Or would I have to create a database on the server side and download the database to the device instead? If I did download the database from the server to the device, does FMDB support an ability to just transfer those tables from one DB to another? I wasn't able to find anything like that while I was googling. Thanks!
It really depends on what you are using the data for. If you are using it to search or for some purpose where you will really need ALL of it, you are basically using this much memory:
270,000 rows * 25 chars/row * 50 files * 1 byte/char = 321 Mb of RAM
I'm guessing about the 25 chars/row part, but when you are using over 100 Mb of memory, you are using a lot of memory. However, the iPad does have about 1GB of RAM for you to use, so you would seem to be within the amount of available memory. But, because of the NSArrays and SQLite FMDB stuff you are using to store it (as well as other NSObjects), the data with encapsulation could amount to over 1GB, when added to other processes running on the iPad.
So, all of this memory might be your problem. You might be better of saving all of this data on a webserver that can handle it, and call whatever functions you would like through said server. It probably would be much faster, and it would allow you to make adjustments to the data easily, if it needed to be updated. It also would allow your app to be used on devices like the iPhone that do not have enough RAM.
If sheer amount of data is your problem, then your solution is the one above. However, I still believe that the iPad is capable of handling 300+ Mb of RAM, so your problem could lie in the code. Make sure you are accessing all of your data correctly, especially when you are separating the data into arrays from the files, because the smallest "out of bounds" array error could cause your program to fail.
Have you also tried running your code with the memory profiler enabled. You can enable this app in the new Xcode by holding down the "Run" round button until you can select an option that says "Profiler." This app will show you how much memory your app is using, so you can make sure its under 1GB.
Unfortunately, I had NSZombieEnabled set on in my scheme to track down an earlier bug. With it turned off, it seems to be now not crashing at this moment.
iPad application
I have a large text file which I want to divide into several pieces and process them in a loop.
I used the following code:
NSArray * contentArray = [stringFromFile componentsSeparatedByString:#" "];
for(...){
contentRange = NSMakeRange(lastPosition , newLength);
NSArray * subContentArray = [contentArray subarrayWithRange:contentRange];
NSString *subContent = [subContentArray componentsJoinedByString:#" "];
....
//Process sub content here...
}
After running, I receive malloc error, code 12
Observing the Activity monitor, the memory and the VM size increases and until system memory is exhausted and the application crashes.
Is there any way to prevent this?
One way to work around this issue will be to use a custom NSAutoreleasePool to constantly clear temporarily allocated memory. Like so:
NSAutoreleasePool *pool = [[NSAutoreleasePool alloc] init];
NSUInteger i=0;
for (...) {
... // your method contents
if (++i % 10 == 0) {
// you might want to play with the frequency of the releases, depending on the size of your loop
[pool release];
pool = [[NSAutoreleasePool alloc] init];
}
}
[pool release];
The autorelease pool thing will take care of the memory problem for the most part, but if the file is any size it might be slow.
It seems like scanning the original string in a loop getting substrings directly from the string would be faster. Also use at least 1/2 the memory, since you duplicate the string in the components array.
The code would use rangeOfString on the original string, until a substring of appropriate length was found, then process the substring, and onto the next one.
iPhone 3G phones only have about 5 - 15 MB of memory total for your application. You are on an iPad, though I see which does give you about 4x that amount.
I am changing the image in UIImageView based on accelerometer input. The images are stored in an array.
The application runs fine for a while and then crashes.
warning: check_safe_call: could not restore current frame
I am not using "UIImage ImageNamed" method when populating the array.
The total size of all images is around 12 Mb. But individual images are very light (<100 kb) and i am using only one image at a time from the array.
Just in case there are any autoreleases, I have allocated a NSAutoreleasePool in view did load and am draining it in the didReceiveMemoryWarning method (which does get called 2, 3 times before the app crashes?).
Following is the code that creates images array:
//create autorelease pool as we are creating many autoreleased objects
NSAutoreleasePool *pool = [[NSAutoreleasePool alloc] init];
NSMutableArray *finalarr = [[NSMutableArray alloc] initWithCapacity:9];
NSLog(#"start loading");
for(int y = 0; y < 100; y+=10)
{
NSMutableArray *arr = [[NSMutableArray alloc] initWithCapacity:10];
for(int x = 0; x < 10; x++)
{
NSString *imageName = [[NSString alloc] initWithFormat:#"0%d",y + x];
UIImage *img = [[UIImage alloc] initWithContentsOfFile:[[NSBundle mainBundle] pathForResource:imageName ofType:#"png"]];
[arr addObject:img];
[imageName release];
[img release];
}
[finalarr addObject:arr];
[arr release];
}
NSLog(#"done loading");
// Override point for customization after app launch
viewController.imagesArray = finalarr;
[finalarr release];
//retain the array of images
[viewController.imagesArray retain];
//release the aurtorelease pool to free memory taken up by objects enqued for release
[pool release];
As the app runs smoothly for a while, which means array creation is definitely not the problem.
After this the following method is called from [accelerometer didAccelerate]
-(BOOL)setImageForX:(int)x andY:(int)y
{
[self.imageView setImage:(UIImage*)[(NSArray*)[self.imagesArray objectAtIndex:y] objectAtIndex:x]];
return TRUE;
}
So the only code being executed is the "UIImageView setImage". But no objects are being created here.
Please let me know if the code seems to have any leaks or i am doing something wrong.
Thanks,
Swapnil
NSAutoreleasePool should be allocated and released in the same method. Allocating it in one place and releasing it in another is usually very bad (the example code you pasted uses it correctly however)
Loading images into memory will decompress them; a 100KB compressed PNG could easily be 2MB uncompressed.
You also seem to have an extra retain call that shouldn't be there: [viewController.imagesArray retain]; (I assume the imagesArray property is marked copy or retain and not assign)
Once again I'm hunting memory leaks and other crazy mistakes in my code. :)
I have a cache with frequently used files (images, data records etc. with a TTL of about
one week and a size limited cache (100MB)). There are sometimes more then 15000 files in a
directory. On application exit the cache writes an control file with the current cache
size along with other useful information. If the applications crashes for some reason
(sh.. happens sometimes) I have in such case to calculate the size of all files on application start to make sure I know the cache size.
My app crashes at this point because of low memory and I have no clue why.
Memory leak detector does not show any leaks at all. I do not see any too. What's wrong
with the code below? Is there any other fast way to calculate the total size of all files
within a directory on iPhone? Maybe without to enumerate the whole contents of the directory?
The code is executed on the main thread.
NSUInteger result = 0;
NSAutoreleasePool *pool = [[NSAutoreleasePool alloc] init];
NSDirectoryEnumerator *dirEnum = [[[NSFileManager defaultManager] enumeratorAtPath:path] retain];
int i = 0;
while ([dirEnum nextObject]) {
NSDictionary *attributes = [dirEnum fileAttributes];
NSNumber* fileSize = [attributes objectForKey:NSFileSize];
result += [fileSize unsignedIntValue];
if (++i % 500 == 0) { // I tried lower values too
[pool drain];
}
}
[dirEnum release];
dirEnum = nil;
[pool release];
pool = nil;
Thanks,
MacTouch
Draining the pool "releases" it, it doesn't just empty it. Think of autorelease pools as stacks, so you have popped yours, meaning that all these new objects are going into the main autorelease pool and not being cleaned up until it gets popped. Instead, move the creation of your autorelease pool to inside the loop. You can do something like
NSAutoreleasePool* pool = [[NSAutoreleasePool alloc] init];
int i = 0;
while( shouldloop ) {
// do stuff
if( ++i%500 == 0 ) {
[pool drain];
pool = [[NSAutoreleasePool alloc] init];
}
}
[pool drain];
I am saving arrays of doubles in an NSData* object that is persisted as a binary property in a Core Data (SQLite) data model. I am doing this to store sampled data for graphing in an iPhone app. Sometimes when there are more than 300 doubles in the binary object not all the doubles are getting saved to disk. When I quit and relaunch my app there may be as few as 25 data points that have persisted or as many as 300.
Using NSSQLitePragmasOption with synchronous = FULL and this may be making a difference. It is hard to tell, as bug is intermittent.
Given the warnings about performance problems as a result of using synchronous = FULL, I am seeking advice and pointers.
Thanks.
[[Edit: here is code.]]
The (as yet unrealized) intent of -addToCache: is to add each new datum to the cache but only flush (fault?) Data object periodically.
From Data.m
#dynamic dataSet; // NSData * attribute of Data entity
- (void) addDatum:(double_t)datum
{
DLog(#"-[Data addDatum:%f]", datum);
[self addToCache:datum];
}
- (void) addToCache:(double_t)datum
{
if (cache == nil)
{
cache = [NSMutableData dataWithData:[self dataSet]];
[cache retain];
}
[cache appendBytes:&datum length:sizeof(double_t)];
DLog(#"-[Data addToCache:%f] ... [cache length] = %d; cache = %p", datum, [cache length], cache);
[self flushCache];
}
- (void) wrapup
{
DLog(#"-[Data wrapup]");
[self flushCache];
[cache release];
cache = nil;
DLog(#"[self isFault] = %#", [self isFault] ? #"YES" : #"NO"); // [self isFault] is always NO.
}
- (void) flushCache
{
DLog(#"flushing cache to store");
[self setDataSet:cache];
DLog(#"-[Data flushCache:] [[self dataSet] length] = %d", [[self dataSet] length]);
}
- (double*) bytes
{
return (double*)[[self dataSet] bytes];
}
- (NSInteger) count
{
return [[self dataSet] length]/sizeof(double);
}
- (void) dump
{
ALog(#"Dump Data");
NSInteger numDataPoints = [self count];
double *data = (double*)[self bytes];
ALog(#"numDataPoints = %d", numDataPoints);
for (int i = 0; i
I was trying to get behavior as if my Core Data entity could have an NSMutableData attribute. To do this my NSManagedObject (called Data) had an NSData attribute and an NSMutableData ivar. My app takes sample data from a sensor and appends each data point to the data set - this is why I needed this design.
On each new data point was appended to the NSMutableData and then the NSData attribute was set to the NSMutableData.
I suspect that because the NSData pointer wasn't changing (though its content was), that Core Data did not appreciate the amount of change. Calling -hasChanged on the NSManagedObjectContext showed that there had been changes, and calling -updatedObjects even listed the Data object as having changed. But the actual data that was being written seems to have been truncated (sometimes).
To work around this I changed things slightly. New data points are still appended to NSMutableData but NSData attribute is only set when sampling is completed. This means that there is a chance that a crash might result in truncated data - but for the most part this work around seems to have solved the problem.
Caveat emptor: the bug was always intermittent, so it is possible that is still there - but just harder to manifest.