iOS Huge JSON (30MB) handling - iphone

Question

I need to download and parse 40mb of json. Right now I'm using AFJSONRequestOperation, which on older devices causes memory warning and crash. How it should be done? I think the only way to do it correctly, is to stream json but I've got no idea how to do it or which library will be best. Please provide examples. Thanks a lot!

Answer 1

For anybody who has the same problem, here's how I solved it:
1. Download JSON file to local storage using AFHTTPRequestOperation's output stream.
2. Parse little chunks of NSData using YAJLParser.
Result: I was testing it on 50mb json on iPad (1), without any memory warnings (memory around 10mb).
Example:
NSError *error = nil;
NSData *data = [NSData dataWithContentsOfFile:path
options:NSDataReadingMappedAlways | NSDataReadingUncached
error:&error];
YAJLParser *parser = [[YAJLParser alloc] initWithParserOptions:YAJLParserOptionsAllowComments];
parser.delegate = self;
[parser parse:data];
parser.delegate = nil;
parser = nil;
YAJLParser delegate:
// first declare in header file NSMutableArray *stack and NSString *mapKey
- (void)parserDidStartDictionary:(YAJLParser *)parser
{
NSString *dictName = mapKey;
if (mapKey == nil)
{
dictName = (stack.count == 0) ? #"" : [stack lastObject];
}
[stack addObject:(dictName)];
}
- (void)parserDidEndDictionary:(YAJLParser *)parser
{
mapKey = nil;
[stack removeLastObject];
}
- (void)parserDidStartArray:(YAJLParser *)parser
{
NSString *arrayName = mapKey;
if (mapKey == nil)
{
arrayName = stack.count == 0 ? #"" : [stack lastObject];
}
[stack addObject:(arrayName)];
if([mapKey isEqualToString:#"something"])
{
// do something
}
}
- (void)parserDidEndArray:(YAJLParser *)parser
{
if([mapKey isEqualToString:#"some1"])
{
// do something
}
mapKey = nil;
[stack removeLastObject];
}
- (void)parser:(YAJLParser *)parser didMapKey:(NSString *)key
{
mapKey = key;
}
- (void)parser:(YAJLParser *)parser didAdd:(id)value
{
if([mapKey isEqualToString:#"id"])
{
// do something
}
}

Answer 2

Write your data to a file, then use NSData's dataWithContentsOfFile:options:error: and specify the NSDataReadingMappedAlways and NSDataReadingUncached flags. This will tell the system to use mmap() to reduce the memory footprint, and not to burden the file system cache with blocks of memory (that makes it slower, but much less of a burden to iOS).
You can find the answer here
iPad - Parsing an extremely huge json - File (between 50 and 100 mb)