I've a video file about 2GB in size. This header of video file is encrypted (approximately 528 bytes encrypted). In order to Decrypt this video file i'm reading all bytes from the file into an NSData object.
As soon as i write this file into NSData object my Application crashes(possibly b'coz max-256MB RAM for iPad).
So how do i go about storing this NSData object into Virtual memory of an iPad/ iPhone temporarily?
Any other approach by means of which i can achieve the same?
Use an NSInputStream to read in the file piece by piece so you aren't loading it all into memory all at once. Specifically you'll want to make use of hasBytesAvailable and read:maxLength:.
Something like:
NSInputStream *myStream = [NSInputStream inputStreamWithFilAtPath:pathToAbsurdlyLargeFile];
[myStream open];
Byte buffer[BUFFER_SIZE];
while ([myStream hasBytesAvailable])
{
int bytesRead = [myStream read:buffer maxLength:BUFFER_SIZE];
NSData *myData = [NSData dataWithBytes:buffer length:bytesRead];
// do other stuff...
}
[myStream close];
Note that you may not need to create an NSData object. You just mentioned you were using it, so I threw it in.
Related
Hello I would like to run a thread and check the current downloaded size of a file.
This is what I use
UIImage *image = [[UIImage alloc] initWithData:[NSData dataWithContentsOfURL:[NSURL URLWithString:#"http://lasp.colorado.edu/home/wp-content/uploads/2011/03/suncombo1.jpg"]]];
NSString *docDir = [NSSearchPathForDirectoriesInDomains(NSDocumentDirectory, NSUserDomainMask, YES) objectAtIndex:0];
NSString *jpegFilePath = [NSString stringWithFormat:#"%#/test.jpeg",docDir];
NSData *data2 = [NSData dataWithData:UIImageJPEGRepresentation(image, 1.0f)];//1.0f = 100% quality
[data2 writeToFile:jpegFilePath atomically:YES];
downloadStatus.text =[NSString stringWithFormat:#"size: %zd", malloc_size(data2)];
[image release];
I have also tried to change malloc_size(data2) into image but again it is not the real result. I know this does not have thread and do not check during the download process but what am I supposed to use here to see the file size?
A couple of observations:
Your question presumed that your attempts to retrieve the size of the NSData were failing. They are not. The correct way to get the size of a NSData is via length.
Your confusion, though, stems from a faulty assumption that taking an externally generated JPEG on a roundtrip through UIImage and UIImageJPEGRepresentation would yield the identical NSData. This would have been extraordinarily unlikely. There are too many different JPG settings that could have changed (see the JPEG Wikipedia page). We certainly don't know what settings that original file used. I know that UIImage and/or UIImageJPEGRepresentation changed the color space of the file. I'd wager it's doing a lot of other things, too.
So your results are correct. The original file was 2.6mb and the resulting file was 4.5mb. If you change the compressionQuality from 1.0 to 0.99, the resulting file is only 1.4mb! But if you want the original file, just save it first (like I do below).
Consider the following code which downloads the image file, saves it, loads it into a UIImage, re-extracts it via UIImageJPEGRepresentation, and saves another copy of the image:
// let's make filenames where we'll store the files
NSString *documentsPath = [NSSearchPathForDirectoriesInDomains(NSDocumentDirectory, NSUserDomainMask, YES) objectAtIndex:0];
NSString *suncomboOrig = [documentsPath stringByAppendingPathExtension:#"suncombo1-orig.jpg"];
NSString *suncomboReprocessed = [documentsPath stringByAppendingPathExtension:#"suncombo1-reprocessed.jpg"];
// let's download the original suncombo1.jpg and save it in Documents and display the size
NSURL *url = [NSURL URLWithString:#"http://lasp.colorado.edu/home/wp-content/uploads/2011/03/suncombo1.jpg"];
NSData *data = [NSData dataWithContentsOfURL:url];
NSLog(#"original = %d", [data length]);
[data writeToFile:suncomboOrig atomically:NO];
// let's load that into a UIImage
UIImage *image = [UIImage imageWithData:data];
// let's extract data out of the image and write that to Documents, too, also logging the size of that
NSData *data2 = UIImageJPEGRepresentation(image, 1.0);
NSLog(#"reprocessed = %d", [data2 length]);
[data2 writeToFile:suncomboReprocessed atomically:NO];
What that does is it reports:
2012-12-13 22:30:39.576 imageapp[90647:c07] original = 2569128
2012-12-13 22:30:40.141 imageapp[90647:c07] reprocessed = 4382876
So the first file I saved (which I suspect is identical to what's on your server) was 2.5mb, and the file after doing a roundtrip to a UIImage and re-extracted via 4.3mb. If I look at the two files that the above code saved, I can confirm that these NSData sizes are correct.
My original answer was predicated on the presumption that the OP was either unable to retrieve the size of a NSData or that there was some subtle issue underlying the simple question (such as wanting to get the size before the download commenced). Anyway, I've expanded my answer above, but I'll keep my original answer for historical purposes:
Original Answer:
The NSData property length tells you how many bytes were downloaded. E.g. [data2 length].
If it's really big, you can use NSURLConnection to download it asynchronously, which, depending upon your web server, may provide total file size before the download commences in the method didReceiveResponse (with the expectedContentLength property in the NSHTTPURLResponse *response parameter).
The other nice thing about NSURLConnection downloading is that you don't have to load the entire file in memory as you're downloading it, but rather you can stream it directly to persistent storage, which is especially useful if you're downloading multiple large files at the same time. If you're downloading a reasonably sized file, using NSURLConnection to download is overkill, but it can be nice when downloading large files and you want a progress indicator (or want to get the file size before the download commences).
But if you just want to know how many bytes were downloaded to your NSData, use length.
You can just use the C FILE class to get the file size.
FILE * handle = fopen([jpegFilePath UTF8String], "r");
fseek(handle, EOF); // seek to end of file
int size = ftell(handle); // returns position in bytes
fclose();
Because you say "current" size and mention a thread, I'm guessing you're trying to determine the file size as it is received. In that case, you can get the thread for free from an NSURLConnection, and you can get the data size from the delegate methods as it's received...
Create an instance variable for the downloaded data:
#property (nonatomic, strong) NSMutableData *data;
Create and launch a connection:
NSURL *url = [NSURL URLWithString:#"http://lasp.colorado.edu/home/wp-content/uploads/2011/03/suncombo1.jpg"];
NSURLRequest *request = [NSURLRequest requestWithURL:url];
self.data = [NSMutableData data];
NSURLConnection *connection = [[NSURLConnection alloc] initWithRequest:request delegate:self];
Implement the required methods in NSURLConnectionDataDelegate. For your question, the special part is this:
- (void)connection:(NSURLConnection *)connection didReceiveData:(NSData *)data {
[self.data appendData:data];
NSUInteger bytesSoFar = [self.data length];
// you're on the app main thread here, so you can do something
// to the UI to indicate progress
downloadStatus.text = [NSString stringWithFormat#"size: %d", bytesSoFar];
}
A good doc on the rest of the protocol is here. When the connection is complete, you can create the image as you did with the dataWithContentsOfURL...
- (void)connectionDidFinishLoading:(NSURLConnection *)connection {
UIImage *image = [[UIImage alloc] initWithData:self.data];
downloadStatus.text = [NSString stringWithFormat#"size: %d", [self.data length]];
}
I'm currently working on an application that has to upload large files (mainly movies/videos) to the web. After reading what I can, I went the the approach of converting the movie to NSData and then including that as the NSURLConnection's HTTPBody. However, upon converting the movie (which was originally an ALAsset) into NSData, I receive a memory warning and then a subsequent crash.
I have no idea how I would go about uploading these types of large files, if that data just causes an instant crash. One solution that I was thinking of is writing to the filesystem and then uploading a file directly from there, but I have not been able to find any information on how one would accomplish this.
Here is the relevant code that I use. If there is something that I'm doing wrong right here, I'd love to know.
ALAssetRepresentation *representation = [asset defaultRepresentation];
Byte *buffer = (Byte *)malloc([representation size]);
NSUInteger buffered = [representation getBytes:buffer fromOffset:0.0 length:[representation size] error:nil];
uploadData = [NSData dataWithBytes:buffer length:buffered];
free(buffer);
Assuming that it makes sense to upload the movie in its native format, you can really make this easier using the BSD (ie Unix) section 3 interface:
given a filePath, open the file and get an int file descriptor (fd)
with fd, get the length of the file
keep track of how much you've loaded so you know where to get more data
use mmap(3) to map in JUST the data you want to upload at any time, and use the void * pointer returned by mmap as the location of the data
when the data has been sent, munmap the old data chunk and mmap a new chunk
after all data is sent, munmap the last chunk, the close(fd).
No temporary memory - no mallocs. I use mmap whenever I have to deal with huge files.
Edit: you can also use NSData dataWithContentsOfFile:options with options set to use mmap. You would then use the byte pointer to read small chunks as you need them.
In case anyone got here and couldn't solve your problems, I figured out a way to do this.
You have to firstly write your ALAssetRepresentation to disk (as described here):
NSUInteger chunkSize = 100 * 1024;
NSString *tempFile = [NSTemporaryDirectory() stringByAppendingPathComponent:#"temp.tmp"];
uint8_t *chunkBuffer = malloc(chunkSize * sizeof(uint8_t));
NSUInteger length = [rep size];
NSFileHandle *fileHandle = [[NSFileHandle fileHandleForWritingAtPath: tempFile] retain];
if(fileHandle == nil) {
[[NSFileManager defaultManager] createFileAtPath:tempFile contents:nil attributes:nil];
fileHandle = [[NSFileHandle fileHandleForWritingAtPath:tempFile] retain];
}
NSUInteger offset = 0;
do {
NSUInteger bytesCopied = [rep getBytes:chunkBuffer fromOffset:offset length:chunkSize error:nil];
offset += bytesCopied;
NSData *data = [[NSData alloc] initWithBytes:chunkBuffer length:bytesCopied];
[fileHandle writeData:data];
[data release];
} while (offset < length);
[fileHandle closeFile];
[fileHandle release];
free(chunkBuffer);
chunkBuffer = NULL;
Then you have to create an NSData object that can map the disk without using memory resources (kind of like David's answer, but inspired by this answer):
NSError *error;
NSData *fileData = [NSData dataWithContentsOfFile:tempFile options:NSDataReadingMappedIfSafe error:&error];
if (!fileData) {
NSLog(#"Error %# %#", error, [error description]);
NSLog(#"%#", tempFile);
//do what you need with the error
}
EDIT Although, if you are uploading the file somewhere, you should open a connection and send small buffers of the file, kind of like what I did above. I had to write a C++ class to handle the socket and the connection
You probably shouldn't be trying to read the whole asset in one shot:
Byte *buffer = (Byte *)malloc([representation size]);
NSUInteger buffered = [representation getBytes:buffer fromOffset:0.0 length:[representation size] error:nil];
Instead, set up a loop and read from the asset in chunks. I've outlined the basic approach. You'll need to fill in a few gaps, but it should solve the memory issue.
You might also want to consider running this in a thread so you don't lock up the UI.
NSError error;
int bufferSize = 1000;
float offset=0.0;
//TODO: Open Connection
while (1)
{
Byte *buffer = (Byte *)malloc(bufferSize);
NSUInteger buffered = [representation getBytes:buffer fromOffset:offset length:bufferSize error:&error];
//TODO: Write data
//TODO: Increment offset, check errors
free(buffer);
//if (done){
//break;
//}
}
//TODO close eonnection
I want to read a large file in objective-c, and used in iphone development
I used the follow code
NSData *data = [[NSData alloc] initWithContentsOfFile:file];
unsigned char *readData[2000];
[data getBytes:readData range:NSMakeRange(1000,3000)]
This maybe get the iphone app crash, because the size of the file is large.
I tried to use NSInputStream, but I can't find the range parameter in any method, can anyone give a help?
Thanks & Best Regards
Timmy
Use NSFileHandle: Seek to the offset where your data starts and start reading from there.
NSFileHandle *fileHandle = [NSFileHandle fileHandleForReadingAtPath:filePath];
[fileHandle seekToFileOffset:1000];
NSData *data = [fileHandle readDataOfLength:2000];
i has the following code which suppose to convert file to data but the problem is the byte size is not the size after the conversion can someone help me on this one? i'm suppose to send the NSdata using NSStream to a server.
NSString *file = [[NSString alloc] initWithContentsOfFile:filepath;
NSLog(#"length = %d",[file length]); // file length is 248563 (Actual file size);
NSdata *data = [file dataUsingEncoding:NSASCIIStringEncoding allowLossyConversion:Yes];
NSLog(#"length = %d",[datalength]);// file length is 249199 (increase in file size);
Thanks in advance.
Why load it into a string at all in the first place?
[NSData dataWithContentsOfFile:]
The problem with your code is that the file probably wasn't ASCII originally, so its reencoding things.
I am reading a very large file using a NSInputStream and sending them to a device in packets. If the receiver does not get a packet, I can send back to the sender with a packet number, representing the starting location in bytes of the packet missing.
I know an NSInputStream cannot rewind and grab the packet, but is there another way to grab the requested byte range without loading the entire large file into memory?
If there was a [NSData dataWithContentsOfFileAtPath:inRange] method, it would be perfect.
I don't think there's a standard function that does that, but you could write one yourself, using a category and the C stdio API:
#interface NSData(DataWithContentsOfFileAtOffsetWithSize)
+ (NSData *) dataWithContentsOfFile:(NSString *)path atOffset:(off_t)offset withSize:(size_t)bytes;
#end
#implementation NSData(DataWithContentsOfFileAtOffsetWithSize)
+ (NSData *) dataWithContentsOfFile:(NSString *)path atOffset:(off_t)offset withSize:(size_t)bytes
{
FILE *file = fopen([path UTF8String], "rb");
if(file == NULL)
return nil;
void *data = malloc(bytes); // check for NULL!
fseeko(file, offset, SEEK_SET);
fread(data, 1, bytes, file); // check return value, in case read was short!
fclose(file);
// NSData takes ownership and will call free(data) when it's released
return [NSData dataWithBytesNoCopy:data length:bytes];
}
#end
Then you can this:
// Read 100 bytes of data beginning at offset 500 from "somefile"
NSData *data = [NSData dataWithContentsOfFile:#"somefile" atOffset:500 withSize:100];
You can rewind with NSInputStream:
[stream setProperty:[NSNumber numberWithInt:offset]
forKey:NSStreamFileCurrentOffsetKey];