Streaming JPEGs, detect end of JPEG - iphone

I have created a java server, which takes screenshots, resizes them, and sends them over TCP/IP to my iPhone application. The application then uses NSInputStream to collect the incoming image data, create an NSMutableData instance with the byte buffer, and then create a UIImage object to display on the iPhone. Screenshare, essentially. My iPhone code to collect the image data is currently as follow:
- (void)stream:(NSStream *)theStream handleEvent:(NSStreamEvent)streamEvent{
if(streamEvent == NSStreamEventHasBytesAvailable && [iStream hasBytesAvailable]){
uint8_t buffer[1024];
while([iStream hasBytesAvailable]){
NSLog(#"New Data");
int len = [iStream read:buffer maxLength:sizeof(buffer)];
[imgdata appendBytes:buffer length:len];
fullen=fullen+len;
/*Here is where the problem lies. What should be in this
if statement in order to make it test the last byte of
the incoming buffer, to tell if it is the End of Image marker
for the end of incoming JPEG file?
*/
if(buffer[len]=='FFD9'){
UIImage *img = [[UIImage alloc] initWithData:imgdata];
NSLog(#"NUMBER OF BYTES: %u", len);
image.image = img;
}
}
}
}
My problem, as indicated by the in-code comment, is figuring out when to stop collecting data in the NSMutableData object, and use the data to create a UIImage. It seems to make sense to look for the JPEG End of File marker--End of Image (EOI) marker (FFD9)--in the incoming bytes, as the image will be ready for display when this is sent. How can I test for this? I'm either missing something about how the data is stored, or about the marker within the JPEG file, but any help in testing for this would be greatly appreciated!
James

You obviously don't want to close the stream because that would kill performance.
Since you control the client server connection, send down the # of bytes in the image before sending the image data. Better yet, send down # of bytes in the image, the image data, and an easily identified serial # at the end so you can quickly verify that the data has actually arrived.
Much easier and more efficient than actually checking for the end of file marker. Though, of course, you could also just check for that after the # of bytes have been received, too. Easy enough.
Of course, all of this is going to be grossly inefficient for screensharing style purposes in all but the unusual cases. In most cases, only a small part of the screen to be mirrored actually changes with each frame. If you try to send the whole screen with every frame, you'll quickly saturate your connection and the client side will be horribly laggy and unresponsive.
Given that this is an extremely mature market, there are tons of solutions and quite a few open source bits from which you can derive a solution to fit your needs (see VNC, for example).

Related

Using GSocketClient, how do I read incoming data without knowing how many incoming bytes there will be?

I am still struggling to be able to read incoming response messages from a piece of hardware my program is communicating with.
I am using a GSocketClient and am able to connect and successfully send messages using g_output_stream_write(). I then want to read the response sent back from the device, but I have no way of knowing how many bytes the reply will be in order to use g_input_stream_read(). I have also tried using g_input_stream_read_all(), but this seems to block the application and never return. I don't know how g_input_stream_read_all() determines that it has reached the end of a stream, but I assume the problem is somewhere there?
I know that there is incoming data because I can use g_input_stream_read() with a made-up byte size like 5 and I then see the first 5 incoming bytes, but the response size will always be different.
So my questions is, is there a way to determine how much data is waiting to be read so that I can plug that into g_input_stream_read() as a variable for the size to read? And if not, what is the correct usage of g_input_stream_read_all() to get it to not block like I am seeing it do?
Does something like the following work?
#define BUF_SIZE 1024
guint8 buffer[BUF_SIZE];
GByteArray *array = g_byte_array_new();
gsize bytes_read;
GError *error = NULL;
while (g_input_stream_read_all(istream, buffer, BUF_SIZE, &bytes_read, NULL, &error))
{
g_byte_array_append(array, buffer, bytes_read);
if (bytes_read < BUF_SIZE)
/* We've reached the end of the stream */
break;
}
if (error)
// error handling code

JPEG encoder super slow, how to Optimize it?

I'm building an App with actionscript 3.0 in my Flash builder. This is a followup question this question.
I need to upload the bytearray to my server, but the function i use to convert the bitmapdata to a ByteArray is super slow, so slow it freezes up my mobile device. My code is as follows:
var jpgenc:JPEGEncoder = new JPEGEncoder(50);
trace('encode');
//encode the bitmapdata object and keep the encoded ByteArray
var imgByteArray:ByteArray = jpgenc.encode(bitmap);
temp2 = File.applicationStorageDirectory.resolvePath("snapshot.jpg");
var fs:FileStream = new FileStream();
trace('fs');
try{
//open file in write mode
fs.open(temp2,FileMode.WRITE);
//write bytes from the byte array
fs.writeBytes(imgByteArray);
//close the file
fs.close();
}catch(e:Error){
Is there a different way to convert it to a byteArray? Is there a better way?
Try to use blooddy library: http://www.blooddy.by . But i didn't test it on mobile devices. Comment if you will have success.
Use BitmapData.encode(), it's faster by orders of magnitude on mobile http://help.adobe.com/en_US/FlashPlatform/reference/actionscript/3/flash/display/BitmapData.html#encode%28%29
You should try to find a JPEG encoder that is capable of encoding asynchronously. That way the app can still be used while the image is being compressed. I haven't tried any of the libraries, but this one looks promising:
http://segfaultlabs.com/devlogs/alchemy-asynchronous-jpeg-encoding-2
It uses Alchemy, which should make it faster than the JPEGEncoder from as3corelib (which I guess is the one you're using at the moment.)
A native JPEG encoder is ideal, asynchronous would be good, but possibly still slow (just not blocking). Another option:
var pixels:ByteArray = bitmapData.getPixels(bitmapData.rect);
pixels.compress();
I'm not sure of native performance, and performance definitely depends on what kind of images you have.
The answer from Ilya was what did it for me. I downloaded the library and there is an example of how to use it inside. I have been working on getting the CameraUI in flashbuilder to take a picture, encode / compress it, then send it over via a web service to my server (the data was sent as a compressed byte array). I did this:
by.blooddy.crypto.image.JPEGEncoder.encode( bmp, 30 );
Where bmp is my bitmap data. The encode took under 3 seconds and was easily able to fit into my flow of control synchronously. I tried async methods but they ultimately took a really long time and were difficult to track for things like when a user moved from cell service to wifi or from tower to tower while an upload was going on.
Comment here if you need more details.

Streaming with MKNetworkKit on iOS

I'm trying to figure out how to make MKNetworkKit working with data from stream. I can see that some data is beeing downloaded (the indicator on status bar), but I don't have any idea what happens with that data after it's actually downloaded. I put a NSLog statement inside body of connection: didReceiveData: but it's not called during streaming. Any pointers how to fix that issue ?
Edit
Sorry my question was inaccurate. I know how to stream to a file but I need to stream to memory (NSData instance preferably). Okay it seems simple again due to NSOutputStream method initWithBytes:capacity:. And my problem is here, my stream has undefined length so there would be enormous impact on memory. I don't know what to do. My perfect solution works like this. Small chunks of data from the stream are processed having been downloaded and then they are discarded.
You could use the outputStreamToBuffer:capacity: method to create the stream.
As for the buffer, you can use a circular buffer, so that you can read from it as the stream writes to it. A great implementation (and explanation) is here.
Streaming a file download is a three line magic with MKNetworkKit.
//Create a MKNetworkOperation for the remote URL.
MKNetworkOperation *op = [self operationWithURLString:remoteURL
params:nil
httpMethod:#"GET"];
// add your output stream, in this case a file
[op addDownloadStream:[NSOutputStream outputStreamToFileAtPath:filePath
append:YES]];
// enqueue the operation to a MKNetworkEngine.
[self enqueueOperation:op];
That's it.

Get amount of memory used by app in iOS

I'm working on an upload app that splits files before upload. It splits the files to prevent being closed by iOS for using too much memory as some of the files can be rather large. It would be great if I could, instead of setting the max "chunk" size, set the max memory usage and determine the size using that.
Something like this
#define MAX_MEM_USAGE 20000000 //20MB
#define MIN_CHUNK_SIZE 5000 //5KB
-(void)uploadAsset:(ALAsset*)asset
{
long totalBytesRead = 0;
ALAssetRepresentation *representation = [asset defaultRepresentation];
while(totalBytesRead < [representation size])
{
long chunkSize = MAX_MEM_USAGE - [self getCurrentMemUsage];
chunkSize = min([representation size] - totalBytesRead,max(chunkSize,MIN_CHUNK_SIZE));//if I can't get 5KB without getting killed then I'm going to get killed
uint8_t *buffer = malloc(chunkSize);
//read file chunk in here, adding the result to totalBytesRead
//upload chunk here
}
}
Is essentially what I'm going for. I can't seem to find a way to get the current memory usage of my app specifically. I don't really care about the amount of system memory left.
The only way I've been able to think of is one I don't like much. Grab the amount of system memory on the first line of main in my app, then store it in a static variable in a globals class then the getCurrentMemUsage would go something like this
-(long)getCurrentMemUsage
{
long sysUsage = [self getSystemMemoryUsed];
return sysUsage - [Globals origSysUsage];
}
This has some serious drawbacks. The most obvious one to me is that another app might get killed in the middle of my upload, which could drop sysUsage lower than origSysUsage resulting in a negative number even if my app is using 10MB of memory which could result in my app using 40MB for a request rather than the maximum which is 20MB. I could always set it up so it clamps the value between MIN_CHUNK_SIZE and MAX_MEM_USAGE, but that would just be a workaround instead of an actual solution.
If there are any suggestions as to getting the amount of memory used by an app or even different methods for managing a dynamic chunk size I would appreciate either.
Now, as with any virtual memory operating system, getting the "memory used" is not very well defined and is notoriously difficult to define and calculate.
Fortunately, thanks to the virtual memory manager, your problem can be solved quite easily: the mmap() C function. Basically, it allows your app to virtually load the file into memory, treating it as if it were in RAM, but it is actually swapped in from storage as it is accessed, and swapped out when iOS is low on memory.
This function is really easy to use in iOS with the Cocoa APIs for it:
- (void) uploadMyFile:(NSString*)fileName {
NSData* fileData = [NSData dataWithContentsOfMappedFile:fileName];
// Work with the data as with any NSData* object. The iOS kernel
// will take care of loading the file as needed.
}

CocoaAsyncSocket and reading data from a socket

On my TCP-socket based server, I send a packets over the stream where packets consist of a header specifying the number of bytes in the packet, followed by that number of bytes. For those familiar with Erlang, I'm simply setting the {packet, 4} option. On the iOS side, I have code that looks like this, assuming I want to figure out the size of the stream for this message:
[asyncSocket readDataToLength:4 withTimeout:-1 tag:HEADER_TAG];
That works fine and the following delegate method callback is invoked:
onSocket:didReadData:withTag:
I figure the next logical step is to figure out the size of the stream, and I do that with:
UInt32 readLength;
[data getBytes:&readLength length:4];
readLength = ntohl(readLength);
After hard coding a string of 12 bytes on the server-side, readLength does indeed read 12 on the client also, so all is good so far. I proceed with the following:
[sock readDataToLength:readLength withTimeout:1 tag:MESSAGE_TAG];
At this point though the callback onSocket:didReadData:withTag: is no longer invoked. Instead timeouts on the read are occurring, probably because I didn't handle the read properly, this delegate method gets invoked:
- (NSTimeInterval)onSocket:(AsyncSocket *)sock shouldTimeoutReadWithTag:(long)tag elapsed:(NSTimeInterval)elapsed bytesDone:(NSUInteger)length
so in total, the server is sending 16 bytes, a 4 byte header and a 12 byte binary stream.
I'm confident that the error is on how I'm using CocoaAsyncSocket. What's the right way to go about reading the rest of the stream after I figure out its size?
** UPDATE **
I changed my client and it seems to be working now. The problem is, I don't understand the point of readDataToLength with the new solution. Here's what I changed my initial read to:
[socket readDataWithTimeout:-1 tag:HEADER_TAG];
Now in my callback, I just do the following:
- (void)onSocket:(AsyncSocket *)sock didReadData:(NSData *)data withTag:(long)tag {
if (tag == HEADER_TAG) {
UInt32 readLength;
[data getBytes:&readLength length:4];
readLength = ntohl(readLength);
int offset = 4;
NSRange range = NSMakeRange(offset, readLength);
char buffer[readLength];
[data getBytes:&buffer range:range];
NSLog(#"buffer %s", buffer);
//[sock readDataToLength:readLength withTimeout:1 tag:MESSAGE_TAG];
} else if (tag == MESSAGE_TAG) {
//[sock readDataToLength:4 withTimeout:1 tag:HEADER_TAG];
}
}
So everything is coming in as one, atomic payload. Perhaps this is because of the way Erlang {packet, 4} works. I hope it is. Otherwise, what's the point of readDataToLength? there's no way to know the length of a message in advance on the client, so what is a good use case to use that method in?
It depends on how you send from the Erlang side, I suppose. The option {packet, 4} will send each data packet with a 4-byte length prefixed to it. Each send operation in Erlang will result in one packet being sent with it's length prefixed (the max size for length 4, for example, is 2 Gb). The relevant part of the Erlang documentation is for setting the socket options using inet:setopts/2.
I'm guessing the data is the total accumulated data read from the socket so far. If that data contains your whole packet, it's fine. But if not, you might want to continue to do a blocked read from the socket using readDataToLength with the remaining data.