I currently have an app where I do a lot of image manipulation. I basically take an image that is 320x320 (or 640x640 on Retina) and scale it down to 128x128 (or 256x256 on Retina) before rounding off its corners and applying a glossy highlight. Everything is done using Core Graphics drawing.
At any one time there could be around 600 images that need this processing so I do around 40 on app launch using a background thread and cache them into a FIFO queue. When an image not in the cache needs processing, I do so and add it to the end of the cache, discarding the first cached image. If that first image is needed again it goes through the same process.
What I would like to know is if it would make more sense, and is ultimately more efficient, to save the discarded images to disk rather than recreate them from scratch the next time they are needed, as I could instead just read them from disk.
These images are also displayed using a CALayer and therefore there may be a overhead when the layers' contents are set because of the conversion from UIImage to CGImage. If I store them on disk I believe they can be read directly as a CGImage?
Any ideas and input on improving the efficiency of this process will be sincerely welcomed.
My personal choice would be to use a disk cache.
However, you say 'which is more efficient' - what do you mean?
If you mean faster then the disk cache is probably going to win.
If you mean more space efficient then recreating them will win.
If you mean lower memory usage then it entirely depends on your implementation!
You will have to try it and see :)
However, the advantage of the disk solution is that the second time your app starts up, it will already have done the processing so will start faster. That's why I'd use the disk.
From my experience, saving and then reading from the disk is faster. I had some memory warnings by doing it over and over, instead of saving and reading. But, the only to know for sure is to try. I was using around 1000 images, so it makes senses in my case to use the disk.
It's also good to give a try github's libs that downloads and caches UIImage/NSData from Internet.
It may by SDWebImage(https://github.com/rs/SDWebImage) or APSmartStorage (https://github.com/Alterplay/APSmartStorage).
APSmartStorage helps to get data from network and automatically caches data on disk or in memory in a smart configurable way. Should be good enough.
Related
I am designing this app and there's a section where the users are allowed to draw with their fingers. As I provide a undo for the operation, when the user starts drawing I have to quickly grab the current drawing content and store it somewhere.
I first tried to store the undo as a CGLayer and also as an image, but my app memory usage went up from 7 to 19 Mb. With 19 MB I am relatively safe, because 24 Mb appears to be the theoretical limit beyond things start to be dangerous. The problem is that I have another section of my app that requires lots of memory and if I run this, the memory peaks from 19 to 28 Mb, that is too dangerous to risk.
Then I decided to save the image on disk. To prevent the little gap that happens when the image has to be saved when the user fires TouchesBegan I refined, to the limits of sanity, the saving to disk method and now I barely don't feel any gap. I said, I barely don't feel, but I still feel a little hair gap, I would say <0.1s that takes to the line start drawing.
What I do is to fire a queue operation to manage the file saving.
Is there any other ways you guys can envision on how could this be improved?
thanks
Use mmap and CGBitmapContextCreate to create images that are backed by a file; the kernel will lazily page in and out parts of the file as they are needed.
Combine this with rjobidon's suggestion by snapshotting every so often and you should have a robust and speedy undo system.
I suggest to store the image as vector data, this way you could play and undo very quickly and divide by thousand times the memory used. For example you can only store:
Gesture coordinates (x, y)
Gesture type (touchBegan, touchEnded)
Pen changes (color, width, effect)
You also have a canvas to render the current image.
Good luck!
How about doing the storing at the end of the previous operation? Then the beginning of each stroke will be instant.
You could even do it on a timer so it only saves an undo if you pause drawing for a few seconds, which would mean the undo takes them back to the previous stage of the drawing, rather than just the previous stroke, which may be desirable.
I have an app that will display a bunch of images in a slideshow. Those images will be part of the bundle, thus distributed with the app.
All the images are photographs or photographic, etc.
I've read that it's preferred to use PNG as the image format, but seeing that the JPG version will be much smaller, I'd rather be using that.
Are there any guidelines which format to use and in which case?
PNG's are pixel perfect (non-lossy), and require very little extra CPU energy to display. However, large PNGs may take longer to read from storage than more compressed image formats, and thus be slower to display.
JPG's are smaller to store, but lossy (amount depends on compression level), and to display them requires a much more complicated decoding algorithm. But the typical compression and image quality is usually quite sufficient for photos.
Use JPG's for photos and for anything large, and PNG's for anything small and/or designed to be displayed "pixel perfect" (e.g. small icons) or as a part of a composited transparent overlay, etc.
Apple optimizes PNG images that are included in your iPhone app bundle. In fact, the iPhone uses a special encoding in which the color bytes are optimized for the hardware. XCode handles this special encoding for you when you build your project. So, you do see additional benefits to using PNG's on an iPhone other than their size consideration. For this reason it is definitely recommended to use PNG's for any images that appear as part of the interface (in a table view, labels, etc).
As for displaying a full screen image such as a photograph you may still reap benefits with PNG's since they are non-lossy and the visual quality should be better than a JPG not to mention resource usage with decoding the image. You may need to decrease the quality of your JPG's in order to see a real benefit in file size but then you are displaying non-optimal images.
File size is certainly a factor but there are other considerations at play as well when choosing an image format.
There is one important thing to think about with PNGs. If a PNG is included in your Xcode build it will be optimized for iOS. This is called PNG crush. If your PNG is downloaded at run time it will not be crushed. Crushed PNGs run about the same as 100% JPGs. Lower quality JPGs run better than higher quality JPGs. So from a performance standpoint from fastest to slowest it would go low quality JPG, high quality JPG, PNG Crushed, PNG.
If you need to download PNGs you should consider crushing the PNGs on the server before the download.
http://www.cocoanetics.com/2011/10/avoiding-image-decompression-sickness/
The Cocoanetics blog published a nice iOS performance benchmark of JPGs at various quality levels, and PNGs, with and without crushing.
From his conclusion:
If you absolutely need an alpha channel or have to go with PNGs then
it is advisable to install the pngcrush tool on your web server and
have it process all your PNGs. In almost all other cases high quality
JPEGs combine smaller file sizes (i.e. faster transmission) with
faster compression and rendering.
It turns out that PNGs are great for small images that you would use
for UI elements, but they are not reasonable to use for any full
screen applications like catalogues or magazines. There you would want
to choose a compression quality between 60 and 80% depending on your
source material.
In terms of getting it all to display you will want to hang onto
UIImage instances from which you have drawn once because those have a
cached uncompressed version of the file in them. And where you don’t
the visual pause for a large image to appear on screen you will have
to force decompression for a couple of images in advance. But bear in
mind that these will take large amounts of RAM and if you are
overdoing it that might cause your app to be terminated. NSCache is a
great place to place frequently used images because this automatically
takes care of evicting the images when RAM becomes scarce.
It is unfortunate that we don’t have any way to know whether or not an
image still needs decompressing or not. Also an image might have
evicted the uncompressed version without informing us as to this
effect. That might be a good Radar to raise at Apple’s bug reporting
site. But fortunately accessing the image as shown above takes no time
if the image is already decompressed. So you could just do that not
only “just in time” but also “just in case”.
Just thought I'd share a bit of decompression performance data...
I'm doing some prototyping of a 360 degree viewer - a carousel where the user can spin through a series of photos taken from different angles, to give the impression of being able to smoothly rotate an object.
I have loaded the image data into an array of NSData's to take file i/o out of the equation, but create NSImage's on the fly. Testing at near max frame rate (~25 fps) and watching in Instruments I see the app is clearly CPU-bound and there's an approximately 10% increase in CPU load showing ~275 kb png's vs. ~75 kb jpg's.
I can't say for sure but my guess is the CPU limit is just from general program execution and moving all the data around in memory, but that image decompression is done on the GPU. Either way and the JPG vs. PNG performance argument looks to favour JPG, especially when the smaller file sizes (and therefore smaller sizes of objects in memory at least in some parts of the chain) is taken into consideration.
Of course every situation is different, there's no substitute for testing...
I have found massive differences in animation performance when using jpegs vs png. For example placing three screen-sized jpegs side by side in a UIScrollView and scrolling horizontally on an iPhone4 results in lag and a thoroughly unpleasant jerky animation. With non-transparent pngs of the same dimensions the scrolling is smooth. I never use jpegs, even if the image is large.
I think if you want to use transparent, you have no choice except PNG. But, if your background is opaque already, then you may use JPG. That is the only difference I can see
'Use JPEG for photos' as mentioned in Human Interface Guidelines under section Produce artwork in the appropriate format.
So I've seen this question asked before and in fact I asked it last night but I thought I'd give it another go just to see if I could get any other unique views on the problem.
The Problem — I have an app with a large number of uiimageviews (with image downloaded to disk) in a scrollview which of course has two large problems facing it: Memory use, and performance. In my app memory use isn't so much a problem because I am employing techniques such as dequeuing and reusing imageviews and such. But performance is another thing entirely. Right now, as a memory saving procedure, I only store image filepaths in memory because it would be ridiculous to store images in memory. But the problem with this is that reading from the disk takes more time than from memory and slows down scrolling on the scrollview immensely.
So, what kind of techniques do any of you suggest for something like this? I've seen three20 but don't want to use it because I need high customizability in my view and that just won't do. Image files are not large, but just thumbnail size so there is no scaling or excess size. There's got to be an intuitive way to handle this. The built in photos app handles up to thousands of photos perfectly with low memory and slick and smooth scrolling performance.
Fundamentally, the problem is that you're probably doing a bunch of disk I/O on your UI thread, which is basically guaranteed to cause performance problems.
You should consider loading your images on a background thread and updating the image views on the main thread when the images are loaded. Depending on your use case you can get more or less clever about how far you preload in advance, etc, so you can have images ready. (There might be some usable source code or even Apple sample code out there that does something like this, but I don't know of it off the top of my head.)
You may notice that some applications (not sure about the Photos app) have an intermediate stage where they load a very small thumb size image for all images, and scale it up to the render size, which acts as a placeholder until the full size version is loaded-- if the user scrolls past that image before the full size is loaded, the visible effect is nearly the same as if the image was there all along.
For the last few weeks I've been tearing my hair out trying to get a tiling mechanism working on an iPhone. I need to scale and crop images of about 150mb so that they can be saved as tiles that will be requested by a scroll view, allowing the user to see the image in high resolution.
The problem is that these images are really pushing the bounds of what an iPhone can handle. It seems quite easy to scale these huge images down to 1000 or so across and do that tiling, but for large zoom levels I need to scale it mid-way, say 4000 across and that's too big. So I hit upon the idea of making medium sized blocks from the full sized image and tiling each of those and the medium zoom.
By creating an autoreleasepool around the inner loops, and draining it after each cycle I can mostly keep the memory under control but sometimes, and to me it seems random, memory is getting leaked, or at least not drained. I'm doing all this on a secondary thread and when it gets back to the first function in that thread I release the thread's own autoreleasepool and only then do the last memory artifacts get cleared. It doesn't seem to bother the simulator but the iPhone is much less forgiving and it crashes before it can complete the whole tiling process. The cropping code I am using is from Hive05
http://www.hive05.com/2008/11/crop-an-image-using-the-iphone-sdk/
Has anyone else had to deal with such massive images before? Is pre-generating tiles the best way to go? Any suggestions on why some loops would increase the memory and some not, or how to force every auto-released thing to clear on the inner pool instead of waiting for the outer pool?
Thanks for reading this far.
for got to add, these images are TIFs, so perhaps a direct reading of the bitmap info would be better than scaling and cropping the entire thing
First of all, I have serious doubts that an 150 MB image would fit in the device's memory, even if we're talking about a 3GS. That one has about 128 MB available memory for 3rd party apps maximum. See the device console messages and look for memory warnings, I guess you'll see that before crashing, the app emits them when trying to load your image. Reading the bitmap info in chunks would seem more sensible, as you'll be managing smaller sections at a time. I don't think Cocoa has a random-access file API, so you'll have to resort to a C function.
I've managed to write the loops that cycle through tiles of 1024x1024 and my iPhone 3G is able to finish the processing. It takes over 30 minutes though so it isn't great but that's what you get for working with 150mb TIF on a cellphone.
To keep the memory usage low I had to drain the AutoReleasepools after each iteration. Apple Tech Support pointed out that since the iPhone is a reference-counted environment rather than a garbage collected environment it is better to create a new AutoReleasePool at the start of each inner loop and drain it at the end of each loop than to create it before any loops start, drain it many times and then release it after the loops are done. Until I made that change my app would crash an iPhone but run fine on the simulator.
I am grabbing an image from the camera roll and then using it for a while as well as save it to disk as a PNG on the iPhone. I am getting the odd crash, presumably due to out of memory.
Does it make a difference if I save it as PNG or JPG (assuming I choose note to degrade the quality in the JPG case)? Specifically:
is more memory then used by the UIImage after I reload it off of disk if I saved it as a PNG?
is it possible the act of saving as PNG uses up more memory transiently during the saving process?
I had been assuming the UIImage was a format neutral representation and it shouldn't matter, but I thought I should verify.
I am getting the odd crash, presumably due to out of memory
Then STOP WHAT YOU ARE DOING RIGHT NOW and first figure out if that's actually the cause of the crash. Otherwise there's a very good chance that you're chasing the wrong problem here, fixing a memory problem that doesn't exist while ignoring the real cause of the crash. If you want to fix a crash, start by figuring out what caused the crash. Following up on what's "presumably" the problem is a recipe for wasted time and effort.
I have an application on the store that needs to save intermediate versions of an image as it's being edited. In the original version, I used PNG format for saving, to avoid quality loss from loading and saving JPEG multiple times.
Sometime around the 2.2 software release, Apple introduced a change into the PNG writing code, such that it takes many times longer to save PNG data from some images. I ended up having to change to saving in JPEG format, because my application was timing out when trying to save images on exit.
Also, you'll run into issues because saving in PNG format doesn't preserve the "orientation" information in the UIImage, so a picture taken in Portrait orientation with the built-in camera will appear rotated after you save and reload it.
It depends on what type of images you're dealing with. If you're dealing with photographic images, JPEGs will almost always be smaller than PNGs, with no discernable loss of detail as can be seen by the human eye.
Conversely, if you're dealing with highly non-photographic images such as GUI elements or images with large blocks of solid colors, then PNGs and JPEGs will be comparable in size, but the PNG will save losslessly whereas the JPEG will be lossy and have very visible artifacts. If you have a really simple image (very large blocks of constant colors, e.g.), then a PNG will very likely be much smaller than a JPEG, and again will not have any compression artifacts.
The act of saving an image as a PNG or JPEG should not take up very much transient memory. When an image is in memory, it is typically stored uncompressed in memory so that it can be drawn to the screen very quickly, as opposed to having to decompress it every time you want to render it. Compared to the size of the uncompressed image, the amount of extra temporary storage you need to compress it is very small. If you can fit the uncompressed image in memory, you don't have to worry about the memory used while compressing it.
And of course, once you write the image to the file system in non-volatile storage and free the in-memory image, it really doesn't matter how big the compressed image is, because it doesn't take up main memory any more. The size of the compressed image only affects how much flash storage it's using, which can be an issue, but it does not affect high likely your app is to run out of memory.
Your crashes may be from a known memory leak in the UIImagePickerController.
This should help you fix that.
I don't have any hard data, but I'd assume that PNGs are preferable because Apple seems to use PNGs virtually everywhere in iPhone OS.
However, if you've already got the code set up for writing PNGs, it shouldn't be too hard to change it to write JPEGs, should it? Just try both methods and see which works better.
Use PNG wherever possible. As part of the compilation XCode runs all PNG files through a utility (pngcrush) to compress and optimize them.
is more memory then used by the UIImage after I reload it off of
disk if I saved it as a PNG?
=> No, it's the same memory size if you import from 2 images that have same resolution and same number of channels. (such as RGBA)
is it possible the act of saving as PNG uses up more memory transiently during the saving process?
=> No, it only effect your disk memory.