I'm developing on the iPhone and the majority of our game is using OpenGL ES, but there are also menus that use CGImage and Quartz in order to be displayed. In OpenGL ES, I know that no matter what image compression goes in (JPG, PNG, etc.), the data stored in memory as a texture is an 8-bit texture, unless I use PVRTC in which case I can get it to 2 or 4 bits. We've been having memory issues due to large CGImages, so my question is... what sort of optimizations and compressions do Quartz and CGImage use? I can't find the details in Apple's docs, when really I want to know if it would make a difference to put a 256-color image in, or a JPG vs a PNG, if having the dimensions at a power of 2 help, etc. Speed is unimportant, memory is the bottleneck here.
Thanks.
Quartz is uncompressed. It is for quickly compositing and rendering pixel accurate content. Once your images have been drawn into a context it doesn't matter where they came from, they take whatever that context takes per pixel for however many pixels they have (generally 4 bytes per pixel in a device if I recall correctly). The one big thing it does is premultiplies the alpha to avoid blending.
Now, some views under memory pressure can evict their contents if not displayed, and reconstitute them as needed. In those cases a CGImage from a compressed source generally ends up taking less memory, but I suspect that is not relevant in the case you described.
Related
I want to implement zooming of sprites with a pinch gesture in Cocos2d.
How do I achieve it without the image getting pixelated?
I tried with vectors but with no success, I'm doomed using raster bitmap images.
Do I need the largest possible image with the highest resolution to make it look
nice?
What is the size limit for pngs in cocos2d?
What other pitfalls do I need to consider?
Yes. For example if the sprite should cover an area of 1024x1024 pixels when zoomed in to maximum, you need to create the image as 1024x1024 and set the scale property to below 1 in order to create a smaller version. If you use scale greater than 1.0 the image will always lose detail and become ever more blurred as scale increases.
There is no size limit in cocos2d, it's the devices that impose the limit. Most devices can handle 2048x2048 except 1st and 2nd generation which support only 1024x1024. You wouldn't normally support these older devices though, so 2048x2048 should be the default. Several newer devices (iPad 2+, iPhone 4S+) can use up to 4096x4096 textures.
Memory consumption. Not sure what you're trying to do, but often developers have little understanding about how much memory textures consume and what amount of memory is available. For instance, 2048x2048 as PNG with 32-Bit color consumes 16 MB of memory. Don't plan on using more than 4-5 of these, unless you're able to reduce color bit depth and use TexturePacker to be able to use the compressed .pvr.ccz format. Read my article about optimizing memory usage for more info.
In my project I am using 5 layers of very wide images, that are on top of each other and cover the whole screen.
In total these images are made out of 20 textures (4 per layer) with 512x512 pixels in size.
I use the PVRTC format to load and compress those textures to save memory. I checked with Instruments and internal memory on devices is not an issue due to the compression but I only get around 10-15fps if I enable and draw all layers.
If I only draw one layer (4 textures) for example I get an ok framerate of around 55fps.
I strongly suspect that my textures are simply too big and cause a bottleneck while rendering. I am not sure however how to counter and solve this bottleneck.
I would be very grateful for any advice and tips on how to handle such big images/textures correctly to get good performance.
I have an Open GL ES game on the iPhone. My framerate is pretty sucky, ~20fps. Using the Xcode OpenGL ES performance tool on an iPhone 3G, it shows:
Renderer Utilization: 95% to 99%
Tiler Utilization: ~27%
I am drawing a lot of pretty large images with a lot of blending. If I reduce the number of images drawn, framerates go from ~20 to ~40, though the performance tool results stay about the same (renderer still maxed). I think I'm being limited by the fill rate of the iPhone 3G, but I'm not sure.
My questions are: How can I determine with more granularity where the bottleneck is? That is my biggest problem, I just don't know what is taking all the time. If it is fillrate, is there anything I do to improve it besides just drawing less?
I am using texture atlases. I have tried to minimize image binds, though it isn't always possible (drawing order, not everything fits on one 1024x1024 texture, etc). Every frame I do 10 image binds. This seem pretty reasonable, but I could be mistaken.
I'm using vertex arrays and glDrawArrays. I don't really have a lot of geometry. I can try to be more precise if needed. Each image is 2 triangles and I try to batch things were possible, though often (maybe half the time) images are drawn with individual glDrawArrays calls. Besides the images, I have ~60 triangles worth of geometry being rendered in ~6 glDrawArrays calls. I often glTranslate before calling glDrawArrays.
Would it improve the framerate to switch to VBOs? I don't think it is a huge amount of geometry, but maybe it is faster for other reasons?
Are there certain things to watch out for that could reduce performance? Eg, should I avoid glTranslate, glColor4g, etc?
I'm using glScissor in a 3 places per frame. Each use consists of 2 glScissor calls, one to set it up, and one to reset it to what it was. I don't know if there is much of a performance impact here.
If I used PVRTC would it be able to render faster? Currently all my images are GL_RGBA. I don't have memory issues.
One of my fullscreen textures is 256x256. Would it be better to use 480x320 so the phone doesn't have to do any scaling? Are there any other general performance advice for texture sizes?
Here is a rough idea of what I'm drawing, in this order:
1) Switch to perspective matrix.
2) Draw a full screen background image
3) Draw a full screen image with translucency (this one has a scrolling texture).
4) Draw a few sprites.
5) Switch to ortho matrix.
6) Draw a few sprites.
7) Switch to perspective matrix.
8) Draw sprites and some other textured geometry.
9) Switch to ortho matrix.
10) Draw a few sprites (eg, game HUD).
Steps 1-6 draw a bunch of background stuff. 8 draws most of the game content. 10 draws the HUD.
As you can see, there are many layers, some of them full screen and some of the sprites are pretty large (1/4 of the screen). The layers use translucency, so I have to draw them in back-to-front order. This is further complicated by needing to draw various layers in ortho and others in perspective.
I will gladly provide additional information if reqested. Thanks in advance for any performance tips or general advice on my problem!
Edit:
I added some logging to see how many glDrawArrays calls I am doing, and with how much data. I do about 20 glDrawArray calls per frame. Usually about 1 to up to 6 of these has about 40 vertices each. The rest of the calls are usually just 2 vertices (one image). I'm just using glVertexPointer and glTexCoordPointer.
Given that the Renderer Utilization is basically at 100%, that indicates that the bottleneck is filling, texturing, and blending pixels. Techniques intended to optimize vertex processing (VBOs and vertex formats) or CPU usage (draw call batching) will likely not help, as they will not speed up pixel processing.
Your best bet is to reduce the number of pixels that you are filling, and also look at different texture formats that make better use of the very limited memory bandwidth available on the first generation devices. Prefer the use of PVRTC textures wherever possible, and 16bit uncompressed textures otherwise.
Look to Apple's "Best Practices for Working with Texture Data" and "Best Practices for Working with Vertex Data" sections of the OpenGL ES Programming Guide for iPhone OS. They highly recommend (as do others) that you use PVRTC for compressing your textures, because they can offer an 8:1 or 16:1 compression ratio over your standard uncompressed textures. Aside from mipmapping, you seem to be doing the other recommended optimization of using a texture atlas.
You do not appear to be geometry-limited, because (as I discovered in this question) the Tiler Utilization statistic seems to indicate how much of a bottleneck is being caused by geometry size. However, the iPhone 3G S (and third-generation iPod touch and iPad) support hardware-accelerated VBOs, so you might give those a shot and see how they affect performance. They might not have as much of an effect as compressing textures would, but they're not hard to implement.
A big win for the (mostly for)3G will also be the texture filtering you are using. Check if you are using "TRILINEAR" filtering, and change it to "BILINEAR".
Make sure you setup the textures like this:
glTexParameteri(GL_TEXTURE_2D, GL_TEXTURE_MIN_FILTER, GL_LINEAR_MIPMAP_NEAREST);
glTexParameteri(GL_TEXTURE_2D, GL_TEXTURE_MAG_FILTER, GL_LINEAR);
and not like this:
glTexParameteri(GL_TEXTURE_2D, GL_TEXTURE_MIN_FILTER, GL_LINEAR_MIPMAP_LINEAR);
glTexParameteri(GL_TEXTURE_2D, GL_TEXTURE_MAG_FILTER, GL_LINEAR);
Harry
I wanted to chime in with an additional answer, change the Framebuffer backing to be a 16bit format as opposed to a 32 bit format.
if ((self = [super initWithCoder:coder]))
{
eaglLayer = (CAEAGLLayer *)self.layer;
eaglLayer.opaque = YES;
eaglLayer.drawableProperties = [NSDictionary dictionaryWithObjectsAndKeys:
[NSNumber numberWithBool:NO],
kEAGLDrawablePropertyRetainedBacking,
kEAGLColorFormatRGB565, //kEAGLColorFormatRGBA8 = large frame buff 32bit // kEAGLColorFormatRGB565 = 16bit frame buffer.
kEAGLDrawablePropertyColorFormat,
nil];
}
What woke me up to this was the XCode profiler. It kept complaining about using to large a frame buffer, eventually I found it in my init section.
http://developer.apple.com/library/ios/#documentation/iPhone/Reference/EAGLDrawable_Ref/EAGLDrawable/EAGLDrawable.html
.
That single change allowed my games on iPad, Retina, and iPods to go to 60 FPS.
I have yet to re-release them with this change as I just found it out 3 days ago :) but I do not think I plan to release at 60fps, 30fps is just fine for casual games, and I found that my sound effects cut the frame rate down, so either resample, play the sfx in another thread, or other solutions to keep that frame rate up if I decide to go with 60fps. Don't forget to discard the buffers that are not used to display..
if (fiOSver >= 4.0f) {
const GLenum discards[] = {GL_DEPTH_ATTACHMENT_OES};
glDiscardFramebufferEXT(GL_FRAMEBUFFER_OES,1,discards);
}
[m_oglContext presentRenderbuffer:GL_RENDERBUFFER_OES];
In a similar situation (porting a 2D Adventure game to the iPad). My 3GS version was running more or less locked at 60FPS, put it on iPad dropped it (and my jaw) to 20fps.
Turns out ONE of the little gotchas involved is that the PVR cards hate GL_ALPHA_TEST; on the PC that actually has a slight positive effect (especially on older intel chips), but they're death on fillrate on the iPhone. Changing that to
glDisable(GL_ALPHA_TEST);
gave me an immediate 100% boost in FPS (up to 40 FPS). Not bad for one line of code.. :)
Allan
The biggest performance killer on the iPhone platform is the number of draw calls and state changes. If you're doing more than 20ish draw calls or state changes, you're going to run into a performance wall.
Batching and texture atlases are your friend.
By my past experience with openGL ES on old windows mobile devices with processing speed around 600mhz
usually reducing the rendering window resolution increase the speed of rendering.
as my tests lately i figured out that i need performance monitoring while rendering frame by frame to collect how many fps can display with current performance and what resolution currently applied
i hope it is a good practice to keep a monitoring algorithm in place of rendering view to balance resolution and frame rate while running a game rendering engine.
depending on the amount of frame rate you wanted, you should sacrifice the rendering view resolution, to perform best and degrade gracefully on most devices with varying hardware and software performence.
you may need to control the resolution menually like explained in this article.
http://www.david-amador.com/2010/09/setting-opengl-view-for-iphone-4-retina-hi-resolution/
Developing a simple game for the iPhone, what gives a better performance?
Using a lot of small (10x10 to 30x30 pixels) PNGs for my UIViews' backgrounds.
Using one large PNG and clipping to the bounds of my UIViews.
My thought is that the first technique requires less memory per individual UIView, but complicates how the iPhone handles the large amount of images, as it tries to combine the images into a larger texture or tries to switch between all the small textures a lot.
The second technique, on the other hand, gives the iPhone the opportunity to handle just one large PNG, but unnessicarily increases the image weight every UIView has to carry.
Am I right about the iPhone's attempts, handling the images the way I described it?
So, what is the way to go?
Seeing the answers thus far, there is still doubt. There seems to be a trade-off with two parameters: Complexity and CPU-intensive coding. What would be my tipping point for deciding what technique to use?
If you end up referring back to the same CGImageRef (for example by sharing a UIImage *), the image won't be loaded multiple times by the different views. This is the technique used by the videowall Core Animation demo at the WWDC 07 keynote. That's OSX code, but UIViews are very similar to CALayers.
The way Core Graphics handles images (from my observation anyway) is heavily tweaked for just-in-time loading and aggressive releasing when memory is tight.
Using a large image you could end up loading the image at draw time if the memory for the decoded image that CGImageRef points to has been reclaimed by the system.
What makes a difference is not how many images you have, but how often the UIKit traverses your code.
Both UIViews and Core Animation CALayers will only repaint if you ask them to (-setNeedsDisplay), and the bottleneck usually is your code plus transferring the rendered content into a texture for the graphics chip.
So my advice is to think your UIView layout in a way that allows portions that change together to be updated all at the same time, which turn into a single texture upload.
One large image mainly gives you more work and more headaches. It's harder to maintain but is probably less ram intensive because there is only one structure + data in memory instead of many structures + data. (though probably not enough to notice).
Looking at the contents of .app bundles on regular Mac OS, it seems the generally approved method of storage is one file/resource per image.
Ofcourse, this is assuming you're not getting the image resources from the web, where the bottleneck would be in http and its specified maximum of two concurrent requests.
One large gives you better performance. (Of cause if you should render all pictures anyway).
One large image will remove any overhead associated with opening and manipulating many images in memory.
I would say there is no authoritative answer to this question. A single large image cuts down (slow) flash access and gets the decode done in one go but a lot of smaller images give you better control over what processing happens when... but it's memory hungry and you do have to slice that image up or mask it.
You will have to implement one solution and test it. If it isn't fast enough and you can't optimise, implement the other. I suggest implementing the version which you personally find easier to imagine implementing because that will be easiest to implement.
I'm trying to solve this flickering problem on the iphone (open gl es game). I have a few images that don't have pow-of-2 dimensions. I'm going to replace them with images with appropriate dimensions... but why do the dimensions need to be powers of two?
The reason that most systems (even many modern graphics cards) demand power-of-2 textures is mipmapping.
What is mipmapping?
Smaller versions of the image will be created in order to make the thing look correctly at a very small size. The image is divided by 2 over and over to make new images.
So, imagine a 256x128 image. This would have smaller versions created of dimensions 128x64, 64x32, 32x16, 16x8, 8x4, 4x2, 2x1, and 1x1.
If this image was 256x192, it would work fine until you got down to a size of 4x3. The next smaller image would be 2x1.5 which is obviously not a valid size. Some graphics hardware can deal with this, but many types cannot.
Some hardware also requires a square image but this isn't very common anymore.
Why do you need mipmapping?
Imagine that you have a picture that is VERY far away, so far away as to be only the size of 4 pixels. Now, when each pixel is drawn, a position on the image will be selected as the color for that pixel. So you end up with 4 pixels that may not be at all representative of the image as a whole.
Now, imagine that the picture is moving. Every time a new frame is drawn, a new pixel is selected. Because the image is SO far away, you are very likely to see very different colors for small changes in movement. This leads to very ugly flashing.
Lack of mipmapping causes problems for any size that is smaller than the texture size, but it is most pronounced when the image is drawn down to a very small number of pixels.
With mipmaps, the hardware will have access to 2x2 version of the texture, so each pixel on it will be the average color of that quadrant of the image. This eliminates the odd color flashing.
http://en.wikipedia.org/wiki/Mipmap
Edit to people who say this isn't true anymore:
It's true that many modern GPUs can support non-power-of-two textures but it's also true that many cannot.
In fact, just last week I had a 1024x768 texture in an XNA app I was working on, and it caused a crash upon game load on a laptop that was only about a year old. It worked fine on most machines though. It's a safe bet that the iPhone's gpu is considerably more simple than a full PC gpu.
Typically, graphics hardware works natively with textures in power-of-2 dimensions. I'm not sure of the implementation/construction details that cause this to be the case, but it's generally how it is everywhere.
EDIT: With a little research, it turns out my knowledge is a little out of date -- a lot of modern graphics cards can handle arbitrary texture sizes now. I would imagine that with the space limitations of a phone's graphics processor though, they'd probably need to omit anything that would require extra silicon like that.
You can find OpenGL ES support info about Apple Ipod/Iphone devices here:
Apple OpenES support
OpenGL ES 2.0 is defined as equal to OpenGL 2.0
The constraint about texture size's has been disappear only from version 2.0
So if you use OpenGL ES with version less then 2.0 - it is normal situation.
I imagine it's a pretty decent optimization in the graphics hardware to assume power-of-2 textures. I bought a new laptop, with latest laptop graphics hardware, and if textures aren't power-of-2 in Maya, the rendering is all messed up.
Are you using PVRTC compression? That requires powers of 2 and square images.
Try implementing wrapping texture-mapping in software and you will quickly discover why power-of-2 sized are desirable.
In short, you will find that if you can assume power-of-2 dimensions then a lot of integer multiplications and divisions turn into bit-shifts.
I would hazard a guess that the recent trend in relaxing this restriction is due to GPUs moving to floating-point maths.
Edit: The "because of mipmapping" answer is incorrect. Mipmapped, non-power-of-two textures are a common feature of modern GPUs.