I'm trying to read the alpha pixel values using glReadPixels. The first thing I did was read the pixels individually. To try to speed things up, I tried reading all the pixels at once :
GLubyte *pixels = new GLubyte[w*h*4];
glReadPixels(0, 0, w, h, GL_RGBA, GL_UNSIGNED_BYTE, pixels);
and it worked, but really slow. Now I'm trying to just retrieve the alpha value, without wasting space to the RGB components :
GLubyte *pixels = new GLubyte[w*h];
glPixelStorei(GL_UNPACK_ALIGNMENT, 1);
glPixelStorei(GL_PACK_ALIGNMENT, 1);
glReadPixels(0, 0, w, h, GL_ALPHA, GL_UNSIGNED_BYTE, pixels);
But I get : OpenGL error 0x0500 in -[EAGLView swapBuffers].
Any idea as to why a INVALID_ENUM (0x0500) is thrown?
According to the documentation on glReadPixels() for OpenGL ES, the only valid enum values for the format parameter are GL_RGBA and GL_IMPLEMENTATION_COLOR_READ_FORMAT_OES. You'd need to check and see what GL_IMPLEMENTATION_COLOR_READ_FORMAT_OES means as a format for the iPhone, but it may not provide support for GL_ALPHA.
In any case, I doubt that going that route will dramatically speed up your reads, because all that will do is discard the RGB components. Your performance issues with glReadPixels() probably lie elsewhere. A good discussion of the reasons for this can be found in the discussion thread here.
Would it be possible for you to render into an offscreen framebuffer that was backed by a texture, then do further processing on the GPU using that texture? This sounds like it would yield better performance than using glReadPixels().
Related
Looking at the docs, I should be able to use BGRA for the internal format of a texture. I am supplying the texture with BGRA data (using GL_RGBA8_OES for glRenderbufferStorage as it seems BGRA there is not allowed). However, the following does not work:
glTexImage2D(GL_TEXTURE_2D, 0, **GL_BGRA**, w, h, 0, GL_BGRA, GL_UNSIGNED_BYTE, buffer);
...
glReadPixels(0, 0, w,h, GL_BGRA, GL_UNSIGNED_BYTE,buffer, 0);
While this gives me a black frame:
glTexImage2D(GL_TEXTURE_2D, 0, GL_RGBA, w, h, 0, GL_BGRA, GL_UNSIGNED_BYTE, buffer);
...
glReadPixels(0, 0, w,h, **GL_BGRA**, GL_UNSIGNED_BYTE,buffer, 0);
And this does work, but the blues/reds are inverted (I supply BGRA data to the texture):
glTexImage2D(GL_TEXTURE_2D, 0, GL_RGBA,w, h, 0, GL_BGRA, GL_UNSIGNED_BYTE, buffer);
...
glReadPixels(0, 0, w,h, **GL_RGBA**, GL_UNSIGNED_BYTE,buffer, 0);
...why can't I just use BGRA throughout? I do notice that glRenderbufferStorage does not seem to accept any BGRA formats...I'm really confused. BGRA is the only suitable format my data is in, as it comes from the iphone's camera.
The third parameter to glTexImage2D() is the number of color components in the texture, not the pixel ordering of the texture. You want to use GL_RGBA here, or it just won't work.
I don't believe GL_BGRA is supported by glReadPixels() on iOS devices as a color format. While providing pixel data to the textures in BGRA format is recommended by Apple when processing video image frames, I think you're fine in reading that back in RGBA format and then encoding that to disk, as you've described elsewhere.
If you want to see a sample project that takes camera video frames in BGRA, sends them to a texture, processes them using shaders, and then reads the resulting pixels back, you can check out the one I built here.
My own looking around for OpenGL ES2 indicates that this works for the BGRA little endian native format under iOS. Here "pixels" points to a buffer of 32 bit BGRA that would come from a CGBitmapContextCreate() or grabbing the image data from out of a CGImageRef.
glTexImage2D(GL_TEXTURE_2D, 0, GL_RGBA, width, height, 0, GL_BGRA_EXT, GL_UNSIGNED_BYTE, pixels);
See the apple ext for this here.
Good discussion of this issue here
Normally, you'd use something like:
glBlendFunc(GL_SRC_ALPHA, GL_ONE_MINUS_SRC_ALPHA);
glEnable(GL_BLEND);
glEnable(GL_LINE_SMOOTH);
glLineWidth(2.0f);
glVertexPointer(2, GL_FLOAT, 0, points);
glEnableClientState(GL_VERTEX_ARRAY);
glDrawArrays(GL_LINE_STRIP, 0, num_points);
glDisableClientState(GL_VERTEX_ARRAY);
It looks good in the iPhone simulator, but on the iPhone the lines get extremely thin and w/o any anti aliasing.
How do you get AA on iPhone?
One can achieve the effect of anti aliasing very cheaply using vertices with opacity 0.
Here's an image example to explain:
Comparison with AA:
You can read a paper about this here:
http://research.microsoft.com/en-us/um/people/hoppe/overdraw.pdf
You could do something along this way:
// Colors is a pointer to unsigned bytes (4 per color).
// Should alternate in opacity.
glColorPointer(4, GL_UNSIGNED_BYTE, 0, colors);
glEnableClientState(GL_COLOR_ARRAY);
// points is a pointer to floats (2 per vertex)
glVertexPointer(2, GL_FLOAT, 0, points);
glEnableClientState(GL_VERTEX_ARRAY);
glDrawArrays(GL_TRIANGLE_STRIP, 0, points_count);
glDisableClientState(GL_VERTEX_ARRAY);
glDisableClientState(GL_COLOR_ARRAY);
Starting in iOS Version 4.0 you have an easy solution, it's now possible to use Antialiasing for the whole OpenGL ES scene with just a few lines of added code. (And nearly no performance loss, at least on the SGX GPU).
For the code please read the following Apple Dev-Forum Thread.
There are also some sample pictures how it looks for me on my blog.
Using http://answers.oreilly.com/topic/1669-how-to-render-anti-aliased-lines-with-textures-in-ios-4/ as a starting point, I was able to get anti-aliased lines like these:
They aren't perfect nor are they as nice as the ones that I had been drawing with Core Graphics, but they are pretty good. I am actually drawing same lines (vertices) twice - once with bigger texture and color, then with smaller texture and translucent white.
There are artifacts when lines overlap too tightly and alphas start to accumulate.
One approach around this limitation is tessellating your lines into textured triangle strips (as seen here).
The problem is that on the iPhone OpenGl renders to a frame buffer object rather than the main frame buffer and as I understand it FBO's don't support multisampling.
There are various tricks that can be done, such as rendering to another FBO at twice the display size and then relying on texture filtering to smooth things out, not something that I've tried though so can't comment on how well this works.
I remember very specifically that I tried this and there is no simple way to do this using OpenGL on the iPhone. You can draw using CGPaths and a CGContextRef, but that will be significantly slower.
Put this in your render method and setUpFrame buffer...
You will get anti-aliased appearance.
/*added*/
//[_context presentRenderbuffer:GL_RENDERBUFFER];
//Bind both MSAA and View FrameBuffers.
glBindFramebuffer(GL_READ_FRAMEBUFFER_APPLE, msaaFramebuffer);
glBindFramebuffer(GL_DRAW_FRAMEBUFFER_APPLE, framebuffer );
// Call a resolve to combine both buffers
glResolveMultisampleFramebufferAPPLE();
// Present final image to screen
glBindRenderbuffer(GL_RENDERBUFFER, _colorRenderBuffer);
[_context presentRenderbuffer:GL_RENDERBUFFER];
/*added*/
I'm using the Texture2D class in an iPhone game using OpenGL ES.
Are their any good tutorials for understanding the Texture2D class?
Specifically I'm looking at the initWithString method for printing text. As the way it is implemented, you get white text when you use it. I would like to modify the method so I could specify the RGB color of the text. Any help / pointers?
Because the class uses an alpha-only texture (read the code!), it will display in whatever color glColor has set. See this line in initWithData (which gets called by initWithString):
glTexImage2D(GL_TEXTURE_2D, 0, GL_ALPHA,
width, height, 0, GL_ALPHA,
GL_UNSIGNED_BYTE, data);
For red text, just call glColor4ub(255, 0, 0, 255) prior to drawing the texture.
Make sure you enable GL_BLEND and GL_COLOR_MATERIAL prior to drawing.
The class is small. I recommend you just read it.
I've slightly modified the iPhone SDK's GLSprite example while learning OpenGL ES and it turns out to be quite slow. Even in the simulator (on the hw worst) so I must be doing something wrong since it's only 400 textured triangles.
const GLfloat spriteVertices[] = {
0.0f, 0.0f,
100.0f, 0.0f,
0.0f, 100.0f,
100.0f, 100.0f
};
const GLshort spriteTexcoords[] = {
0,0,
1,0,
0,1,
1,1
};
- (void)setupView {
glViewport(0, 0, backingWidth, backingHeight);
glMatrixMode(GL_PROJECTION);
glLoadIdentity();
glOrthof(0.0f, backingWidth, backingHeight,0.0f, -10.0f, 10.0f);
glMatrixMode(GL_MODELVIEW);
glClearColor(0.3f, 0.0f, 0.0f, 1.0f);
glVertexPointer(2, GL_FLOAT, 0, spriteVertices);
glEnableClientState(GL_VERTEX_ARRAY);
glTexCoordPointer(2, GL_SHORT, 0, spriteTexcoords);
glEnableClientState(GL_TEXTURE_COORD_ARRAY);
// sprite data is preloaded. 512x512 rgba8888
glGenTextures(1, &spriteTexture);
glBindTexture(GL_TEXTURE_2D, spriteTexture);
glTexImage2D(GL_TEXTURE_2D, 0, GL_RGBA, width, height, 0, GL_RGBA, GL_UNSIGNED_BYTE, spriteData);
free(spriteData);
glTexParameteri(GL_TEXTURE_2D, GL_TEXTURE_MIN_FILTER, GL_LINEAR);
glEnable(GL_TEXTURE_2D);
glBlendFunc(GL_ONE, GL_ONE_MINUS_SRC_ALPHA);
glEnable(GL_BLEND);
}
- (void)drawView {
..
glClear(GL_COLOR_BUFFER_BIT);
glLoadIdentity();
glTranslatef(tx-100, ty-100,10);
for (int i=0; i<200; i++) {
glTranslatef(1, 1, 0);
glDrawArrays(GL_TRIANGLE_STRIP, 0, 4);
}
..
}
drawView is called every time the screen is touched or the finger on the screen is moved and tx,ty are set to the x,y coordinates where that touch happened.
I've also tried using GLBuffer, when translation was pre-generated and there was only one DrawArray but gave the same performance (~4 FPS).
===EDIT===
Meanwhile I've modified this so that much smaller quads are used (sized: 34x20) and much less overlapping is done. There are ~400 quads->800 triangles spread on the whole screen. Texture size is 512x512 atlas and RGBA_8888 while the texture coordinates are in float.
The code is very ugly in terms of API efficiency: there are two MatrixMode change along with two loads and two translation then a drawarrays for a triangle strip (quad).
Now this produces ~45 FPS.
(I know this is very late, but I couldn't resist. I'll post anyway, in case other people come here looking for advice.)
This has nothing to do with the texture size. I don't know why people rated up Nils. He seems to have a fundamental misunderstanding of the OpenGL pipeline. He seems to think that for a given triangle, the entire texture is loaded and mapped onto that triangle. The opposite is true.
Once the triangle has been mapped into the viewport, it is rasterized. For every on-screen pixel the your triangle covers, the fragment shader is called. The default fragment shader (OpenGL ES 1.1, which you are using) will lookup the texel that most closely maps (GL_NEAREST) to the pixel you are drawing. It might look up 4 texels since you are using the higher quality GL_LINEAR method to average the best texel. Still, if the pixel count in your triangle is, say 100, then the most texture bytes you will have to read is 4(lookups) * 100(pixels) * 4(bytes per color. Far far less than what Nils was saying. It's amazing that he can make it sound like he actually knows what he's talking about.
WRT the tiled architecture, this is common in embedded OpenGL devices to preserve locality of reference. I believe that each tile gets exposed to each drawing operation, quickly culling most of them. Then the tile decides what to draw on itself. This is going to be much slower when you have blending turned on, as you do. Because you are using large triangles that might overlap and blend with other tiles, the GPU has to do a lot of extra work. If, instead of rendering the example square with alpha edges, you were to render an actual shape (instead of a square picture of the shape), then you could turn off blending for this part of the scene and I bet that would speed things up tremendously.
If you want to try it, just turn off blending and see how much things speed up, even if the don't look right. glDisable(GL_BLEND);
Your texture is 512*512*4 bytes per pixel. That's a megabyte of data. If you render it 200 times per frame you generate a bandwidth load of 200 megabytes per frame.
With roughly 4 fps you consume 800mb/second just for texture reads alone. Frame- and Zbuffer writes need bandwidth as well. Then there is the CPU, and don't underestimate the bandwidth requirements of the display as well.
RAM on embedded systems (e.g. your iphone) is not as fast as on a Desktop-PC. What you see here is a bandwidth starvation effect. The RAM simply can't handle the data faster.
How to cure this problem:
pick a sane texture-size. On average you should have 1 texel per pixel. This gives crisp looking textures. I know - it's not always possible. Use common sense.
use mipmaps. This takes up 33% of extra space but allows the graphic chip to pick use a lower resolution mipmap if possible.
Try smaller texture formats. Maybe you can use the ARGB4444 format. This would double the rendering speed. Also take a look at the compressed texture formats. Decompression does not cause a performance drop as it's done in hardware. Infact the opposite is true: Due to the smaller size in memory the graphic chip can read the texture-data faster.
I guess my first try was just a bad (or very good) test.
iPhone has a PowerVR MBX Lite which has a tile based graphics processor. It subdivides the screen into smaller tiles and renders them parallel. Now in the first case above the subdivision might got a bit exhausted because of the very high overlapping. More over, they couldn't be clipped because of the same distance and so all texture coordinates had to calculated (This could be easily tested by changing the translation in the loop).
Also because of the overlapping the parallelism couldn't be exploited and some tiles were sitting doing nothing and the rest (1/3) were working a lot.
So I think, while memory bandwidth could be a bottleneck, this wasn't the case in this example. The problem is more because of how the graphics HW works and the setup of the test.
I'm not familiar with the iPhone, but if it doesn't have dedicated hardware for handling floating point numbers (I suspect it doesn't) then it'd be faster to use integers whenever possible.
I'm currently developing for Android (which uses OpenGL ES as well) and for instance my vertex array is int instead of float. I can't say how much of a difference it makes, but I guess it's worth a try.
Apple is very tight-lipped about the specific hardware specs of the iPhone, which seems very strange to those of us coming from a console background. But people have been able to determine that the CPU is a 32-bit RISC ARM1176JZF. The good news is that it have a full floating-point unit, so we can continue writing math and physics code the way we do in most platforms.
http://gamesfromwithin.com/?p=239
I asked before about pixel-pushing, and have now managed to get far enough to get noise to show up on the screen. Here's how I init:
CGDataProviderRef provider;
bitmap = malloc(320*480*4);
provider = CGDataProviderCreateWithData(NULL, bitmap, 320*480*4, NULL);
CGColorSpaceRef colorSpaceRef;
colorSpaceRef = CGColorSpaceCreateDeviceRGB();
ir = CGImageCreate(
320,
480,
8,
32,
4 * 320,
colorSpaceRef,
kCGImageAlphaNoneSkipLast,
provider,
NULL,
NO,
kCGRenderingIntentDefault
);
Here's how I render each frame:
for (int i=0; i<320*480*4; i++) {
bitmap[i] = rand()%256;
}
CGRect rect = CGRectMake(0, 0, 320, 480);
CGContextDrawImage(context, rect, ir);
Problem is this is awfully awfully slow, around 5fps. I think my path to publish the buffer must be wrong. Is it even possible to do full-screen pixel-based graphics that I could update at 30fps, without using the 3D chip?
The slowness is almost certainly in the noise generation. If you run this in Instruments you'll probably see that a ton of time is spent sitting in your loop.
Another smaller issue is your colorspace. If you use the screen's colorspace, you'll avoid a colorspace conversion which is potentially expensive.
If you can use CoreGraphics routines for your drawing, you'd be better served by creating a CGLayer for the drawing context instead of creating a new object each time.
The bytesPerRow component is also important for performance. It should be a factor of 32 IIRC. There's some code available link text that shows how to compute it.
And yeah, for raw performance, OpenGL.
I suspect doing 614400 (320*480*4) memory writes, random number generation and making a new object each frame is slowing you down.
Have you tried just writing a static bitmap to screen and seeing how fast that is? Have you perhaps tried profiling the code? Do you also need to make a new CGRect each time?
If you just want to give the effect of randomness, there is probably no need to regenerate the entire bitmap each time.
To my knowledge, OpenGL is supposed to be the fastest way to do graphics on the iPhone. This includes 2D and 3D. A UIView is backed by a core animation layer, which ends up drawing with OpenGL anyway. So why not skip the middle-man.
You can avoid the trip through CGContextDrawImage by assigning your CGImageRef to -[CALayer setContents:], just be sure not to free bitmap while you're still using it.
[[view layer] setContents:(id)ir];
Yes, I know this is old, I stumbled upon it from Google