I'm developing an video player for iPhone. I'm using ffmpeg libraries to decode frames of video and I'm using opengl 2.0 to render the frames to the screen.
But my render method is very slowly.
A user told me:
iOS 5 includes a new way to do this fast. The trick is to use AVFoundation and link a Core Video pixel buffer directly to an OpenGL texture.
My problem now is that my video player send to render method a uint8_t* type that I use then with glTexSubImage2D.
But if I want to use CVOpenGLESTextureCacheCreateTextureFromImage I need a CVImageBufferRef with the frame.
The question is: How I can create CVImageBufferRef from uint8_t buffer?
This is my render method:
- (void) render: (uint8_t*) buffer
{
NSLog(#"render");
[EAGLContext setCurrentContext:context];
glBindFramebuffer(GL_FRAMEBUFFER, defaultFramebuffer);
glViewport(0, 0, backingWidth, backingHeight);
glClearColor(0.0f, 0.0f, 0.0f, 1.0f);
glClear(GL_COLOR_BUFFER_BIT | GL_DEPTH_BUFFER_BIT);
// OpenGL loads textures lazily so accessing the buffer is deferred until draw; notify
// the movie player that we're done with the texture after glDrawArrays.
glTexSubImage2D(GL_TEXTURE_2D, 0, 0, 0, mFrameW, mFrameH, GL_RGB,GL_UNSIGNED_SHORT_5_6_5, buffer);
glDrawArrays(GL_TRIANGLE_STRIP, 0, 4);
[moviePlayerDelegate bufferDone];
glBindRenderbuffer(GL_RENDERBUFFER, colorRenderbuffer);
[context presentRenderbuffer:GL_RENDERBUFFER];
}
Thanks,
I am trying to do something similar.
Apparently, you need to create CVPixelBufferRef, and substitute it for CVImageBufferRef. I.e., you first create CVPixelBufferRef, as described at here (for download), and then get access to pixel buffer:
CVPixelBufferLockBaseAddress(renderTarget, 0);
_rawBytesForImage = (GLubyte *)CVPixelBufferGetBaseAddress(renderTarget);
(Code not mine).
For an actual working example that shows how video data can be passed directly to an OpenGL view see my blog post on the subject. The problem with looking at a series of "code pieces" around online is that you will not find actual complete working examples for iOS.
Related
For my non-app store app, I've been using the private framework Core Surface to draw directly to the screen of the iPhone. However, this can be rather slow on older devices because it heavily uses the CPU to do its drawing. To fix this, I've decided to try to use OpenGLES to render the pixels to the screen.
Currently (and I have no way of changing this), I have a reference to an unsigned short * variable called BaseAddress, and essential 3rd party code accesses BaseAddress and updates it with the new pixel data.
I've set up a GLKViewController, and implemented the viewDidLoad as follows:
- (void)viewDidLoad {
[super viewDidLoad];
self.context = [[EAGLContext alloc] initWithAPI:kEAGLRenderingAPIOpenGLES2];
if (!self.context) {
NSLog(#"Failed to create ES context");
}
[EAGLContext setCurrentContext:self.context];
GLKView *view = (GLKView *)self.view;
view.context = self.context;
glGenBuffers(1, &screenBuffer);
glBindBuffer(GL_ARRAY_BUFFER, screenBuffer);
glBufferData(GL_ARRAY_BUFFER, sizeof(BaseAddress), BaseAddress, GL_DYNAMIC_DRAW);
}
where screenBuffer is an instance variable. In the glkView:drawInRect: method I have the following:
- (void)glkView:(GLKView *)view drawInRect:(CGRect)rect {
glDrawElements(GL_ARRAY_BUFFER, sizeof(BaseAddress)/sizeof(BaseAddress[0]), GL_UNSIGNED_SHORT, BaseAddress);
}
Unfortunately, only a black screen appears when I run the app. If I go back to using Core Surface, the app works fine. So basically, how can I draw the pixels to the screen using OpenGLES?
I think that it might be best to use a texture and for your case I'd try to find some older ES1 template for iOS devices. Basically what you need is a frame buffer and a color buffer made from your UIView layer:
glGenFramebuffers(1, &viewFramebuffer);
glBindFramebuffer(GL_FRAMEBUFFER, viewFramebuffer);
glGenRenderbuffers(1, &viewColorBuffer);
glBindRenderbuffer(GL_RENDERBUFFER, viewColorBuffer);
[context renderbufferStorage:GL_RENDERBUFFER fromDrawable:(CAEAGLLayer*)self.layer];
glFramebufferRenderbuffer(GL_FRAMEBUFFER, GL_COLOR_ATTACHMENT0, GL_RENDERBUFFER, viewColorBuffer);
glGetRenderbufferParameteriv(GL_RENDERBUFFER, GL_RENDERBUFFER_WIDTH, &backingWidth);
glGetRenderbufferParameteriv(GL_RENDERBUFFER, GL_RENDERBUFFER_HEIGHT, & backingHeight);
As for projection matrix I suggest you use glOrthof(.0f, backingWidth, backingHeight, .0f, -1.0f, 1.0f); that will make your GL coordinates same as your view coordinates.
Next on some initialization generate a texture, bind it and give it dimensions of power of 2 (textureWidth = 1; while(textureWidth < backingWidth) textureWidth = textureWidth << 1;) and pass it NULL for data pointer (all in function "glTexImage2D")
Then generate vertex array for a square same as texture, from (0,0) to (textureWidth, textureHeight) and texture coordinates from (0,0) to (1,1)
When you get the data to your pointer and are ready to be pushed to texture use glTexSubImage2D to update it: You can update only a segment of a texture if you get data for it or to update a whole screen use rect (0,0,screenWidth, screenHeight)
Now just draw those 2 triangles with your texture..
Note that there is a limit on texture size: most active iOS devices 1<<11 (2048) iPad3 1<<12
Do not forget to set texture parameters when creating it: glTexParameteri
Do check for retina display and set content scale of CAEAGLLayer to 2 if needed
When I try to attach a texture to a framebuffer, glCheckFramebufferStatus reports GL_FRAMEBUFFER_UNSUPPORTED for certain texture sizes. I've tested on both a 2nd and 4th generation iPod Touch. The sizes of texture that fail are not identical between the two models.
Here are some interesting results:
2nd generation - 8x8 failed, 16x8 failed, but 8x16 succeeded!
4th generation - 8x8 succeeded, 8x16 succeeded, but 16x8 failed!
Here's some code I used to test attaching textures of different sizes:
void TestFBOTextureSize(int width, int height)
{
GLuint framebuffer, texture;
// Create framebuffer
glGenFramebuffersOES(1, &framebuffer);
glBindFramebufferOES(GL_FRAMEBUFFER_OES, framebuffer);
// Create texture
glGenTextures(1,&texture);
glBindTexture(GL_TEXTURE_2D,texture);
glTexImage2D(GL_TEXTURE_2D, 0, GL_RGBA, width, height, 0, GL_RGBA, GL_UNSIGNED_BYTE, NULL);
glBindTexture(GL_TEXTURE_2D,0);
// Attach texture to framebuffer
glFramebufferTexture2DOES(GL_FRAMEBUFFER_OES, GL_COLOR_ATTACHMENT0_OES, GL_TEXTURE_2D, texture, 0);
GLenum error = glGetError();
GLenum status = glCheckFramebufferStatusOES(GL_FRAMEBUFFER_OES);
if (status==GL_FRAMEBUFFER_COMPLETE_OES)
NSLog(#"%dx%d Succeeded!",width,height,status);
else
NSLog(#"%dx%d Failed: %x %x %d %d",width,height,status,error,texture,framebuffer);
// Cleanup
glFramebufferTexture2DOES(GL_FRAMEBUFFER_OES, GL_COLOR_ATTACHMENT0_OES, GL_TEXTURE_2D, 0, 0);
glDeleteTextures(1, &texture);
glBindFramebufferOES(GL_FRAMEBUFFER_OES, 0);
glDeleteFramebuffersOES(1, &framebuffer);
}
void TestFBOTextureSizes()
{
int width,height;
for (width=1; width<=1024; width<<=1)
{
for (height=1; height<=1024; height<<=1)
TestFBOTextureSize(width,height);
}
}
It seems that as long as both dimensions are at least 16 pixels then everything works ok on both devices. The thing that bothers me, though, is that I haven't seen anything written about texture size requirements for attaching to a framebuffer object. One solution, for now, would be to restrict my texture sizes to be at least 16 pixels, but might this break in the future or already be broken on some device I haven't tried? I could also perform this test code at startup in order to dynamically figure out which texture sizes are allowed, but that seems a bit hokey.
I have experienced similar problem, when I'm trying to render to texture with size 480x320 (full screen w/o resolution scale) on iPod touch 4. When I call glCheckFramebufferStatus() it returns GL_FRAMEBUFFER_UNSUPPORTED. My code:
glGenTextures(1, &texture);
glBindTexture(GL_TEXTURE_2D, texture);
glTexImage2D(GL_TEXTURE_2D, 0, GL_RGB, 480, 320, 0, GL_RGB, GL_UNSIGNED_SHORT_5_6_5, 0);
glBindTexture(GL_TEXTURE_2D, 0);
glGenFramebuffers(1, &frameBuffer);
glBindFramebuffer(GL_FRAMEBUFFER, frameBuffer);
glFramebufferTexture2D(GL_FRAMEBUFFER, GL_COLOR_ATTACHMENT0, GL_TEXTURE_2D, texture, 0);
GLenum status = glCheckFramebufferStatus(GL_FRAMEBUFFER);
if (status != GL_FRAMEBUFFER_COMPLETE) {
// report error
}
Investigating this problem I have found that GL_TEXTURE_2D has to be a valid OpenGL ES object if we want it to use in render-to-texture mechanism. This means texture should be ready for bound and use. So to fix an error I have to set some texture parameters. Because I use non-POT texture I have to set GL_TEXTURE_WRAP_ to GL_CLAMP_TO_EDGE (default value is GL_REPEAT) and GL_TEXTURE_MIN_FILTER to GL_NEAREST or GL_LINEAR (default value is GL_NEAREST_MIPMAP_LINEAR) to use this texture.
I couldn't find what's the problem with 16x8, but 16x9 and 17x8 works fine if this parameters are set. I hope this information will be helpful for you.
I am writing a GLPaint-esque drawing application for the iPad, however I have hit a stumbling block. Specifically, I am trying to implement two things at the moment:
1) A background image that can be drawn onto.
2) The ability to draw temporary shapes, e.g. you might draw a line, but the final shape would only be committed once the finger has lifted.
For the background image, I understand the idea is to draw the image into a VBO and draw it right before every line drawing. This is fine, but now I need to add the ability to draw temporary shapes... with kEAGLDrawablePropertyRetainedBacking set to YES (as in GLPaint) the temporary are obviously not temporary! Turning the retained backing property to NO works great for the temporary objects, but now my previous freehand lines aren't kept.
What is the best approach here? Should I be looking to use more than one EAGLLayer? All the documentation and tutorials I've found seem to suggest that most things should be possible with a single layer. They also say that retained backing should pretty much always be set to NO. Is there a way to work my application in such a configuration? I tried storing every drawing point into a continually expanding vertex array to be redrawn each frame, but due to the sheer number of sprites being drawn this isn't working.
I would really appreciate any help on this one, as I've scoured online and found nothing!
I've since found the solution to this problem. The best way appears to be to use custom framebuffer objects and render-to-texture. I hadn't heard of this before asking the question, but it looks like an incredibly useful tool for the OpenGLer's toolkit!
For those that may be wanting to do something similar, the idea is that you create a FBO and attach a texture to it (instead of a renderbuffer). You can then bind this FBO and draw to it like any other, the only difference being that the drawings are rendered off-screen. Then all you need to do to display the texture is to bind the main FBO and draw the texture to it (using a quad).
So for my implementation, I used two different FBOs with a texture attached to each - one for the "retained" image (for freehand drawing), and the other for the "scratch" image (for temporary drawings). Each time a frame is rendered, I first draw a background texture (in my case I just used the Texture2D class), then draw the retained texture, and finally the scratch texture if required. When drawing a temporary shape everything is rendered to the scratch texture, and this is cleared at the start of every frame. Once it is finished, the scratch texture is drawn to the retained texture.
Here are a few snippets of code that might be of use to somebody:
1) Create the framebuffers (I have only shown a couple here to save space!):
// ---------- DEFAULT FRAMEBUFFER ---------- //
// Create framebuffer.
glGenFramebuffersOES(1, &viewFramebuffer);
glBindFramebufferOES(GL_FRAMEBUFFER_OES, viewFramebuffer);
// Create renderbuffer.
glGenRenderbuffersOES(1, &viewRenderbuffer);
glBindRenderbufferOES(GL_RENDERBUFFER_OES, viewRenderbuffer);
// Get renderbuffer storage and attach to framebuffer.
[context renderbufferStorage:GL_RENDERBUFFER_OES fromDrawable:layer];
glFramebufferRenderbufferOES(GL_FRAMEBUFFER_OES, GL_COLOR_ATTACHMENT0_OES, GL_RENDERBUFFER_OES, viewRenderbuffer);
glGetRenderbufferParameterivOES(GL_RENDERBUFFER_OES, GL_RENDERBUFFER_WIDTH_OES, &backingWidth);
glGetRenderbufferParameterivOES(GL_RENDERBUFFER_OES, GL_RENDERBUFFER_HEIGHT_OES, &backingHeight);
// Check for completeness.
status = glCheckFramebufferStatusOES(GL_FRAMEBUFFER_OES);
if (status != GL_FRAMEBUFFER_COMPLETE_OES) {
NSLog(#"Failed to make complete framebuffer object %x", status);
return NO;
}
// Unbind framebuffer.
glBindFramebufferOES(GL_FRAMEBUFFER_OES, 0);
// ---------- RETAINED FRAMEBUFFER ---------- //
// Create framebuffer.
glGenFramebuffersOES(1, &retainedFramebuffer);
glBindFramebufferOES(GL_FRAMEBUFFER_OES, retainedFramebuffer);
// Create the texture.
glColor4f(0.0f, 0.0f, 0.0f, 0.0f);
glGenTextures(1, &retainedTexture);
glEnable(GL_TEXTURE_2D);
glBindTexture(GL_TEXTURE_2D, retainedTexture);
glTexImage2D(GL_TEXTURE_2D, 0, GL_RGBA, 1024, 1024, 0, GL_RGBA, GL_UNSIGNED_BYTE, NULL);
glTexParameteri(GL_TEXTURE_2D, GL_TEXTURE_MIN_FILTER, GL_LINEAR);
glTexParameteri(GL_TEXTURE_2D, GL_TEXTURE_MAG_FILTER, GL_LINEAR);
glBindTexture(GL_TEXTURE_2D, 0);
// Attach the texture as a renderbuffer.
glFramebufferTexture2DOES(GL_FRAMEBUFFER_OES, GL_COLOR_ATTACHMENT0_OES, GL_TEXTURE_2D, retainedTexture, 0);
// Check for completeness.
status = glCheckFramebufferStatusOES(GL_FRAMEBUFFER_OES);
if (status != GL_FRAMEBUFFER_COMPLETE_OES) {
NSLog(#"Failed to make complete framebuffer object %x", status);
return NO;
}
// Unbind framebuffer.
glBindFramebufferOES(GL_FRAMEBUFFER_OES, 0);
2) Draw to the render-to-texture FBO:
// Ensure that we are drawing to the current context.
[EAGLContext setCurrentContext:context];
glBindFramebufferOES(GL_FRAMEBUFFER_OES, retainedFramebuffer);
glViewport(0, 0, 1024, 1024);
// DRAWING CODE HERE
3) Render the various textures to the main FBO, and present:
glBindFramebufferOES(GL_FRAMEBUFFER_OES, viewFramebuffer);
glViewport(0, 0, backingWidth, backingHeight);
glClearColor(1.0f, 1.0f, 1.0f, 1.0f); // Clear to white.
glClear(GL_COLOR_BUFFER_BIT);
glEnableClientState(GL_VERTEX_ARRAY);
glEnableClientState(GL_TEXTURE_COORD_ARRAY);
[self drawBackgroundTexture];
[self drawRetainedTexture];
[self drawScratchTexture];
glDisableClientState(GL_VERTEX_ARRAY);
glDisableClientState(GL_TEXTURE_COORD_ARRAY);
glBindRenderbufferOES(GL_RENDERBUFFER_OES, viewRenderbuffer);
[context presentRenderbuffer:GL_RENDERBUFFER_OES];
For example, drawing drawing the retained texture using [self drawRetainedTexture] would use the following code:
// Bind the texture.
glBindTexture(GL_TEXTURE_2D, retainedTexture);
// Destination coords.
GLfloat retainedVertices[] = {
0.0, backingHeight, 0.0,
backingWidth, backingHeight, 0.0,
0.0, 0.0, 0.0,
backingWidth, 0.0, 0.0
};
// Source coords.
GLfloat retainedTexCoords[] = {
0.0, 1.0,
1.0, 1.0,
0.0, 0.0,
1.0, 0.0
};
// Draw the texture.
glVertexPointer(3, GL_FLOAT, 0, retainedVertices);
glTexCoordPointer(2, GL_FLOAT, 0, retainedTexCoords);
glDrawArrays(GL_TRIANGLE_STRIP, 0, 4);
// Unbind the texture.
glBindTexture(GL_TEXTURE_2D, 0);
A lot of code, but I hope that helps somebody. It certainly had me stumped for a while!
I remember running into this problem when I started using OpenGL in OS X. Eventually I solved it, but I think that was just by using glut and c++ instead of Objective-C...
The lines of code I have in init for the ES1Renderer are as follows:
glEnable(GL_DEPTH_TEST);
glDepthFunc(GL_LEQUAL);
Then in the render method, I have this:
glClear(GL_COLOR_BUFFER_BIT | GL_DEPTH_BUFFER_BIT);
I assume I'm missing something specific to either the iPhone or ES. What other steps are required to enable the depth test?
Thanks
The instructions are here, if anyone else has this problem. The code is also below:
glGenRenderbuffersOES(1, &depthRenderbuffer);
glBindRenderbufferOES(GL_RENDERBUFFER_OES, depthRenderbuffer);
glRenderbufferStorageOES(GL_RENDERBUFFER_OES, GL_DEPTH_COMPONENT16_OES, 320, 480);
glFramebufferRenderbufferOES(GL_FRAMEBUFFER_OES, GL_DEPTH_ATTACHMENT_OES, GL_RENDERBUFFER_OES, depthRenderbuffer);
GLenum status = glCheckFramebufferStatusOES(GL_FRAMEBUFFER_OES) ;
if(status != GL_FRAMEBUFFER_COMPLETE_OES) {
NSLog(#"failed to make complete framebuffer object %x", status);
}
You need to allocate the depth buffer itself. Allocate a new renderbuffer with the internal format DEPTH_COMPONENT16 or DEPTH_COMPONENT24, and attach it to the framebuffer object.
#define USE_DEPTH_BUFFER 1 if you're using the OpenGL ES project template. This sets up a depth buffer somewhere in EAGLView.m.
I'm trying to render to a texture, then draw that texture to the screen using OpenGL ES on the iPhone. I'm using this question as a starting point, and doing the drawing in a subclass of Apple's demo EAGLView.
Instance variables:
GLuint textureFrameBuffer;
Texture2D * texture;
To initialize the frame buffer and texture, I'm doing this:
GLint oldFBO;
glGetIntegerv(GL_FRAMEBUFFER_BINDING_OES, &oldFBO);
// initWithData results in a white image on the device (works fine in the simulator)
texture = [[Texture2D alloc] initWithImage:[UIImage imageNamed:#"blank320.png"]];
// create framebuffer
glGenFramebuffersOES(1, &textureFrameBuffer);
glBindFramebufferOES(GL_FRAMEBUFFER_OES, textureFrameBuffer);
// attach renderbuffer
glFramebufferTexture2DOES(GL_FRAMEBUFFER_OES, GL_COLOR_ATTACHMENT0_OES, GL_TEXTURE_2D, texture.name, 0);
if(glCheckFramebufferStatusOES(GL_FRAMEBUFFER_OES) != GL_FRAMEBUFFER_COMPLETE_OES)
NSLog(#"incomplete");
glBindFramebufferOES(GL_FRAMEBUFFER_OES, oldFBO);
Now, if I simply draw my scene to the screen as usual, it works fine:
glClear(GL_COLOR_BUFFER_BIT | GL_DEPTH_BUFFER_BIT);
// draw some triangles, complete with vertex normals
[contentDelegate draw];
[self swapBuffers];
But, if I render to 'textureFrameBuffer', then draw 'texture' to the screen, the resulting image is upside down and "inside out". That is, it looks as though the normals of the 3d objects are pointing inward rather than outward -- the frontmost face of each object is transparent, and I can see the inside of the back face. Here's the code:
GLint oldFBO;
glGetIntegerv(GL_FRAMEBUFFER_BINDING_OES, &oldFBO);
glBindFramebufferOES(GL_FRAMEBUFFER_OES, textureFrameBuffer);
glClear(GL_COLOR_BUFFER_BIT | GL_DEPTH_BUFFER_BIT);
// draw some polygons
[contentDelegate draw];
glBindFramebufferOES(GL_FRAMEBUFFER_OES, oldFBO);
glClear(GL_COLOR_BUFFER_BIT | GL_DEPTH_BUFFER_BIT);
glEnableClientState(GL_TEXTURE_COORD_ARRAY);
glEnable(GL_TEXTURE_2D);
glColor4f(1, 1, 1, 1);
[texture drawInRect:CGRectMake(0, 0, 320, 480)];
glDisableClientState(GL_TEXTURE_COORD_ARRAY);
glDisable(GL_TEXTURE_2D);
[self swapBuffers];
I can flip the image rightside-up easily enough by reordering the (glTexCoordPointer) coordinates accordingly (in Texture2D's drawInRect method), but that doesn't solve the "inside-out" issue.
I tried replacing the Texture2D texture with a manually created OpenGL texture, and the result was the same. Drawing a Texture2D loaded from a PNG image works fine.
As for drawing the objects, each vertex has a unit normal specified, and GL_NORMALIZE is enabled.
glVertexPointer(3, GL_FLOAT, 0, myVerts);
glNormalPointer(GL_FLOAT, 0, myNormals);
glDrawArrays(GL_TRIANGLES, 0, numVerts);
Everything draws fine when it's rendered to the screen; GL_DEPTH_TEST is enabled and is working great.
Any suggestions as to how to fix this? Thanks!
The interesting part of this is that you're seeing a different result when drawing directly to the backbuffer. Since you're on the iPhone platform, you're always drawing to an FBO, even when you're drawing to the backbuffer.
Make sure that you have a depth buffer attached to your offscreen FBO. In your initialization code, you might want to add the following snippet right after the glBindFramebufferOES(...).
// attach depth buffer
GLuint depthRenderbuffer;
glGenRenderbuffersOES(1, &depthRenderbuffer);
glBindRenderbufferOES(GL_RENDERBUFFER_OES, depthRenderbuffer);
glRenderbufferStorageOES(GL_RENDERBUFFER_OES, GL_DEPTH_COMPONENT16_OES, width, height);
glFramebufferRenderbufferOES(GL_FRAMEBUFFER_OES, GL_DEPTH_ATTACHMENT_OES, GL_RENDERBUFFER_OES, depthRenderbuffer);