Using GPUImage with another EAGLContext - iphone

I'm trying to use GPUImage with my OpenGL based photo editing app. The problem is that it seems like the two OpenGL contexts conflict which means that GPUImage doesn't function properly.
I've tried adding this before using GPUImage:
[EAGLContext setCurrentContext:[GPUImageContext sharedImageProcessingContext].context];
But it's not helping. The problem that I'm seeing is that when I try to apply a filter to an image, it applies the filter to the previously filtered image i.e.
1) Filter image 1 - output image 2
2) Reload my OpenGL image canvas
3) Filter image 2 - output image 2 (it should be image 3 i.e. the filtered version of image 2)
This happens the first three times I apply the filter, then it starts working correctly. Is there a way to reset the GPUImage context? Or to dispose of the GPUImage context singleton and start afresh?

First, you'll want to make sure that your OpenGL-based drawing application is rendering its scene to a texture and that you're using a texture input (if passing into GPUImage). GPUImage needs to work with textures, not render buffers themselves. If going out from GPUImage, use a texture output and its rendering callback block to know when rendering has finished.
To share content between GPUImage and another OpenGL context, you need to make GPUImage's context part of the same share group as your own OpenGL context. It's probably best not to share the GPUImage context directly, since it performs operation internally on a background dispatch queue and you can't guarantee thread safety if rendering to that in your own code.
The CubeExample sample application on the iOS side shows how to do this for OpenGL ES. In particular, it uses the following line:
context = [[EAGLContext alloc] initWithAPI:kEAGLRenderingAPIOpenGLES2 sharegroup:[[[GPUImageContext sharedImageProcessingContext] context] sharegroup]];
to set up the share group for the OpenGL ES context used for the non-GPUImage rendering there. I'd have to look up what the equivalent is for the Mac and desktop OpenGL, though.

Related

How can I load a Gigapixel image as a material in SceneKit?

I’m trying to create an AR image to project on a wall from a Gigapixel image. Obviously Xcode crashes if I try to load the image as a material. Is there an efficient way to load only parts of the image that the user is looking at?
I'm using Swift 4.
This may not do exactly what you want, and you might need to roll-your-own way of parsing and passing data between Core Animation and SceneKit, but this is native, and is designed to handle large images and texture data sources, and feed them out asynchronously, and/or on a demand based basis:
https://developer.apple.com/documentation/quartzcore/catiledlayer

Apply custom camera filters on live camera preview - Swift

I'm looking to make a native iPhone iOS application in Swift 3/4 which uses the live preview of the back facing camera and allows users to apply filters like in the built in Camera app. The idea was for me to create my own filters by adjusting Hue/ RGB/ Brightness levels etc. Eventually I want to create a HUE slider which allows users to filter for specific colours in the live preview.
All of the answers I came across for a similar problem were posted > 2 years ago and I'm not even sure if they provide me with the relevant, up-to-date solution I am looking for.
I'm not looking to take a photo and then apply a filter afterwards. I'm looking for the same functionality as the native Camera app. To apply the filter live as you are seeing the camera preview.
How can I create this functionality? Can this be achieved using AVFoundation? AVKit? Can this functionality be achieved with ARKit perhaps?
Yes, you can apply image filters to the camera feed by capturing video with the AVFoundation Capture system and using your own renderer to process and display video frames.
Apple has a sample code project called AVCamPhotoFilter that does just this, and shows multiple approaches to the process, using Metal or Core Image. The key points are to:
Use AVCaptureVideoDataOutput to get live video frames.
Use CVMetalTextureCache or CVPixelBufferPool to get the video pixel buffers accessible to your favorite rendering technology.
Draw the textures using Metal (or OpenGL or whatever) with a Metal shader or Core Image filter to do pixel processing on the CPU during your render pass.
BTW, ARKit is overkill if all you want to do is apply image processing to the camera feed. ARKit is for when you want to know about the camera’s relationship to real-world space, primarily for purposes like drawing 3D content that appears to inhabit the real world.

Apply a filter algorithm for each frame of the camera

I am working on an Iphone application.
I need to do the following: when the user clicks on the "Camera Tab" the camera open inside the view with circle overlays.
I want to apply a filtering algorithm on the camera.
I am looking for the best way to do this. Is there a library that can help?
What I am doing currently:
I am using the OpenCV Library.
I define a timer.
For each timer tick I call cvCaptureFromCam() method from the OpenCV
framework (This will capture the picture with a camera and return
it).
I apply the algorithm on the image captured.
i display the image in a UIImageView
The idea is that on each timer tick I get the image, filter it and put it in the UIImageView. If the timer tick is fast enough it will appear as continuous.
However the cvCaptureFromCam is a little slow and this whole process is taking too much memory.
Any suggestions of a better way is greatly appreciated. Thanks
Anything that's based on CPU-bound processing, such as OpenCV, is probably going to be too slow for live video filtering on current iOS devices. As I state in this answer, I highly recommend looking to OpenGL ES for this.
As mentioned by CSmith, I've written an open source framework called GPUImage for doing this style of GPU-based filtering without having to worry about the underlying OpenGL ES involved. Most of the filters in this framework can be applied to live video at 640x480 at well over the 30 FPS framerate of the iOS camera. I've been gradually adding filters with the goal of replacing all of those present in Core Image, as well as most of the image processing functions of OpenCV. If there's something I'm missing from OpenCV that you need, let me know on the issues page for the project.
Build and run the FilterShowcase example application to see a full listing of the available filters and how they perform on live video sources, and look at the SimplePhotoFilter example to see how you can apply those filters to preview video and photos taken by the camera.

Generating graphics at runtime with Cocos2D - How to display?

I'm trying to create dynamic graphics for my game, which I'm building with Cocos2D. The graphics generation will occur at predictable, finite points, such as level loading. I'm having a hard time figuring out how to actually draw this at runtime. From what I can tell, the easiest way would be to draw into a PNG file at runtime and then load an AtlasSprite based on the PNG file, but I can't seem to figure out if this is indeed the best way or how to go about doing it. Any suggestions?
I'm not sure how Cocos2D loads Sprites or Atlases so this is a more general answer.
It might be worth taking a look at the Texture2D class that comes with the old CrashLanding example app. It uses a bitmap graphics context to generate a texture of a string for drawing with OpenGL. The code uses the CGBitmapContextCreate function to create a context. You can draw whatever you want onto it.
Then once you've finished drawing, you can either save the file as a PNG or you can call glTexImage2D on the data to use it with OpenGL.
There's more information about it in the Graphics and Drawing
documentation, specifically the section: Creating and Drawing Images.
Edit: It looks like Cocos2D comes with Texture2D so you should be in good shape. Check out the initWithString method here.

Performance and background images for OpenGL ES/iPhone

I'm developing a 2D game for the iPhone using OpenGL ES and I'd like to use a 320x480 bitmapped image as a persistent background.
My first thought was to create a 320x480 quad and then map a texture onto it that represents the background. So... I created a 512x512 texture with a 320x480 image on it. Then I mapped that to the 320x480 quad.
I draw this background every frame and then draw animated sprites on top of it. This works fine except that the drawing of all of these objects (background + sprites) is too slow.
I did some testing and discovered that my slowdown is in the pixel pipeline. Not surprisingly, the large background image is the main culprit. To prove this, I removed the background draw and everything else rendered very fast.
I am looking for advice on how to keep my background and also improve performance.
Here's some more info:
1) I am currently testing on the Simulator (still waiting on Apple for the license)
2) The background is a PVR texture squeezed down to 128k
3) I had hoped that there might be a way to cache this background into a color buffer but haven't had any luck with that. that may be due to my inexperience with OpenGL ES or it just might be a stupid idea that won't work :)
4) I realize that the entire background does not always have to refresh, just the parts that have been drawn over by the moving sprites. I started to look into techniques for refreshing (as necessary) parts of the the background either as separate textures or with a scissor box, however this seems less than elegant.
Any tips/advice would be greatly appreciated...
Thank you.
Do not do performance testing on the simulator. Ever!
The differences to the real hardware are huge. In both directions.
If you draw the background every frame:
Do not clear the framebuffer. The background will overdraw the whole thing anyway.
Do you really need a background texture ?
What about using a color gradient via vertex colors ?
Try using the 2bit mode for the texture.
Turn of all render steps that you do not need for the background.
E.g.: Lighting, Blending, Depth-Test, ...
If you could post some of your drawing code it would be a lot easier to help you.
If you're making a 2D game, is there any reason you aren't using an existing library? Specifically, the cocos2d for iPhone may be worth your time. I can't answer your question about how to fix the issue doing it all yourself, but I can say that I've done exactly what you're talking about (having one full screen background with sprites on top) with cocos2d and it works great. (Assuming 60 fps is fast enough for you.) You may have your reasons for doing it yourself, but if you can, I would highly suggest at least doing a quick prototype with cocos2d and seeing if that doesn't help you along. (Details and source for the iPhone version are here: http://code.google.com/p/cocos2d-iphone/)
Thanks to everyone who provided info on this. All of the advice helped out in one way or another.
However, I wanted to make it clear that the main issue here turned out to be the behavior of simulator itself (as implied by Andreas in his response). Once I was able to get the application on the device, it performed much, much better. I mention this because, prior to developing my game, I had seen a lot of posts that indicated that the device was much slower than the simulator. This might be true in some instances (e.g. general application logic) but in my experience, animation (particularly 3d transformations) are much faster on the device.
I dont have much experience with OpenGL ES, but this problem occurs generally.
Your idea about the 'color buffer' is good intuition, essentially you want to be storing your background as a frame buffer and loading it directly onto your rendering buffer before drawing the foreground.
In OpenGL this is fairly straight forward with Frame Buffer Objects (FBO's). Unfortunatly I dont think OpenGL ES supports them, but it might give you somewhere to start looking.
you may want to try using VBOs (Vertex Buffer Objects) and see if that speeds up things. Tutorial is here
In addition, I just saw, that since OpenGL ES v1.1, there is a function called glDrawTex (Draw Texture) that is designed for
fast rendering of background paintings, bitmapped font glyphs, and 2D framing elements in games
You could use frame buffer objects similar to the GLPaint example from Apple.
Use a texture atlas to minimize the number of draw calls you make. You can use glTexCoordPointer for setting your texture coordinates that maps each image to its correct position. Remember to set your vertex buffer too. Ideally one draw call will render your entire 2D scene.
Avoid enabling/disabling states where possible.