I have three steps process for my application which display h264 frame on iPhone screen.
decode using ffmpeg.
scale and colorspace conversion (scale to 256 X 256 Opengl ES 1 texture and convert colospace from yuv420p to rgb565 using sws_Scale from ffmpeg).
Render opengl 1 texture to frame buffer to render buffer
after these three step process, I got my picture on iPhone screen.
When I was testing the performance for 720 X 576 resolution frames, I obtain very poor FPS. It is reaching max to 180 milliseconds and hence resulting into 5 to 6 FPS.
Any direction will be grateful.
Related
I can use AVCaptureDevice as background.contents of SCNScene. And it works, but there is one problem with it. I would like to use video format that has resolution 1920x1080, and 60 FPS. But I can clearly see, that different format is used, it is 30 FPS. I am configuring used device before applying it to background, but somehow SCNScene is changing it. SceneView itself works in 60 FPS, but camera preview is a different story. Can I somehow force SCNScene to use video format I choose?
I know I could just add layer with camera preview under SceneView, but I have reasons why this approach is not working correctly, so I need to use this background property od the scene.
Sample project is here: https://www.dropbox.com/s/b820wxlg8voya58/SampleApp.zip?dl=1
In terminal you can clearly see, that after starting SceneView active format for device changes:
Selected format is:
<AVCaptureDeviceFormat: 0x282d58d00 'vide'/'420v' 1280x 720, { 3- 60 fps}, HRSI:4096x2304, fov:58.632, supports vis, max zoom:120.00 (upscales #2.91), AF System:2, ISO:23.0-736.0, SS:0.000013-0.333333>
2018-10-10 14:47:35.009890+0200 SampleApp[6799:1110910] [SceneKit] Error: Could not get pixel buffer (CVPixelBufferRef)
Format after 3 seconds is:
<AVCaptureDeviceFormat: 0x282d58fb0 'vide'/'420v' 1920x1080, { 3- 30 fps}, HRSI:4096x2304, fov:58.632, supports vis, max zoom:16.00 (upscales #1.94), AF System:2, ISO:23.0-736.0, SS:0.000013-0.333333>
I use raspicam library from here. I can change frame rate at src/private/private_impl.cpp file. After the frame rate to 60, I can receive the frame rate 60, but the object size in the image is changed. I attached two images one is captured using 30fps and another one is captured using 60fps.
Why I have bigger object size using 60fps and how can I have normal object size (same as using 30fps)?
The first image is usign 30fps and second image is using 60fps.
According to description here, the higher frame rate modes require cropping on the sensor for 8M pixel camera. At the default 30fps the GPU code will have chosen the 1640x922 mode, so gives full field of view (FOV). Exceed 40fps and it will switch to the cropped 1280x720 mode. In either case the GPU will then resize it to the size you requested. Resize a smaller FOV to the same size and any object in the scene will use more pixels.Can use 5M pixel camera if no cropping is required.
I should use Field of view, zoom or cropping rather than object size is bigger.
It is also possible to keep the image the same size at higher frame rates by explicitly choosing a camera mode that does "binning" (which combines multiple sensor pixels into one image pixel) for both lower- and higher-rate capture. Binning is helpful because it effectively increases the sensitivity of your camera.
See https://www.raspberrypi.org/blog/new-camera-mode-released/ for details when the "new" higher frame rates were announced.
Also, the page in the other answer has a nice picture with the various frame sizes, and a good description of the available camera modes. In particular, modes 4 and higher are binning, starting with 2x2 binning (so 4 sensor pixels contribute to 1 image pixel) and ending with 4x4 (so 16 sensor pixels contribute to 1 image pixel).
Use the sensor_mode parameter to the PiCamera constructor to choose a mode.
I need to read the pixel data from the frame buffer in OpenGL ES 2.0. I know that can be done easily with glReadPixels but since iOS 5 we can use the TextureCached objects for faster reading.
I have implemented the solution proposed by Brad Larson ( I will be always thankful to him, I think he is doing a great job for the community sharing so much knowledge...) in Faster alternative to glReadPixels in iPhone OpenGL ES 2.0
Everything seems to work, I get the proper data and if I compare it with glReadPixels, the data is identical.
My problem came when I measure the performance of this 2 possible solutions (consumed time while retrieving the data).
Here my results :
(framebuffer and texture size 320x480 pixels)
GPUImageProcessingDemo[1252:707] glReadPixels 2750 us
GPUImageProcessingDemo[1252:707] Texture reading 1276 us
GPUImageProcessingDemo[1252:707] glReadPixels 2443 us
GPUImageProcessingDemo[1252:707] Texture reading 1263 us
GPUImageProcessingDemo[1252:707] glReadPixels 2494 us
GPUImageProcessingDemo[1252:707] Texture reading 1375 us
Which seems very interesting since it is almost half of the time needed when using glReadPixels.
The problem is when I change the texture size to something a little bit bigger I get this results:
(framebuffer and texture size 480x620 pixels)
GPUImageProcessingDemo[1077:707] glReadPixels 2407 us
GPUImageProcessingDemo[1077:707] Texture reading 2842 us
GPUImageProcessingDemo[1077:707] glReadPixels 2392 us
GPUImageProcessingDemo[1077:707] Texture reading 3040 us
GPUImageProcessingDemo[1077:707] glReadPixels 2224 us
Does this make sense? Or should I expect to get better results always?
I am newbie to iOS, but I have implemented an FFmpeg-based playback routine on Android. I am planning to do it again on iOS.
It appears that I can use OpenGL ES 1.1 / 2.0 to draw frames from FFmpeg video to the screen. However, OpenGL ES seems difficult. Additionally, I ran into some limits due to the width of the video texture. I had to split the video frame into many images and draw them to the screen to compose the frame.
Is there an easier way to render this video using OpenGL ES on iOS? Is there any other way to draw 2-D video frames to the screen quickly in iOS?
Ah. So you want to render a non-POT source. This can be done without splitting to multiple textures - by creating the closest POT-sized texture, rendering to that and blitting only the part that actually contains the image.
Have a look here for an example (C++). The relevant parts:
//Calculating the texture size
double exp = ceil(log((double)max(Texture.HardwareHeight, Texture.HardwareWidth))/log(2.0));
texsize = min(pow(2, exp), (double)maxSize);
then
//draw the original frame 1:1 into the (larger) texture
float ymax = min(2.0f*((float)Texture.HardwareHeight/(float)texsize) - 1.0f, 1.0f); //clamping is for cards with
float xmax = min(2.0f*((float)Texture.HardwareWidth/(float)texsize) - 1.0f, 1.0f); //smaller max hardware texture than the current frame size
Use these maximum values instead of 1.0 when as texture coordinates when rendering.
I've tried using pngs with gradients as textures in my OpenGL ES based iPhone game. The gradients are not drawn correctly. How can I fix this?
By "not drawn correctly" I mean the gradients are not smooth and seem to degrade to sections of a particular color rather a smooth transition.
The basic problem is having too few bits of RGB in your texture or (less likely) your frame buffer. The PNG file won't be used directly by the graphics hardware -- it must be converted into some internal format. I do not know the OpenGL ES API, but presumably you either hand it the .PNG file directly or you first do some kind of conversion step and hand the converted data to Open GL ES. In either case consult the relevant documentation to ensure that the internal format used is of sufficient depth. For example, a 256 color palettized image will be sufficient as would a 24 bit RGB or 32 bit RGBA format. I strongly suspect your PNG is converted to RGB15 or RGB16 which has only 5 or 6 bits per color component -- not nearly enough to display a smooth gradient.