In iphone i have a mpeg-4 video file, which when played & when i click in the middle or any portion of the video, its goes back & starts from beginning rather continuing from the place where i clicked.
Whereas the same file plays when the same behavior is extorted(ie. play, click in the middle of the video) works fine in quicktime player/safari.
Do someone have any thoughts on this?
The video proabably only has one keyframe at the beginning of the clip.
That means to recreate the image in the middle of the video, the player needs to go back to the first frame and apply the rules from every spanning frame. A good video player on powerful hardware would be able to calculate what the 20,000th frame on the fly essentially by playing the entire video to that point. However, that is quite a lot of calculation, so a less powerful video player (like one on a phone), would not be able to make all of the necessary calculations in a timely manner much faster than real time, so it might simply go back to the beginning.
One solution would be to put key-frames more frequently. That way you have a higher seek precision. Every 1-2 seconds is pretty standard and used by DirectTV (That is why it can sometimes take a second to change the station. Th box can't render the image until it receives a keyframe.)
A little explanation of keyframes...
If you don't know what a keyframe is, you need to understand a little about video compression. An uncompressed video would store the complete image for each frame. So, 30 frames would be the equivalent of jpgs (or other image format).
One of the larges gains in compression comes when, instead of storing the complete image in each frame, you only store the information that was different. For example, if you have a static scene and a ball flies through it, you only have to describe the path of the ball.
There are many rules for this calculation. Some of the rules are philosophically similar to the manner in which traditional animators layered cels above a static background. Other rules more resemble the way William Kentridge modifies the previous frame in his charcoal animation by panning, erasing, or drawing over.
Here is a good explanation of the mpeg-4 standard
Related
for a project I am working on I will need to automatically decide if a video is a VR (360) video, and if so what format it is. Is there any way to tell? I was thinking the metadata but could not find any information on this.
Checking size is pointless, most properly encoded movies are using standard sizes like 1080p (1920x1080), WQHD (2560×1440) or 4K (3840×2160) because of better hardware decoding. To do that they don't have square pixels. Therefore you shouldn't guess anything by ratio.
What you should do, is check the presence of zenith and nadir. That is check for the topmost and bottommost region of image if it is the same color (assuming the most standard equirectangular projection).
This approach will need some adjusting if you have stereoscopy involved. You would have to repeat this procedure for each eye region. As a bonus, you can also deduce some stereoscopy types - for example you could differentiate top-bottom, mono and left-right. Unfortunately you couldn't guess which image is for which eye, so you would have to assume the more common situation where the left eye is usually the top or left one in the image.
There is an RFC for Metadata to describe spherical data in MP4 videos:
https://github.com/google/spatial-media/blob/master/docs/spherical-video-v2-rfc.md
This includes the introduction of a new spherical video header box, svhd, which you can test for the presence of to detect if a video is a VR 360 video.
This is not ubiquitous yet but it does have support from key players like Google, and as you have discovered, something like this is necessary so it seems likely its use will spread.
I know that APNG is an extension of PNG, while MNG is more of its own format (albeit developed by the original PNG developers). MNG is barely supported in any browser at all, while APNG almost only has native support in Firefox (for various backward compatibility- and decoding-related reasons, it seems).
Except all of these behind-the-scenes things, what are the differences between APNG and MNG? Does one have features the other doesn't (for example, storing only parts that are modified instead of always whole frames)? Does one have better performance or file size than the other?
APNG can create a new frame by replacing the entire image or by overlaying or blending a smaller image over part of it. To display a "pong" game you'd need a new image of the ball in each different location. APNG has essentially the same capabilities as animated GIF, but also allowing 24bit RGB and 8-bit alpha.
MNG can do that, plus it can also retrieve an image that was previously defined in the datastream and place it over the previous frame in a new location. To display your "pong" game you'd only need to transmit one image of the ball and use it like a sprite.
Much more detail is available in the specifications:
apng: (https://wiki.mozilla.org/APNG_Specification)
mng: (http://www.libpng.org/pub/mng/spec/mng-lc.html)
I am working on an Iphone application.
I need to do the following: when the user clicks on the "Camera Tab" the camera open inside the view with circle overlays.
I want to apply a filtering algorithm on the camera.
I am looking for the best way to do this. Is there a library that can help?
What I am doing currently:
I am using the OpenCV Library.
I define a timer.
For each timer tick I call cvCaptureFromCam() method from the OpenCV
framework (This will capture the picture with a camera and return
it).
I apply the algorithm on the image captured.
i display the image in a UIImageView
The idea is that on each timer tick I get the image, filter it and put it in the UIImageView. If the timer tick is fast enough it will appear as continuous.
However the cvCaptureFromCam is a little slow and this whole process is taking too much memory.
Any suggestions of a better way is greatly appreciated. Thanks
Anything that's based on CPU-bound processing, such as OpenCV, is probably going to be too slow for live video filtering on current iOS devices. As I state in this answer, I highly recommend looking to OpenGL ES for this.
As mentioned by CSmith, I've written an open source framework called GPUImage for doing this style of GPU-based filtering without having to worry about the underlying OpenGL ES involved. Most of the filters in this framework can be applied to live video at 640x480 at well over the 30 FPS framerate of the iOS camera. I've been gradually adding filters with the goal of replacing all of those present in Core Image, as well as most of the image processing functions of OpenCV. If there's something I'm missing from OpenCV that you need, let me know on the issues page for the project.
Build and run the FilterShowcase example application to see a full listing of the available filters and how they perform on live video sources, and look at the SimplePhotoFilter example to see how you can apply those filters to preview video and photos taken by the camera.
I am building an iphone app that needs to record grayscale video and save it to the camera roll. I'm stumped at how best to approach this.
I am thinking along the following lines:
Use a shader and opengl to transform the video to grayscale
Use AVFoundation (AVAssetWriter with an AVAssetWriterInputPixelBufferAdaptor) to write the video to the file.
My questions are:
Is this the right approach (simplest, best performance)?
If so, what would be the best way to go from opengl output to a CVPixelBufferRef input for the AVAssetWriterInputPixelBufferAdaptor?
If not, what would be a better approach?
Any nudge in the right direction is much appreciated!
In general, I'd agree with this approach. Doing your processing in an OpenGL ES 2.0 shader should be the most performant way of doing video frame alteration like this, but it won't be very simple. Fortunately, you can start from a pre-existing template that already does this.
You can use the sample application I wrote here (and explained here) as a base. I use custom shaders in this example to track colors in an image, but you could easily alter this to convert the video frames to grayscale (I even saw someone do this once). The code for feeding camera video into a texture and processing it could be used verbatim from that sample.
In one of the display options within that application, I render the processed image first to a framebuffer object, then use glReadPixels() to pull the resulting image back into bytes that I can work with on the CPU. You could use this to get the raw image data back after the GPU has processed a frame, then feed those bytes into CVPixelBufferCreateWithBytes() to generate your CVPixelBufferRef for writing to disk.
(Edit: 2/29/2012) As an update to this, I just implemented this kind of video recording in my open source GPUImage framework, so I can comment on the specific performance for the encoding part of this. It turns out that you can capture video from the camera, perform live filtering on it, grab it from OpenGL ES using glReadPixels(), and write that out as live H.264 video in 640x480 frames on an iPhone 4 at 30 FPS (the maximum camera framerate).
There were a few things that I needed to do in order to get this recording speed. You need to make sure that you set your AVAssetWriterInputPixelBufferAdaptor to use kCVPixelFormatType_32BGRA as its color format for input pixel buffers. Then, you'll need to re-render your RGBA scene using a color-swizzling shader to provide BGRA output when using glReadPixels(). Without this color setting, your video recording framerates will drop to 5-8 FPS on an iPhone 4, where with it they are easily hitting 30 FPS. You can look at the GPUImageMovieWriter class source code to see more about how I did this.
Using the GPUImage framework, your above filtering and encoding task can be handled by simply creating a GPUImageVideoCamera, attaching a target of a GPUImageSaturationFilter with the saturation set to 0, and then attaching a GPUImageMovieWriter as a target of that. The framework will handle the OpenGL ES interactions for you. I've done this, and it works well on all iOS devices I've tested.
I have a problem when try to load the big animation with about 54 images (320x480 each image) into CCSpriteFrameCache, I can't use plist for this. How can I make the animation work? At this time, my animation can't work on iPhone 2G, 3G, and iPod.
Thank for your help,
John
You won´t be able to do it...
Consider playing a video or just animating an small portion of the screen.
Your best bet is to determine why the animation has 54 images that are all the width/height of the screen. This is an unnecessary number of images.
Break the animation down:
Is the background 'static' (does it move around, change constantly, etc?)
If it moves around a bit, but is really part of a much larger "canvas" then simply export out the entire background canvas and perform the movements yourself using the Cocos2D Actions available to you (CCMoveTo, CCJumpTo, CCDelayTime, CCSequence, etc)
What in the animation moves around, and how does it move around?
Can it be broken into much smaller bits and the frames for the various "characters"
or "movable objects" within the scene be exported out onto a sprite sheet (saved out
via Zwoptex?)
A good animation sequence should be a series of much smaller images, all working together in unison to create the final "animation sequence".
If you break it down, I wouldn't be surprised if you were able to reduce your 54 images at 320x480 each down to a handful of 512x512 spritesheets (ala Zwoptex).
If your having trouble breaking it down, I would be available to look at the final animation and help you determine what could be minimized to reduce the overhead.