What's the best way to encode video (with audio) on the iPhone? It looks like QTKit isn't available... so I might have to link with ffmpeg, but ffmpeg doesn't look like it encodes H.264 (judging from their home page.)
If it is possible, I'm also curious how fast I can expect it to perform on the ARM. I imagine it might take minutes to encode a 20sec movie.
Both ffmpeg and mencoder will encode H2.264 videos when combined with x264, but I'd imagine getting it all running iPhone would be an absolute nightmare, let alone the performance of it once you've got it running.
A while ago I wrote an AVI encoder for the iPhone that used raw file I/O. I just started work on a QuickTime encoder that encodes BMP data into a quicktime container. If it is H.264 you want to encode, I would try making a server that uses QTKit and having your app connect to that for conversion.
Related
I receive raw H.264 NALUs from an IP camera (via Live555) and I want to decode them using hardware because FFmpeg is great but it's too slow (the camera sensor is large).
The only solution I see is to write the NALUs to some movie container file such as MPEG-4, and then read and decode that file using an AVAssetReader.
Am I off in the weeds? Is anyone having success decoding H.264 NALUs from a stream? Does anyone have any tips for writing NALUs to an MPEG-4 file? Other ideas?
Like Matt mentioned, there is no direct access to Apple's H264 decoder.
However, I have had success with ffmpeg and h264 decoding. Like you mentioned, I have built ffmpeg with LGPL I was able to decode H264 streams all the way to real-time HD stream with no latency on both ipad and iphone. Nothing fancy is required from ffmpeg, you can find bunch of standard decoding c++ code that will work just fine on iOS. Also, in my case H264 NALUs were delivered via RTP/RTSP in real-time.
Also, if I was you I would run your app through xcode instruments to truly see where you bottleneck is, but I would be highly surprised it is in ffmpeg decoding step.. Hopefully this info helps.
Unfortunately, you cannot do this at present. Feel free to file a radar with Apple about wanting this sort of access to the hardware decoder. It'll certainly be resolved as a duplicate :-). I assume it is for licensing reasons why they can't give this sort of access to the hardware codec.
So, you're going to have to use a software decoder. Please be aware that if you're going to ship to the App Store then you need something with a non-GPL license (unless you want to open source your app as well).
I have a webservice returning .flv file, it has to be played in iphone application, how do i play a .flv (flash file) in iphone?
Does anyone has faced this scenario? Programmatically is it possible to convert to some format and play in iphone?
Thanks.
IPhone doesn't and judging by the Apple official statements won't ever (or at least in the forseeable future) support flash content.
Converting the content to another format on the server side should be easy to do and would allow content playback on an iDevice.
SInce the video is probably already h.264 encoded inside the FLV container, you may want to try FLV Extract on the server to avoid recompression:
http://www.videohelp.com/tools/FLV_Extract
Basically you just need to run it once for each of the videos on the server and keep the results around.
I would recommend setting up your webservice to use something like ffmpeg ( http://www.ffmpeg.org/ ) to convert the .flv file to an mp4 file which can be played directly from the iPhone's web browser.
Pioto and Josaih are on the right track in suggesting that you should convert the video server-side using a tool like FFMpeg. As far as I know there is zero support for flv in any part of iOS, so you'd be unable to transcode it locally. Even if you could, it would make your users angry, since transcoding is a resource-intensive process that would kill their battery life and take a significant amount of time.
So, your solution is to transcode your videos to h.264 server-side. However, I'd caution against transcoding from flv->h.264 if there are any other options available. If you have the original, uncompressed (or at least less-compressed) source video available, you'll get higher-quality video by transcoding that to h.264. Each time lossy compression (eg, squeeze or h.264) is used on a file, you lose some information and quality. If you've ever seen a 3rd or 4th generation copy of a VHS tape, you can understand what I'm getting at.
Once you have a h.264 formatted video, you can play it on iOS. Not sure about the exact details of this.
You may be able to use ffmpeg or something on your server to transcode it to H.264. I'm not so sure you would really want to do that transcoding on the phone. Given Apple's current stance on Flash, this is probably your best option.
For FLV files, what I do is I upload them on Google Drive and watch them from Google Drive app.
I need to convert image sequences(ie,png) to video file in iPhone. How i can convert the images to video.
Regards,
Just ignore bad advice like "use ffmpeg". That would work on the desktop, but the license issue makes including ffmpeg source code in your iPhone app legally questionable. Apple provides a class named AVAssetWriter that you would use in your app to encode a series of images as h.264 stored in a .m4v quicktime container file. While the apple provided logic does work, it is not so easy to actually use and you will need to read quite a lot of documentation to get the code working. If you want to skip implementing it yourself (and likely save yourself 3 or 4 days of work), please consider using my AVAnimator library for iOS as the h.264 encoding logic is already implemented in the class AVAssetWriterConvertFromMaxvid. Once encoded as h.264, the video can be played with the standard player and it is small enough to upload to a remote server.
You are likely going to need something like FFMPEG
I'm having a hard time trying to record something other than linear PCM on the iPhone :-(
The samples I've found (SDK's SpeakHere, Zdziarski's and Sadun's books and the one at trailsinthesand.com) all use linear PCM but I'd like a commonly used compressed format instead (no ima4 or whatever the name is...).
I just cannot figure out how to tweak the sample code to be used with, for example AAC, MP3 or AMR instead. Any suggestions and hints for how to do that are much appreciated!
(Btw, I do not think an MP3-encoder nor AMR-encoder are available due to licensing issues, but AAC does exist, or???)
Edit/Update: I stumbled upon the following text in Apple's "iPhone Application Programming Guide", 2009-01-06, page 137, section: Recording Audio:
"You can record audio in any of the formats listed in “Preferred Audio Formats in iPhone OS” (page 140)", and as preferred audio formats on page 140 are: "For compressed audio when playing one sound at a time, and when you don’t need to play audio simultaneously with the iPod application, use the AAC format packaged in a CAF or m4a file."
Thus, I interpret that as a clear indication that it is indeed, not only possible, but even preferable, to record audio in AAC format wrapped up in a m4a file, which is just what I want. But still, I am not able to achieve that?!
Thanks,
/John
Keep looking at those docs. In "Core Audio Essentials", the section "Core Audio Plug-ins: Audio Units and Codecs" notes that:
iPhone OS contains the recording
codecs listed in Table 2-5. As you can
see, neither MP3 nor AAC recording is
available. This is due to the high CPU
overhead, and consequent battery
drain, of these formats.
Table 2-5 lists several formats, but as the text notes does not include the ones you're looking for. If you want those formats you'll have to bring your own encoder.
It is possible in iPhone 3GS and iPod 2nd generation and above. They have a hardware enconder for AAC.
There is an Apple example project for this that does exactly what you want to do:
iPhoneExtAudioFileConvertTest
A tip for using faac to convert from iPhone-output linear pcm files to aac format:
I found by experimentation that you have to use the -X flag on the command-line version of faac (running on Mac OS X), which "swaps the input bytes." (I guess it changes the endian-ness.)
So, for example, if you audio recorded the file linear.pcm on your iPhone, you could then run:
faac -XP linear.pcm
on your mac to convert that into the aac file linear.aac.
I'm guessing this means you might be able to use faac within your app to do the conversion, if you wanted to, since it's a C library.
Also note that, technically, the files output by iPhone's audio services are usually CAF files, Core Audio Format, which is sort of a wrapper format around a bunch of possible encodings: Apple's docs on CAF. The above command line works fine for me even though linear.pcm starts with the caff header.
Have you tried Lame mp3 encoder? Or faac/faad AAC codecs? You can embed them in ffmpeg for even more audio codecs.
Why don't try TPAACAudioConverter written by the legendary Michael Tyson ?
TPAACAudioConverter is a simple Objective-C class that performs the
conversion of any audio file to an AAC-encoded m4a, asynchronously
with a delegate, or converts any audio provided by a data source class
(which provides for recording straight to AAC).
This is related to my another question
Here I'd like to ask if it is in theory (according to video file formats and codecs, etc) possible to have such scenario:
1) Client on iPhone has a reference to video in flv format. It sends http request to converting "proxy" like http://convproxy.com?source=url_of_original_video.flv by just clicking such link in Safari
2) Converting proxy starts downloading that flv file and converting it to mp4 (which iphone understands) on the fly, returning converted portion as http response, so iPhone can immediately start playing it, before entire flv is downloaded and converted.
I was playing with ffmpeg trying to do such thing, and it indeed converts flv and produces mp4 file, however that mp4 file can not be played until convertion is finished or ffmpeg is stopped. If I just kill ffmpeg process the mp4 file can not be played. If I let it finish or press ctrl-c to stop it, the part that was downloaded and converted can be played. Seems like ffmpeg does some job after it receives stop signal. Is that a necessary part of mp4 format or it can be done differently? I see that iPhone can stream video, by starting playing before the entire file is downloaded to it, so in general it seems like possible scenario for me.
I short words, I can convert flv file to mp4 file, and the question is if I can convert flv stream to mp4 stream.
According to wikipedia, the MP4 container format requires a separate "hint track" to enable streaming. I assume ffmpeg writes this at the end of the conversion. If the iPhone OS requires this track to stream, I don't see a way to stream live video outside of using a different format and having a custom decoder on the iPhone side similar to how the Orb client for iPhone does it.