I want to start with an audio file of a modest filesize, and finish with an array of unsigned chars that can be loaded into OpenAL with alBufferData. My trouble is the steps that happen in the middle.
I thought AAC would be the way to go, but according to Apple representative Rincewind (circa 12/08):
Currently hardware assisted compression formats are not supported for decode on iPhone OS. These formats are AAC, MP3 and ALAC.
Using ExtAudioFile with a client format set generates PERM errors, so he's not making things up.
So, brave knowledge-havers, what are my options here? Package the app with .wav's and just suck up having a massive download? Write my own decoder?
Any links to resources or advice you might have would be greatly appreciated.
Offline rendering of compressed audio is now possible, see QA1562.
While Vorbis and the others suggested are good, they can be fairly slow on the iPhone as there is no hardware acceleration.
One codec that is natively supported (but has only a 4:1 compression ratio) is ADPCM, aka ima4. It's handled through the ExtAudioFile interface and is only the tiniest bit slower than loading .wav's directly.
There are some good open source audio decoding libraries that you could use:
mpg123
FAAC
Both are licensed under LGPL, meaning you can use them in closed source applications provided modifications to the library, if any, are open sourced.
You could always make your wave files mono and hence cut your wave file size in half. But that might not be the best alternative for you
Another option for doing your own decoding would be Ogg Vorbis. There's even a low-memory version of their library for integer processors called "Tremor".
Related
I receive raw H.264 NALUs from an IP camera (via Live555) and I want to decode them using hardware because FFmpeg is great but it's too slow (the camera sensor is large).
The only solution I see is to write the NALUs to some movie container file such as MPEG-4, and then read and decode that file using an AVAssetReader.
Am I off in the weeds? Is anyone having success decoding H.264 NALUs from a stream? Does anyone have any tips for writing NALUs to an MPEG-4 file? Other ideas?
Like Matt mentioned, there is no direct access to Apple's H264 decoder.
However, I have had success with ffmpeg and h264 decoding. Like you mentioned, I have built ffmpeg with LGPL I was able to decode H264 streams all the way to real-time HD stream with no latency on both ipad and iphone. Nothing fancy is required from ffmpeg, you can find bunch of standard decoding c++ code that will work just fine on iOS. Also, in my case H264 NALUs were delivered via RTP/RTSP in real-time.
Also, if I was you I would run your app through xcode instruments to truly see where you bottleneck is, but I would be highly surprised it is in ffmpeg decoding step.. Hopefully this info helps.
Unfortunately, you cannot do this at present. Feel free to file a radar with Apple about wanting this sort of access to the hardware decoder. It'll certainly be resolved as a duplicate :-). I assume it is for licensing reasons why they can't give this sort of access to the hardware codec.
So, you're going to have to use a software decoder. Please be aware that if you're going to ship to the App Store then you need something with a non-GPL license (unless you want to open source your app as well).
I tried WAV but that failed, although I've seen example code using WAV.
I found an AIFF in another example which worked.
I'm planning on buying sounds for my app, and I want to know which format to go for.
You can also take a look at AVAudioPlayer, that’s a fairly nice sound playback API. If your WAV did not work, did it have the right endianity and sample resolution? You should be safe with little-endian sampled at 44.1 kHz, mono or stereo. You can buy in almost any format, the conversion quite very easy and can be done using free tools like afconvert.
You can use .WAV format sounds if the audio properties are acceptable to iOS. What makes them acceptable you ask? Well, I don't know the full list of metrics, but I do know I am successfully using files sampled at 8kHz, 16-bit PCM, mono.
According to this What Techniques Are Best To Live Stream iPhone Video Camera Data To a Computer? is possible to get compressed data from iphone camera, but as I've been reading in the AVFoundation reference you only get uncompressed data.
So the questions are:
1) How to get compressed frames and audio from iPhone's camera?
2) Encoding uncompressed frames with ffmpeg's API is fast enough for real-time streaming?
Any help will be really appreciated.
Thanks.
You most likely already know....
1) How to get compressed frames and audio from iPhone's camera?
You can not do this. The AVFoundation API has prevented this from every angle. I even tried named pipes, and some other sneaky unix foo. No such luck. You have no choice but to write it to file. In your linked post a user suggest setting up the callback to deliver encoded frames. As far as I am aware this is not possible for H.264 streams. The capture delegate will deliver images encoded in a specific pixel format. It is the Movie Writers and AVAssetWriter that do the encoding.
2) Encoding uncompressed frames with ffmpeg's API is fast enough for real-time streaming?
Yes it is. However, you will have to use libx264 which gets you into GPL territory. That is not exactly compatible with the app store.
I would suggest using AVFoundation and AVAssetWriter for efficiency reasons.
I agree with Steve. I'd add that on trying with Apple's API, you're going to have to do some seriously nasty hacking. AVAssetWriter by default spends a second before spilling its buffer to file. I haven't found a way to change that with settings. The way around that seems to be to force small file writes and file close with the use of multiple AVAssetWriters. But then that introduces lots of overhead. It's not pretty.
Definitely file a new feature request with Apple (if you're an iOS developer). The more of us that do, the more likely they'll add some sort of writer that can write to a buffer and/or to a stream.
One addition I'd make to what Steve said on the x264 GPL issue is that I think you can get a commercial license for that which is better than GPL, but of course costs you money. But that means you could still use it and get pretty OK results, and not have to open up your own app source. Not as good as an augmented Apple API using their hardware codecs, but not bad.
I know this is not a specific programming question but I hope someone can give me a suggestion. My applications (iPhone and Blackberry applications) use a lot of audio files. I need a solution for my applications in order to save some spaces.
Is it right that .aac is the most suitable audio format for iPhone? Is it the smallest one? It it also suitable for Blackberry?
Is there any way to make the audio files smaller without losing a lot of quality of the sounds? How about the bitrate, sampling freq and channels? Are they really matter?
AAC is a good format for the iPhone. The iOS is optimized to play AAC.
Yes, things like bitrate, sampling frequency and number of channels are all factors in the audio file's size.
What you should do is take your audio and convert it to different formats with different settings and then just play them on a real device to see if the quality is acceptable.
Sorry, there is no simple answer. Experiment.
Depends on what type of audio you're encoding. For speech, AMR is supported by all major smartphones, and will generally give the smallest file sizes. Quality degredation is noticeable enough that it's not suitable for music, but it's optimized for voice recording (the voice notes app on the BlackBerry uses it as its file format) so it'll give you very nice results with spoken audio.
I'm having a hard time trying to record something other than linear PCM on the iPhone :-(
The samples I've found (SDK's SpeakHere, Zdziarski's and Sadun's books and the one at trailsinthesand.com) all use linear PCM but I'd like a commonly used compressed format instead (no ima4 or whatever the name is...).
I just cannot figure out how to tweak the sample code to be used with, for example AAC, MP3 or AMR instead. Any suggestions and hints for how to do that are much appreciated!
(Btw, I do not think an MP3-encoder nor AMR-encoder are available due to licensing issues, but AAC does exist, or???)
Edit/Update: I stumbled upon the following text in Apple's "iPhone Application Programming Guide", 2009-01-06, page 137, section: Recording Audio:
"You can record audio in any of the formats listed in “Preferred Audio Formats in iPhone OS” (page 140)", and as preferred audio formats on page 140 are: "For compressed audio when playing one sound at a time, and when you don’t need to play audio simultaneously with the iPod application, use the AAC format packaged in a CAF or m4a file."
Thus, I interpret that as a clear indication that it is indeed, not only possible, but even preferable, to record audio in AAC format wrapped up in a m4a file, which is just what I want. But still, I am not able to achieve that?!
Thanks,
/John
Keep looking at those docs. In "Core Audio Essentials", the section "Core Audio Plug-ins: Audio Units and Codecs" notes that:
iPhone OS contains the recording
codecs listed in Table 2-5. As you can
see, neither MP3 nor AAC recording is
available. This is due to the high CPU
overhead, and consequent battery
drain, of these formats.
Table 2-5 lists several formats, but as the text notes does not include the ones you're looking for. If you want those formats you'll have to bring your own encoder.
It is possible in iPhone 3GS and iPod 2nd generation and above. They have a hardware enconder for AAC.
There is an Apple example project for this that does exactly what you want to do:
iPhoneExtAudioFileConvertTest
A tip for using faac to convert from iPhone-output linear pcm files to aac format:
I found by experimentation that you have to use the -X flag on the command-line version of faac (running on Mac OS X), which "swaps the input bytes." (I guess it changes the endian-ness.)
So, for example, if you audio recorded the file linear.pcm on your iPhone, you could then run:
faac -XP linear.pcm
on your mac to convert that into the aac file linear.aac.
I'm guessing this means you might be able to use faac within your app to do the conversion, if you wanted to, since it's a C library.
Also note that, technically, the files output by iPhone's audio services are usually CAF files, Core Audio Format, which is sort of a wrapper format around a bunch of possible encodings: Apple's docs on CAF. The above command line works fine for me even though linear.pcm starts with the caff header.
Have you tried Lame mp3 encoder? Or faac/faad AAC codecs? You can embed them in ffmpeg for even more audio codecs.
Why don't try TPAACAudioConverter written by the legendary Michael Tyson ?
TPAACAudioConverter is a simple Objective-C class that performs the
conversion of any audio file to an AAC-encoded m4a, asynchronously
with a delegate, or converts any audio provided by a data source class
(which provides for recording straight to AAC).