I want to use the MTAudioProcessingTap's functionality, but instead of using the AVPlayer like CHris in his tutorial i want to use the iPhones microphone.
Can this be done and / or is it documented anywhere?
The way I would proceed with this is to set the session to AVAudioSessionCategoryPlayAndRecord.
The tutorial says that you can apply the MTAudioProcessingTap to modify any file on your phone. If you follow this tutorial, it shows you how to create a file you recorded using AVFoundation and then play it back.
Right now AV Foundation is not set up to do real-time audio processing as you record audio. It can only modify the audio in real time while it is being played back or it can do offline audio processing as detailed in the Audio Session Programming Guide.
I also do not recommend doing a destructive process to a sound as it is being processed. Best practice for audio creation is to leave the master untouched and to change the sound after you capture it.
As of the beginning of 2014 there is a great deal of information about AV Foundation that is not yet documented. There is a new audio session category that has not been included in the Audio Session Programming Guide. In a few month a whole book on AV Foundation will be published and hopefully that book will provide more solutions to some of these questions.
Related
I have an audio app in which all of the sound generating work is accomplished by pure data (using libpd).
I've coded a special sequencer in swift which controls the start/stop playback of multiple sequences, played by the synth engines in pure data.
Until now, I've completely avoided using Core Audio or AVFoundation for any aspect of my app, because I know nothing about them, and they both seem to require C or Objective C coding, which I know nearly nothing about.
However, I've been told from a previous q&a on here, that I need to use Core Audio or AVFoundation to get accurate timing. Without it, I've tried everything else, and the timing is totally messed up (laggy, jittery).
All of the tutorials and books on Core Audio seem overwhelmingly broad and deep to me. If all I need from one of these frameworks is accurate timing for my sequencer, how do you suggest I achieve this as someone who is a total novice to Core Audio and Objective-C, but otherwise has a 95% finished audio app?
If your sequencer is Swift code that depends on being called just-in-time to push audio, it won't work with good timing accuracy. e.g. you can't get the timing you need.
Core Audio uses a real-time pull-model (which excludes Swift code of any interesting complexity). AVFoundation likely requires you to create your audio ahead of time, and schedule buffers. An iOS app needs to be designed nearly from the ground up for one of these two solutions.
Added: If your existing code can generate audio samples a bit ahead of time, enough to statistically cover using a jittery OS timer, you can schedule this pre-generated output to be played a few milliseconds later (e.g. when pulled at the correct sample time).
AudioKit is an open source audio framework that provides Swift access to Core Audio services. It includes a Core Audio based sequencer, and there is plenty of sample code available in the form of Swift Playgrounds.
The AudioKit AKSequencer class has the transport controls you need. You can add MIDI events to your sequencer instance programmatically, or read them from a file. You could then connect your sequencer to an AKCallbackInstrument which can execute code upon receiving MIDI noteOn and noteOff commands, which might be one way to trigger your generated audio.
I've just started working on a project that requires me to do lots of audio related stuff on iOS.
This is the first time I'm working in the realm of audio, and have absolutely no idea how to go about it. So, I googled for documents, and was mostly relying on Apple docs. Firstly, I must mention that the documents are extremely confusing, and often, misleading.
Anyways, to test a recording, I used AVAudioSession and AVAudioRecorder. From what I understand, these are okay for simple recording and playback. So, here are a couple of questions I have regarding doing anything more complex:
If I wish to do any real-time processing with the audio, while recording is in progress, do I need to use Audio Queue services?
What other options do I have apart from Audio Queue Services?
What are Audio Units?
I actually got Apple's Audio Queue Services programming guide, and started writing an audio queue for recording. The "diagram" on their audio queue services guide (pg. 19 of the PDF) shows recording being done using an AAC codec. However, after some frustration and wasting a lot of time, I found out that AAC recording is not available on iOS - "Core Audio Essentials", section "Core Audio Plug-ins: Audio Units and Codecs".
Which brings me to my another two question:
What's a suitable format for recording, given Apple Lossless, iLBC, IMA/ADPCM, Linear PCM, uLaw/aLaw? Is there some chart somewhere that someone might be able to refer to?
Also, if MPEG4AAC (.m4a) recording is not available using an audio queue, how is it that I can record an MPEG4AAC (.m4a) using AVAudioRecorder?!
Super thanks a ton in advance for helping me out on this. I'll super appreciate any links, directions and/or words of wisdom.
Thanks again and cheers!
For your first question, Audio Queue services or using the RemoteIO Audio Unit are the appropriate APIs for real-time audio processing, with RemoteIO allowing lower and more deterministic latency, but with stricter real-time requirements than Audio Queues.
For creating aac recordings, one possibility is to record to raw linear PCM audio, then later use AV file services to convert buffered raw audio into your desired compressed format.
I want to know if AudioToolbox.framework will suppport an array of pageviewcontrollers to change with audio synchronization programmtically. From the apple documentation I found that Audio Queue Services lets you record, play, pause, loop, and synchronize audio but when they say synchronize audio are they referring to page change synchronization or something else?
I already have AVFoundation.framework in my app to play an audio file.
It's a 'Core' AV library. You'll have to you your own program to interact with or manipulate your UI. The library does not rely on either AppKit or UIKit.
Sync: consider it along the lines of accurate timing of audio playback.
I am building an App that allows our customers to record, save, and play recorded sound as a basic functionality. This should be straight using AVFoundation Framework. What I also allow users are
Fast Forward and Reverse functionality.
User should also able to manipulate the sound. I mean they allow to insert the sound in between their recorded sound later.
Could anyone please tell me how could I achieve these? Is there any good open-source library for this?
The AVAudioPlayer supports manipulating the playback speed via the rate and enableRate properties, but it only allows forward playing.
The MPMoviePlayerController conforms to the MPMediaPlayback protocol which allows you to specify any rate (even reverse). Though this method will result in choppy audio for some rates.
As far as merging audio files, I think your best bet is to convert your samples to linear PCM. Then you can insert additional samples anywhere in your stream.
On iOS, is it possible to get the user's audio stream in a decompressed format? For example, the MP3 is returned as a WAV that can be used for audio analysis? I'm relatively new to the iOS platform, and I remember seeing that this wasn't possible in older iOS versions. I read that iOS 4 brought in some advanced APIs but I'm not sure where I can find documentations/samples for these.
If you don't mind using API for iOS 4.1 and above, you could try using the AVAssetReader class and friends. In this similar question you have a full example on how to extract video frames. I would expect the same to work for audio, and the nice thing is that the reader deals with all the details of decompression. You can even do composition with AVComposition to merge several streams.
These classes are part of the AVFramework, which allows not only reading but also creating your own content.
Apple has an OpenAL example at http://developer.apple.com/library/mac/#samplecode/OpenALExample/Introduction/Intro.html where Scene.m should interest you.
The Apple documentation has this picture where the Core Audio framework clearly shows that it gives you MP3 out. It also states that you can access audio units in a more radical way if you so need.
The same Core Audio document gives also some information about using MIDI if it may help you.
Edit:
You're in luck today.
In this example an audio file is loaded and fed into an AudioUnit graph. You could fairly easily write an AudioUnit of your own to put into this graph and which analyzes the PCM stream as you see fit. You can even do it in the callback function, although that's probably not a good idea because callbacks are encouraged to be as simple as possible.