Recently I've bumped into next problem.
I use CoreAudio AudioUnit (RemoteI/O) to play/record sound stream in an iOS app.
Sound stream which goes into audio unit is 2 channel LPCM, 16 bit, signed integer, interleaved (I also configure an output recording stream which is basically the same but has only one channel and 2 bytes per packet and frame).
I have configured my input ASBD as follows (I get no error when I set it and when I initialize unit):
ASBD.mFormatFlags = kLinearPCMFormatFlagIsSignedInteger | kLinearPCMFormatFlagIsPacked;
ASBD.mBytesPerPacket = 4;
ASBD.mFramesPerPacket = 1;
ASBD.mBytesPerFrame = 4;
ASBD.mChannelsPerFrame = 2;
ASBD.mBitsPerChannel = 16;
In my render callback function I get AudioBufferList with one buffer (as I understand, because the audio stream is interleaved).
I have a sample stereo file for testing which is 100% stereo with 2 obvious channels. I translate it into stream which corresponds to ASBD and feed to audio unit.
When I play sample file I hear only left channel.
I would appreciate any ideas why this happens. If needed I can post more code.
Update: I've tried to set
ASBD.mFormatFlags = kLinearPCMFormatFlagIsSignedInteger | kLinearPCMFormatFlagIsPacked | kLinearPCMFormatFlagIsNonInterleaved;
ASBD.mBytesPerPacket = 2;
ASBD.mFramesPerPacket = 1;
ASBD.mBytesPerFrame = 2;
ASBD.mChannelsPerFrame = 2;
ASBD.mBitsPerChannel = 16;
ASBD and I've got buffer list with two buffers. I deinterleaved my stream into 2 channels(1 channel for 1 buffer) and got the same result. I tried with headset and speaker on iPad (I know that speaker is mono).
Ok. So I've check my code and spotted that I use VoiceProcessingIO audio unit (instead of RemoteIO which is in the question) which is basically correct for my app since documentation says "The Voice-Processing I/O unit (subtype kAudioUnitSubType_VoiceProcessingIO) has the characteristics of the Remote I/O unit and adds echo suppression for two-way duplex communication. It also adds automatic gain correction, adjustment of voice-processing quality, and muting"
When I changed audio unit type to RemoteIO I've immediately got the stereo playback. I didn't have to change stream properties.
Basically VoiceProcessingIO audio unit downfalls to mono and disregards stream properties.
I've posted a question on Apple Developer forum regarding stereo output using VoiceProcessingIO audio unit but haven't got any answer yet.
It seems pretty logical for me to downfall to mono in order to do some signal processing like echo cancelation because iOS devices can record only mono sound without specific external accessories. Although this is not documented anywhere in Apple documentation. I've also come across a post of guy who claimed that stereo worked for VoiceProcessingIO AU prior to iOS5.0.
Anyway thanks for your attention. Any other comments on the matter would be greatly appreciated.
Related
I'm doing an audio processing project. I have to record audio from a microphone and have to process those samples real time. While the recording process is carried out, I must process the samples and play them back. I am using the code below to record my audio.
% Setting parameters
Fs = 44100;
nbits = 16;
dev_id = getfield(getfield(audiodevinfo, 'input'), 'ID');
% Creating the audiorecorder object
arec = audiorecorder(Fs, nbits, 2, dev_id);
display('Start speaking.');
recordblocking(arec, 15.0);
display('End of Recording.');
data = getaudiodata(arec);
display(data);
soundsc(data, Fs);
plot(data);
wavwrite(data,Fs,nbits,'changed.wav');
Can real time processing can be done? Can someone please give me an idea?
Real time processing usually means that you receive a streaming block of data, process it, and pass it to the output without having any gaps or stops in the stream. As you might have discovered, that is not what is happening with your code above.
Unfortunately standard Matlab does not not support real time processing of signals, but if you look into the DSP Systems Toolbox addon, you'll find the necessary functionality, along with some examples.
Another option is to implement a the real time processing through a MEX interface instead.
I'm developing a bass guitar pitch detection app on iphone 3GS.
I found I can't get sound data lower than 150Hz with RemoteIO.
However bass guitar may generate tones lower than 50hz.
According to the report "iPhone 4 Headset Input Frequency Response", http://blog.faberacoustical.com/2010/iphone/iphone-4-audio-and-frequency-response-limitations/
There is a sharp drop-off below 150 Hz.
Here shows how I setup the AudioUnit.
// set audio unit
{
// create AudioUnit
{
AudioComponentDescription desc;
desc.componentType = kAudioUnitType_Output;
desc.componentSubType = kAudioUnitSubType_RemoteIO;
desc.componentManufacturer = kAudioUnitManufacturer_Apple;
desc.componentFlags = 0;
desc.componentFlagsMask = 0;
AudioComponent comp = AudioComponentFindNext(NULL, &desc);
OSAssert(AudioComponentInstanceNew(comp, &m_AudioUnit));
}
//enable input on the remote I/O unit (output is default enabled, but input is not)
{
UInt32 one = 1;
OSAssert(AudioUnitSetProperty(m_AudioUnit, kAudioOutputUnitProperty_EnableIO,
kAudioUnitScope_Input, 1, &one, sizeof(one)));
}
//set render callback function
{
AURenderCallbackStruct callbackInfo;
callbackInfo.inputProc=staticPerformThru;
callbackInfo.inputProcRefCon=this;
OSAssert(AudioUnitSetProperty(m_AudioUnit,
kAudioUnitProperty_SetRenderCallback,
kAudioUnitScope_Input,
0, &callbackInfo, sizeof(callbackInfo)));
}
//set in/output format
{
CAStreamBasicDescription outFormat;
outFormat.SetAUCanonical(channels, false);
outFormat.mSampleRate = sampleRate;
OSAssert(AudioUnitSetProperty(m_AudioUnit, kAudioUnitProperty_StreamFormat, kAudioUnitScope_Input, 0, &outFormat, sizeof(outFormat)));
OSAssert(AudioUnitSetProperty(m_AudioUnit, kAudioUnitProperty_StreamFormat, kAudioUnitScope_Output, 1, &outFormat, sizeof(outFormat)));
}
//Initialize remote I/O unit
OSStatus r=AudioUnitInitialize(m_AudioUnit);
OSAssert(r);
}
//start audio output
OSAssert(AudioOutputUnitStart(m_AudioUnit));
This is the callback function.
OSStatus AudioThruWorker::staticPerformThru(
void *inRefCon,
AudioUnitRenderActionFlags *ioActionFlags,
const AudioTimeStamp *inTimeStamp,
UInt32 inBusNumber,
UInt32 inNumberFrames,
AudioBufferList *ioData)
{
AudioUnitRender(((AudioThruWorker*)inRefCon)->m_AudioUnit, ioActionFlags, inTimeStamp, 1, inNumberFrames, ioData);
//Detect pitch here...
return 0;
}
To identify the root cause,
I modified my callback function to just bypass input data to output.
Use Mac to generate white noise
Use iRig to redirect the signal from Mac's headset to iPhone3Gs which is running my program.
Use iRig to redirect output of iPhone back to Mac.
Record data on Mac.
The output data spectrum shows in below figure.
You can see the sharp drop at 150Hz.
To identify whether the problem is at input side or output side, I changed the callback function to ignore input data and output a white noise. Here is the result.
Very clear that there is NO drop at 150 Hz. Therefore the problem should at the input side.
I thought this is a hardware limitation.
HOWEVER I tried the app "Amplitube" on the same device, turn off all effects ,input white noise and analyze output. It has NO drop at 150Hz. Here is the result.
This means the drop-off problem is NOT a hardware limitation. There must be some way software can do to avoid the problem.
Does anyone know the secret?
Thanks.
Well, it is a phone, a device optimized for speech supposedly. And devices optimized for speech usually have some sort of LF cut filter to avoid rumble and distortion.
This filter is probably on the input side of the phone, that is why you can generate and output a wider range of frequencies, it is probably a hardware/discrete element filter, since those are simple to create using only a few components and will work in real time without any strain on processing.
I don't think it makes sense to cut lows in software, I know I wouldn't do it, well, for a DAW application you can do it, but for a device optimized to filter lows...
Considered the fact amplitube devs might have been aware of this issue and added extra low boost to try and make it up for the hardware limitation?
On the other hand, it may very well be possible to chose different "signal paths" according to usage scenario, maybe there is some os handle apps can touch and say "hey, I am not voice, don't cut my lows" - if so, this feature should be somewhere in the api description.
Interesting question. I am not aware of any such filter... you can set an input callback on the remote IO unit and get a stream of floats.
I haven't seen any documentation that this float stream is in some way already processed.
I have written a pitch detector which successfully picks up notes at the low end of my singing range (~80Hz)
Maybe post code -- what are you doing inside this callback?
Can I recommend you re-ask this question, with an appropriate title ( Maybe something like ' how to prevent iPhone from filtering low frequencies ( < 150Hz ) ' ) and containing all of the necessary information, also stick in that picture, that is an important part of the question. it's very easy to embed a picture.
you haven't even stated what it is you are trying to accomplish.
It looks like you're new to the site, and it looks like a very interesting question.
But you are not being at all clear.
Firstly, this is an issue with record or playback or both?
Now, what exactly are you doing in your experiment? Western scientific method... lay it out.
You are generating static in your remoteIO render callback? No, it sounds like you are using passthrough, you say you are putting white noise into the iPhone. how are you setting up your audio unit?
so the white noise goes into the iPhone, straight out, and... now what does that graph represent? FFT / Spectral analysis of the output of the iPhone? how are you doing that? You're feeding this into some OSX graphing software that can take input from the line in?
also, you could do with narrowing it down. If you are indeed passing sound right through the device, you have no idea where the filtering is occurring. for all you know there could be filtering occurring in the device that generates the white noise, as well as the device that graphs the iPhone output.
how about generating your white noise from inside the remote I/O render callback?
and then generating it externally and analysing the data that arrives from within this callback -- you could perform an FFT.
anyway, for input wouldn't you do better to feed in sine waves of different frequencies?
In iPhone SDK 4.3 I would like to record what is being played out through speaker via Remote IO and also record the mic input. I was wondering if the best way is to record each separately to a different channel in an audio file. If so which apis allow me to do this and what audio format should I use. I was planning on using ExtAudioFileWrite to do the actual writing to the file.
Thanks
If both tracks that you have is mono, 16bit integer with the same sample rate:
format->mFormatFlags = kLinearPCMFormatFlagIsSignedInteger | kLinearPCMFormatFlagIsPacked;
format->mBitsPerChannel = 16;
you can combine those tracks to the 2 channels PCM by just alternating sample from one track with sample from another.
[short1_track1][short1_track2][short2_track1][short2_track2] and so on.
After that you can write this samples to the output file using ExtAudioFileWrite. That file should be 2 channel kLinearPCMFormatFlagIsSignedInteger | kLinearPCMFormatFlagIsPacked of course.
If one of tracks is stereo (I don't think that it is reasonable to record stereo from iphone mic), you can convert it to the mono by taking average from 2 channels or by skipping every second sample of it.
You can separately save PCM data from the play and record callback buffers of the RemoteIO Audio Unit, then mix them using your own mixer code (DSP code) before writing the mixed result to a file.
You may or may not need to do your own echo cancellation (advanced DSP code) as well.
I've currently got my output audio on the iPhone setup as this :
AudioStreamBasicDescription audioFormat;
audioFormat.mSampleRate = 48000;
audioFormat.mFormatID = kAudioFormatLinearPCM;
audioFormat.mFormatFlags = kAudioFormatFlagIsSignedInteger | kAudioFormatFlagIsPacked;
audioFormat.mFramesPerPacket = 1;
audioFormat.mChannelsPerFrame = 2;
audioFormat.mBitsPerChannel = 16;
audioFormat.mBytesPerPacket = 4;
audioFormat.mBytesPerFrame = 4;
However, when I examine my performance figures through shark I am seeing functions such as :
SRC_Convert_table_i32_scalar_stereo
take a fair chunk of time.
This made me think - what is the ideal and suggested output format for the iPhone? The one that requires as little work for the device to play.
Shark does work with the iPhone. You can enable iPhone profiling by selecting "Sampling > Network/iPhone Profiling..." in the menu.
Definitely try using a 44100 Hz sampling rate. With 48000 I see the same function that you posted appearing in the callstacks -- no such function shows up when using 44100. The canonical audio format for Audio Units on the iPhone is non-interleaved 8.24 linear PCM:
streamFormat.mFormatID = kAudioFormatLinearPCM;
streamFormat.mFormatFlags =
kAudioFormatFlagIsSignedInteger
| kAudioFormatFlagsNativeEndian
| kLinearPCMFormatFlagIsNonInterleaved
| (24 << kLinearPCMFormatFlagsSampleFractionShift);
streamFormat.mSampleRate = mixing_rate;
streamFormat.mBitsPerChannel = 32;
streamFormat.mChannelsPerFrame = 2;
streamFormat.mFramesPerPacket = 1;
streamFormat.mBytesPerFrame = ( streamFormat.mBitsPerChannel / 8 );
streamFormat.mBytesPerPacket = streamFormat.mBytesPerFrame *
streamFormat.mFramesPerPacket;
From iphone dev centre (requires login) the hardware suppoorted codecs are
iPhone Audio Hardware Codecs
iPhone OS applications can use a wide
range of audio data formats, as
described in the next section. Some of
these formats use software-based
encoding and decoding. You can
simultaneously play multiple sounds in
these formats. Moreover, your
application and a background
application (notably, the iPod
application) can simultaneously play
sounds in these formats.
Other iPhone OS audio formats employ a
hardware codec for playback. These
formats are:
AAC
ALAC (Apple Lossless)
MP3
48000 is a weird format for audio in general. While it's marginally (and imperceptibly) better than CD standard 44.1Khz it's not worth the size.
Is there any compelling reason in your app to use high quality stereo sound? In other words is the app likely to be played on speakers or good headphones?
LinearPCM format is hardware native so it should be optimal. (No decompression to fiddle with.) So that call may be a downsample to 44.1Khz.
A voice recorder doesn't need uncompressed Linear PCM audio. Compressed AMR would do fine. The iPhone framework built for recording audio is simple enough, but the only examples I've found for setting up the audio format (which come from Apple) use LinearPCM. I've tried various other combinations of values, but can't seem to get anything to work.
Does anybody have any code that actually records AMR?
Edit:
The AMR format is one of the options for setting the data type, but the other options (packet size, frame size, etc.) don't seem to match up no matter what I set them to.
Edit: Here's what I have for the PCM version:
/*
If we want to use AMR instead of PCM:
AMR Format:
Sampling Frequency: 8 kHz/13-bit (160 samples for 20 ms frames), filtered to 200-3400 Hz
eight source codecs : 12.2, 1.2, 7.95, 7.40, 6.70, 5.90, 5.15, 4.75 kbit/s
generated frame length: 244, 204, 159, 148, 134, 118, 103, 95 bits per frame
*/
format->mFormatID = kAudioFormatLinearPCM;
format->mSampleRate = 8000.0; //8 kHz
format->mFramesPerPacket = 1; //1 frame per packet
format->mChannelsPerFrame = 1; //Mono
format->mBytesPerFrame = 2; //8/bits per frame (round up)
format->mBytesPerPacket = 2; //Same as bytes per frame
format->mBitsPerChannel = 16; //16-bit audio
format->mReserved = 0; //always 0
format->mFormatFlags = kLinearPCMFormatFlagIsBigEndian |
kLinearPCMFormatFlagIsSignedInteger |
kLinearPCMFormatFlagIsPacked;
AMR codec is NOT supported for encoding/recording on the iPhone, albeit it is supported for playback: this is the reason the kAudioFormatAMR constant exists.
Official api says that supported encoding formats are:
ALAC (Apple Lossless) ~> kAudioFormatAppleLossless
iLBC (internet Low Bitrate Codec, for speech) ~> kAudioFormatiLBC
IMA/ADPCM (IMA4) ~> kAudioFormatAppleIMA4
linear PCM ~> kAudioFormatLinearPCM
ยต-law ~> kAudioFormatULaw
a-law ~> kAudioFormatALaw
You may try one of these formats or use an open source AMR encoder as goldenmean suggests.
edit: Updated Official api link
To update olegueret's link to the official documentation (why do they hide this stuff?)
http://developer.apple.com/library/ios/#qa/qa2008/qa1615.html
I guess AMR codec format is not supported my iPhone voice recorder app.
May be one can try integrating some open-source, implementation of AMR encoder into the apples' iPhone application framework and try making the voice recorder store the audio in AMR encoded format. (i dont know if thats allowed by apple by their NDA/license).
-AD
You can record audio to a uncompressed Linear PCM buffer (circular or ring), and, in another thread, convert data in this buffer, using your own AMR (or other) compression engine, before saving the compressed audio data to a file.