Thank you for taking the time and reading my question.
Target:
I am getting some audio data from UDP server and I have to convert them into PCM format to play the actual audio realtime.
Query:
For some reason using below code did not work for the format of my
AVAudioPCMBuffer and yielded no sound, or a memory error. However, I
am able to get white noise/static to play from the speakers by
converting the NSData to a AVAudioPCMBuffer. I have a feeling that I
am getting close to the answer, because whenever I speak into the
microphone I hear a different frequency of static.
Here is my code:
1. Converting received data to PCM Buffer:
func toPCMBuffer(format: AVAudioFormat, data: NSData) -> AVAudioPCMBuffer {
let PCMBuffer = AVAudioPCMBuffer(pcmFormat: format, frameCapacity: UInt32(data.count) / format.streamDescription.pointee.mBytesPerFrame)
PCMBuffer!.frameLength = PCMBuffer!.frameCapacity
let channels = UnsafeBufferPointer(start: PCMBuffer?.int16ChannelData, count: Int(PCMBuffer!.format.channelCount))
data.getBytes(UnsafeMutableRawPointer(channels[0]) , length: data.count)
return PCMBuffer!
}
2. Playing the converted buffer into player.
if let receivedData = notification.userInfo?["data"] as? NSData {
let inputFormat = AVAudioFormat(commonFormat: AVAudioCommonFormat.pcmFormatInt16, sampleRate: 8000, channels: 1, interleaved: false)
let pcmBufferData = self.toPCMBuffer(format: inputFormat!, data: receivedData) // converting raw data to pcm buffer
let mainMixer = audioEngine.inputNode
audioEngine.attach(audioFilePlayer)
let outputFormat = mainMixer.outputFormat(forBus: 1)
audioEngine.connect(audioFilePlayer, to:audioEngine.outputNode, format: outputFormat)
audioFilePlayer.scheduleBuffer(pcmBufferData, at: nil, completionHandler: nil)
audioEngine.prepare()
do {
try audioEngine.start()
} catch _{
print("error")
}
audioFilePlayer.play()
}
I don't understand what the issue is? Your help will be much appreciated. Thanks :)
Edit 1:
As # Vladimir's suggestion, I updated my code as below:
func configAudioEngine(){ // Config audio Engine only once.
let mainMixer = audioEngine.inputNode
audioEngine.attach(audioFilePlayer)
let outputFormat = mainMixer.outputFormat(forBus: 0)
// let outputFormat = mainMixer.outputFormat(forBus: bus)
audioEngine.connect(audioFilePlayer, to:audioEngine.outputNode, format: outputFormat)
audioEngine.prepare()
do {
try audioEngine.start()
} catch _{
print("error")
}
}
// Receiving Data in notification from UDP socket delegate, converting it to pcm buffer and then playing.
// Note : This callback gets called everytime I receive data from UDP
#objc func didReceiveData(notification:NSNotification) {
if let receivedData = notification.userInfo?["data"] as? NSData
{
let inputFormat = AVAudioFormat(commonFormat: AVAudioCommonFormat.pcmFormatInt16, sampleRate: 80000, channels: 1, interleaved: false)
let pcmBufferData = self.toPCMBuffer(format: inputFormat!, data: receivedData)
audioFilePlayer.play()
}
}
// Still getting the white noise without any improvement in Audio.
//Here is my received Data in UDP
//incoming message: 1600 bytes
<48003c00 36003400 36002800 1600f5ff d2ffb8ff afffb6ff d1fff3ff 19003900 45004200 26000600 e1ffc2ff b7ffbbff c8ffd5ff dfffe3ff e1ffe9ff f4ff1200 3c006f00 93009900 88006000 2400e1ff a9ff8aff 87ff8eff a9ffc1ff d1ffdaff dcffe9ff 04002c00 52006c00 74005e00 2900e3ff 9fff76ff 78ffa2ff ebff2d00 5e007400 69005700 3e003000 2d002f00 28000a00 d7ff93ff 51ff24ff 23ff59ff a8ff1200 6700a100 af009500 77004a00 36002100 1e001700 0300deff abff77ff 5eff6fff 9cffebff 2d006800 80007500 4f001900 f3ffdbff d1ffd6ff d2ffd5ff ceffc8ff d1ffe0ff 0c003800 6e009900 b000a800 7e003e00 f8ffb6ff 7cff55ff 3fff3fff 50ff71ff 99ffc8ff 00003000 64008b00 ab00b000 a0008000 55002b00 f7ffdbff b2ffb1ff c2ffe2ff 0b002800 41003c00 30000b00 e8ffbdff 96ff86ff 74ff74ff 6bff72ff 7eff9aff daff1900 6500a500 da00f200 f400dc00 af007900 44001700 efffcaff a7ff87ff 77ff71ff 75ff87ff 9bffbbff d8fff4ff fbfff4ff daffc3ff bdffc1ff ddfff4ff 17003d00 5a007800 88009900 a0009f00 9a008500 69003700 0000caff 95ff79ff 66ff5fff 69ff77ff 8affa6ff bdffd5ff e9fffbff 12001f00 2d002900 2b002700 27003000 36004200 46004900 47003800 30002300 1a000b00 f7ffe3ff c8ffc0ff b1ffb4ff b8ffbeff ceffdeff f4ff0000 10001700 19001300 0c000300 fffffdff f7fff9ff fcff0c00 12001500 11001500 11001500 19001200 11000100 f3ffe8ff e2ffe7ff f2ff0700 1c002900 2d002700 20001100 0500fcff f2ffecff e5ffd7ff c7ffbaff b1ffb3ff bcffd7ff f4ff1500 2b003600 3b002e00 2e002600 2b003200 34003600 27001b00 0400f4ff ecffecff faff0100 0b000900 f9ffe9ff d9ffcdff c7ffc3ff c6ffc3ff c3ffc3ff c3ffd2ff eaff0f00 35005600 73007800 75006100 4c003600 23001100 ffffefff e1ffd6ff cbffc7ff c4ffd2ff dbffebff efffedff e5ffd9ff d9ffd1ff daffe3ff eefff8ff 06001100 17002500 32004400 57006300 68005f00 48002c00 0c00edff cdffb8ff a7ff9dff 96ff97ff a0ffb2ff d1fff0ff 0a002100 2e003200 2d001f00 17000c00 0a000f00 15001f00 24002400 25002600 25002400 1f001800 0b00fbff e7ffceff bbffacff a2ffa2ff acffbdff d5fff2ff 0f002400 31003900 3f004200 42004300 42003a00 28001200 ffffefff deffd1ff d1ffd5ff e0ffe6ff effff5ff f7fff5ff f5fff3ff effff0ff ecffedff effff2ff fbff0200 0d001e00 2c003b00 45004a00 42003300 1d000500 efffdbff c6ffb8ff a7ff9fff a1ffafff cefff0ff 1a003e00 5d006b00 69005e00 49002d00 0f00f3ff deffcdff c6ffc3ff c6ffcfff ddfff4ff 09002100 2e002c00 1e000600 e9ffd3ff bfffadff a9ffacff bbffdcff 05003c00 65008700 a100a600 a3008d00 65002c00 f0ffb6ff 82ff5eff 4aff47ff 56ff76ff a5ffd1ff 06003000 51006200 67005c00 47003100 1700ffff e6ffdeff e4fff1ff 0a002600 3b004e00 5b006000 5e004d00 2a00feff cbff97ff 6aff53ff 42ff45ff 59ff79ff aeffeaff 2c006100 84009d00 a300a800 95008200 5d003100 0700e3ff cbffc4ff c4ffc5ff ceffd2ff e5fff4ff 0d001300 0a00f4ff d7ffc3ff b6ffadff adffa7ff b3ffc1ff e9ff1600 3b006000 70007f00 7e008200 7b006700 4b002600 feffdcff bbffadff 9eff94ff 8fff95ff adffc4ff e2ffedff f7ffffff 02000d00 1b002200 1e001800 12001300 16001e00 28003000 38003900 42003a00 36002900 12000100 e8ffd5ff bbffabff 9cff99ff 9bffa2ff b5ffc9ff ddfff3ff 06002100 37004800 59005d00 5c005500 48004000 2b002b00 19001300 0100f0ff eaffd4ff dfffc2ff dfffbeff d1ffb6ff afffb1ff 9fffbdff b3ffe4ff e4ff1800 24004200 63005d00 80007100 83007500 6a005f00 41003100 0500e8ff baff9bff 85ff76ff 76ff7eff 89ff97ff a6ffc1ff d1fff5ff 04002300 32003d00 51004700 56004f00 5b005800 56006200 4b006200 3f003c00 1f000100 e3ffbcff a3ff7dff 76ff62ff 67ff6aff 7bff91ff acffc7ff eaff0300 28004000 68008300 9e00b000 a200a000 7a006700 3f002700 0d00edff d8ffb5ff a4ff93ff 8cff95ff 92ffa4ff b7ffc7ff d8ffedff eefff6ff f7ffebff edfff0ff 0c002000 42005a00 73008500 8c008c00 76006400 45002000 ffffd7ff b4ff97ff 83ff79ff 75ff70ff 7eff94ff b4ffe1ff ffff2b00>
Edit 2:
As # Vladimir's suggestion, I updated my configAudioEngine() as below:
func configAudioEngine(){
let mainMixer = audioEngine.outputNode
audioEngine.attach(audioFilePlayer)
let outputFormat = mainMixer.outputFormat(forBus: 0)
// let outputFormat = mainMixer.outputFormat(forBus: bus)
audioEngine.connect(audioFilePlayer, to:mainMixer, format: outputFormat)
audioEngine.prepare()
do {
try audioEngine.start()
} catch _{
print("error")
}
NotificationCenter.default.addObserver(self, selector: #selector(didReceiveData), name: Global.inSocket.notificationAlertMessage, object: nil)
}
This has resolved my exception and I am receiving data. But No sound is coming. It's mute.
Related
I'm trying to convert input node format to S16LE format. I've tried it with AVAudioMixerNode
First I create audio session
do {
try audioSession.setCategory(.record)
try audioSession.setActive(true)
} catch {
...
}
//Define formats
let inputNodeOutputFormat = audioEngine.inputNode.outputFormat(forBus: 0)
guard let wantedFormat = AVAudioFormat(commonFormat: AVAudioCommonFormat.pcmFormatInt16, sampleRate: 16000, channels: 1, interleaved: false) else {
return;
}
//Create mixer node and attach it to the engine
audioEngine.attach(mixerNode)
//Connect the input node to mixer node and mixer node to mainMixerNode
audioEngine.connect(audioEngine.inputNode, to: mixerNode, format: inputNodeOutputFormat)
audioEngine.connect(mixerNode, to: audioEngine.mainMixerNode, format: wantedFormat)
//Install the tab on the output of the mixerNode
mixerNode.installTap(onBus: 0, bufferSize: bufferSize, format: wantedFormat) { (buffer, time) in
let theLength = Int(buffer.frameLength)
var bufferData: [Int16] = []
for i in 0 ..< theLength
{
let char = Int16((buffer.int16ChannelData?.pointee[i])!)
bufferData.append(char)
}
}
I get the following error.
Exception 'I[busArray objectAtindexedSubscript:
(NSUlnteger)element] setFormat:format
error:&nsErr]: returned false, error Error
Domain=NSOSStatusErrorDomain Code=-10868
"(null)"' was thrown
What part of the graph did I mess up?
You have to set the format of nodes to match the actual format of the data. Setting the node's format doesn't cause any conversions to happen, except that mixer nodes can convert sample rates (but not data formats). You'll need to use an AVAudioConverter in your tap to do the conversion.
As an example of what this code would look like, to handle arbitrary conversions:
let inputNode = audioEngine.inputNode
let inputFormat = inputNode.inputFormat(forBus: 0)
let outputFormat = AVAudioFormat(... define your format ...)
guard let converter = AVAudioConverter(from: inputFormat, to: outputFormat) else {
throw ...some error...
}
inputNode.installTap(onBus: 0, bufferSize: 1024, format: inputFormat) {[weak self] (buffer, time) in
let inputBlock: AVAudioConverterInputBlock = {inNumPackets, outStatus in
outStatus.pointee = AVAudioConverterInputStatus.haveData
return buffer
}
let targetFrameCapacity = AVAudioFrameCount(outputFormat.sampleRate) * buffer.frameLength / AVAudioFrameCount(buffer.format.sampleRate)
if let convertedBuffer = AVAudioPCMBuffer(pcmFormat: outputFormat, frameCapacity: targetFrameCapacity) {
var error: NSError?
let status = converter.convert(to: convertedBuffer, error: &error, withInputFrom: inputBlock)
assert(status != .error)
let sampleCount = convertedBuffer.frameLength
let rawData = convertedBuffer.int16ChannelData![0]
// ... and here you have your data ...
}
}
If you don't need to change the sample rate, and you're converting from uncompressed audio to uncompressed audio, you may be able to use the simpler convert(to:from:) method in your tap.
Since iOS 13, you can also do this with AVAudioSinkNode rather than a tap, which can be more convenient.
my record work normally, but the problem is after WebRTC call, i got crash
required condition is false: format.sampleRate == hwFormat.sampleRate
here is how i start crash and installTap:
func startRecord() {
self.filePath = nil
print("last format: \(audioEngine.inputNode.inputFormat(forBus: 0).sampleRate)")
let session = AVAudioSession.sharedInstance()
do {
try session.setCategory(.playAndRecord, options: .mixWithOthers)
} catch {
print("======== Error setting setCategory \(error.localizedDescription)")
}
do {
try session.setPreferredSampleRate(44100.0)
} catch {
print("======== Error setting rate \(error.localizedDescription)")
}
do {
try session.setPreferredIOBufferDuration(0.005)
} catch {
print("======== Error IOBufferDuration \(error.localizedDescription)")
}
do {
try session.setActive(true, options: .notifyOthersOnDeactivation)
} catch {
print("========== Error starting session \(error.localizedDescription)")
}
let format = AVAudioFormat(commonFormat: AVAudioCommonFormat.pcmFormatInt16,
sampleRate: 44100.0,
// sampleRate: audioEngine.inputNode.inputFormat(forBus: 0).sampleRate,
channels: 1,
interleaved: true)
audioEngine.connect(audioEngine.inputNode, to: mixer, format: format)
audioEngine.connect(mixer, to: audioEngine.mainMixerNode, format: format)
let dir = NSSearchPathForDirectoriesInDomains(.documentDirectory, .userDomainMask, true).first! as String
filePath = dir.appending("/\(UUID.init().uuidString).wav")
_ = ExtAudioFileCreateWithURL(URL(fileURLWithPath: filePath!) as CFURL,
kAudioFileWAVEType,(format?.streamDescription)!,nil,AudioFileFlags.eraseFile.rawValue,&outref)
mixer.installTap(onBus: 0, bufferSize: AVAudioFrameCount((format?.sampleRate)!), format: format, block: { (buffer: AVAudioPCMBuffer!, time: AVAudioTime!) -> Void in
let audioBuffer : AVAudioBuffer = buffer
_ = ExtAudioFileWrite(self.outref!, buffer.frameLength, audioBuffer.audioBufferList)
})
try! audioEngine.start()
startMP3Rec(path: filePath!, rate: 128)
}
func stopRecord() {
self.audioFilePlayer.stop()
self.audioEngine.stop()
self.mixer.removeTap(onBus: 0)
self.stopMP3Rec()
ExtAudioFileDispose(self.outref!)
try? AVAudioSession.sharedInstance().setActive(false)
}
func startMP3Rec(path: String, rate: Int32) {
self.isMP3Active = true
var total = 0
var read = 0
var write: Int32 = 0
let mp3path = path.replacingOccurrences(of: "wav", with: "mp3")
var pcm: UnsafeMutablePointer<FILE> = fopen(path, "rb")
fseek(pcm, 4*1024, SEEK_CUR)
let mp3: UnsafeMutablePointer<FILE> = fopen(mp3path, "wb")
let PCM_SIZE: Int = 8192
let MP3_SIZE: Int32 = 8192
let pcmbuffer = UnsafeMutablePointer<Int16>.allocate(capacity: Int(PCM_SIZE*2))
let mp3buffer = UnsafeMutablePointer<UInt8>.allocate(capacity: Int(MP3_SIZE))
let lame = lame_init()
lame_set_num_channels(lame, 1)
lame_set_mode(lame, MONO)
lame_set_in_samplerate(lame, 44100)
lame_set_brate(lame, rate)
lame_set_VBR(lame, vbr_off)
lame_init_params(lame)
DispatchQueue.global(qos: .default).async {
while true {
pcm = fopen(path, "rb")
fseek(pcm, 4*1024 + total, SEEK_CUR)
read = fread(pcmbuffer, MemoryLayout<Int16>.size, PCM_SIZE, pcm)
if read != 0 {
write = lame_encode_buffer(lame, pcmbuffer, nil, Int32(read), mp3buffer, MP3_SIZE)
fwrite(mp3buffer, Int(write), 1, mp3)
total += read * MemoryLayout<Int16>.size
fclose(pcm)
} else if !self.isMP3Active {
_ = lame_encode_flush(lame, mp3buffer, MP3_SIZE)
_ = fwrite(mp3buffer, Int(write), 1, mp3)
break
} else {
fclose(pcm)
usleep(50)
}
}
lame_close(lame)
fclose(mp3)
fclose(pcm)
self.filePathMP3 = mp3path
}
}
func stopMP3Rec() {
self.isMP3Active = false
}
as first time run app, i log the last format using
print("last format: \(audioEngine.inputNode.inputFormat(forBus: 0).sampleRate)")
--> return 0 -> record normally
next time return 44100 -> record normally
but after webrtc call, i got 48000, then it make crash in this line
self.audioEngine.connect(self.audioEngine.inputNode, to: self.mixer, format: format)
i spend 4 hour in stackoverflow but no solution work for me.
i dont want 48000 format, because i have set the sample to
sampleRate: audioEngine.inputNode.inputFormat(forBus: 0).sampleRate,
-> my output is hard to hear, i can recognize my voice :(
So i think 44100 is the best
can someone give me some advices? Thanks
The down sample part , more vivid on your case.
let bus = 0
let inputNode = audioEngine.inputNode
let inputFormat = inputNode.outputFormat(forBus: bus)
let outputFormat = AVAudioFormat(commonFormat: .pcmFormatFloat32, sampleRate: 44100, channels: 1, interleaved: true)!
let converter = AVAudioConverter(from: inputFormat, to: outputFormat)!
inputNode.installTap(onBus: bus, bufferSize: 1024, format: inputFormat){ (buffer: AVAudioPCMBuffer, when: AVAudioTime) in
var newBufferAvailable = true
let inputCallback: AVAudioConverterInputBlock = { inNumPackets, outStatus in
if newBufferAvailable {
outStatus.pointee = .haveData
newBufferAvailable = false
return buffer
} else {
outStatus.pointee = .noDataNow
return nil
}
}
let convertedBuffer = AVAudioPCMBuffer(pcmFormat: outputFormat, frameCapacity: AVAudioFrameCount(outputFormat.sampleRate) * buffer.frameLength / AVAudioFrameCount(buffer.format.sampleRate))!
var error: NSError?
let status = converter.convert(to: convertedBuffer, error: &error, withInputFrom: inputCallback)
// 44100 Hz buffer
print(convertedBuffer.format)
}
This line bugs.
let format = AVAudioFormat(commonFormat: AVAudioCommonFormat.pcmFormatInt16, ...
AVAudioCommonFormat.pcmFormatInt16 not works by default.
You should use .pcmFormatFloat32
And the xcode tip is obvious,
the crash line
self.audioEngine.connect(self.audioEngine.inputNode, to: self.mixer, format: format)
You know it by print mixer.inputFormat(forBus: 0 )
then you got sample rate 48000 by the actual device. you can get 44100 by converting
just use AVAudioConverter to do down sample audio buffer.
let input = engine.inputNode
let bus = 0
let inputFormat = input.outputFormat(forBus: bus )
guard let outputFormat = AVAudioFormat(commonFormat: .pcmFormatFloat32, sampleRate: 44100, channels: 1, interleaved: true), let converter = AVAudioConverter(from: inputFormat, to: outputFormat) else{
return
}
if let convertedBuffer = AVAudioPCMBuffer(pcmFormat: outputFormat, frameCapacity: AVAudioFrameCount(outputFormat.sampleRate) * buffer.frameLength / AVAudioFrameCount(buffer.format.sampleRate)){
var error: NSError?
let status = converter.convert(to: convertedBuffer, error: &error, withInputFrom: inputCallback)
assert(status != .error)
print(convertedBuffer.format)
}
Only saw this in the iOS simulator:
I spent over an hour going down a rat hole on this. I had worked on a Logic audio session with some headphones (at 48K) on and then went over to my iOS simulator to work on my audio code for my app and started getting this crash. Unplugged my headphones, still crashed. Rebooted simulator, deleted app from the simulator, restarted XCode and machine, still crashed.
Finally I went to system preference on my mac, selected:
Sound & Input, plugged in my headphones so it says "External Microphone".
Also went to the simulator I/O settings for audio input set to "Internal Microphone"
Now my app was able to startup in the simulator without crashing while trying to create an AKMicrophone()...
I tried the accepted answer but it didn't work for me.
I was able to fix it by declaring audioEngine instance variable as optional. Right before when I need to monitor or record sound. I used to assign a new object of type AVAudioEngine to it.
Upon ending the recording session. I call audioEngine!.stop and then assign it to nil to deallocate the object.
I'm having trouble converting a linear PCM buffer to a compressed AAC ELD (Enhanced Low Delay) buffer.
I got some working code for the conversion into ilbc format from this question:
AVAudioCompressedBuffer to UInt8 array and vice versa
This approach worked fine.
I changed the input for the format to this:
let packetCapacity = 8
let maximumPacketSize = 96
lazy var capacity = packetCapacity * maximumPacketSize // 768
let convertedSampleRate: Double = 16000
lazy var aaceldFormat: AVAudioFormat = {
var descriptor = AudioStreamBasicDescription(mSampleRate: convertedSampleRate, mFormatID: kAudioFormatMPEG4AAC_ELD, mFormatFlags: 0, mBytesPerPacket: 0, mFramesPerPacket: 0, mBytesPerFrame: 0, mChannelsPerFrame: 1, mBitsPerChannel: 0, mReserved: 0)
return AVAudioFormat(streamDescription: &descriptor)!
}()
The conversion to a compressed buffer worked fine and I was able to convert the buffer to a UInt8 Array.
However, the conversion back to a PCM Buffer didn't work. The input block for the conversion back to a buffer looks like this:
func convertToBuffer(uints: [UInt8], outcomeSampleRate: Double) -> AVAudioPCMBuffer? {
// Convert to buffer
let compressedBuffer: AVAudioCompressedBuffer = AVAudioCompressedBuffer(format: aaceldFormat, packetCapacity: AVAudioPacketCount(packetCapacity), maximumPacketSize: maximumPacketSize)
compressedBuffer.byteLength = UInt32(capacity)
compressedBuffer.packetCount = AVAudioPacketCount(packetCapacity)
var compressedBytes = uints
compressedBytes.withUnsafeMutableBufferPointer {
compressedBuffer.data.copyMemory(from: $0.baseAddress!, byteCount: capacity)
}
guard let audioFormat = AVAudioFormat(
commonFormat: AVAudioCommonFormat.pcmFormatFloat32,
sampleRate: outcomeSampleRate,
channels: 1,
interleaved: false
) else { return nil }
guard let uncompressor = getUncompressingConverter(outputFormat: audioFormat) else { return nil }
var newBufferAvailable = true
let inputBlock : AVAudioConverterInputBlock = {
inNumPackets, outStatus in
if newBufferAvailable {
outStatus.pointee = .haveData
newBufferAvailable = false
return compressedBuffer
} else {
outStatus.pointee = .noDataNow
return nil
}
}
guard let uncompressedBuffer: AVAudioPCMBuffer = AVAudioPCMBuffer(pcmFormat: audioFormat, frameCapacity: AVAudioFrameCount((audioFormat.sampleRate / 10))) else { return nil }
var conversionError: NSError?
uncompressor.convert(to: uncompressedBuffer, error: &conversionError, withInputFrom: inputBlock)
if let err = conversionError {
print("couldnt decompress compressed buffer", err)
}
return uncompressedBuffer
}
The error block after the convert method triggers and prints out "too few bits left in input buffer". Also, it seems like the input block only gets called once.
I've tried different codes and this seems to be one of the most common outcomes. I'm also not sure if the problem is in the initial conversion from the pcm buffer to uint8 array although I get an UInt8 Array filled with 768 values every 0.1 seconds (Sometimes the array contains a few zeros at the end, which doesn't happen in ilbc format.
Questions:
1. Is the initial conversion from pcm buffer to uint8 array done with the right approach? Are the packetCapacity, capacity and maximumPacketSize valid? -> Again, seems to work
2. Am I missing something at the conversion back to pcm buffer? Also, am I using the variables in the right way?
3. Has anyone achieved this conversion without using C in the project?
** EDIT: ** I also worked with the approach from this post:
Decode AAC to PCM format using AVAudioConverter Swift
It works fine with AAC format, but not with AAC_LD or AAC_ELD
I am using AVAudioEngine to take a measurement. I play a stimulus sound out of my interface, and use a micTap to record the returned signal.
I am now looking at different Audio Interfaces which support a multitude of different formats. I am converting the input format of the inputNode via a mixer for two different reasons:
to downsample from the interfaces' preferred sampleRate to the sampleRate at which my app is working
to convert the incoming channel count to a single mono channel
I try this, however it does not always seem to work as expected. If my interface is running 96k and my app is running 48k, doing a format change via a mixer ends up with the following:
This looks like it is only getting one side of a stereo interleaved channel. Below is my audioEngine code:
func initializeEngine(inputSweep:SweepFilter) {
buf1current = 0
buf2current = 0
in1StartTime = 0
in2startTime = 0
in1firstRun = true
in2firstRun = true
in1Buf = Array(repeating:0, count:1000000)
in2Buf = Array(repeating:0, count:1000000)
engine.stop()
engine.reset()
engine = AVAudioEngine()
numberOfSamples = 0
var time:Int = 0
do {
try AVAudioSession.sharedInstance().setCategory(.playAndRecord)
try AVAudioSession.sharedInstance()
.setPreferredSampleRate(Double(sampleRate))
} catch {
assertionFailure("AVAudioSession setup failed")
}
let format = engine.outputNode.inputFormat(forBus: 0)
let stimulusFormat = AVAudioFormat(commonFormat: .pcmFormatFloat32,
sampleRate: Double(sampleRate),
channels: 1,
interleaved: false)
let outputFormat = engine.outputNode.inputFormat(forBus: 0)
let inputFormat = engine.inputNode.outputFormat(forBus: 0)
let srcNode = AVAudioSourceNode { _, timeStamp, frameCount, AudioBufferList -> OSStatus in
let ablPointer = UnsafeMutableAudioBufferListPointer(AudioBufferList)
if self.in2firstRun == true {
let start2 = CACurrentMediaTime()
self.in2startTime = Double(CACurrentMediaTime())
self.in2firstRun = false
}
if Int(frameCount) + time >= inputSweep.stimulus.count{
self.running = false
print("AUDIO ENGINE STOPPED")
}
if (Int(frameCount) + time) <= inputSweep.stimulus.count {
for frame in 0..<Int(frameCount) {
let value = inputSweep.stimulus[frame + time] * Float(outputVolume)
for buffer in ablPointer {
let buf: UnsafeMutableBufferPointer<Float> = UnsafeMutableBufferPointer(buffer)
buf[frame] = value
}
}
time += Int(frameCount)
} else {
for frame in 0..<Int(frameCount) {
let value = 0
for buffer in ablPointer {
let buf: UnsafeMutableBufferPointer<Float> = UnsafeMutableBufferPointer(buffer)
buf[frame] = Float(value)
}
}
}
return noErr
}
engine.attach(srcNode)
engine.connect(srcNode, to: engine.mainMixerNode, format: stimulusFormat)
engine.connect(engine.mainMixerNode, to: engine.outputNode, format: format)
let requiredFormat = AVAudioFormat(commonFormat: .pcmFormatFloat32,
sampleRate: Double(sampleRate),
channels: 1,
interleaved: false)
let formatMixer = AVAudioMixerNode()
engine.attach(formatMixer)
engine.connect(engine.inputNode, to: formatMixer, format: inputFormat)
let MicSinkNode = AVAudioSinkNode() { (timeStamp, frames, audioBufferList) ->
OSStatus in
if self.in1firstRun == true {
let start1 = CACurrentMediaTime()
self.in1StartTime = Double(start1)
self.in1firstRun = false
}
let ptr = audioBufferList.pointee.mBuffers.mData?.assumingMemoryBound(to: Float.self)
var monoSamples = [Float]()
monoSamples.append(contentsOf: UnsafeBufferPointer(start: ptr, count: Int(frames)))
if self.buf1current >= 100000 {
self.running = false
}
for frame in 0..<frames {
self.in1Buf[self.buf1current + Int(frame)] = monoSamples[Int(frame)]
}
self.buf1current = self.buf1current + Int(frames)
return noErr
}
engine.attach(MicSinkNode)
engine.connect(formatMixer, to: MicSinkNode, format: requiredFormat)
engine.prepare()
assert(engine.inputNode != nil)
running = true
try! engine.start()
}
My sourceNode is an array of Floats synthesised to use the stimulusFormat. If I listen to this audioEngine with my interface at 96k, the output stimulus sounds completely clean. However this broken up signal is what is coming from the micTap. Physically the output of the interface is routed. directly to the input, so not going through any other device.
Further to this, I have the following function, which records my arrays to WAV files so that I can visually inspect in a DAW.
func writetoFile(buff:[Float], name:String){
let SAMPLE_RATE = sampleRate
let outputFormatSettings = [
AVFormatIDKey:kAudioFormatLinearPCM,
AVLinearPCMBitDepthKey:32,
AVLinearPCMIsFloatKey: true,
AVLinearPCMIsBigEndianKey: true,
AVSampleRateKey: SAMPLE_RATE,
AVNumberOfChannelsKey: 1
] as [String : Any]
let fileName = name
let DocumentDirURL = try! FileManager.default.url(for: .documentDirectory, in: .userDomainMask, appropriateFor: nil, create: true)
let url = DocumentDirURL.appendingPathComponent(fileName).appendingPathExtension("wav")
//print("FilePath: \(url.path)")
let audioFile = try? AVAudioFile(forWriting: url, settings: outputFormatSettings, commonFormat: AVAudioCommonFormat.pcmFormatFloat32, interleaved: false)
let bufferFormat = AVAudioFormat(settings: outputFormatSettings)
let outputBuffer = AVAudioPCMBuffer(pcmFormat: bufferFormat!, frameCapacity: AVAudioFrameCount(buff.count))
for i in 0..<buff.count {
outputBuffer?.floatChannelData!.pointee[i] = Float(( buff[i] ))
}
outputBuffer!.frameLength = AVAudioFrameCount( buff.count )
do{
try audioFile?.write(from: outputBuffer!)
} catch let error as NSError {
print("error:", error.localizedDescription)
}
}
If I set my interface to be 48k, and my app is working at 48k, if I inspect my reference signal and. my measurement signal, i get the following:
The measured signal is clearly a lot longer than the original stimulus. The physical file size. is the same as it is initialised as an empty array of fixed size. However at some point doing the format conversion, it is not correct.
If I put my interface at 44.1k and my app runs at 48k, I can see the regular 'glitches' in audio. So the format convert here is not working as it should do.
Can anyone see anything obviously wrong?
put the non-interleaved option "AVLinearPCMIsNonInterleaved" inside the format settings:
let outputFormatSettings = [
**AVLinearPCMIsNonInterleaved: 0,**
AVFormatIDKey:kAudioFormatLinearPCM,
AVLinearPCMBitDepthKey:32,
AVLinearPCMIsFloatKey: true,
AVLinearPCMIsBigEndianKey: true,
AVSampleRateKey: SAMPLE_RATE,
AVNumberOfChannelsKey: 1
] as [String : Any]
it worked for me, let me know
I get the callbacks from camera with for audio with data in the format of CMSampleBuffer but I am unable to convert this data to PCM data.
I followed the docs provided by Apple copyPCMData, UnsafeMutablePointer, AudioBufferList but all I get is 0.0 at the end.
Here is my code:
private let pcmBufferPointer = UnsafeMutablePointer<AudioBufferList>.allocate(capacity: 1024)
init(....){
//...
let unsafeRawPointer = UnsafeMutableRawPointer.allocate(byteCount: 4, alignment: 0)
let audioBuffer = AudioBuffer(mNumberChannels: 1, mDataByteSize: 4, mData: unsafeRawPointer)
let audioBufferList = AudioBufferList(mNumberBuffers: 0, mBuffers: audioBuffer)
self.pcmBufferPointer.initialize(repeating: audioBufferList, count: 1024)
}
//CMSampleBuffer obtained from AVCaptureAudioDataOutputSampleBufferDelegate
private func audioFrom(sampleBuffer: CMSampleBuffer) -> Void {
let status = CMSampleBufferCopyPCMDataIntoAudioBufferList(sampleBuffer, 0, 1024, pcmBufferPointer)
if status == 0 {
Logger.log(key: "Audio Sample Buffer Status", message: "Buffer copied to pointer")
let dataValue = pcmBufferPointer[0].mBuffers.mData!.load(as: Float32.self) //Tried with Int, Int16, Int32, Int64 and Float too
Logger.log(key: "PCM Data Value", message: "Data value : \(dataValue)") //prints 0.0
}else{
Logger.log(key: "Audio Sample", message: "Buffer allocation failed with status \(status)")
}
}
Finally got it working.
Had to add extra step for the conversion of AudioBufferList pointer to AudioList pointer
if status == 0 {
let inputDataPtr = UnsafeMutableAudioBufferListPointer(pcmBufferPointer)
let mBuffers : AudioBuffer = inputDataPtr[0]
if let bufferPointer = UnsafeMutableRawPointer(mBuffers.mData){
let dataPointer = bufferPointer.assumingMemoryBound(to: Int16.self)
let dataArray = Array(UnsafeBufferPointer.init(start: dataPointer, count: 1024))
pcmArray.append(contentsOf: dataArray)
}else{
Logger.log(key: "Audio Sample", message: "Failed to generate audio sample")
}
}else{
Logger.log(key: "Audio Sample", message: "Buffer allocation failed with status \(status)")
}
Above code works only for single channel PCM data. For 2 channels data refer the following GIST - https://gist.github.com/hotpaw2/ba815fc23b5d642705f2b1dedfaf0107