I've been trying to use Apple's CoreAudio from swift.
I found many examples on how to enumerate streams and channels on a device.
However, all of them seem to use incorrect size when calling UnsafeMutablePointer<AudioBufferList>.allocate().
They first request property data size, which returns the number of bytes.
Then they use this number of bytes to allocate an (unsafe) AudioBufferList of that size (using the number of bytes as the size of the list!).
Please see my comments inline below:
var address = AudioObjectPropertyAddress(
mSelector:AudioObjectPropertySelector(kAudioDevicePropertyStreamConfiguration),
mScope:AudioObjectPropertyScope(kAudioDevicePropertyScopeInput),
mElement:0)
var propsize = UInt32(0);
var result:OSStatus = AudioObjectGetPropertyDataSize(self.id, &address, 0, nil, &propsize);
if (result != 0) {
return false;
}
// ABOVE: propsize is set to number of bytes that property data contains, typical number are 8 (no streams), 24 (1 stream, 2 interleaved channels)
// BELOW: propsize is used for AudioBufferList capacity (in number of buffers!)
let bufferList = UnsafeMutablePointer<AudioBufferList>.allocate(capacity:Int(propsize))
result = AudioObjectGetPropertyData(self.id, &address, 0, nil, &propsize, bufferList);
if (result != 0) {
return false
}
let buffers = UnsafeMutableAudioBufferListPointer(bufferList)
for bufferNum in 0..<buffers.count {
if buffers[bufferNum].mNumberChannels > 0 {
return true
}
}
This works all of the time, because it allocates much more memory than needed for UnsafeMutablePointer<AudioBufferList>, but this is obviously wrong.
I've been searching for a way to correctly allocate UnsafeMutablePointer<AudioBufferList> from the number of bytes that is returned by AudioObjectGetPropertyDataSize(), but I cannot find anything for the whole day. Please help ;)
to correctly allocate UnsafeMutablePointer<AudioBufferList> from the number of bytes that is returned by AudioObjectGetPropertyDataSize()
You should not allocate UnsafeMutablePointer<AudioBufferList>, but allocate raw bytes of the exact size and cast it to UnsafeMutablePointer<AudioBufferList>.
Some thing like this:
let propData = UnsafeMutableRawPointer.allocate(byteCount: Int(propsize), alignment: MemoryLayout<AudioBufferList>.alignment)
result = AudioObjectGetPropertyData(self.id, &address, 0, nil, &propsize, propData);
if (result != 0) {
return false
}
let bufferList = propData.assumingMemoryBound(to: AudioBufferList.self)
I fully agree with the accepted answer of using UnsafeMutableRawPointer.allocate(byteCount:alignment:) (though it should also be paired with a call to deallocate() for getting the device stream configuration, but just wanted to share another option for completeness (this shouldn't be upvoted for this question)
If you truly need to calculate the number of buffers from the number of bytes (I'm not sure there is actually any such need), it can be done.
When first converting code to Swift, I used something like :
let numBuffers = (Int(propsize) - MemoryLayout<AudioBufferList>.offset(of: \AudioBufferList.mBuffers)!) / MemoryLayout<AudioBuffer>.size
if numBuffers == 0 { // Avoid trying to allocate zero buffers
return false
}
let bufferList = AudioBufferList.allocate(maximumBuffers: numBuffers)
defer { bufferList.unsafeMutablePointer.deallocate() }
err = AudioObjectGetPropertyData(id, &address, 0, nil, &propsize, bufferList.unsafeMutablePointer)
Again, I do NOT actually recommend this approach to get the stream configuration - it's unnecessarily complex IMO, and I've since adopted something like the accepted answer. So this may not have value other than as an academic exercise.
Related
I'm trying to read information from AudioUnit:
var value = AudioChannelLayoutTag()
var size = UInt32(MemoryLayout<AudioChannelLayoutTag>.size)
let status = AudioUnitGetProperty(audioUnit, kAudioUnitProperty_SupportedChannelLayoutTags, kAudioUnitScope_Global, 0, &value, &size)
it always brings me back a mistake:
kAudioUnitErr_InvalidElement (err code: -10877)
can you advise me please?
The kAudioUnitProperty_SupportedChannelLayoutTags uses a scope of either kAudioUnitScope_Input or kAudioUnitScope_Output. That is where the error is from.
If the kAudioUnitScope_Global is changed to either kAudioUnitScope_Input or kAudioUnitScope_Output, you code will work as expected. But it will only return the first element of an array of tags that the audio unit supports.
When performing AudioUnitGetProperty on a kAudioUnitProperty_SupportedChannelLayoutTags, it puts the tags into the array with the size you specified. It also modifies the size you specified to the valid buffer's size.
Unfortunately, there is no way to know what the number of tags before trying to access it. So we can only access it progressively.
The following code returns the number of tags supported by the output of an audio unit. It tries to access 4 elements first, then expands that number by doubling until the returned element size is smaller than specified.
func supportedOutputChannelLayoutTags() -> [AudioChannelLayoutTag] {
let elementSize = MemoryLayout<AudioChannelLayoutTag>.size
var elementCount = 4
while true {
var layoutTags = [AudioChannelLayoutTag](repeating: kAudioChannelLayoutTag_Unknown, count: elementCount)
let ioSize = UInt32(elementSize * layoutTags.count)
var returnedIoSize = ioSize
AudioUnitGetProperty(self, kAudioUnitProperty_SupportedChannelLayoutTags, kAudioUnitScope_Output, 0, &layoutTags, &returnedIoSize)
if returnedIoSize != ioSize {
layoutTags.removeLast(Int(ioSize - returnedIoSize) / elementSize)
return layoutTags
}
//Guard against cases where there is no tags.
//The `AudioUnitGetProperty` will not modify &returnedIoSize if no tag is copied.
if layoutTags.first == kAudioChannelLayoutTag_Unknown { return [] }
elementCount *= 2
}
}
I'm trying to read an MP4 file in chunks of 1024 bytes. I've made a code that - almost - works. I'm doing the following:
let audioFilePath = Bundle.main.path(forResource: "video", ofType: "mp4")!
var chunks = [[UInt8]]()
if let stream: InputStream = InputStream(fileAtPath: audioFilePath) {
var buf: [UInt8] = [UInt8](repeating: 0, count: 1024)
stream.open()
while stream.hasBytesAvailable {
stream.read(&buf, maxLength: 1024)
chunks.append(buf)
}
stream.close()
}
print(chunks.count)
The problem with the code above is that I'm reading an MP4 file of size 15.948.514 bytes. It means that it should finish in exactly 15.574 chunks (the last chunk may have less than 1024, but this is not a problem), but the code prints 15.576 chunks, and all of them of size 1024. What is wrong with the code above?
hasBytesAvailable can also return true if a read must be attempted in order to determine the availability of bytes. That is what happens in your case: The final read returns zero for “end of file.”
hasBytesAvailable can be useful with input streams like TCP sockets to avoid a blocking read(), but is not really needed for reading from files. In any case, you must check the return value of read() which can be zero (end of file) or -1 (read error) or the actual number of bytes read into the buffer (which can be less than the number of bytes requested).
Note also that you always append a chunk with 1024 bytes to the chunks array, even if the buffer is only partially filled with bytes from the input stream.
if let stream = InputStream(fileAtPath: audioFilePath) {
var buf = [UInt8](repeating: 0, count: 1024)
stream.open()
while case let amount = stream.read(&buf, maxLength: 1024), amount > 0 {
// print(amount)
chunks.append(Array(buf[..<amount]))
}
stream.close()
}
I have a function that is passed an optional MTLBuffer. My goal is to iteratively change values in that buffer using an index into a Typed pointer to the same buffer. However, when I run the code, I'm getting an error "Thread 1: EXC_BAD_ACCESS (code=2, address=0x1044f1000)".
Am I converting to the Typed UnsafeMutablePointer correctly?
Would it be better to covert to a Typed UnsafeMutableBufferPointer? If so, how would I convert from the MTLBuffer to Typed UnsafeMutableBufferPointer?
Any idea why I'm getting this error?
Note: I've removed most guard checks to keep this simple. I've confirmed the MTLDevice (via device), bufferA allocation, dataPtr and floatPtr are all non-nil. floatPtr and dataPtr do point to the same memory address.
This is how I allocate the buffer:
bufferSize = 16384
bufferA = device?.makeBuffer(length: bufferSize, options: MTLResourceOptions.storageModeShared)`
Here's my code operating on the buffer:
guard let dataPtr = bufferA?.contents() else {
fatalError("error retrieving buffer?.contents() in generateRandomFloatData()")
}
let floatPtr = dataPtr.bindMemory(to: Float.self, capacity: bufferA!.length)
for index in 0...bufferSize - 1 {
floatPtr[index] = 1.0 // Float.random(in: 0...Float.greatestFiniteMagnitude)
}
Thank you!
Am I converting to the Typed UnsafeMutablePointer correctly?
NO.
When you call makeBuffer(length:options:) you pass the length in bytes.
But, a Float occupies 4 bytes in memory.
So, you may need to modify some parts related to number of elements:
let floatPtr = dataPtr.bindMemory(to: Float.self, capacity: bufferA!.length/MemoryLayout<Float>.stride)
for index in 0..<bufferSize/MemoryLayout<Float>.stride {
floatPtr[index] = 1.0 // Float.random(in: 0...Float.greatestFiniteMagnitude)
}
I am receiving audio buffers and I am converting them into a conventional array for ease of use. This code has always been reliable. However, recently it began crashing quite frequently. I am using Airpods when it crashes, which may or may not be part of the problem. The mic object is an AKMicrophone object from AudioKit.
func tap(){
let recordingFormat = mic.outputNode.inputFormat(forBus: 0)
mic.outputNode.removeTap(onBus: 0)
mic.outputNode.installTap(onBus: 0,
bufferSize: UInt32(recordingBufferSize),
format: recordingFormat)
{ (buffer, when) in
let stereoDataUnsafePointer = buffer.floatChannelData!
let monoPointer = stereoDataUnsafePointer.pointee
let count = self.recordingBufferSize
let bufferPointer = UnsafeBufferPointer(start: monoPointer, count: count)
let array = Array(bufferPointer) //CRASHES HERE
}
mic.start()
}
When running on iPhone 7 with airpods, this crashes about 7/10 times with one of two different error messages:
EXC_BAD_ACCESS
Fatal error: UnsafeMutablePointer.initialize overlapping range
If the way I was converting the array was wrong I would expect it to crash every time. I speculate that the recording sample rate could be an issue.
I figured out the answer. I hardcoded the buffer size to 10000, and when initiating the tap, specified buffer size of 10000. However, the device ignored this buffer size and instead sent me buffers that were 6400. This meant when I tried to initialize an array of with size 10000 it went off the end. I modified code to check the actual buffer size, not the size I requested:
let stereoDataUnsafePointer = buffer.floatChannelData!
let monoPointer = stereoDataUnsafePointer.pointee
let count = buffer.frameLength //<--Check actual buffer size
let bufferPointer = UnsafeBufferPointer(start: monoPointer, count: Int(count))
let array = Array(bufferPointer)
I am writing an application that needs to detect a frequency in the audio stream. I have read about a million articles and am having problems crossing the finish line. I have my audio data coming to me in this function via the AVFoundation Framework from Apple.
I am using Swift 4.2 and have tried playing with the FFT functions, but they are a little over my head at the current moment.
Any thoughts?
// get's the data as a call back for the AVFoundation framework.
public func captureOutput(_ output: AVCaptureOutput, didOutput sampleBuffer: CMSampleBuffer, from connection: AVCaptureConnection) {
// prints the whole sample buffer and tells us alot of information about what's inside
print(sampleBuffer);
// create a buffer, ready out the data, and use the CMSampleBufferGetAudioBufferListWithRetainedBlockBuffer method to put
// it into a buffer
var buffer: CMBlockBuffer? = nil
var audioBufferList = AudioBufferList(mNumberBuffers: 1,
mBuffers: AudioBuffer(mNumberChannels: 1, mDataByteSize: 0, mData: nil))
CMSampleBufferGetAudioBufferListWithRetainedBlockBuffer(sampleBuffer, bufferListSizeNeededOut: nil, bufferListOut: &audioBufferList, bufferListSize: MemoryLayout<AudioBufferList>.size, blockBufferAllocator: nil, blockBufferMemoryAllocator: nil, flags: UInt32(kCMSampleBufferFlag_AudioBufferList_Assure16ByteAlignment), blockBufferOut: &buffer);
let abl = UnsafeMutableAudioBufferListPointer(&audioBufferList)
var sum:Int64 = 0
var count:Int = 0
var bufs:Int = 0
var max:Int64 = 0;
var min:Int64 = 0
// loop through the samples and check for min's and maxes.
for buff in abl {
let samples = UnsafeMutableBufferPointer<Int16>(start: UnsafeMutablePointer(OpaquePointer(buff.mData)),
count: Int(buff.mDataByteSize)/MemoryLayout<Int16>.size)
for sample in samples {
let s = Int64(sample)
sum = (sum + s*s)
count += 1
if(s > max) {
max = s;
}
if(s < min) {
min = s;
}
print(sample)
}
bufs += 1
}
// debug
print("min - \(min), max = \(max)");
// update the interface
DispatchQueue.main.async {
self.frequencyDataOutLabel.text = "min - \(min), max = \(max)";
}
// stop the capture session
self.captureSession.stopRunning();
}
After much research I found that the answer is to use an FFT method (Fast Fourier Transform). It takes the raw input from the iPhone's code above and converts it into an array of values representing magnitude of each frequency in bands.
Much props to the open code here https://github.com/jscalo/tempi-fft that created a visualizer that captures the data, and displays it. From there, it was a matter of manipulating it to meet the needs. In my case I was looking for frequencies very high above human hearing (20kHz range). By scanning the later half of the array in the tempi-fft code I was able to determine if frequencies I was looking for were present and loud enough.