I am trying to do some computation on the raw PCM samples of a mp3 files I'm playing with an AVAudioEngine graph. I have a closure every 44100 samples that provides an AVAudioPCMBuffer. It has a property channelData of type UnsafePointer<UnsafeMutablePointer<Float>>?. I have not worked with pointers in Swift 3 and so I'm unclear how to access these Float values.
I have the following code but there are many issues:
audioPlayerNode.installTap(onBus: 0,
bufferSize: 1024,
format: audioPlayerNode.outputFormat(forBus: 0)) { (pcmBuffer, time) in
let numChans = Int(pcmBuffer.format.channelCount)
let frameLength = pcmBuffer.frameLength
if let chans = pcmBuffer.floatChannelData?.pointee {
for a in 0..<numChans {
let samples = chans[a]// samples is type Float. should be pointer to Floats.
for b in 0..<flength {
print("sample: \(b)") // should be samples[b] but that gives error as "samples" is Float
}
}
}
For instance, how do I iterate through the UnsafeMutablePointer<Floats which are N float pointers where N is the number of channels in the buffer. I could not find discussion on accessing buffer samples in the Apple Docs on this Class.
I think the main problem is let samples = chans[a]. Xcode says chans is of type UnsafeMutablePointer<Float>. But that should be NumChannels worth of those pointers. Which is why I use a in 0..<numChans to subscript it. Yet I get just Float when I do.
EDIT:
hm, seems using chans.advanced(by: a) instead of subscripting fixed things
Here is what I've found:
let arraySize = Int(buffer.frameLength)
let samples = Array(UnsafeBufferPointer(start: buffer.floatChannelData![0], count:arraySize))
This is assuming buffer is the name of your AVAudioPCMBuffer.
This way you can avoid pointers, which is likely much simpler. Now you can actually search through the data using a for loop.
Related
I have written a sample project in Swift to try out the relatively new Core Audio V3 API stuff. Everything seems to work around creating a custom Audio Unit and loading it in process. But the actual audio rendering isn't going so well. I've often read that the rendering code needs to be in C or C++ but I've also heard Swift is fast and thought I could write some minimal audio rendering code in it.
the rendering code
override var internalRenderBlock: AUInternalRenderBlock {
get {
return {
(_ actionFlags: UnsafeMutablePointer<AudioUnitRenderActionFlags>,
_ timeStamp: UnsafePointer<AudioTimeStamp>,
_ frameCount: AUAudioFrameCount,
_ outputBusNumber: Int,
_ bufferList: UnsafeMutablePointer<AudioBufferList>,
_ renderEvent: UnsafePointer<AURenderEvent>?,
_ pull: AudioToolbox.AURenderPullInputBlock?) -> AUAudioUnitStatus in
let bufferList = bufferList.pointee
let theBuffers = bufferList.mBuffers // only one (AudioBuffer) ??
guard let theBufferData = theBuffers.mData?.assumingMemoryBound(to: Float.self) else {
return 1 // come up with better error?
}
let amountFrames = Int(frameCount)
for frame in 0...amountFrames / 2 {
let frame = theBufferData.advanced(by: frame)
frame.pointee = sin(self.phase)
self.phase += 0.0001
}
return noErr
}
}
}
Sounds Bad
The resulting sound is not what I'd expect. My initial thoughts are that Swift is the wrong choice. Yet Interestingly, AudioToolbox does provide a typealias for this AUAudioUnit's rendering property which looks like:
public typealias AUInternalRenderBlock = (UnsafeMutablePointer<AudioUnitRenderActionFlags>, UnsafePointer<AudioTimeStamp>, AUAudioFrameCount, Int, UnsafeMutablePointer<AudioBufferList>, UnsafePointer<AURenderEvent>?, AudioToolbox.AURenderPullInputBlock?) -> AUAudioUnitStatus
This would lead me to believe that it is perhaps possible to write rendering code in Swift.
observed problems
But still, there are a few things going wrong here. (aside from my obvious lack of competency with Swift memory management stuff).
A) despite theBuffers saying that its mNumberOfBuffers is 2, theBuffers winds up not being an array but rather of type (AudioBuffer). I don't understand the need for parenthesis. I can't find a second AudioBuffer.
B) more importantly, when I write a basic sin wave to the one AudioBuffer I can access, the resulting sound is distorted and inconsistent. Could this be Swift's fault? Is it just impossible to write any audio unit rendering code in Swift? Or have a made some assumptions here that is breaking my rendering somehow?
Finally
If it is simply the case that writing this part in Swift is infeasible, then I would like to have some resources on interoperating Swift and C for Audio Unit rendering blocks. So, could the property returning the closure be written in Swift, but the closure's implementation calls down into C? or does the property have to simply return a C function whose prototype matches the closure's type?
Thanks in advance.
The rest of this project can be seen here for context.
The main reason that you were listening a distorted sound was that the phase increment of 0.0001 is too small, which would take 62832 samples to fill up one period of the sine wave -- merely 0.70 hertz! (Assuming your sample rate is 44100)
In addition to the ultra-low-frequency sine wave, you were listening to a sound of about 44100 / 512 = 86.1 Hz, because you were filling only the half of the audio buffer (amountFrames / 2). So the sound was a near-rectangular wave of the period of your audio rendering period, with slowly varying amplitude in about 0.70 Hz.
I could write a working sine wave generator unit based on your code:
override var internalRenderBlock: AUInternalRenderBlock {
return { ( _, _, frameCount, _, bufferList, _, _) in
let srate = Float(self.bus.format.sampleRate)
var phase = self.phase
for buffer in UnsafeMutableAudioBufferListPointer(bufferList) {
phase = self.phase
assert(buffer.mNumberChannels == 1, "interleaved channel not supported")
let frames = buffer.mData!.assumingMemoryBound(to: Float.self)
for i in 0 ..< Int(frameCount) {
frames[i] = sin(phase)
phase += 2 * .pi * 440 / srate // 440 Hz
if phase > 2 * .pi {
phase -= 2 * .pi // to avoid floating point inaccuracy
}
}
}
self.phase = phase
return noErr
}
}
Regarding the observed problem A, the AudioBufferList is a wrapper for variable length C struct, where the first field mNumberBuffers indicates the number of buffers (i.e. number of non-interleaved channels), and the second field is a variable length array:
typedef struct AudioBufferList {
UInt32 mNumberBuffers;
AudioBuffer mBuffers[1];
} AudioBufferList;
The user of this struct, in Objective-C or C++, is expected to allocate mNumberBuffers * sizeof(AudioBuffer) bytes, which is enough for storing multiple mBuffers. Since C does not perform boundary checks on arrays, the users could just write mBuffers[1] or mBuffers[2] to access the second or third buffer.
Because Swift doesn't have this variable length array feature, Apple provides UnsafeMutableAudioBufferListPointer, which can be used like a Swift collection of AudioBuffers; I used this in the outer for loop above.
Finally, I tried not to access self in the innermost loop in the code, because accessing a Swift or Objective-C object might involve unexpected lags, which was the reason why Apple recommends writing rendering loop in C/C++. But for simple cases like this, I would say writing in Swift is a lot easier and the latency is still manageable.
I'm trying to initialize Data from an UnsafeBufferPointer, but it's throwing an EXC_BAD_ACCESS when it hits the third line. Help appreciated.
let pointer = UnsafePointer<UInt8>(UnsafePointer<UInt8>(bitPattern: 15)!)
let buffer = UnsafeBufferPointer<UInt8>(start: pointer, count: 1)
let data = Data(bytes: Array(buffer)) // EXC_BAD_ACCESS
My end goal is to convert bits of data to some human readable format (eg., convert a bit pattern of 15 to "F"). I was hoping to initialize a String from the data object as a hex value. I'm open to better and correct ways of going about this.
I am playing with C and Swift 3.0 code using vecLib and Accelerate framework from Apple as dynamic lib + my code in C lang based project and Swift playground.
And in situation with calling Apple's wrapper from framework of SIMD instruction with 1 or < 4 elements computation function like vvcospif() from framework is slower than simple standart cos(x * PI) when functions calls from loop near 1.000 times as example.
I know about difference between vvcospif() and cos(), I should use exactly vvcospif() for x * PI.
Example in playground, you can just copy code and run it:
import Cocoa
import Accelerate
func cosine_interpolate(alpha: Float, a: Float, b: Float) -> Float {
let ft: Float = alpha * 3.1415927;
let f: Float = (1 - cos(ft)) * 0.5;
return a + f*(b - a);
}
var start: Date = NSDate() as Date
var interp: Float;
for index in 0..<1000 {
interp = cosine_interpolate(alpha: 0.25, a: 1.0, b: 0.75)
}
var end = NSDate();
var timeInterval: Double = end.timeIntervalSince(start);
print("cosine_interpolate in \(timeInterval) seconds")
func fast_cosine_interpolate(alpha: Float, a: Float, b: Float) -> Float {
var x: Float = alpha
var count: Int32 = 1
var result: Float = 0
vvcospif(&result, &x, &count)
let SINSIN_HALF_X: Float = (1 - result) * 0.5;
return a + SINSIN_HALF_X * (b - a);
}
start = NSDate() as Date
for index in 0..<1000 {
interp = fast_cosine_interpolate(alpha: 0.25, a: 1.0, b: 0.75)
}
end = NSDate();
timeInterval = end.timeIntervalSince(start);
print("fast_cosine_interpolate in \(timeInterval) seconds")
My question is:
Why vvcospif() is slow in this example?
May be because vvcospif() it is wrapper under Objective-C runtime and converting data structures / copying of memory from Intel SIMD -> Objective-C -> Swift runtime is slower then tiny cos()?
I also have performance issue with C code +
#include <Accelerate/Accelerate.h>
vvcospif(resultVector, inputVector, &count);
when inputVector and resultVector is small arrays with 1 or 2 elements or just float variable, and calls in loop with ~ 1.000.000 times.
cos(x * PI) computation time near 20 ms.
and
vvcospif(x) with processing one float or float array[2] - computation time near 80 ms! Where is Acceleration? :)
Yes, in Xcode I use compiler -O -whole-module-optimization optimisation with whole module opt. enabled.
For a more detailed discussion with examples, see "Introduction to Fast Bezier (and Trying the Accelerate.framework)".
The first, fundamental problem is that non-inlined function calls are extremely expensive. You don't want function calls if you can possibly help it in performance-critical code. Within a module, the compiler can often inline functions for you, and parts of stdlib can be inlined for you. But when you start crossing module barriers, Swift generally cannot optimize out the call.
The point of SIMD functions is that you set up all your data in the right format, and then call them just one time. That way the cost of the function call is made up by the SIMD optimized code you're calling.
But remember, you don't have to call into Accelerate to get SIMD optimizations. The compiler is perfectly capable of noticing you've written a loop and turning it into an inline SIMD algorithm itself (and it does this all the time). So for many simple problems, the compiler is going to win anyway. Think about it: if calling vvcospif with a count of 1 were faster than calling cos, wouldn't they just implement cos that way?
I haven't played with your code much, but if you want to improve its performance with Accelerate, you want to think about how to arrange all your input data so you can call vvcospif one time with a large N. It's quite possible in that case that it will be much faster that a loop (since cos is not trivial).
If you want an example of Accelerate in practice, and how you need to organize your data, see PinchText. This code is computing offsets for a page full of a few thousand glyphs based on up to 10 touches in real-time, with animations (see PinchText.mov for what the result looks like). In particular look at adjustViewPositions:count:forTouchPoint:. Notice how count is large, and the data is transformed step by step with no loops. Even throwing in a (very expensive) ObjC method call into that method doesn't matter very much because it's only made one time. Getting rid of function calls in loops is a huge part of performance programming.
I'm trying to fill an AVAudioPCMBuffer programmatically in Swift to build a metronome. This is the first real app I'm trying to build, so it's also my first audio app. Right now I'm experimenting with different frameworks and methods of getting the metronome looping accurately.
I'm trying to build an AVAudioPCMBuffer with the length of a measure/bar so that I can use the .Loops option of the AVAudioPlayerNode's scheduleBuffer method. I start by loading my file(2 ch, 44100 Hz, Float32, non-inter, *.wav and *.m4a both have same issue) into a buffer, then copying that buffer frame by frame separated by empty frames into the barBuffer. The loop below is how I'm accomplishing this.
If I schedule the original buffer to play, it will play back in stereo, but when I schedule the barBuffer, I only get the left channel. As I said I'm a beginner at programming, and have no experience with audio programming, so this might be my lack of knowledge on 32 bit float channels, or on this data type UnsafePointer<UnsafeMutablePointer<float>>. When I look at the floatChannelData property in swift, the description makes it sound like this should be copying two channels.
var j = 0
for i in 0..<Int(capacity) {
barBuffer.floatChannelData.memory[j] = buffer.floatChannelData.memory[i]
j += 1
}
j += Int(silenceLengthInSamples)
// loop runs 4 times for 4 beats per bar.
edit: I removed the glaring mistake i += 1, thanks to hotpaw2. The right channel is still missing when barBuffer is played back though.
Unsafe pointers in swift are pretty weird to get used to.
floatChannelData.memory[j] only accesses the first channel of data. To access the other channel(s), you have a couple choices:
Using advancedBy
// Where current channel is at 0
// Get a channel pointer aka UnsafePointer<UnsafeMutablePointer<Float>>
let channelN = floatChannelData.advancedBy( channelNumber )
// Get channel data aka UnsafeMutablePointer<Float>
let channelNData = channelN.memory
// Get first two floats of channel channelNumber
let floatOne = channelNData.memory
let floatTwo = channelNData.advancedBy(1).memory
Using Subscript
// Get channel data aka UnsafeMutablePointer<Float>
let channelNData = floatChannelData[ channelNumber ]
// Get first two floats of channel channelNumber
let floatOne = channelNData[0]
let floatTwo = channelNData[1]
Using subscript is much clearer and the step of advancing and then manually
accessing memory is implicit.
For your loop, try accessing all channels of the buffer by doing something like this:
for i in 0..<Int(capacity) {
for n in 0..<Int(buffer.format.channelCount) {
barBuffer.floatChannelData[n][j] = buffer.floatChannelData[n][i]
}
}
Hope this helps!
This looks like a misunderstanding of Swift "for" loops. The Swift "for" loop automatically increments the "i" array index. But you are incrementing it again in the loop body, which means that you end up skipping every other sample (the Right channel) in your initial buffer.
I am very new to Swift 2.0 and am pretty stuck here...
I have created a random number generator based off a user set range for low and high limits. This function worked great with my test function when I could set the variables for low and high to Uint32
Once I tried to store the low/high range in NSUserDefaults as an integer. Now when I try to load the values set by the user for the low and high to use in my function below it gives me the error that it needs to be a UInt32 value.
The problem is.... I cannot seem to figure out how to save them in NSUser as that type. Or unwrap it as that type coming out of the UserDefaults.
My specific error message is...
ERROR IS Binary Operator + cannot be applied to operands of type UInt32 and In
Thanks so much ahead of time for your help and time I truly appreciate it! You guys are awesome here!
Let me know if there is anything else you need from me :)
//CREATE SAVE SETTINGS FOR LOW AND HIGH RANGE
NSUserDefaults.standardUserDefaults().integerForKey("CatLowRange")
NSUserDefaults.standardUserDefaults().integerForKey("CatHighRange")
//Variables to load in values from NSUserDefaults on load
var catLowRange = 0 //Load In Low Range From NSUser
var catHighRange = 40 //Load In High Range From NSUser
//LOAD IN VALUES TO VARIABLES FROM NSUSERDEFAULTS
catLowRange = NSUserDefaults.standardUserDefaults().integerForKey("CatLowRange")
catHighRange = NSUserDefaults.standardUserDefaults().integerForKey("CatHighRange")
**//My Function To Generate RandNumber...**
*ERROR IS Binary Operator + cannot be applied to operands of type UInt32 and Int*
randomNumber = arc4random_uniform (catHighRange - catLowRange) + catLowRange
For anyone wondering... Thanks to Adam we have an answer... Here is the code broken down how I used his suggestion to make this thing work :)
//WORKING METHOD!?
//vars for use
let catLowRange : UInt32 = 0
let catHighRange : UInt32 = 40
var randNumGenerated : UInt32 = 0
//SAVEPOINTS (STEP 1 - save the UInt32s as int for saving)
NSUserDefaults.standardUserDefaults().setInteger(Int(catLowRange), forKey: "CatLowRange")
NSUserDefaults.standardUserDefaults().setInteger(Int(catHighRange), forKey: "CatHighRange")
//LOAD (STEP 2 - Take Int save as a UInt32 now)
let catLowRangeLoad = UInt32(NSUserDefaults.standardUserDefaults().integerForKey("CatLowRange"))
let catHighRangeLoad = UInt32(NSUserDefaults.standardUserDefaults().integerForKey("CatHighRange"))
//FUNCTION TEST
randNumGenerated = arc4random_uniform (catHighRangeLoad - catLowRangeLoad) + catLowRangeLoad
//CHECK
print ("catLowRange = \(catLowRange)")
print ("catLowRangeLoad = \(catLowRangeLoad)")
print ("catHighRange = \(catHighRange)")
print ("catHighRangeLoad = \(catHighRangeLoad)")
print ("randNumGen = \(randNumGenerated)")
Use this code (updated for swift 3):
let number : UInt32 = 8
UserDefaults.standard.set(NSNumber(value: number), forKey: "key")
let number2 = (UserDefaults.standard.value(forKey: "key") as? NSNumber)?.uint32Value