I've hunted high and low and cannot find a solution to this problem. I am looking for a method to change the input/output devices which an AVAudioEngine will use on macOS.
When simply playing back an audio file the following works as expected:
var outputDeviceID:AudioDeviceID = xxx
let result:OSStatus = AudioUnitSetProperty(outputUnit, kAudioOutputUnitProperty_CurrentDevice, kAudioUnitScope_Global, 0, &outputDeviceID, UInt32(MemoryLayout<AudioObjectPropertyAddress>.size))
if result != 0 {
print("error setting output device \(result)")
return
}
However if I initialize the audio input (with let input = engine.inputNode) then I get an error once I attempt to start the engine:
AVAEInternal.h:88 required condition is false: [AVAudioEngine.mm:1055:CheckCanPerformIO: (canPerformIO)]
I know that my playback code is OK since, if I avoid changing the output device then I can hear the microphone and the audio file, and if I change the output device but don't initialize the inputNode the file plays to the specified destination.
Additionally to this I have been trying to change the input device, I understood from various places that the following should do this:
let result1:OSStatus = AudioUnitSetProperty(inputUnit, kAudioOutputUnitProperty_CurrentDevice, kAudioUnitScope_Output, 0, &inputDeviceID, UInt32(MemoryLayout<AudioObjectPropertyAddress>.size))
if result1 != 0 {
print("failed with error \(result1)")
return
}
However, this doesn't work - in most cases it throws an error (10853) although if I select a sound card that has both inputs and outputs it succeeds - it appears that when I am attempting to set the output or the input node it is actually setting the device for both.
I would think that this meant that an AVAudioEngine instance can only deal with one device, however it is quite happy working with the default devices (mic and speakers/headphones) so I am confident that isn't the issue. Looking at some solutions I have seen online people simply change the default input, but this isn't a massively nice solution.
Does anyone have any ideas as to whether this is possible?
It's worth noting that kAudioOutputUnitProperty_CurrentDevice is the only property available, there is not an equivalent kAudioInputUnitProperty_CurrentDevice key, due to the fact that as I understand it both the inputNode and outputNode are classed as "Output Units" (as they both emit sound somewhere).
Any ideas would be much appreciated as this is very very frustrating!!
Thanks
So I filed a support request with apple on this and another issue and the response confirms that an AVAudioEngine can only be assigned to a single Aggregate device (that is, a device with both input and output channels) - the system default units create effectively an aggregate device internally which is why they work, although I've found an additional issue in that if the input device also has output capabilities (and you activate the inputNode) then that device has to be both the input and output device as otherwise the output appears not to work.
So answer is that I think there is no answer..
Related
My aim is to write an audio app for low latency realtime audio analysis on OSX. This will involve connecting to one or more USB interfaces and taking specific channels from these devices.
I started with the learning core audio book and writing this using C. As I went down this path it came to light that a lot of the old frameworks have been deprecated. It appears that the majority of what I would like to achieve can be written using AVAudioengine and connecting AVAudioUnits, digging down into core audio level only for the lower things like configuring the hardware devices.
I am confused here as to how to access two devices simultaneously. I do not want to create an aggregate device as I would like to treat the devices individually.
Using core audio I can list the audio device ID for all devices and change the default system output device here (and can do the input device using similar methods). However this only allows me one physical device, and will always track the device in system preferences.
static func setOutputDevice(newDeviceID: AudioDeviceID) {
let propertySize = UInt32(MemoryLayout<UInt32>.size)
var deviceID = newDeviceID
var propertyAddress = AudioObjectPropertyAddress(
mSelector: AudioObjectPropertySelector(kAudioHardwarePropertyDefaultOutputDevice),
mScope: AudioObjectPropertyScope(kAudioObjectPropertyScopeGlobal),
mElement: AudioObjectPropertyElement(kAudioObjectPropertyElementMaster))
AudioObjectSetPropertyData(AudioObjectID(kAudioObjectSystemObject), &propertyAddress, 0, nil, propertySize, &deviceID)
}
I then found that the kAudioUnitSubType_HALOutput is the way to go for specifying a static device only accessible through this property. I can create a component of this type using:
var outputHAL = AudioComponentDescription(componentType: kAudioUnitType_Output, componentSubType: kAudioUnitSubType_HALOutput, componentManufacturer: kAudioUnitManufacturer_Apple, componentFlags: 0, componentFlagsMask: 0)
let component = AudioComponentFindNext(nil, &outputHAL)
guard component != nil else {
print("Can't get input unit")
exit(-1)
}
However I am confused about how you create a description of this component and then find the next device that matches the description. Is there a property where I can select the audio device ID and link the AUHAL to this?
I also cannot figure out how to assign an AUHAL to an AVAudioEngine. I can create a node for the HAL but cannot attach this to the engine. Finally is it possible to create multiple kAudioUnitSubType_HALOutput components and feed these into the mixer?
I have been trying to research this for the last week, but nowhere closer to the answer. I have read up on channel mapping and everything I need to know down the line, but at this level getting the audio at. lower level seems pretty undocumented, especially when using swift.
I'm using the (very cool) AudioKit framework to process audio for a macOS music visualizer app. My audio source ("mic") is iTunes 12 via Rogue Amoeba Loopback.
In the Xcode debug window, I'm seeing the following error message each time I launch my app:
kAudioUnitErr_TooManyFramesToProcess : inFramesToProcess=513, mMaxFramesPerSlice=512
I've gathered from searches that this is probably related to sample rate, but I haven't found a clear description of what this error indicates (or if it even matters). My app is functioning normally, but I'm wondering if this could be affecting efficiency.
EDIT: The error message does not appear if I use Audio MIDI Setup to set the Loopback device output to 44.1kHz. (I set it initially to 48.0kHz to match my other audio devices, which I keep configured to the video standard.)
Keeping Loopback at 44.1kHz is an acceptable solution, but now my question would be: Is it possible to avoid this error even with a 48.0kHz input? (I tried AKSettings.sampleRate = 48000 but that made no difference.) Or can I just safely ignore the error in any case?
AudioKit is initialized thusly:
AKSettings.audioInputEnabled = true
mic = AKMicrophone()
do {
try mic.setDevice(AudioKit.inputDevices![inputDeviceNumber])
}
catch {
AKLog("Device not set")
}
amplitudeTracker = AKAmplitudeTracker(mic)
AudioKit.output = AKBooster(amplitudeTracker, gain: 0)
do {
try AudioKit.start()
} catch {
AKLog("AudioKit did not start")
}
mic.start()
amplitudeTracker?.start()
This string saved my app
try? AVAudioSession.sharedInstance().setPreferredIOBufferDuration(0.02)
I'm running an mlmodel that is coming from keras on an iPhone 6. The predictions often fails with the error Error computing NN outputs. Does anyone know what could be the cause and if there is anything I can do about it?
do {
return try model.prediction(input1: input)
} catch let err {
fatalError(err.localizedDescription) // Error computing NN outputs error
}
EDIT: I tried apple's sample project and that one works in the background so it seems it's specific to either our project or model type.
I got the same error myself at similar "seemingly random" times. A bit of debug tracing established that it was caused by the app sometimes trying to load its coreml model when it was sent to background, then crashing or freezing when reloaded into foreground.
The message Error computing NN outputs error was preceded by:
Execution of the command buffer was aborted due to an error during execution. Insufficient Permission (to submit GPU work from background) (IOAF code 6)
I didn't need (or want) the model to be used when the app was in background, so I detected when the app was going in / out of background, set a flag and used a guard statement before attempting to call the model.
Detect when going into background using applicationWillResignActive within the AppDelegate.swift file and set a Bool flag e.g. appInBackground = true. See this for more info: Detect iOS app entering background
Detect when app re-enters foreground using applicationDidBecomeActive in the same AppDelegate.swift file, and reset flag appInBackground = false
Then in the function where you call the model, just before calling model, use a statement such as:
guard appInBackground == false else { return } // new line to add
guard let model = try? VNCoreMLModel(for modelName.model) else { fatalError("could not load model") // original line to load model
I doubt this is the most elegant solution, but it worked for me.
I haven't established why the attempt to load the model in background only happens sometimes.
In the Apple example you link to, it looks like their app only ever calls the model in response to a user input, so it will never try to load the model when in background. Hence the difference in my case ... and possibly yours as well?
In the end it was enough for us to set the usesCPUOnly flag. Using the GPU in the background seems prohibited in iOS. Apple actually wrote about this in their documentation as well. To specify this flag we couldn't use the generated model class anymore but had to call the raw coreml classes instead. I can imagine this changing in a future version however. The snippet below is taken from the generated model class, but with the added MLPredictionOptions specified.
let options = MLPredictionOptions()
options.usesCPUOnly = true // Can't use GPU in the background
// Copied from from the generated model class
let input = model_input(input: mlMultiArray)
let output = try generatedModel.model.prediction(from: input, options: options)
let result = model_output(output: output.featureValue(for: "output")!.multiArrayValue!).output
Given the following code if I use the first method in the if branch to obtain a MIDIDestination the code works correctly, and MIDI data is sent. If I use the second method from the else branch, no data is sent.
var client = MIDIClientRef()
var port = MIDIPortRef()
var dest = MIDIEndpointRef()
MIDIClientCreate("jveditor" as CFString, nil, nil, &client)
MIDIOutputPortCreate(client, "output" as CFString, &port)
if false {
dest = MIDIGetDestination(1)
} else {
var device = MIDIGetExternalDevice(0)
var entity = MIDIDeviceGetEntity(device, 0)
dest = MIDIEntityGetDestination(entity, 0)
}
var name: Unmanaged<CFString>?
MIDIObjectGetStringProperty(dest, kMIDIPropertyDisplayName, &name)
print(name?.takeUnretainedValue() as! String)
var gmOn : [UInt8] = [ 0xf0, 0x7e, 0x7f, 0x09, 0x01, 0xf7 ]
var pktlist = MIDIPacketList()
var current = MIDIPacketListInit(&pktlist)
current = MIDIPacketListAdd(&pktlist, MemoryLayout<MIDIPacketList>.stride, current, 0, gmOn.count, &gmOn)
MIDISend(port, dest, &pktlist)
In both cases the printed device name is correct, and the status of every call is noErr.
I have noticed that if I ask for the kMIDIManufacturerName property that I get different results - specifically using the first method I get Generic, from the USB MIDI interface to which the MIDI device is connected, and with the second method I get the value of Roland configured via the Audio MIDI Setup app.
The reason I want to use the second method is specifically so that I can filter out devices that don't have the desired manufacturer name, but as above I can't then get working output.
Can anyone explain the difference between these two methods, and why the latter doesn't work, and ideally offer a suggestion as to how I can work around that?
It sounds like you want to find only the MIDI destination endpoints to talk to a certain manufacturer's devices. Unfortunately that isn't really possible, since there is no protocol for discovering what MIDI devices exist, what their attributes are, and how they are connected to the computer.
(Remember that MIDI is primitive 1980s technology. It doesn't even require bidirectional communication. There are perfectly valid MIDI setups with MIDI devices that you can send data to, but can never receive data from, and vice versa.)
The computer knows what MIDI interfaces are connected to it (for instance, a USB-MIDI interface). CoreMIDI calls these "Devices". You can find out how many there are, how many ports each has, etc. But there is no way to find out anything about the physical MIDI devices like keyboards and synthesizers that are connected to them.
"External devices" are an attempt to get around the discovery problem. They are the things that appear in Audio MIDI Setup when you press the "Add Device" button. That's all!
Ideally your users would create an external device for each physical MIDI device in their setup, enter all the attributes of each one, and set up all the connections in a way that perfectly mirrors their physical MIDI cables.
Unfortunately, in reality:
There may not be any external devices. There is not much benefit to creating them in Audio MIDI Setup, and it's a lot of boring data entry, so most people don't bother.
If there are external devices, you can't trust any of the information that the users added. The manufacturer might not be right, or might be spelled wrong, for instance.
It's pretty unfriendly to force your users to set things up in Audio MIDI Setup before they can use your software. Therefore, no apps do that... and therefore nobody sets anything up in Audio MIDI Setup. It's a chicken-and-egg problem.
Even if there are external devices, your users might want to send MIDI to other endpoints (like virtual endpoints created by other apps) that are not apparently connected to external devices. You should let them do what they want.
The documentation for MIDIGetDevice() makes a good suggestion:
If a client iterates through the devices and entities in the system, it will not ever visit any virtual sources and destinations created by other clients. Also, a device iteration will return devices which are "offline" (were present in the past but are not currently present), while iterations through the system's sources and destinations will not include the endpoints of offline devices.
Thus clients should usually use MIDIGetNumberOfSources, MIDIGetSource, MIDIGetNumberOfDestinations and MIDIGetDestination, rather iterating through devices and entities to locate endpoints.
In other words: use MIDIGetNumberOfDestinations and MIDIGetDestination to get the possible destinations, then let your users pick one of them. That's all.
If you really want to do more:
Given a destination endpoint, you can use MIDIEndpointGetEntity and MIDIEndpointGetDevice to get to the MIDI interface.
Given any MIDI object, you can find its connections to other objects. Use MIDIObjectGetDataProperty to get the value of property kMIDIPropertyConnectionUniqueID, which is an array of the unique IDs of connected objects. Then use MIDIObjectFindByUniqueID to get to the object. The outObjectType will tell you what kind of object it is.
But that's pretty awkward, and you're not guaranteed to find any useful information.
Based on a hint from Kurt Revis's answer, I've found the solution.
The destination that I needed to find is associated with the source of the external device, with the connection between them found using the kMIDIPropertyConnectionUniqueID property of that source.
Replacing the code in the if / else branch in the question with the code below works:
var external = MIDIGetExternalDevice(0)
var entity = MIDIDeviceGetEntity(external, 0)
var src = MIDIEntityGetSource(entity, 0)
var connID : Int32 = 0
var dest = MIDIObjectRef()
var type = MIDIObjectType.other
MIDIObjectGetIntegerProperty(src, kMIDIPropertyConnectionUniqueID, &connID)
MIDIObjectFindByUniqueID(connID, &dest, &type)
A property dump suggests that the connection Unique ID property is really a data property (perhaps containing multiple IDs) but the resulting CFData appears to be in big-endian format so reading it as an integer property instead seems to work fine.
I've been looking around Swift documentation to save an audio output from AVAudioEngine but I couldn't find any useful tip.
Any suggestion?
Solution
I found a way around thanks to matt's answer.
Here a sample code of how to save an audio after passing it through an AVAudioEngine (i think that technically it's before)
newAudio = AVAudioFile(forWriting: newAudio.url, settings: nil, error: NSErrorPointer())
//Your new file on which you want to save some changed audio, and prepared to be bufferd in some new data...
var audioPlayerNode = AVAudioPlayerNode() //or your Time pitch unit if pitch changed
//Now install a Tap on the output bus to "record" the transformed file on a our newAudio file.
audioPlayerNode.installTapOnBus(0, bufferSize: (AVAudioFrameCount(audioPlayer.duration)), format: opffb){
(buffer: AVAudioPCMBuffer!, time: AVAudioTime!) in
if (self.newAudio.length) < (self.audioFile.length){//Let us know when to stop saving the file, otherwise saving infinitely
self.newAudio.writeFromBuffer(buffer, error: NSErrorPointer())//let's write the buffer result into our file
}else{
audioPlayerNode.removeTapOnBus(0)//if we dont remove it, will keep on tapping infinitely
println("Did you like it? Please, vote up for my question")
}
}
Hope this helps !
One issue to solve:
Sometimes, your outputNode is shorter than the input: if you accelerate the time rate by 2, your audio will be 2 times shorter. This is the issue im facing for now since my condition for saving the file is (line 10)
if(newAudio.length) < (self.audioFile.length)//audiofile being the original(long) audio and newAudio being the new changed (shorter) audio.
Any help here?
Yes, it's quite easy. You simply put a tap on a node and save the buffer into a file.
Unfortunately this means you have to play through the node. I was hoping that AVAudioEngine would let me process one sound file into another directly, but apparently that's impossible - you have to play and process in real time.
Offline rendering Worked for me using GenericOutput AudioUnit. Please check this link, I have done mixing two,three audios offline and combine it to a single file. Not the same scenario but it may help you for getting some idea. core audio offline rendering GenericOutput