I am trying to install a tap on an AVAudioEngine. I have the current code:
guard let engine = engine, let input = engine.inputNode else {
print("error!")
return
}
let format = input.inputFormat(forBus: 0)
let bufferSize = 4096
input.installTap(onBus: 0, bufferSize: AVAudioFrameCount(bufferSize), format: format, block: )
I am unsure on what goes in the block. There isn't much documentation on this. I have found this: https://developer.apple.com/reference/avfoundation/avaudionodetapblock?language=objc
Could someone explain how to use this?
Thanks,
Feras A.
You'd better check the Swift version of the reference, if you want to write it in Swift.
Declaration
typealias AVAudioNodeTapBlock = (AVAudioPCMBuffer, AVAudioTime) -> Void
You need to pass a closure taking two arguments and returning nothing, so you can write it as:
input.installTap(onBus: 0, bufferSize: AVAudioFrameCount(bufferSize), format: format, block: {buffer, when in
//...
})
The type of two arguments buffer and when are AVAudioPCMBuffer and AVAudioTime respectively.
So, if you want to record the tapped audio into an audio file, you can write something like this:
input.installTap(onBus: 0, bufferSize: AVAudioFrameCount(bufferSize), format: format, block: {buffer, when in
do {
try self.audioFile?.write(from: buffer)
} catch {
print(error)
}
})
(Assume audioFile is an instance property of type AVAudioFile?.)
Anyway, you need to know how to use AVAudioPCMBuffer.
I'm not sure if input.inputFormat(forBus: 0) can be an appropriate format in your case, but that may be another issue.
Related
I'm using TinySoundFont to use SF2 files on watchOS. I want to play the raw audio generated by the framework in real time (which means calling tsf_note_on as soon as the corresponding button is pressed and calling tsf_render_short as soon as new data is needed). I'm using an AVAudioSourceNode to achieve that.
Despite the sound rendering fine when I render it into a file, it's really noisy when played using the AVAudioSourceNode. (Based on the answer from Rob Napier, this might be because I ignore the timestamp property - I'm looking for a solution that addresses that concern.) What causes this issue and how can I fix it?
I'm looking for a solution that renders audio realtime and not precalculates it, since I want to handle looping sounds correctly as well.
You can download a sample GitHub project here.
ContentView.swift
import SwiftUI
import AVFoundation
struct ContentView: View {
#ObservedObject var settings = Settings.shared
init() {
settings.prepare()
}
var body: some View {
Button("Play Sound") {
Settings.shared.playSound()
if !settings.engine.isRunning {
do {
try settings.engine.start()
} catch {
print(error)
}
}
}
}
}
Settings.swift
import SwiftUI
import AVFoundation
class Settings: ObservableObject {
static let shared = Settings()
var engine: AVAudioEngine!
var sourceNode: AVAudioSourceNode!
var tinySoundFont: OpaquePointer!
func prepare() {
let soundFontPath = Bundle.main.path(forResource: "GMGSx", ofType: "sf2")
tinySoundFont = tsf_load_filename(soundFontPath)
tsf_set_output(tinySoundFont, TSF_MONO, 44100, 0)
setUpSound()
}
func setUpSound() {
if let engine = engine,
let sourceNode = sourceNode {
engine.detach(sourceNode)
}
engine = .init()
let mixerNode = engine.mainMixerNode
let audioFormat = AVAudioFormat(
commonFormat: .pcmFormatInt16,
sampleRate: 44100,
channels: 1,
interleaved: false
)
guard let audioFormat = audioFormat else {
return
}
sourceNode = AVAudioSourceNode(format: audioFormat) { silence, timeStamp, frameCount, audioBufferList in
guard let data = self.getSound(length: Int(frameCount)) else {
return 1
}
let ablPointer = UnsafeMutableAudioBufferListPointer(audioBufferList)
data.withUnsafeBytes { (intPointer: UnsafePointer<Int16>) in
for index in 0 ..< Int(frameCount) {
let value = intPointer[index]
// Set the same value on all channels (due to the inputFormat, there's only one channel though).
for buffer in ablPointer {
let buf: UnsafeMutableBufferPointer<Int16> = UnsafeMutableBufferPointer(buffer)
buf[index] = value
}
}
}
return noErr
}
engine.attach(sourceNode)
engine.connect(sourceNode, to: mixerNode, format: audioFormat)
do {
try AVAudioSession.sharedInstance().setCategory(.playback)
} catch {
print(error)
}
}
func playSound() {
tsf_note_on(tinySoundFont, 0, 60, 1)
}
func getSound(length: Int) -> Data? {
let array = [Int16]()
var storage = UnsafeMutablePointer<Int16>.allocate(capacity: length)
storage.initialize(from: array, count: length)
tsf_render_short(tinySoundFont, storage, Int32(length), 0)
let data = Data(bytes: storage, count: length)
storage.deallocate()
return data
}
}
The AVAudioSourceNode initializer takes a render block. In the mode you're using (live playback), this is a real-time callback, so you have a very tight deadline to fill the block with the requested data and return it so it can be played. You don't have a ton of time to do calculations. You definitely don't have time to access the filesystem.
In your block, you're re-computing an entire WAV every render cycle, then writing it to disk, then reading it from disk, then filling in the block that was requested. You ignore the timestamp requested, and always fill the buffer starting at sample zero. The mismatch is what's causing the buzzing. The fact that you're so slow about it is probably what's causing the pitch-drop.
Depending on the size of your files, the simplest way to implement this is to first decode everything into memory, and fill in the buffers for the timestamps and lengths requested. It looks like your C code already generates PCM data, so there's no need to convert it into a WAV file. It seems to already be in the right format.
Apple provides a good sample project for a Signal Generator that you should use as a starting point. Download that and make sure it works as expected. Then work to swap in your SF2 code. You may also find the video on this helpful: What’s New in AVAudioEngine.
The easiest tool to use here is probably an AVAudioPlayerNode. Your SoundFontHelper is making things much more complicated, so I've removed it and just call TSF directly from Swift. To do this, create a file called tsf.c as follows:
#define TSF_IMPLEMENTATION
#include "tsf.h"
And add it to BridgingHeader.h:
#import "tsf.h"
Simplify ContentView to this:
import SwiftUI
struct ContentView: View {
#ObservedObject var settings = Settings.shared
init() {
// You'll want error handling here.
try! settings.prepare()
}
var body: some View {
Button("Play Sound") {
settings.play()
}
}
}
And that leaves the new version of Settings, which is the meat of it:
import SwiftUI
import AVFoundation
class Settings: ObservableObject {
static let shared = Settings()
var engine = AVAudioEngine()
let playerNode = AVAudioPlayerNode()
var tsf: OpaquePointer
var outputFormat = AVAudioFormat()
init() {
let soundFontPath = Bundle.main.path(forResource: "GMGSx", ofType: "sf2")
tsf = tsf_load_filename(soundFontPath)
engine.attach(playerNode)
engine.connect(playerNode, to: engine.mainMixerNode, format: nil)
updateOutputFormat()
}
// For simplicity, this object assumes the outputFormat does not change during its lifetime.
// It's important to watch for route changes, and recreate this object if they occur. For details, see:
// https://developer.apple.com/documentation/avfaudio/avaudiosession/responding_to_audio_session_route_changes
func updateOutputFormat() {
outputFormat = engine.mainMixerNode.outputFormat(forBus: 0)
}
func prepare() throws {
// Start the engine
try AVAudioSession.sharedInstance().setCategory(.playback)
try engine.start()
playerNode.play()
updateOutputFormat()
// Configure TSF. The only important thing here is the sample rate, which can be different on different hardware.
// Core Audio has a defined format of "deinterleaved 32-bit floating point."
tsf_set_output(tsf,
TSF_STEREO_UNWEAVED, // mode
Int32(outputFormat.sampleRate), // sampleRate
0) // gain
}
func play() {
tsf_note_on(tsf,
0, // preset_index
60, // key (middle C)
1.0) // velocity
// These tones have a long falloff, so you want a lot of source data. This is 10s.
let frameCount = 10 * Int(outputFormat.sampleRate)
// Create a buffer for the samples
let buffer = AVAudioPCMBuffer(pcmFormat: outputFormat, frameCapacity: AVAudioFrameCount(frameCount))!
buffer.frameLength = buffer.frameCapacity
// Render the samples. Do not mix. This buffer has been extended to
// the needed size by the assignment to `frameLength` above. The call to
// `assumingMemoryBound` is known to be correct because the format is Float32.
let ptr = buffer.audioBufferList.pointee.mBuffers.mData?.assumingMemoryBound(to: Float.self)
tsf_render_float(tsf,
ptr, // buffer
Int32(frameCount), // samples
0) // mixing (do not mix)
// All done. Play the buffer, interrupting whatever is currently playing
playerNode.scheduleBuffer(buffer, at: nil, options: .interrupts)
}
}
You can find the full version at my fork. You can also see the first commit, which is another approach that maintains your SoundFontHelper and does conversions to deal with it, but it's much simpler to just render the audio correctly in the first place.
I would like to use some C code that uses a file descriptor.
Background is that I would like to read some data from cgraph library.
public extension UnsafeMutablePointer where Pointee == Agraph_t {
func saveTo(fileName: String) {
let f = fopen(cString(fileName), cString("w"))
agwrite(self,f)
fsync(fileno(f))
fclose(f)
}
}
I would like to have the file output, but without writing to a temp file. Hence, I would like to do something like this:
public extension UnsafeMutablePointer where Pointee == Agraph_t {
var asString: String {
let pipe = Pipe()
let fileDescriptor = UnsafeMutablePointer<Int32>.allocate(capacity: 1)
fileDescriptor.pointee = pipe.fileHandleForWriting.fileDescriptor
agwrite(self, fileDescriptor)
let data = pipe.fileHandleForReading.readDataToEndOfFile()
if let output = String(data: data, encoding: .utf8) {
return output
}
return ""
}
}
But it doesn't work, resulting in a EXC_BAD_ACCESS within agwrite(,). What do I need to do instead?
Many thanks in advance!
File descriptors and file pointers are not the same thing. It's confusing, and made even more frustrating by the fact that FILE * is really hard to Google because of the symbol.
You need to fdopen the file descriptor (pipe.fileHandleForWriting.fileDescriptor), to receive a FILE * (UnsafeMutablePointer<FILE> in Swift). This is what you then pass to agwrite.
It's important to fclose the file pointer when you're done writing to it, otherwise .readDataToEndOfFile() will never terminate. I made a helper function to ensure the fclose can't be forgetten. It's possible that agwrite closes the file pointer itself, internally. If that's the case, you should delete this code and just give it the result of fdopen, plain and simple.
import Foundation
public typealias Agraph_t = Int // Dummy value
public struct AGWriteWrongEncoding: Error { }
func agwrite(_: UnsafeMutablePointer<Agraph_t>, _ filePointer: UnsafeMutablePointer<FILE>) {
let message = "This is a stub."
_ = message.withCString { cString in
fputs(cString, stderr)
}
}
#discardableResult
func use<R>(
fileDescriptor: Int32,
mode: UnsafePointer<Int8>!,
closure: (UnsafeMutablePointer<FILE>) throws -> R
) rethrows -> R {
// Should prob remove this `!`, but IDK what a sensible recovery mechanism would be.
let filePointer = fdopen(fileDescriptor, mode)!
defer { fclose(filePointer) }
return try closure(filePointer)
}
public extension UnsafeMutablePointer where Pointee == Agraph_t {
func asString() throws -> String {
let pipe = Pipe()
use(fileDescriptor: pipe.fileHandleForWriting.fileDescriptor, mode: "w") { filePointer in
agwrite(self, filePointer)
}
let data = pipe.fileHandleForReading.readDataToEndOfFile()
guard let output = String(data: data, encoding: .utf8) else {
throw AGWriteWrongEncoding()
}
return output
}
}
let ptr = UnsafeMutablePointer<Agraph_t>.allocate(capacity: 1) // Dummy value
print(try ptr.asString())
Several other things:
Throwing an error is probably a better choice than returning "". Empty strings aren't a good error handling mechanism. Returning an optional would also work, but it's likely to always be force unwrapped, anyway.
readDataToEndOfFile is a blocking call, which can lead to a bad use experience. It's probably best that this code be run on a background thread, or use a FileHandle.readabilityHandler to asynchronously consume the data as it comes in.
I need to get the data from AKMicrophone in raw form so i can make a custom plot. All of the examples from AudioKit use their built in plots, but I need to use a plot I made. The input my plot is expecting is an array of Doubles, but im not very worried about typing since I can change that. I just cant get a tap to access the data working correctly. I have already looked at these:
https://groups.google.com/forum/#!searchin/audiokit/tap%7Csort:date/audiokit/V16xV7zZzPM/wuJjmhG8BwAJ
AudioKit - How to get Real Time floatChannelData from Microphone?
but these answers really just show examples from the audiokit examples which aren't helpful for what i need
Here is my attempt, which instantly crashes saying "required condition is false: [AVAEGraphNode.mm:851:CreateRecordingTap: (nullptr == Tap())]
2018-12-27 13:13:25.628188-0700"
mic.avAudioNode.installTap(onBus: 0, bufferSize:
AVAudioFrameCount(bufferSize), format: nil) { [weak self] (buffer, _) in
guard let strongSelf = self else {
AKLog("Unable to create strong reference to self")
return
}
buffer.frameLength = AVAudioFrameCount(strongSelf.bufferSize)
let offset = Int(buffer.frameCapacity - buffer.frameLength)
if let tail = buffer.floatChannelData?[0] {
print(tail)
}
}
I'm having trouble writing wav files in 24bits with AVAudioEngine in swift.
For my usage, my input is an array of Float.
I have the audio format of the input file (retrieved with AVAudioFile).
So, I need to convert my input Float array to a value that will be writable for the buffer. Also, I want to find the right channel to write my data.
My code is working with 16bit and 32 bit files, but I don't know how to handle 24 bit files...
Here it is :
//Static func to write audiofile
fileprivate func writeAudioFile(to outputURL : URL,
withFormat format : AVAudioFormat,
fromSamples music : [Float] )
{
var outputFormatSettings = format.settings
guard let bufferFormat = AVAudioFormat(settings: outputFormatSettings) else{
return
}
var audioFile : AVAudioFile?
do{
audioFile = try AVAudioFile(forWriting: outputURL,
settings: outputFormatSettings,
commonFormat: format.commonFormat,
interleaved: true)
} catch let error as NSError {
print("error:", error.localizedDescription)
}
let frameCount = music.count / Int(format.channelCount)
let outputBuffer = AVAudioPCMBuffer(pcmFormat: bufferFormat,
frameCapacity: AVAudioFrameCount(frameCount))
//We write the data in the right channel
guard let bitDepth = (outputFormatSettings["AVLinearPCMBitDepthKey"] as? Int) else {
return
}
switch bitDepth {
case 16:
for i in 0..<music.count {
var floatValue = music[i]
if(floatValue > 1){
floatValue = 1
}
if(floatValue < -1){
floatValue = -1
}
let value = floatValue * Float(Int16.max)
outputBuffer?.int16ChannelData!.pointee[i] = Int16(value)
}
case 24:
//Here I am not sure of what I do ... Could'nt find the right channel !
for i in 0..<music.count {
outputBuffer?.floatChannelData!.pointee[i] = music[i]
}
case 32:
for i in 0..<music.count {
outputBuffer?.floatChannelData!.pointee[i] = music[i]
}
default:
return
}
outputBuffer?.frameLength = AVAudioFrameCount( frameCount )
do{
try audioFile?.write(from: outputBuffer!)
} catch let error as NSError {
print("error:", error.localizedDescription)
return
}
}
Thanks by advance if someone have an idea of how to handle this !
Representing a 24 bit int in C isn't fun so in Swift I'm sure it's downright painful, and none of the API's support it anyway. Your best bet is to convert to a more convenient format for processing.
AVAudioFile has two formats and an internal converter to convert between them. Its fileFormat represents the format of the file on disk, while its processingformat represents the format of the lpcm data when it is read from, and the format of the lpcm data that it will accept when being written to.
The typical workflow is choose a standard processingFormat, do all of your processing using this format, and let AVAudioFile convert to and from the file format for reading and writing to disk. All of the Audio Unit APIs accept non-interleaved formats, so I tend to use non interleaved for all of my processing formats.
Here's an example that copies the first half of an audio file. It doesn't address your existing code, but illustrates a more common approach:
func halfCopy(src: URL, dst: URL) throws {
let srcFile = try AVAudioFile(forReading: src) //This opens the file for reading using the standard format (deinterleaved floating point).
let dstFile = try AVAudioFile(forWriting: dst,
settings: srcFile.fileFormat.settings,
commonFormat: srcFile.processingFormat.commonFormat,
interleaved: srcFile.processingFormat.isInterleaved) //AVAudioFile(forReading: src) always returns a non-interleaved processing format, this will be false
let frameCount = AVAudioFrameCount(srcFile.length) / 2 // Copying first half of file
guard let buffer = AVAudioPCMBuffer(pcmFormat: srcFile.processingFormat,
frameCapacity: frameCount) else {
fatalError("Derp")
}
try srcFile.read(into: buffer, frameCount: frameCount)
try dstFile.write(from: buffer)
}
I am trying to create a multiplayer game that will send moves between players using Game Center. I'm still learning a lot about programming, so please excuse me if my question is ill-formed. Also, I am not very familiar with Obj-C, so a Swift answer would be great.
In my toy program to try and teach myself, I am trying to follow the strategy used by Shayne Meyer using the GameKitHelper class here: https://github.com/shaynemeyer/SwiftCircuitRacer/tree/master/SwiftCircuitRacer
Using this approach, Shayne sends messages to other players online using structs sent as NSData. I am able to send integers (e.g., the ILoveYou message) but not messages that carry a string property (e.g., the Thanks message). In this latter case I get "Thread 1: EXC_BAD_ACCESS(code=1, address=0x78674100)" at the line "var messageThanks = UnsafePointer,MesssageThanks>(data.bytes).memory"
Eventually, I would like to send game moves that provide both strings and integers together. How does one send a message struct as NSData when properties also include a string? Secondly, I would be appreciative if someone could help me understand fundamentally what is going on when the data is packaged and how what UnsafePointer is doing as it related to sending data via Game Center.
Thank you.
Cliff
enum MessageType: Int {
case ILoveYou, Thanks
}
struct Message {
let messageType: MessageType
}
struct MessageILoveYou {
let message: Message
let messageSenderNumber: UInt32
}
struct MessageThanks {
let message: Message
let messageSenderName: String
let messageSenderNumber: UInt32
}
func sendILoveYou() {
println("sendILoveYou:")
let nameNumber = UInt32(56)
var message = MessageILoveYou(message: Message(messageType: MessageType.ILoveYou), messageSenderNumber: nameNumber)
let data = NSData(bytes: &message, length: sizeof(MessageILoveYou))
sendData(data)
}
func sendThanks() {
println("sendThanks:")
let nameString = "Don J"
let senderNumberInt = UInt32(88)
var message = MessageThanks(message: Message(messageType: MessageType.Thanks), messageSenderName: nameString, messageSenderNumber: senderNumberInt)
let data = NSData(bytes: &message, length: sizeof(MessageThanks))
sendData(data)
}
func matchReceivedData(match: GKMatch, data: NSData, fromPlayer player: String) {
println("matchReceivedData:")
var message = UnsafePointer<Message>(data.bytes).memory
if message.messageType == MessageType.ILoveYou {
println("messageType == ILoveYou")
let messageILoveYou = UnsafePointer<MessageILoveYou>(data.bytes).memory
iLoveYouThanksDelegate?.iLoveYouReceived(from: messageILoveYou.messageSenderNumber)
} else if message.messageType == MessageType.Thanks {
println("messageType == Thanks")
var messageThanks = UnsafePointer<MessageThanks>(data.bytes).memory
iLoveYouThanksDelegate?.thanksReceived(from: messageThanks.messageSenderName)
}
}
func sendData(data: NSData) {
var sendDataError: NSError?
let gameKitHelper = GameKitHelper.sharedInstance
if let multiplayerMatch = gameKitHelper.multiplayerMatch {
let success = multiplayerMatch.sendDataToAllPlayers(data, withDataMode: .Reliable, error: &sendDataError)
if !success {
if let error = sendDataError {
println("Error:\(error.localizedDescription)")
matchEnded()
}
}
}
}
The problem here is that when you create a String in Swift, it allocates a bit of memory itself, and then uses that memory to store the actual characters of the string. All that the string value really holds is some data representing a pointer to that memory and some other info (like how much memory has been allocated, so that it can be freed properly.
You can see this here:
let str = "This is quite a long string, certainly more than 24 bytes"
sizeofValue(str) // and yet this only returns 24
When you stuff variables into an NSData object, the initializer takes a pointer to the memory of the string variable that is holding those pointers, not the characters itself:
// only storing those 24 bytes, not the actual string
let data = NSData(bytes: &str, length: sizeofValue(str))
Note, the type of the bytes argument is UnsafePointer<Void>. This is an indication that you are heading into tricky territory.
Then, when you unmarshal the data at the other end, all your receiver is going to get is some pointers to random memory (sadly, memory on the other user’s device!)
If you want to put string values into an NSData object, you are going to need to marshal them first into raw data. For example, you could encode them into an array:
let data = Array(str.utf8).withUnsafeBufferPointer { buf in
NSData(bytes: buf.baseAddress, length: buf.count)
}
As it happens, since this is a common thing to want to do, there’s a method to do this directly:
let data = str.dataUsingEncoding(NSUTF8StringEncoding)
Then, to unpack the data, you can use NSString’s constructor from an NSData object:
let newStr = NSString(data: data, encoding: NSUTF8StringEncoding)
edit: if you wanted to encode more than just a string in a single NSData, you could do something along these lines… I should say, I’ve never had to do this myself so I’m in no way familiar with the standard practices for this, there could be much better techniques or helper classes/functions. Hopefully someone with more experience can edit to show how to do this properly :)
var type = MessageType.Thanks
// start the data with the type
let data = NSMutableData(bytes: &type, length: sizeofValue(type))
// then append the string
data.appendData(Array(str.utf8).withUnsafeBufferPointer { buf in
NSMutableData(bytes: buf.baseAddress, length: buf.count)
})
switch UnsafePointer<MessageType>(data.bytes).memory {
case .ILoveYou:
// ...
case .Thanks:
let str = NSString(data: data.subdataWithRange(NSMakeRange(1, data.length-1)), encoding: NSUTF8StringEncoding)
}