Cannot load default library in Metal using Swift - swift

For my project (being compiled as a framework) I have a file ops.metal:
kernel void add(device float *lhs [[buffer(0)]],
device float *rhs [[buffer(1)]],
device float *result [[buffer(2)]],
uint id [[ thread_position_in_grid ]])
{
result[id] = lhs[id] + rhs[id];
}
and the following Swift code:
#available(OSX 10.11, *)
public class MTLContext {
var device: MTLDevice!
var commandQueue:MTLCommandQueue!
var library:MTLLibrary!
var commandBuffer:MTLCommandBuffer
var commandEncoder:MTLComputeCommandEncoder
init() {
if let defaultDevice = MTLCreateSystemDefaultDevice() {
device = defaultDevice
print("device created")
} else {
print("Metal is not supported")
}
commandQueue = device.makeCommandQueue()
library = device.newDefaultLibrary()
if let defaultLibrary = device.newDefaultLibrary() {
library = defaultLibrary
} else {
print("could not load default library")
}
commandBuffer = commandQueue.makeCommandBuffer()
commandEncoder = commandBuffer.makeComputeCommandEncoder()
}
deinit {
commandEncoder.endEncoding()
}
}
When I try to create an instance of MTLContext in a unit test, the device is created, but the default library cannot be created ("could not load default library"). I've checked that the compiled framework has a default.metallib in Resources (which is the most common reason given for newDefaultLibrary).
Unfortunately I haven't been able to find any working examples that are creating compute kernels in a Metal shader file (there are a few examples using the performance shaders, but they don't need to make kernels in the shader file).
Any suggestions would be greatly appreciated!

newDefaultLibrary() loads from the main bundle of the currently running application. It doesn't search any embedded frameworks or other locations for libraries.
If you want to use a metallib that was compiled into an embedded framework, the easiest thing to do is to get a reference to its containing Bundle and ask for the default library of that bundle instead:
let frameworkBundle = Bundle(for: SomeClassFromMyShaderFramework.self)
guard let defaultLibrary = try? device.makeDefaultLibrary(bundle: frameworkBundle) else {
fatalError("Could not load default library from specified bundle")
}
This does require that you have at least one publicly-visible class in the framework containing your shaders, but that can be as simple as declaring an empty class strictly for the purpose of doing the bundle look-up:
public class SomeClassFromMyShaderFramework {}

Related

webrtc macOS change input source

I'm trying to change the audio input source (microphone) in my macOS app like this:
var engine = AVAudioEngine()
private func activateNewInput(_ id: AudioDeviceID) {
let input = engine.inputNode
let inputUnit = input.audioUnit!
var inputDeviceID: AudioDeviceID = id
let status = AudioUnitSetProperty(inputUnit,
kAudioOutputUnitProperty_CurrentDevice,
kAudioUnitScope_Global,
0,
&inputDeviceID,
UInt32(MemoryLayout<AudioDeviceID>.size))
if status != 0 {
NSLog("Could not change input: \(status)")
}
/*
engine.prepare()
do {
try engine.start()
}
catch {
NSLog("\(error.localizedDescription)")
}*/
}
The status of the AudioUnitSetProperty is succesful (zero) however webrtc keeps on using the default input source no matter what.
On iOS it is typically done using. AVAudioSession which is missing on macOS. That's why I use AudioUnitSetProperty from Audio Toolbox framework. I also checked RTCPeerConnectionFactory and it has no constructor to inject ADM (audio device module) nor any APIs to control audio input. I use webrtc branch 106.
Any ideas, hints are much appreciated.
Thanks

Noisiness when playing sound using an AVAudioSourceNode

I'm using TinySoundFont to use SF2 files on watchOS. I want to play the raw audio generated by the framework in real time (which means calling tsf_note_on as soon as the corresponding button is pressed and calling tsf_render_short as soon as new data is needed). I'm using an AVAudioSourceNode to achieve that.
Despite the sound rendering fine when I render it into a file, it's really noisy when played using the AVAudioSourceNode. (Based on the answer from Rob Napier, this might be because I ignore the timestamp property - I'm looking for a solution that addresses that concern.) What causes this issue and how can I fix it?
I'm looking for a solution that renders audio realtime and not precalculates it, since I want to handle looping sounds correctly as well.
You can download a sample GitHub project here.
ContentView.swift
import SwiftUI
import AVFoundation
struct ContentView: View {
#ObservedObject var settings = Settings.shared
init() {
settings.prepare()
}
var body: some View {
Button("Play Sound") {
Settings.shared.playSound()
if !settings.engine.isRunning {
do {
try settings.engine.start()
} catch {
print(error)
}
}
}
}
}
Settings.swift
import SwiftUI
import AVFoundation
class Settings: ObservableObject {
static let shared = Settings()
var engine: AVAudioEngine!
var sourceNode: AVAudioSourceNode!
var tinySoundFont: OpaquePointer!
func prepare() {
let soundFontPath = Bundle.main.path(forResource: "GMGSx", ofType: "sf2")
tinySoundFont = tsf_load_filename(soundFontPath)
tsf_set_output(tinySoundFont, TSF_MONO, 44100, 0)
setUpSound()
}
func setUpSound() {
if let engine = engine,
let sourceNode = sourceNode {
engine.detach(sourceNode)
}
engine = .init()
let mixerNode = engine.mainMixerNode
let audioFormat = AVAudioFormat(
commonFormat: .pcmFormatInt16,
sampleRate: 44100,
channels: 1,
interleaved: false
)
guard let audioFormat = audioFormat else {
return
}
sourceNode = AVAudioSourceNode(format: audioFormat) { silence, timeStamp, frameCount, audioBufferList in
guard let data = self.getSound(length: Int(frameCount)) else {
return 1
}
let ablPointer = UnsafeMutableAudioBufferListPointer(audioBufferList)
data.withUnsafeBytes { (intPointer: UnsafePointer<Int16>) in
for index in 0 ..< Int(frameCount) {
let value = intPointer[index]
// Set the same value on all channels (due to the inputFormat, there's only one channel though).
for buffer in ablPointer {
let buf: UnsafeMutableBufferPointer<Int16> = UnsafeMutableBufferPointer(buffer)
buf[index] = value
}
}
}
return noErr
}
engine.attach(sourceNode)
engine.connect(sourceNode, to: mixerNode, format: audioFormat)
do {
try AVAudioSession.sharedInstance().setCategory(.playback)
} catch {
print(error)
}
}
func playSound() {
tsf_note_on(tinySoundFont, 0, 60, 1)
}
func getSound(length: Int) -> Data? {
let array = [Int16]()
var storage = UnsafeMutablePointer<Int16>.allocate(capacity: length)
storage.initialize(from: array, count: length)
tsf_render_short(tinySoundFont, storage, Int32(length), 0)
let data = Data(bytes: storage, count: length)
storage.deallocate()
return data
}
}
The AVAudioSourceNode initializer takes a render block. In the mode you're using (live playback), this is a real-time callback, so you have a very tight deadline to fill the block with the requested data and return it so it can be played. You don't have a ton of time to do calculations. You definitely don't have time to access the filesystem.
In your block, you're re-computing an entire WAV every render cycle, then writing it to disk, then reading it from disk, then filling in the block that was requested. You ignore the timestamp requested, and always fill the buffer starting at sample zero. The mismatch is what's causing the buzzing. The fact that you're so slow about it is probably what's causing the pitch-drop.
Depending on the size of your files, the simplest way to implement this is to first decode everything into memory, and fill in the buffers for the timestamps and lengths requested. It looks like your C code already generates PCM data, so there's no need to convert it into a WAV file. It seems to already be in the right format.
Apple provides a good sample project for a Signal Generator that you should use as a starting point. Download that and make sure it works as expected. Then work to swap in your SF2 code. You may also find the video on this helpful: What’s New in AVAudioEngine.
The easiest tool to use here is probably an AVAudioPlayerNode. Your SoundFontHelper is making things much more complicated, so I've removed it and just call TSF directly from Swift. To do this, create a file called tsf.c as follows:
#define TSF_IMPLEMENTATION
#include "tsf.h"
And add it to BridgingHeader.h:
#import "tsf.h"
Simplify ContentView to this:
import SwiftUI
struct ContentView: View {
#ObservedObject var settings = Settings.shared
init() {
// You'll want error handling here.
try! settings.prepare()
}
var body: some View {
Button("Play Sound") {
settings.play()
}
}
}
And that leaves the new version of Settings, which is the meat of it:
import SwiftUI
import AVFoundation
class Settings: ObservableObject {
static let shared = Settings()
var engine = AVAudioEngine()
let playerNode = AVAudioPlayerNode()
var tsf: OpaquePointer
var outputFormat = AVAudioFormat()
init() {
let soundFontPath = Bundle.main.path(forResource: "GMGSx", ofType: "sf2")
tsf = tsf_load_filename(soundFontPath)
engine.attach(playerNode)
engine.connect(playerNode, to: engine.mainMixerNode, format: nil)
updateOutputFormat()
}
// For simplicity, this object assumes the outputFormat does not change during its lifetime.
// It's important to watch for route changes, and recreate this object if they occur. For details, see:
// https://developer.apple.com/documentation/avfaudio/avaudiosession/responding_to_audio_session_route_changes
func updateOutputFormat() {
outputFormat = engine.mainMixerNode.outputFormat(forBus: 0)
}
func prepare() throws {
// Start the engine
try AVAudioSession.sharedInstance().setCategory(.playback)
try engine.start()
playerNode.play()
updateOutputFormat()
// Configure TSF. The only important thing here is the sample rate, which can be different on different hardware.
// Core Audio has a defined format of "deinterleaved 32-bit floating point."
tsf_set_output(tsf,
TSF_STEREO_UNWEAVED, // mode
Int32(outputFormat.sampleRate), // sampleRate
0) // gain
}
func play() {
tsf_note_on(tsf,
0, // preset_index
60, // key (middle C)
1.0) // velocity
// These tones have a long falloff, so you want a lot of source data. This is 10s.
let frameCount = 10 * Int(outputFormat.sampleRate)
// Create a buffer for the samples
let buffer = AVAudioPCMBuffer(pcmFormat: outputFormat, frameCapacity: AVAudioFrameCount(frameCount))!
buffer.frameLength = buffer.frameCapacity
// Render the samples. Do not mix. This buffer has been extended to
// the needed size by the assignment to `frameLength` above. The call to
// `assumingMemoryBound` is known to be correct because the format is Float32.
let ptr = buffer.audioBufferList.pointee.mBuffers.mData?.assumingMemoryBound(to: Float.self)
tsf_render_float(tsf,
ptr, // buffer
Int32(frameCount), // samples
0) // mixing (do not mix)
// All done. Play the buffer, interrupting whatever is currently playing
playerNode.scheduleBuffer(buffer, at: nil, options: .interrupts)
}
}
You can find the full version at my fork. You can also see the first commit, which is another approach that maintains your SoundFontHelper and does conversions to deal with it, but it's much simpler to just render the audio correctly in the first place.

What is the best way to get BSD drive names on macOS (Swift)?

What's the best way I can get a list of BSD names of all USB devices (and maybe including internal Mac drives) without using a diskutil CLI wrapper?
I don't want to use any wrappers that interact with the CLI interface, as this way of interacting is quite slow and unreliable:
This is an example of why I'm not happy with using CLI wrappers
(Compare 'Time elapsed for DiskUtil CLI Wrapper.' and 'Time elapsed for Disk Arbitration')
What is the best way to implement the solution for my problem?
Use the data from IOReg?
If yes, how can I get a list of BSD names of connected devices using it?
Here is an example what I want to get:
["disk0", "disk0s1", "disk0s2", "disk0s3", "disk1", "disk1s1", "disk1s2", "disk1s3", "disk1s4", "disk2", "disk2s1", "disk2s2", "disk3", "disk3s1", "disk3s1s1", "disk3s2", "disk3s3", "disk3s4", "disk3s5", "disk3s6", "disk4", "disk4s1", "disk4s2", "disk5", "disk5s1", "disk5s2", "disk6", "disk6s1", "disk6s2", "disk10", "disk10s1", "disk10s2", "disk11", "disk11s1"]
At the moment, I have the following:
static func getMountedBSDNames() -> [String] {
guard let session = DASessionCreate(nil) else { return [] }
guard let mountedVolumeURLs = FileManager.default.mountedVolumeURLs(includingResourceValuesForKeys: nil) else { return [] }
var BSDNames: [String] = []
for volumeURL in mountedVolumeURLs {
if let disk = DADiskCreateFromVolumePath(kCFAllocatorDefault, session, volumeURL as CFURL), let BSDName = DADiskGetBSDName(disk) {
BSDNames.append(
String(cString: BSDName)
)
}
}
return BSDNames
}
But in this case, only mounted are returning.
I want there to have even those, that were ejected
I achieved the desired result using the IOReg lookup method:
Elapsed Time
func getDriveBSDNames() -> [String] {
var iterator: io_iterator_t = 0
let matching: CFDictionary = IOServiceMatching(kIOServicePlane)
// Use 'kIOMasterPortDefault' for macOS older than 12.0 Monterey
IOServiceGetMatchingServices(kIOMainPortDefault, matching, &iterator)
var child: io_object_t = IOIteratorNext(iterator)
var BSDNames: [String] = []
while child > 0 {
if let BSDNameAnyObject = IORegistryEntryCreateCFProperty(child, "BSD Name" as CFString, kCFAllocatorDefault, IOOptionBits(kIORegistryIterateRecursively)) {
if let BSDNameString = (BSDNameAnyObject.takeRetainedValue() as? String), BSDNameString.starts(with: "disk") {
BSDNames.append(
BSDNameString
)
}
}
child = IOIteratorNext(iterator)
}
return BSDNames
}
In this case, it was also necessary to filter the output of the results using:
BSDNameString.starts(with: "disk")
(otherwise, some unnecessary devices were added, such as en0, anpi0, llw0, etc.)
Note that while the Disk Arbitration framework doesn't have a function for synchronously enumerating all disks, it does effectively support asynchronous enumeration by registering a callback for disk appearance. This may or may not be useful depending on your use case - when providing the user with an interactive list of devices, this is usually exactly what you want though, as you'll automatically be notified of newly added devices.
I don't do Swift, sorry, but the following C code should be easy enough to understand to come up with something similar in other languages.
#include <DiskArbitration/DiskArbitration.h>
#include <stdio.h>
static void disk_appeared(DADiskRef disk, void* context)
{
printf("%s\n", DADiskGetBSDName(disk) ?: "(null)");
}
int main()
{
DASessionRef session = DASessionCreate(kCFAllocatorDefault);
DASessionSetDispatchQueue(session, dispatch_get_main_queue());
DARegisterDiskAppearedCallback(session, NULL, disk_appeared, NULL /*context*/);
dispatch_main();
}
Note that the callback will also be called for APFS snapshots, which don't have a BSD name, so DADiskGetBSDName returns NULL and you'll have to do a little bit of filtering.

LiDAR: export ARReferenceObject as .obj

Since LiDAR has been built into the newest 2020+ iOS devices (iPhone 12 Pro, iPad Pro).
ARKit has more possibilities than ever, including support for exporting to .obj.
Here is the code for exporting a ARReferenceObject to .arobject
guard let testRun = self.testRun,
let object = testRun.referenceObject,
let name = object.name
else {
print("Error: Missing scanned object.")
return
}
let documentURL = FileManager.default.temporaryDirectory
.appendingPathComponent(name + ".arobject")
DispatchQueue.global().async {
do {
try object.export(to: documentURL,
previewImage: testRun.previewImage)
} catch {
fatalError("Failed to save the file to \(documentURL)")
}
}
How do you export as .obj?
Sparse point cloud contained in .arobject file can't be exported as 3D geometry.
So the answer is: NO.

Swift, Extracting Glyphs from a Font by Name

I am attempting to glyphs from the SF Symbol front using the code sample below provided by an extract https://github.com/davedelong/sfsymbols. I have isolated the code that I can't get to work with values extracted from a run time.
I have condensed the code down to only the statements required to produce the problem.
This issue seems to be the name I am providing to the final statement in CTFontGetGlyphWithName, Every combination of values I have attempted returns 0.
Please note that you have to have the SF Symbols application installed for it to acquire the initial bundle.
Please excuse the code quality, this is just to create the shortest possible example of code to reproduce the problem.
thanks.
I have followed quite a few snipped and tutorials but there doesn't seem to be a great deal of practical examples of these more advanced functions available.
public class StackOverflow {
public func Test() {
let maybeCFURLs = LSCopyApplicationURLsForBundleIdentifier("com.apple.SFSymbols" as CFString, nil)?.takeRetainedValue()
let cfURLs = maybeCFURLs as? Array<URL>
let fontFile = cfURLs!.compactMap { url -> URL? in
guard let appBundle = Bundle(url: url) else { return nil }
return appBundle.url(forResource: "SFSymbolsFallback", withExtension: "ttf")
}
let attributes = [
kCTFontTraitsAttribute: [
kCTFontWeightTrait: NSFont.Weight.regular.rawValue
]
]
let provider = CGDataProvider(url: fontFile.first! as CFURL)
let cgFont = CGFont(provider!)
let ctAttributes = CTFontDescriptorCreateWithAttributes(attributes as CFDictionary)
let font = CTFontCreateWithGraphicsFont(cgFont!, 32, nil, ctAttributes)
// let name = "uni.plus.square.fill.medium" as CFString - Does not work
//var name = "square.fill.medium" as CFString - Does not work
let glyphName = "plus.square.fill.medium" as CFString
var glyph = CTFontGetGlyphWithName(font, glyphName )
}
}