SwiftUI: Detect finger position on Mac trackpad - swift

Question

I'm making a SwiftUI app for macOS and I'd like to use the trackpad as an (x, y) input by detecting the position of the user's fingers. I wanna be able to detect multiple fingers that are resting on the trackpad (not dragging). How do I do that?
A similar question has been asked before, but I'm asking again because that was from nearly 10 years ago, the answers are all in Obj-C (one in Swift 3), and I'm wondering if there's an updated methodology. Most importantly, I've no clue how to implement the Obj-C code into my SwiftUI app so if there isn't any updated methodology, I'd appreciate if someone could just explain how to implement the old Obj-C code.
To demonstrate what I mean, this video demo of the AudioSwift app does exactly what I want. macOS itself also uses this for hand-writing Chinese (although I don't need to recognize characters).

Answer 1

Always split your task into smaller ones and do them one by one. Ask in the same way and avoid broad questions touching lot of topics.
Goal
Track pad view (gray rectangle)
Circles on top of it showing fingers physical position
Step 1 - AppKit
SwiftUI doesn't provide all the required information
AppKit & NSTouch does - normalizedPosition.
First step is to create a simple AppKitTouchesView forwarding required touches via a delegate.
import SwiftUI
import AppKit
protocol AppKitTouchesViewDelegate: AnyObject {
// Provides `.touching` touches only.
func touchesView(_ view: AppKitTouchesView, didUpdateTouchingTouches touches: Set<NSTouch>)
}
final class AppKitTouchesView: NSView {
weak var delegate: AppKitTouchesViewDelegate?
override init(frame frameRect: NSRect) {
super.init(frame: frameRect)
// We're interested in `.indirect` touches only.
allowedTouchTypes = [.indirect]
// We'd like to receive resting touches as well.
wantsRestingTouches = true
}
required init?(coder: NSCoder) {
fatalError("init(coder:) has not been implemented")
}
private func handleTouches(with event: NSEvent) {
// Get all `.touching` touches only (includes `.began`, `.moved` & `.stationary`).
let touches = event.touches(matching: .touching, in: self)
// Forward them via delegate.
delegate?.touchesView(self, didUpdateTouchingTouches: touches)
}
override func touchesBegan(with event: NSEvent) {
handleTouches(with: event)
}
override func touchesEnded(with event: NSEvent) {
handleTouches(with: event)
}
override func touchesMoved(with event: NSEvent) {
handleTouches(with: event)
}
override func touchesCancelled(with event: NSEvent) {
handleTouches(with: event)
}
}
Step 2 - Simplified touch structure
Second step is to create a simple custom Touch structure which holds all the required information only and is SwiftUI compatible (not flipped y).
struct Touch: Identifiable {
// `Identifiable` -> `id` is required for `ForEach` (see below).
let id: Int
// Normalized touch X position on a device (0.0 - 1.0).
let normalizedX: CGFloat
// Normalized touch Y position on a device (0.0 - 1.0).
let normalizedY: CGFloat
init(_ nsTouch: NSTouch) {
self.normalizedX = nsTouch.normalizedPosition.x
// `NSTouch.normalizedPosition.y` is flipped -> 0.0 means bottom. But the
// `Touch` structure is meants to be used with the SwiftUI -> flip it.
self.normalizedY = 1.0 - nsTouch.normalizedPosition.y
self.id = nsTouch.hash
}
}
Step 3 - Wrap it for the SwiftUI
NSViewRepresentable documentation
Binding documentation
Third step is to create a SwiftUI view wrapping our AppKit AppKitTouchesView view.
struct TouchesView: NSViewRepresentable {
// Up to date list of touching touches.
#Binding var touches: [Touch]
func updateNSView(_ nsView: AppKitTouchesView, context: Context) {
}
func makeNSView(context: Context) -> AppKitTouchesView {
let view = AppKitTouchesView()
view.delegate = context.coordinator
return view
}
func makeCoordinator() -> Coordinator {
Coordinator(self)
}
class Coordinator: NSObject, AppKitTouchesViewDelegate {
let parent: TouchesView
init(_ view: TouchesView) {
self.parent = view
}
func touchesView(_ view: AppKitTouchesView, didUpdateTouchingTouches touches: Set<NSTouch>) {
parent.touches = touches.map(Touch.init)
}
}
}
Step 4 - Make a TrackPadView
Fourth step is to create a TrackPadView which internally does use our
TouchesView and draws circles on it representing physical location of fingers.
struct TrackPadView: View {
private let touchViewSize: CGFloat = 20
#State var touches: [Touch] = []
var body: some View {
ZStack {
GeometryReader { proxy in
TouchesView(touches: self.$touches)
ForEach(self.touches) { touch in
Circle()
.foregroundColor(Color.green)
.frame(width: self.touchViewSize, height: self.touchViewSize)
.offset(
x: proxy.size.width * touch.normalizedX - self.touchViewSize / 2.0,
y: proxy.size.height * touch.normalizedY - self.touchViewSize / 2.0
)
}
}
}
}
}
Step 5 - Use it in the main ContentView
Fifth step is to use it in our main view with some aspect ratio which is close to the real trackpad aspect ratio.
struct ContentView: View {
var body: some View {
TrackPadView()
.background(Color.gray)
.aspectRatio(1.6, contentMode: .fit)
.padding()
.frame(maxWidth: .infinity, maxHeight: .infinity)
}
}
Complete project
Open Xcode
Create a new project (macOS App & Swift & SwiftUI)
Copy & paste ContentView.swift from this gist