Custom Carbon key event handler fails after mouse events - swift

Question

I'm trying to write a custom NSMenu which will be able to list for key input and intercept the necessary events. This is to provide a simple search-as-you-type functionality for my open source clipboard manager.
It seems like the only way to do this is to install a custom Carbon event handler which will listen for key events and handler them accordingly, but it seems like there is an issue with such a custom handler.
Normally, I can propagate events downwards to other handlers (e.g. system ones) and they should be gracefully handled. This can be done by a simple callback:
let eventHandlerCallback: EventHandlerUPP = { eventHandlerCallRef, eventRef, userData in
let response = CallNextEventHandler(eventHandlerCallRef, eventRef!)
print("Response \(response)")
return response
}
This callback works perfectly and prints Response 0 all the time. This response means that the event is handled correctly.
However, things get weird once we send mouse events before keyboard events. In such case, the callback fails and prints Response -9874. This response means that the event was not handled correctly.
It seems like the event fails to be handled somewhere below my custom view and I don't know where exactly or how to overcome this issue.
To reproduce, I've uploaded the code to Gist which can be added to XCode playground and run. Once you see menu popup, press some keys (preferably arrow keys as they won't close the menu) and observe Response 0 in the console. After that, move cursor inside the menu and press more arrow keys. You should see Response -9874 in the console now.

Answer 1

Unclear if you have an NSTextField as your menu view, but if you use one then it's easy to setup a delegate for that text field that can get the current contents of the field as the user types (this takes care of them moving backwards with the arrow keys and then deleting characters, using the delete key, etc). Your delegate implements the appropriate delegate method and gets called each time the text changes:
extension CustomMenuItemViewController: NSTextFieldDelegate {
func controlTextDidChange( _ obj: Notification) {
if let postingObject = obj.object as? NSTextField {
let text = postingObject.stringValue
print("the text is now: \(text)")
}
}
}
Just to confirm this works as expected, I created the ViewController class for the custom menu item views (label + edit field) in a xib file and then dynamically built a simple test menu with a single menu item that has the custom view controller's view and added it to the menubar inside my app delegate:
func installCustomMenuItem() {
let menuBarItem = NSMenuItem(title: "Test", action: nil, keyEquivalent: "")
let menu = NSMenu(title: "TestMenu" )
let subMenuBarItem = NSMenuItem(title: "Custom View", action: nil, keyEquivalent: "")
subMenuBarItem.view = menuItemVC.view
menu.addItem(subMenuBarItem)
menuBarItem.submenu = menu
NSApp.mainMenu?.addItem(menuBarItem)
}
Looks like this after I typed "hello":
And you can from the console that my handler got called for every character typed:
the text is now: H
the text is now: He
the text is now: Hel
the text is now: Hell
the text is now: Hello
Your situation is probably a little different, but it seems like this approach is very clean and might work for you. If it won't for some reason, add a clarifying comment and we'll see if we can't make it work for you.
Addition:
It occurred to me that you might wish to not use NSTextField and so I was curious if it was as easy to do this with a custom view and it's relatively easy.
Make a subclass of NSView:
class CustomMenuView: NSView {
override var acceptsFirstResponder: Bool {
return true
}
override func draw(_ dirtyRect: NSRect) {
super.draw(dirtyRect)
// Drawing code here.
}
override func keyDown(with event: NSEvent) {
print("key down with character: \(String(describing: event.characters)) " )
}
}
Set the class of your root view in the custom view controller to be of this type of class and then all done as before - view controller loaded in applicationDidFinishLaunching and menus built and view controller's view (which is now a CustomMenuView) is set as the menuBarItem.view.
That's it. You now get your keyDown method called for every key down when the menu is dropped down.
key down with character: Optional("H")
key down with character: Optional("e")
key down with character: Optional("l")
key down with character: Optional("l")
key down with character: Optional("o")
key down with character: Optional(" ")
key down with character: Optional("T")
key down with character: Optional("h")
key down with character: Optional("i")
key down with character: Optional("s")
key down with character: Optional(" ")
key down with character: Optional("i")
key down with character: Optional("s")
key down with character: Optional(" ")
key down with character: Optional("c")
key down with character: Optional("o")
key down with character: Optional("o")
key down with character: Optional("l")
:)
Now your custom view (and subviews if you like) can do their own drawing and so on.
Addition with requested sample without the ViewController:
// Simple swift playground test
// The pop-up menu will show up onscreen in the playground at a fixed location.
// Click in the popup and then all key commands will be logged.
// The ViewController in my example above may be taking care of putting the custom view in the responder chain, or the fact that it's in a menubar and being invoked via a MenuItem might be.
// I'd suggest trying it in the actual environment rather than in a playground. In my test app you click the menu name in the menubar to drop down the menu and it is added to the responder chain and works as expected without having to click in the menu first to get the events flowing.
// There is no reason you need to be hooking events either with carbon events or the newer format. If you're in the responder chain of and implement the necessary, method then you'll get the key events you're looking for.
import AppKit
class CustomMenuView: NSView {
override var acceptsFirstResponder: Bool {
return true
}
override func draw(_ dirtyRect: NSRect) {
super.draw(dirtyRect)
// Drawing code here.
}
override func keyDown(with event: NSEvent) {
print("key down with character: \(String(describing: event.characters)) " )
}
}
func installCustomMenuItem() -> NSMenu {
// let menuBarItem = NSMenuItem(title: "Test", action: nil, keyEquivalent: "")
let resultMenu = NSMenu(title: "TestMenu" )
let subMenuBarItem = NSMenuItem(title: "Custom View", action: nil, keyEquivalent: "")
subMenuBarItem.view = CustomMenuView(frame: NSRect(x: 0, y: 0, width: 40, height: 44))
resultMenu.addItem(subMenuBarItem)
// menuBarItem.submenu = menu
return resultMenu
}
var menu = installCustomMenuItem()
menu.popUp(positioning: nil, at: NSPoint(x:600,y:400), in: nil)

Answer 2

I didn't manage to figure out why this issue was happening or how to fix it, but I understood that it's possible to work around this issue by intercepting all the keys and simulating their behavior manually.
For example, this is how I now handle down arrow key which is supposed to select next item in menu list:
class Menu: NSMenu {
func selectNext() {
var indexToHighlight = 1
if let item = highlightedItem {
indexToHighlight = index(of: item) + 1
}
if let itemToHighlight = self.item(at: indexToHighlight) {
let highlightItemSelector = NSSelectorFromString("highlightItem:")
perform(highlightItemSelector, with: itemToHighlight)
if itemToHighlight.isSeparatorItem || !itemToHighlight.isEnabled || itemToHighlight.isHidden {
selectNext()
}
}
}
}
This way, when I receive a key down event with down arrow key - I can just call the function and return true to prevent the event from reaching default NSMenu handler. Similarly, up arrow key can be done.
In case of a return key, I ended up with the following code:
class Menu: NSMenu {
func select() {
if let item = highlightedItem {
performActionForItem(at: index(of: item))
cancelTracking()
}
}
}
The full commit implementing this is https://github.com/p0deje/Maccy/commit/158610d1d.