I need to select text in a WebKit view of another application (Apple Mail) using accessibility APIs.
For regular text fields, I do something like this:
func selectText(withRange range: CFRange) throws {
var range = range
guard let newValue: AXValue = AXValueCreate(AXValueType.cfRange, &range) else { return }
AXUIElementSetAttributeValue(self, kAXSelectedTextRangeAttribute as CFString, newValue)
}
However, in the composing window of Apple Mail every text seems to be of type Static Text which doesn't come with the necessary AXSelectedTextRange
It has AXSelectedTextMarkerRange, though, which requires an AXTextMarker. I just don't get how to create one of these. I have no trouble reading the text from a user created selection using this here, but I'm unable to select text via the accessibility APIs.
Thanks to the hint from Willeke I was able to figure it out. It is indeed possible to do it using AXTextMarkerForIndex. Knowing that it's actually pretty straightforward.
Here's my code:
func getTextMarker(forIndex index: CFIndex) throws -> AXTextMarker? {
var textMarker: AnyObject?
guard AXUIElementCopyParameterizedAttributeValue(self,"AXTextMarkerForIndex" as CFString, index as AnyObject, &textMarker) == .success else { return nil }
return textMarker as! AXTextMarker
}
func selectStaticText(withRange range: CFRange) throws {
guard let textMarkerStart = try? getTextMarker(forIndex: range.location) else { return }
guard let textMarkerEnd = try? getTextMarker(forIndex: range.location + range.length) else { return }
let textMarkerRange = AXTextMarkerRangeCreate(kCFAllocatorDefault, textMarkerStart, textMarkerEnd)
AXUIElementSetAttributeValue(self, "AXSelectedTextMarkerRange" as CFString, textMarkerRange)
}
Related
I am trying to make a document based app in swiftUI with a custom UI. I want iCloud capabilities in my app. I am trying to use iCloud Document (No cloudKit) way for storing data on iCloud container. I am using UIDocument and it's working. It's storing data to iCloud and I am able to retrieve it back.
Now the thing is when I run the app on two devices (iphone and iPad) and make changes to a file on one device, the changes are not reflecting on the other device while the file or say app is open. I have to close the app and relaunch it to see the changes.
I know I have to implement NSMetadataQuery to achieve this but I am struggling with it. I don't know any objective-C. I have been searching on the internet for a good article but could not find any. Can you please tell how do I implement this feature in my app. I have attach the working code of UIDocument and my Model class.
Thank you in advance !
UIDocument
class NoteDocument: UIDocument {
var notes = [Note]()
override func load(fromContents contents: Any, ofType typeName: String?) throws {
if let contents = contents as? Data {
if let arr = try? PropertyListDecoder().decode([Note].self, from: contents) {
self.notes = arr
return
}
}
//if we get here, there was some kind of problem
throw NSError(domain: "NoDataDomain", code: -1, userInfo: nil)
}
override func contents(forType typeName: String) throws -> Any {
if let data = try? PropertyListEncoder().encode(self.notes) {
return data
}
//if we get here, there was some kind of problem
throw NSError(domain: "NoDataDomain", code: -2, userInfo: nil)
}
}
Model
class Model: ObservableObject {
var document: NoteDocument?
var documentURL: URL?
init() {
let fm = FileManager.default
let driveURL = fm.url(forUbiquityContainerIdentifier: nil)?.appendingPathComponent("Documents")
documentURL = driveURL?.appendingPathComponent("savefile.txt")
document = NoteDocument(fileURL: documentURL!)
}
func loadData(viewModel: ViewModel) {
let fm = FileManager.default
if fm.fileExists(atPath: (documentURL?.path)!) {
document?.open(completionHandler: { (success: Bool) -> Void in
if success {
viewModel.notes = self.document?.notes ?? [Note]()
print("File load successfull")
} else {
print("File load failed")
}
})
} else {
document?.save(to: documentURL!, for: .forCreating, completionHandler: { (success: Bool) -> Void in
if success {
print("File create successfull")
} else {
print("File create failed")
}
})
}
}
func saveData(_ notes: [Note]) {
document!.notes = notes
document?.save(to: documentURL!, for: .forOverwriting, completionHandler: { (success: Bool) -> Void in
if success {
print("File save successfull")
} else {
print("File save failed")
}
})
}
func autoSave(_ notes: [Note]) {
document!.notes = notes
document?.updateChangeCount(.done)
}
}
Note
class Note: Identifiable, Codable {
var id = UUID()
var title = ""
var text = ""
}
This is a complex topic. Apple do provide some sample swift code, the Document-Based App Programming Guide for iOS and iCloud Design Guide.
There is also some good third party guidance: Mastering the iCloud Document Store.
I would recommend reading the above, and then return to the NSMetaDataQuery API. NSMetaDataQuery has an initial gathering phase and a live-update phase. The later phase can remain in operation for the lifetime of your app, allowing you to be notified of new documents in your app's iCloud container.
I have custom keyboard which I added as textfield.inputView. But my shouldChangeCharactersIn at UITextFieldDelegate doesn’t work? Any idea?
You have to implement the call to the delegate method yourself prior to inserting the text into the control.
For example, let’s assume you have some function for handling tap of a button, which calls UIKeyInput method insertText. Just check that the delegate implements the method and that it did not return false:
#objc func didTapButton(_ sender: ...) {
guard let range = target?.selectedRange else { return } // assumes `target` was defined to conform to `UITextInput`, using extension shared below
let string = ...
if let textField = target as? UITextField, textField.delegate?.textField?(textField, shouldChangeCharactersIn: range, replacementString: string) == false {
return
}
if let textView = target as? UITextView, textView.delegate?.textView?(textView, shouldChangeTextIn: range, replacementText: string) == false {
return
}
target?.insertText(string) // assumes `target` was defined to conform to `UIKeyInput`
}
Where:
extension UITextInput {
var selectedRange: NSRange? {
guard let textRange = selectedTextRange else { return nil }
let location = offset(from: beginningOfDocument, to: textRange.start)
let length = offset(from: textRange.start, to: textRange.end)
return NSRange(location: location, length: length)
}
}
Clearly, the details are dependent upon your custom keyboard implementation (the above is based upon https://stackoverflow.com/a/57275689/1271826), but hopefully it illustrates the basic idea.
I'm trying to share a record with other users in CloudKit but I keep getting an error. When I tap one of the items/records on the table I'm presented with the UICloudSharingController and I can see the iMessage app icon, but when I tap on it I get an error and the UICloudSharingController disappears, the funny thing is that even after the error I can still continue using the app.
Here is what I have.
Code
var items = [CKRecord]()
var itemName: String?
func tableView(_ tableView: UITableView, didSelectRowAt indexPath: IndexPath) {
let item = items[indexPath.row]
let share = CKShare(rootRecord: item)
if let itemName = item.object(forKey: "name") as? String {
self.itemName = item.object(forKey: "name") as? String
share[CKShareTitleKey] = "Sharing \(itemName)" as CKRecordValue?
} else {
share[CKShareTitleKey] = "" as CKRecordValue?
self.itemName = "item"
}
share[CKShareTypeKey] = "bundle.Identifier.Here" as CKRecordValue
prepareToShare(share: share, record: item)
}
private func prepareToShare(share: CKShare, record: CKRecord){
let sharingViewController = UICloudSharingController(preparationHandler: {(UICloudSharingController, handler: #escaping (CKShare?, CKContainer?, Error?) -> Void) in
let modRecordsList = CKModifyRecordsOperation(recordsToSave: [record, share], recordIDsToDelete: nil)
modRecordsList.modifyRecordsCompletionBlock = {
(record, recordID, error) in
handler(share, CKContainer.default(), error)
}
CKContainer.default().privateCloudDatabase.add(modRecordsList)
})
sharingViewController.delegate = self
sharingViewController.availablePermissions = [.allowPrivate]
self.navigationController?.present(sharingViewController, animated:true, completion:nil)
}
// Delegate Methods:
func cloudSharingControllerDidSaveShare(_ csc: UICloudSharingController) {
print("saved successfully")
}
func cloudSharingController(_ csc: UICloudSharingController, failedToSaveShareWithError error: Error) {
print("failed to save: \(error.localizedDescription)")// the error is generated in this method
}
func itemThumbnailData(for csc: UICloudSharingController) -> Data? {
return nil //You can set a hero image in your share sheet. Nil uses the default.
}
func itemTitle(for csc: UICloudSharingController) -> String? {
return self.itemName
}
ERROR
Failed to modify some records
Here is what I see...
Any idea what could be wrong?
EDIT:
By the way, the error is generated in the cloudSharingController failedToSaveShareWithError method.
Looks like you're trying to share in the default zone which isn't allowed. From the docs here
Sharing is only supported in zones with the
CKRecordZoneCapabilitySharing capability. The default zone does not
support sharing.
So you should set up a custom zone in your private database, and save your share and records there.
Possibly it is from the way you're trying to instantiate the UICloudSharingController? I cribbed my directly from the docs and it works:
let cloudSharingController = UICloudSharingController { [weak self] (controller, completion: #escaping (CKShare?, CKContainer?, Error?) -> Void) in
guard let `self` = self else {
return
}
self.share(rootRecord: rootRecord, completion: completion)
}
If that's not the problem it's something with either one or both of the records themselves. If you upload the record without trying to share it, does it work?
EDIT TO ADD:
What is the CKShareTypeKey? I don't use that in my app. Also I set my system fields differently:
share?[CKShare.SystemFieldKey.title] = "Something"
Try to add this to your info.plist
<key>CKSharingSupported</key>
<true/>
I am using MLVision cloud text recognition for my app. I capture/upload a photo and then I start the process. When it recognises the image and extract the text, then I separate it and append every separated block into an array.
The code below is for the whole process.
lazy var vision = Vision.vision()
var textRecognizer: VisionTextRecognizer!
var test = [] as Array<String>
override func viewDidLoad() {
super.viewDidLoad()
let options = VisionCloudTextRecognizerOptions()
options.languageHints = ["en","hi"]
textRecognizer = vision.cloudTextRecognizer(options: options)
}
//where pickedImage is the image that user captures.
let visionImage = VisionImage(image: pickedImage)
textRecognizer.process(visionImage, completion: { (features, error) in
guard error == nil, let features = features else {
self.resultView.text = "Could not recognize any text"
self.dismiss(animated: true, completion: nil)
return
}
for block in features.blocks {
for line in block.lines{
//for element in line.elements{
self.resultView.text = self.resultView.text + "\(line.text)"
}
}
self.separate()
})
func separate(){
let separators = CharacterSet(charactersIn: (":)(,•/·]["))
let ofWordsArray = self.resultView.text.components(separatedBy: separators)
for word in ofWordsArray{
let low = word.trimmingCharacters(in: .whitespacesAndNewlines).lowercased()
if low != ""{
test.append(low)
}
}
print(test)
}
Everything works fine and I get the result that I want.The problem is that I think is really slow. It takes about 20sec for the entire process.Is there a way to make it faster?
Thanks in advance.
You are using the VisionCloudTextRecognizer. Speed will depend on your connection, in my case it was only few seconds. Your other option is to use on-device text recognition or use a hybrid approach, where you first detect on-device, then correct with Cloud API later.
Below is a snippet that prints highlighted text, however, it does not work with all apps. For instance, one app I know it does not work with is Mail.
What attributes should I be searching for to get the highlighted text in apps like Mail :)
func getHighlightedText() -> AnyObject? {
let systemWideElement = AXUIElementCreateSystemWide()
var focusedElement: AnyObject?
let focusedCode = AXUIElementCopyAttributeValue(systemWideElement, "AXFocusedUIElement" as CFString, &focusedElement)
if (focusedCode == AXError.success) {
var selectedText: AnyObject?
let textCode = AXUIElementCopyAttributeValue(focusedElement as! AXUIElement, "AXSelectedText" as CFString, &selectedText)
if (textCode == AXError.success) {
return selectedText
}
}
return nil
}
sleep(3) #enough time to switch to another app and highlight the text
print(getHighlightedText())