As many other developers, I have plunged myself into Apple's new ARKit technology. It's great.
For a specific project however, I would like to be able to recognise (real-life) images in the scene, to either project something on it (just like Vuforia does with its target images), or to use it to trigger an event in my application.
In my research on how to accomplish this, I stumbled upon the Vision and CoreML frameworks by Apple. This seems promising, although I have not yet been able to wrap my head around it.
As I understand it, I should be able to do exactly what I want by finding rectangles using the Vision framework and feeding those into a CoreML model that simply compares it to the target images that I predefined within the model. It should then be able to spit out which target image it found.
Although this sounds good in my head, I have not yet found a way to do this. How would I go about creating a model like that, and is it even possible at all?
I found this project on Github some weeks ago:
AR Kit Rectangle Detection
I think that is exactly what you are looking for...
As of ARKit 1.5 (coming with IOS 11.3 in the spring of 2018), a feature seems to be implemented directly on top of ARKit that solves this problem.
ARKit will fully support image recognition.
Upon recognition of an image, the 3d coordinates can be retrieved as an anchor, and therefore content can be placed onto them.
Vision's ability to detect images was implemented in ARKit beginning from iOS 11.3+, so since then ARKit has ARImageAnchor subclass that extends ARAnchor parent class and conforms to ARTrackable protocol.
// Classes hierarchy and Protocol conformance...
ObjectiveC.NSObject: NSObjectProtocol
↳ ARKit.ARAnchor: ARAnchorCopying
↳ ARKit.ARImageAnchor: ARTrackable
ARWorldTrackingConfiguration class has a detectionImages instance property that is actually a set of images that ARKit attempts to detect in the user's environment.
open var detectionImages: Set<ARReferenceImage>!
And ARImageTrackingConfiguration class has a trackingImages instance property that is a set as well, it serves the same purpose – ARKit attempts to detect and track it in the user's environment.
open var trackingImages: Set<ARReferenceImage>
So, having a right configuration and an ability to automatically get ARImageAnchor in a ARSession, you can tether any geometry to that anchor.
P.S. If you want to know how to implement image detection feature in your ARKit app please look at this post.
Related
I tried out object detection in SceneKit but the object is only detected in the beginning once virtual objects are placed the object is no longer detected or tracked. I remember reading that object detection is not a continuous process but can not find that resource anymore. Please share resources that might be useful. Thank you.
Update:
ARKit 4 introduces two new types that conform to ARTrackable (ARAppClipCodeAnchor, and ARGeoAnchor), but no changes in ARObjectAnchor.
Also, while I was researching a little more about the subject, I found that Vision framework provides object tracking. This might be useful in some situations.
First, I will try to clarify some concepts.
SceneKit is a rendering framework, so its work is to render virtual objects on the screen.
RealityKit is also a rendering framework. This one was created by Apple to help developers in creating better AR experiences.
That said, rendering frameworks don't actually track anything in the real environment, like physical objects or images. The responsible for understanding the real world is ARKit. ARKit can detect real objects, images, etc., in the physical world.
Going back to your question, in order to be tracked, the physical object should have a correspondent anchor (ARAnchor) in your app that conforms to ARTrackable. As you can see in the documentation, the conforming types of ARTrackable are ARBodyAnchor, ARFaceAnchor, and ARImageAnchor. Object anchors are of the type ARObjectAnchor, and, as you can see in the docs, this class doesn't conform to ARTrackable, meaning that physical objects can't be tracked on ARKit 3.0. ARKit only detects the object, but doesn't track it. In the future, object tracking might be supported.
I need to integrate animations to an iOS SpriteKit game. The artist provides them in Spine format. Such animation consists of a set of images, a .spine file describing the animations and a .json file describing the metadata.
I see two possible ways to solve this problem:
Find (or make) a module allowing to add nodes and related animations from Spine files to a SpriteKit scene.
Convert the animation to a different format which can be imported to SpriteKit easily.
Please, share your experience. Any help is welcomed!
There is a library that allows you to do exactly that. Installing & usage examples are in the description of the library.
https://github.com/maxgribov/Spine
I am new to iPhone app development (but experienced developer in other languages) and I am making an ARKit application in which I need to track an image's position and display a rectangle around this image.
I could do it in C++ with OpenCV and make the appropriate "bridging" classes to be able to call this code from swift. Now, I need to get the images from ARKit and pass them to this function.
How do I suscribe a function which handles the ARFrames from the ARKit scene? I found that I could get some ARFrame from sceneView.session.currentFrame but I did not find how to make a function that would be called for each frame (or each time my function has ended and is ready to receive the next frame).
Also, I discovered the Vision Framework but it seems to only be able to track an element on which the user tapped. Is that right or is there a combination of predefined functions which could be used to this purpose?
You can get each captured frame by the camera with the session(:didUpdate:) method from the ARSessionDelegate protocol. Note that this method might be useful even if you don't use the provided ARFrame; for instance, I use it to "poll" if there's any virtual content added currently visible.
Regarding the tracking image question, I believe you could create an ARAnchor or SCNNode and position it over the image and ARKit would track it
Is it possible to use SceneKit for GPU accelerated graphics within a cocoa Application?
I'm looking for a way to do 2D animations in a PowerPoint like application, and not sure which direction to take to have GPU accelerated animations.
Picked up xCode about a week ago and went straight into swift with no Objective-C background. Didn't find any information on the topic, hoping to get this answered here.
Your screenshot depicts the template selection in the project creator. What template you use has no impact on what features is possible in your application. It is perfectly possible to create a Cocoa app and simply drag and drop a SCNView and set it to run a SceneKit scene.
That being said, SceneKit or SpriteKit is definitely overkill just for doing "2D animations like in PowerPoint". You should look into Core Animation:
https://developer.apple.com/library/mac/documentation/Cocoa/Conceptual/CoreAnimation_guide/Introduction/Introduction.html
I'm planing on making an utility app that could help people create table plans for sitting guests.
The thing is, the only way I see this is by letting users create first their own room, with the correct number of tables and sits, then populate the room with the guest list.
But to do that, I was thinking on using the Sprite Kit technology, because it seems to be an quick and easy way to make 2D schemes, use sprites and rely on coordinates.
Here comes the questions :
Is it possible to make an utility app with a "game" core ? Would Apple allow me to do so ?
If Sprite Kit is not an option, and regarding the fact I'm actually working on xcode 6 + Swift, what would be the alternative to make this process of letting the user create a room that uses X,Y coordinates?
I hope I'm clear, if you have any questions, I'll be happy to complete my requests. Thanks folks !
Nicolas
It depends on the design of your application and what it's requirements in terms of user interface. With SpriteKit, many of the interface elements provided by UIKit are not available and you'd have to build your own if you wanted to use SpriteKit exclusively.
However, there is nothing stopping you from also using UIKit for certain elements such as a UITableView, but it would be a separate view and input would need to be handled via UIKit system. With SpriteKit you don't have a bunch of UI elements to work with.
SpriteKit would mainly be most useful if you need animation and game-like functionality in your utility app.
Apple wouldn't restrict you from using SpriteKit for any kind of app.
In terms of an x,y coordinates, any element you create via SpriteKit or otherwise is going to allow you to position it in that way.
From SpriteKit programming guide, first sentence actually:
"Sprite Kit provides a graphics rendering and animation infrastructure that you can use to animate arbitrary textured images, or sprites."
That is what makes it ideal for games, however not limited to that.