I have an ARKit app that uses image recognition to trigger SceneKit/3D Object files, specifically for art exhibitions.
I've recently began implementing Firebase Storage in order to reduce the overall download size and rather download the 3D files on demand, per which exhibition the user is using!
I've successfully set up the app to download the SceneKit files to the mobile storage, but where I am stuck now is figuring out how to read the file from the specific downloaded location and continue working it in the image recognition/AR process accordingly.
Before, when the Scenekit files were included in the initial download via Scenekit Catalog in the app folder, I would read them like so:
let ShipScene = SCNScene(named: "art.scnassets/ship.scn")
Now, I've updated it to read the same string as the download URL, but the image recognition is not working.
I assume my problem is that it is not reading the right location, or I may be using the wrong function. I've put the app through to my phone through TestFlight and still no luck.
// Downloading from firebase to device URL
let shipURL = documentsURL.appendingPathComponent("file:///var/mobile/Containers/Data/Application/QZ_Gallery/SceneKitFiles/ship.scn", isDirectory: true)
let shipDownload = shipRef.write(toFile: shipURL)
// Attempting to pull the file from the downloaded location
let ShipScene = SCNScene(named:"file:///var/mobile/Containers/Data/Application/QZ_Gallery/SceneKitFiles/ship.scn" )
*** My question is in the final line of code I included. I am trying to make sure I am searching in the right place to retrieve the files that were downloaded, or if the function I am using is even proper for retrieving files stored locally on a mobile device.
Solved by first adding a function to find the scenepath of the container which has all my scenes, and then using the scene path in the fetching of the individual .scn file. Scene downloads from Firebase Storage (No use of Realtime Database) and integrates properly with the image recognition.
let documentsURL = FileManager.default.urls(for: .documentDirectory, in: .userDomainMask).first!
let scenePath = documentsURL.appendingPathComponent("Scenes")
if(!FileManager.default.fileExists(atPath: scenePath.absoluteString)) {
do {
try FileManager.default.createDirectory(atPath: scenePath.absoluteString, withIntermediateDirectories: true, attributes: nil)
} catch {
print(error.localizedDescription);
}
}
// Start Download, writing to a file
let sceneURL = scenePath.appendingPathComponent("Ship.scn", isDirectory: true)
Related
I added an USDZ file to my Xcode project. The USDZ file is showing as normal in the project in Xcode open on my mac, however, when I try to load it via code using modelIO library it loads the mesh but it doesn't load the texture.
In the snapshots below there is a bee model that I try to load. In the second there is a bee and aircraft, bee textures are not rendered whereas aircraft textures are loaded without problem. I also convert the USDZ file to the scn file and it worked properly. I think the problem is not about the file but the code I am using can't load the USDZ file properly. I am trying to load a USDZ file via code and which library I am using not much important. If you have other suggestions I am open to listen them.
I use the following code to load my model:
let fileName = “bee”
let ext = “usdz”
let asset = MDLAsset(url: Bundle.main.url(forResource:fileName, withExtension: ext)!)
let loadedScene = SCNScene(mdlAsset: asset)
for child in loadedScene.rootNode.childNodes {
child.geometry?.firstMaterial?.lightingModel = .physicallyBased
}
Here are some snapshots to demonstrate results I get:
- snapshot from Xcode viewer on my mac, successfully showing the bee with texture:
snapshot from the app on mobile phone, bee and aircraft, bee textures are not visible:
on the Xcode console I get following error logs:
What am I missing here to load the model file entirely, including textures, to my scene?
I found the solution. There is a method in MDLAsset as MDLAsset.loadTextures() to load the textures. When you call it then it loads the textures.
I have created a prototype application in unity using vuforia where I upload an image to myserver the server then sends the image (and associated assetbundle's link in metadata) to vuforia cloud to add it to the image target database. then in unity when camera tracks the image target I download the asset bundle to augment it.
public void OnNewSearchResult(TargetFinder.TargetSearchResult targetSearchResult)
{
TargetFinder.TargetSearchResult cloudRecoSearchResult =
(TargetFinder.TargetSearchResult)targetSearchResult;
mTargetMetadata = cloudRecoSearchResult.MetaData;
Debug.Log(mTargetMetadata);
mCloudRecoBehaviour.CloudRecoEnabled = false;
// Build augmentation based on target
if (ImageTargetTemplate)
{
Debug.Log("Image target activated");
// enable the new result with the same ImageTargetBehaviour:
ObjectTracker tracker = TrackerManager.Instance.GetTracker<ObjectTracker>();
ImageTargetBehaviour imageTargetBehaviour =
(ImageTargetBehaviour)tracker.TargetFinder.EnableTracking(
targetSearchResult, ImageTargetTemplate.gameObject);
JsonData jd = JsonMapper.ToObject(mTargetMetadata);
string url = jd["content-url"].ToString();
Debug.Log("video url :"+ "http://192.168.2.92/arads/" + url);
vidPlayer.url = "http://192.168.2.92/arads/"+url;
vidPlayer.Prepare();
if(!vidPlayer.isPlaying)
vidPlayer.Play();
}
}
the above code is to get associated video from the server. Can I get similar functionality with arcore or arfoundation, I read that arcore's refrence image database can have 1000 images,
what if the image I am tracking is not in the current database, can I switch to different db in that case ?
do I have to download and add the image to database in the applicatoin whenever I upload a new image on the server ?
can these images in arcore have meta data like in vuforia ?
The difference between ARCore and Vuforia is in ARCore you can add images to database in run time so you do not have to use any server.
You can switch to a different database by modifying Session config using this: GoogleARCore.ARCoreSessionConfig.AugmentedImageDatabase
As i said you can add images to database in run time so as long as you have the image in your project hierarchy you can add images to database.
I do not think having a meta data is possible only information you can get is database index of the image.
Good Luck!
I'm trying to play a local video file and keep getting the following log:
[framework] CUICatalog: Invalid asset name supplied: '(null)'
My video file is in the project directory and also in the main bundle resources. I've tried multiple versions of the syntax to play the video. Here's the code I have for now in a test project:
#IBAction func buttonAction(_ sender: Any) {
if let path = Bundle.main.path(forResource: "slipMovement", ofType:
"mp4") {
let video = AVPlayer(url: URL(fileURLWithPath: path))
let videoPlayer = AVPlayerViewController()
videoPlayer.player = video
present(videoPlayer, animated: true, completion: {
video.play()
})
}
}
When I use a AVplayer and PlayerLayer, I don't get the 'null' messages. It only happens when I use AVPlayerViewController.
I'm fairly new to programming, any help would be very much appreciated as I just can't find a working solution online.
I experienced this same problem when playing downloaded files from AWS S3.
I don't have an explanation for the meaning behind the error message.
But in my case it was because the file was not actually playable.
1. Make sure the path is correct.
I did not use Bundle.
I put the file under the Documents directory and I used FileManager to set the path:
let documentsUrl: URL = FileManager.default.urls(for: .documentDirectory, in: .userDomainMask).first!
let localFileName: String = "recording.mp4"
let localFileUrl = documentsUrl.appendingPathComponent(localFileName)
print("Playing \(localFileUrl.absoluteString)")
That would print something like this:
file:///var/mobile/Containers/Data/Application/D8AF92BD-D736-4B86-9FF4-C60ABA6C741B/Documents/recording.mp4
2. Make sure the file is actually playable.
For this, I wanted to check what the file is actually like when it's stored on the device.
It turned out that file was "corrupted" (ex. copy/download interrupted), so it really wasn't playable.
Connect the iOS device to your Mac.
From Xcode, go to Window > Devices and Simulators
Look for your device from the Connected list on the left.
Look for your app from the Installed Apps list.
Click the gears on the bottom of the list.
Select Download Container..
Download the *.xcappdata to a local directory
Open the directory in Finder
Right-click on the *.xcappdata, then select Show Package Contents
It should show something like the image below
Check the file's properties and try to play it
3. Make sure the file format is actually supported by AVPlayer
To see the supported AVPlayer formats, use print(AVURLAsset.audiovisualTypes()).
Reference:
https://developer.apple.com/documentation/avfoundation/avurlasset/1386800-audiovisualtypes
Once the path, the file, and the format were all OK, and I was able to play it normally using AVPlayer and AVPlayerViewController.
let player = AVPlayer(url: localFileUrl)
let playerViewController = AVPlayerViewController()
playerViewController.player = player
self.present(playerViewController, animated: true) {
playerViewController.player!.play()
}
I’m having no luck trying to selectively extract certain elements of a scene file. I’m using SCNSceneSource class. I’m running this swift code inside the Playgrounds App on my ipad pro. I’ve tried multiple dae, scn files using different ways. From the Resources directory.
I’ve also tried different class types, SCNScene, SCNNode, SCNGeometry, etc
Any help would be greatly appreciated.
guard let url = Bundle.main.url(forResource: "file", withExtension: "scn") else {
fatalError("Failed to find model file")
}
let sceneSource = SCNSceneSource(url: url as URL, options: nil)!
//List identifiers
let identifiers = sceneSource.identifiersOfEntries(withClass: SCNGeometry.self) as [String]
for identifier in identifiers {
print(identifiers)
}
Edit:
Btw, I’m using SketchUp to create the original Collada (DAE) files which I import into Xcode. SketchUp supports Collada 1.4 specification and schema. In a past SO answer, it was recommended to avoid Collada version 1.5 as its not supported by Scenekit.
When I review the Collada dae files in Xcode using scene graph (a hierarchy of nodes with attached geometries, lights, cameras etc. ) The attributes window corresponding to each node/geometry is showing “no identifier” when clicking on the different geometry/nodes in the scene graph. I was able to add some names by manually editing the xml schema in the dae file using a text editor (or clicking on the scene graph parts and renaming, on the relevant nodes/geometry... but even though each geometry element has an id=“geometry1” etc.... in the orignal Collada dae file schema.. that is still not showing up as the identifier in the attributes menu of Xcode.
Does anyone have a sample validated DAE file they can provide or link to, in others words, sceneSource.identifiersOfEntries(withClass: SCNGeometry.self) will return the SCNGeometry identifiers... this will help me troubleshoot any problems with SketchUp Collada dae files not being formatted properly by the SketchUp exporter.
Apple’s own scenekit documentation does not seemed to go into any details about how each scn class reconciles with the Collada schema.
NOTE: I’M not having a problem loading or using dae files in an iOS application. Or converting to an SCN file format. This problem only relates to extracting id’s from the sceneSource via “identfiersOfEntries”
I have this app in which the main purpose is to record video using AVFoundation, and then saving the outputfile delegate.
On first any normal run the app could properly record and save the video, and I could load the resulting outputfile URL into Avplayer I have on the next viewcontroller.
The AVPlayer would play just fine normally if the Xcode is not re-run. What I mean with rerunning is tapping into the Command+R again or clicking on the Play button on xcode to rebuild re-run.
What happens when I run the app again is that I could still see the files and list them via print, but when I'm trying to access the recorded file the app would no longer play the video. It would seem that the video wasn't loaded properly or anything even if it had already been working earlier. After this happens I could still go on record new ones and play the newly recorded ones, but not the old files.
The old files before rebuild of app would just look like this on AVPlayer
Also the AVPlayer would successfully return .readyToPlay status on my observer.
I'm confused how should I go on fixing this one.
if let videoPreviewController = storyBoard.instantiateViewController(withIdentifier: "VideoPreviewController") as? VideoPreviewController {
let url = NSURL(fileURLWithPath: "file:///var/mobile/Containers/Data/Application/61E182ED-D490-4C7D-BAB7-C90D095C7E43/Documents/Thursday,%208%20June%202017%20at%201:47:51%20PM%20Philippine%20Standard%20Time.mov")
videoPreviewController.player = AVPlayer()
let playerItem = AVPlayerItem.init(url: url as URL)
videoPreviewController.player?.replaceCurrentItem(with: playerItem)
videoPreviewController.report = relationship
videoPreviewController.appointment = self.appointment
navigation.pushViewController(videoPreviewController, animated: true)
}
Your problem is this line:
let url = NSURL(fileURLWithPath: "file:///var/mobile/Containers/Data/Application/61E182ED-D490-4C7D-BAB7-C90D095C7E43/Documents/Thursday,%208%20June%202017%20at%201:47:51%20PM%20Philippine%20Standard%20Time.mov")
That path will change every time your restart your app -- you can't hard code it. You have to ask the OS for the path to your documents folder and append your filename to it.
let userDocumentsURL = FileManager.default.urls(for: .documentDirectory, in: .userDomainMask)[0]
let fileName = "Thursday,%208%20June%202017%20at%201:47:51%20PM%20Philippine%20Standard%20Time.mov"
let movieURL = userDocumentsURL.appendingPathComponent(fileName)
And by the way, that's probably how you saved the file in the first place... get the path, append your filename and save it.