RealityKit – Get a model entity from USDZ file - swift

I haave a file (exists in main bundle with target membership checked) named matrix.usdz and need to load it with
do {
let path = Bundle.main.path(forResource: "Matrix", ofType: "usdz")!
let url = URL(fileURLWithPath: path)
let assetsLoader = try Entity.load(contentsOf: url)
}
catch {
print(error)
}
But it crashes with
Thread 1: signal SIGABRT
on this line
let assetsLoader = try Entity.load(contentsOf: url)
Preview

You have to create an anchor if you need to load an entity into your scene. In order to get a ModelEntity, you need to grab it from the scene hierarchy using .children[X] subscript.
import RealityKit
class ViewController: UIViewController {
#IBOutlet var arView: ARView!
override func viewDidLoad() {
super.viewDidLoad()
do {
let path = Bundle.main.path(forResource: "Matrix", ofType: "usdz")!
let url = URL(fileURLWithPath: path)
// Scene
let scene = try Entity.load(contentsOf: url)
print(scene)
// Entity
let entity = scene.children[0].........children[0] as! ModelEntity
entity.model?.materials[0] = UnlitMaterial(color: .red)
let anchor = AnchorEntity(plane: .any)
anchor.addChild(scene)
arView.scene.anchors.append(anchor)
} catch {
print(error)
}
}
}
You can also get a model this way:
let modelEntity = try Entity.loadModel(contentsOf: url)
modelEntity.model?.materials[0] = UnlitMaterial(color: .red)
P.S.
I should say that you have an obvious naming error – "Matrix" vs "matrix". Also Matrix.rcproject and Matrix.usdz are not the same. To load Matrix.rcproject (Reality Composer project) use the following approach:
// .rcproject
let scene = try! Matrix.loadCircle()
let circleEntity = scene.children[0]...........children[0] as! ModelEntity
to load a USDZ model use this one:
// .usdz
let model = try! Entity.loadModel(named: "Matrix", in: nil)
But as far as I know, you do not need an RC project, so export USDZ from Reality Composer.
To load .reality file use the following approach:
// .reality
let carModel = try! Entity.loadAnchor(named: "car")
print(carModel)
arView.scene.addAnchor(carModel)
Here's your USDZ model on iOS simulator:

Related

How to create ARQuickLookPreviewItem with a ModelEntity with RealityKit?

I'm trying to Load a model from a usdz file, change the material colour of it and then display the updated model with AR QuickLook. I have got my code to the point that it loads the original model in the quick look preview. How do i use the updated ModelEntity?
guard let path = Bundle.main.path(forResource: "atm", ofType: "usdz") else {
fatalError("Couldn't find the supported asset file.")
}
let url = URL(fileURLWithPath: path)
guard let modelEntity = try? Entity.loadModel(contentsOf: url) else {
fatalError("Entity not found")
}
var newMaterial = SimpleMaterial()
newMaterial.color.tint = UIColor.cyan
for i in 0...(modelEntity.model?.materials.count ?? 1) - 1 {
modelEntity.model?.materials[i] = newMaterial
}
//let temporaryDirectory = FileManager.default.temporaryDirectory
//let temporaryFileURL = temporaryDirectory.appendingPathComponent("model.usdz")
let previewItem = ARQuickLookPreviewItem(fileAt: url) // how do i use modelEntity here?
previewItem.allowsContentScaling = allowsContentScaling
previewItem.canonicalWebPageURL = nil
return previewItem
}
I just need to change the colour of the model at runtime and show it. TIA

How do I load a Data object into a SCNScene?

I want to load a 3d usdz blob into a view, but since I only have the data object, I'm trying to initialize the scene with that with no luck.
To that, I initialize the SCNSceneSource() and then open it using .scene().
Now what I don't understand:
If I use a URL and load the scene directly - it works.
If I use a Data object on the same URL it doesn't.
Apple docs says, the data should be of type NSData but that seems wrong.
import SceneKit
let url = URL(string: "file:///Users/thilo/Desktop/Input/UU2.usdz")!
// working
let src_ok = SCNSceneSource(url: url)
let scn_ok = src_ok?.scene(options: nil, statusHandler: {
a,b,c,d in print("OK: \(a) \(b) \(String(describing: c)) \(d) ")
})
print("Ok: \(scn_ok)")
// Not working?
let data = try! Data(contentsOf: url)
let src_bad = SCNSceneSource(data: data)
let scn_bad = src_bad?.scene(options: nil, status handler: {
a,b,c,d in print("BAD: \(a) \(b) \(String(describing: c)) \(d) ")
})
print("Failed: \(scn_bad)")
running on Playground says:
Ok: Optional(<SCNScene: 0x6000038e1200>)
BAD: 0.0 SCNSceneSourceStatus(rawValue: 4) nil 0x000000016fa948bf
BAD: 0.0 SCNSceneSourceStatus(rawValue: 4) nil 0x000000016fa942af
BAD: 0.0 SCNSceneSourceStatus(rawValue: -1) Optional(Error Domain=NSCocoaErrorDomain Code=260 "Could not load the scene" UserInfo={NSLocalizedDescription=Could not load the scene, NSLocalizedRecoverySuggestion=An error occurred while parsing the COLLADA file. Please check that it has not been corrupted.}) 0x000000016fa942af
Failed: nil
What am I missing?
SCNSceneSource doesn't support .usdz in Data context
Official documentation says that SCNSceneSource object supports only .scn, .dae and .abc file formats. But it turns out that SceneKit doesn't support URL-loading of .usdz only in the context of working with Data. Thus, when working with Data, use files in the .scn format.
import SceneKit
import Cocoa
class GameViewController : NSViewController {
override func viewDidLoad() {
super.viewDidLoad()
if let url = URL(string: "file:///Users/swift/Desktop/ship.scn") {
let data = try! Data(contentsOf: url)
let source = SCNSceneSource(data: data)
let sceneView = self.view as! SCNView
sceneView.scene = source?.scene()
}
}
}
To load .usdz using URL, try SCNSceneSource.init?(url: URL)
class GameViewController : NSViewController {
override func viewDidLoad() {
super.viewDidLoad()
if let url = URL(string: "file:///Users/swift/Desktop/ship.usdz") {
let source = SCNSceneSource(url: url)
let sceneView = self.view as! SCNView
sceneView.scene = source?.scene()
}
}
}
Or use SCNScene object to load .usdz model
class GameViewController : NSViewController {
override func viewDidLoad() {
super.viewDidLoad()
let url = URL(fileURLWithPath: "/Users/swift/Desktop/ship.usdz")
do {
let scene = try SCNScene(url: url)
let sceneView = self.view as! SCNView
sceneView.scene = scene
sceneView.autoenablesDefaultLighting = true
} catch {
print(error.localizedDescription)
}
}
}
Gathering from the comment "does not support usdz" my solution is:
to create a temporary file ( .usdz) seems to be required by the API...
and then manually remove the temporary file after loading.
First extend FileManager with the below code:
public extension FileManager {
func temporaryFileURL(fileName: String = UUID().uuidString,ext: String) -> URL? {
return URL(fileURLWithPath: NSTemporaryDirectory(), isDirectory: true)
.appendingPathComponent(fileName + ext)
}
}
For a limited hard-coded use case:
let fm = FileManager.default
let tempusdz = fm.temporaryFileURL(ext:".usdz")!
fm.createFile(atPath: tempusdz.path(), contents: sceneData)
let src = SCNSceneSource(url: tempusdz)
if let scene = src?.scene(options: nil) {
....
}
try? fm.removeItem(at: tempusdz)
of course this is a hack, because it will only work if the data is in usdz format.
Since usdz is a ZIP archive, maybe testing for a zip and then just doing the below is a better option:
let sceneData:Data? = data
var sceneSrc: SCNSceneSource? = nil
var tempURL:URL? = nil
if let dataStart = sceneData?.subdata(in: 0..<4),
let dataMagic = String(data: dataStart, encoding: String.Encoding.utf8) as String?,
dataMagic == "PK\u{3}\u{4}" {
let fm = FileManager.default
tempURL = fm.temporaryFileURL(ext: ".usdz")
if let tempURL {
fm.createFile(atPath: tempURL.path(), contents: sceneData)
sceneSrc = SCNSceneSource(url: tempURL)
}
} else {
sceneSrc = SCNSceneSource(data: sceneData!)
}
let scene = sceneSrc?.scene()
if let tempURL {
try? FileManager.default.removeItem(at: tempURL)
}
Does anyone knows a better solution?
Is there an easy way to check the type of the Data ?
potential solution could be to verify the format of the data object and ensure that it is a valid COLLADA file.
import Foundation
let url = URL(string: "file:///Users/thilo/Desktop/Input/UU2.usdz")!
let data = try! Data(contentsOf: url)
print("Data size: \(data.count)")
print("Data format: \(data.description)")
you usually get these types of errors when the data wasn't properly formatted

How to play two audio files at once with Swift using AudioKit

I have two short wav audio files that I'm trying to play at the same time. Using AudioKit, I have an AudioEngine(), and I'm assuming I should use a MultiSegmentAudioPlayer() as the output along with the scheduleSegments().
Here is what I have:
class AudioPlayClass : ObservableObject {
var player = MultiSegmentAudioPlayer()
let engine = AudioEngine()
init(){}
func playFiles(){
self.engine.output = player
do {
try AVAudioSession.sharedInstance().setCategory(.playAndRecord, mode: .default, options: .defaultToSpeaker)
try AVAudioSession.sharedInstance().setActive(true)
try engine.start()
guard let url = Bundle.main.url(forResource: note1, withExtension: "wav", subdirectory: instrumentDirectory) else {return}
guard let url2 = Bundle.main.url(forResource: note2, withExtension: "wav", subdirectory: instrumentDirectory) else {return}
let audioFile = try AVAudioFile(forReading: url)
let audioFile2 = try AVAudioFile(forReading: url2)
let fileSampleRate = audioFile.processingFormat.sampleRate
let file2SampleRate = audioFile2.processingFormat.sampleRate
let fileNumberOfSamples = audioFile.length
let file2NumberOfSamples = audioFile2.length
let audioFileEndTime = Double(fileNumberOfSamples)/fileSampleRate
let audioFile2EndTime = Double(file2NumberOfSamples)/file2SampleRate
let segment1 = segment(audioFile: audioFile,
playbackStartTime: 0.0, fileStartTime: 0, fileEndTime: audioFileEndTime)
let segment2 = segment(audioFile: audioFile2,
playbackStartTime: 0.0, fileStartTime: 0, fileEndTime: audioFile2EndTime)
player2.scheduleSegments(audioSegments: [segment1, segment2])
player2.play()
} catch {
print(error.localizedDescription.debugDescription)
}
}
}
public struct segment : StreamableAudioSegment {
public var audioFile: AVAudioFile
public var playbackStartTime: TimeInterval
public var fileStartTime: TimeInterval
public var fileEndTime: TimeInterval
public var completionHandler: AVAudioNodeCompletionHandler?
}
I just have a basic understanding of playing audio in Swift and using AudioKit so any feedback would be greatly appreciated. Thanks!
I think MultiSegmentAudioPlayer is mostly for playing sounds sequentially. You probably just want 2 AudioPlayer() and play them both at the same time.
Solved this by creating two separate instances of an AudioEngine(), each with their own respective AudioPlayer(), and loaded and played them immediately one after the other.

Mac - Swift 3 - queuing audio files and playing

I would like to write an app in swift 3 in order to play queued audio files without any gap, crack or noise when passing from one to another.
My first try was using AvAudioPlayer and AvAudioDelegate (AVAudioPlayer using array to queue audio files - Swift), but I don't know how to preload the next song to avoid gap. Even if I know how to do it, I am not certain it is the best way to achieve my goal.
AVQueuePlayer seems to be a better candidate for the job, it is made for that purpose, but I don't find any example to help me out.
Maybe it is only a problem of preloading or buffering? I am a bit lost in this ocean of possibilities.
Any suggestion is welcomed.
It is far to be perfect, specially if you want to do it twice or more ("file exist" error), but it can serve as a base.
What it does is taking two files (mines are aif samples of ap. 4 sec.), encode them in one file and play the resulting files. If you have hundreds of them, assembled aleatory or not, it can make great fun.
All credits for the mergeAudioFiles function goes to #Peyman and #Pigeon_39. Concatenate two audio files in Swift and play them
Swift 3
import Cocoa
import AVFoundation
var action = AVAudioPlayer()
let path = Bundle.main.path(forResource: "audiofile1.aif", ofType:nil)!
let url = URL(fileURLWithPath: path)
let path2 = Bundle.main.path(forResource: "audiofile2.aif", ofType:nil)!
let url2 = URL(fileURLWithPath: path2)
let array1 = NSMutableArray(array: [url, url2])
class ViewController: NSViewController, AVAudioPlayerDelegate
{
#IBOutlet weak var LanceStop: NSButton!
override func viewDidLoad()
{
super.viewDidLoad()
}
override var representedObject: Any?
{
didSet
{
// Update the view, if already loaded.
}
}
#IBAction func Lancer(_ sender: NSButton)
{
mergeAudioFiles(audioFileUrls: array1)
let url3 = NSURL(string: "/Users/ADDUSERNAMEHERE/Documents/FinalAudio.m4a")
do
{
action = try AVAudioPlayer(contentsOf: url3 as! URL)
action.delegate = self
action.numberOfLoops = 0
action.prepareToPlay()
action.volume = 1
action.play()
}
catch{print("error")}
}
func audioPlayerDidFinishPlaying(_ player: AVAudioPlayer, successfully flag: Bool)
{
if flag == true
{
}
}
var mergeAudioURL = NSURL()
func mergeAudioFiles(audioFileUrls: NSArray) {
//audioFileUrls.adding(url)
//audioFileUrls.adding(url2)
let composition = AVMutableComposition()
for i in 0 ..< audioFileUrls.count {
let compositionAudioTrack :AVMutableCompositionTrack = composition.addMutableTrack(withMediaType: AVMediaTypeAudio, preferredTrackID: CMPersistentTrackID())
let asset = AVURLAsset(url: (audioFileUrls[i] as! NSURL) as URL)
let track = asset.tracks(withMediaType: AVMediaTypeAudio)[0]
let timeRange = CMTimeRange(start: CMTimeMake(0, 600), duration: track.timeRange.duration)
try! compositionAudioTrack.insertTimeRange(timeRange, of: track, at: composition.duration)
}
let documentDirectoryURL = FileManager.default.urls(for: .documentDirectory, in: .userDomainMask).first! as NSURL
self.mergeAudioURL = documentDirectoryURL.appendingPathComponent("FinalAudio.m4a")! as URL as NSURL
let assetExport = AVAssetExportSession(asset: composition, presetName: AVAssetExportPresetAppleM4A)
assetExport?.outputFileType = AVFileTypeAppleM4A
assetExport?.outputURL = mergeAudioURL as URL
assetExport?.exportAsynchronously(completionHandler:
{
switch assetExport!.status
{
case AVAssetExportSessionStatus.failed:
print("failed \(assetExport?.error)")
case AVAssetExportSessionStatus.cancelled:
print("cancelled \(assetExport?.error)")
case AVAssetExportSessionStatus.unknown:
print("unknown\(assetExport?.error)")
case AVAssetExportSessionStatus.waiting:
print("waiting\(assetExport?.error)")
case AVAssetExportSessionStatus.exporting:
print("exporting\(assetExport?.error)")
default:
print("Audio Concatenation Complete")
}
})
}
}

Create/Load SKTilemapnode without using SKS/Scene files

I create my sprite kit games programmatically and the sks files end up just confusing me.
Is there a way to load or create a tile map node without having to use the sks/scene file?
I figured out a way to to do it:
1) Create a SKS file called tilemaps
2) Create all your maps in there
3) Add this extension to load that scene file in your current scene:
extension SKNode {
class func unarchiveFromFile(file : NSString) -> SKNode? {
if let path = Bundle.main.path(forResource: file as String, ofType: "sks") {
let sceneData = NSData(contentsOfFile: path)
let archiver = NSKeyedUnarchiver(forReadingWith: sceneData as! Data)
archiver.setClass(self.classForKeyedUnarchiver(), forClassName: "SKScene")
let scene = archiver.decodeObject(forKey: NSKeyedArchiveRootObjectKey) as! SKNode
archiver.finishDecoding()
return scene
} else {
return nil
}
}
}
Then grab the tile map from the sks file. Weird trick is that you have to remove it from its parent first:
guard
let tileScene = SKScene.unarchiveFromFile(file: "TileMaps"),
let testMap = tileScene.childNode(withName: "Dungeon1")
as? SKTileMapNode else {
fatalError("Background node not loaded")
}
self.testMap = testMap
self.testMap.removeFromParent()
self.testMap.zPosition = 1200
self.addChild(testMap)