I'm looking for efficient way to generate multiple stars at random places (for example 10-15 of them). With this code below I can easily achieve one star but my question would be how to generate more than one star.
let filter = StarShineGenerator()
filter.center = CIVector(values: [CGFloat.random(in: 50.0...finalImage.size.width), CGFloat.random(in: 50.0...finalImage.size.height)], count: 2)
let ciFilter = filter.filter()
guard let ciFilter = ciFilter else { return }
let result = ciFilter.outputImage
guard let result = result else { return }
finalImage = UIImage(cgImage: context.createCGImage(result, from: CIImage(image: finalImage)!.extent)!)
The CIStarShineGenerator creates a single starburst. If you want to generate lots of them, you'd have to call it repeatedly and composite the resulting images together.
Core Image also has compositing filters that will combine images; you could use one of those to combine your different starbursts. I don't know if it would be fast enough.
You might also install your image into multiple CALayers, apply rotation and shift transforms to those layers, add them to the backing layer of a UIView, and then capture the combined view's contents into an image.
Related
I created an ML Model for simple object detection. When I used it in the Xcode "Preview" tab, it perfectly identified and put a bounding box around the object. However, when I try to do it programmatically, I end up with an MLMultiArray (or a similar type depending on what I try) which I cannot use to create a bounding box. Here is the relevant part of my code:
func performAnalysis(frame: CVImageBuffer) {
guard let input = try? IdentifyBoomInput(imagePath: frame) else { return }
guard let result = try? identifyBoom.prediction(input: input) else { return }
if result.coordinates.count == 0 { return }
var coords = result.***????????????????***
print(coords)
}
I've tried every member of result (see here), but I'm unable to get anything useful for creating a bounding box. Any help would be greatly appreciated.
Update:
I had originally created the model using CreateML. I thought that maybe CreateML was the issue, so I used PyTorch with YOLOv8. When that didn't work, I tried YOLOv5, which also didn't work. My annotations and training data are clearly fine– as is my original model– because the interface Xcode provides for testing allows me to use it just fine. Thoughts?
Did you take a look at the .mlmodel specification in XCode? Screenshot of my .mlmodel details. I have trained the model with YOLOv5 PyTorch and converted the .pt file into .ml file format.
After that, in the image
3 is how I get the element of my MLMultiArray (1 x 25200 x 5+C), in my situation I only have one class --> (1 x 25200 x 6). As I read in this thread https://github.com/ultralytics/yolov5/issues/7011 :
private func getOutput(image: UIImage) {
let image = image.resizeImageTo(size: CGSize(width: 640, height: 640))
let buffer = image!.convertToBuffer()
let output = try? model.prediction(image: buffer!)
let prediction = output!.var_903
x = Double(truncating: prediction[0])
y = Double(truncating: prediction[1])
width = Double(truncating: prediction[2])
height = Double(truncating: prediction[3])
Hopefully, this could help
I'm kind of stuck right now when it comes to applying a specific texture on my 3d obj model.
Easiest solution of all would be to do let test = SCNScene(named: "models.scnassets/modelFolder/ModelName.obj"), but this requires that the mtl file maps the texture file directly inside of it which is not something that's possible with my current workflow.
With my current understanding, this leaves me with the option of using a scattering function to apply textures to a specific semantic, something like such :
if let url = URL(string: obj) {
let asset = MDLAsset(url: url)
guard let object = asset.object(at: 0) as? MDLMesh else {
print("Failed to get mesh from asset.")
self.presentAlert(title: "Warning", message: "Could not fetch the model.", firstBtn: "Ok")
return
}
// Create a material from the various textures with a scatteringFunction
let scatteringFunction = MDLScatteringFunction()
let material = MDLMaterial(name: "material", scatteringFunction: scatteringFunction)
let property = MDLMaterialProperty(name: "texture", semantic: .baseColor, url: URL(string: self.textureURL))
material.setProperty(property)
// Apply the texture to every submesh of the asset
object.submeshes?.forEach {
if let submesh = $0 as? MDLSubmesh {
submesh.material = material
}
}
// Wrap the ModelIO object in a SceneKit object
let node = SCNNode(mdlObject: object)
let scene = SCNScene()
scene.rootNode.addChildNode(node)
// Set up the SceneView
sceneView.scene = scene
...
}
The actual problem is the semantics. The 3d models are made on Unreal and for many models there's a png texture which has 3 semantics inside of it, namely Ambient Occlusion, Roughness and Metallic. Ambient Occlusion would need to be applied on the red channel, Roughness on the greed channel and Metallic on the blue channel.
How could I achieve this? An MdlMaterialSemantic has all of these possible semantics, but metallic, ambient occlusion and roughness are all separate. I tried simply applying the texture on each, but obviously this did not work very well.
Considering that my .png texture has all of those 3 "packaged" in it under a different channel, how can I work with this? I was thinking that maybe I could somehow use a small script to add mapping to the texture in the mtl file on my end in the app directly, but this seems sketchy lol..
What are my other options if there's no way of doing this? I've also been trying to use fbx files with assimpKit, but I couldn't manage to load any textures, just the model in black...
I am open to any suggestion, if more info is needed, please let me know! Thank you very much!
Sorry, I don't have enough rep to comment, but this might be more of a comment than an answer!
Have you tried loading the texture png image separately (as a NS/UI/CGImage) and then splitting it into three channels manually, then applying these channels separately? (Splitting into three separate channels is not as simple as it could be... but you could use this grayscale conversion for guidance, and just do one channel at a time.)
Once you have your objects in SceneKit, it is possibly slightly easier to modify these materials. Once you have a SCNNode with a SCNGeometry with a SCNMaterial you can access any of these materials and set the .contents property to almost anything (including a XXImage).
Edit:
Here's an extension you can try to extract the individual channels from a CGImage using Accelerate. You can get a CGImage from an NSImage/UIImage depending on whether you're on Mac or iOS (and you can load the file directly into one of those image formats).
I've just adapted the code from the link above, I am not very experienced with the Accelerate framework, so use at your own risk! But hopefully this puts you on the right path.
extension CGImage {
enum Channel {
case red, green, blue
}
func getChannel(channel: Channel) -> CGImage? {
// code adapted from https://developer.apple.com/documentation/accelerate/converting_color_images_to_grayscale
guard let format = vImage_CGImageFormat(cgImage: cgImage) else {return nil}
guard var sourceImageBuffer = try? vImage_Buffer(cgImage: cgImage, format: format) else {return nil}
guard var destinationBuffer = try? vImage_Buffer(width: Int(sourceImageBuffer.width), height: Int(sourceImageBuffer.height), bitsPerPixel: 8) else {return nil}
defer {
sourceImageBuffer.free()
destinationBuffer.free()
}
let redCoefficient: Float = channel == .red ? 1 : 0
let greenCoefficient: Float = channel == .green ? 1 : 0
let blueCoefficient: Float = channel == .blue ? 1 : 0
let divisor: Int32 = 0x1000
let fDivisor = Float(divisor)
var coefficientsMatrix = [
Int16(redCoefficient * fDivisor),
Int16(greenCoefficient * fDivisor),
Int16(blueCoefficient * fDivisor)
]
let preBias: [Int16] = [0, 0, 0, 0]
let postBias: Int32 = 0
vImageMatrixMultiply_ARGB8888ToPlanar8(&sourceImageBuffer,
&destinationBuffer,
&coefficientsMatrix,
divisor,
preBias,
postBias,
vImage_Flags(kvImageNoFlags))
guard let monoFormat = vImage_CGImageFormat(
bitsPerComponent: 8,
bitsPerPixel: 8,
colorSpace: CGColorSpaceCreateDeviceGray(),
bitmapInfo: CGBitmapInfo(rawValue: CGImageAlphaInfo.none.rawValue),
renderingIntent: .defaultIntent) else {return nil}
guard let result = try? destinationBuffer.createCGImage(format: monoFormat) else {return nil}
return result
}
}
I have an array of UIImages that I want to composite via an MTKView using a variety of specific comp modes (source-over, erase, etc). With the approach I describe below, I find that the biggest overhead seems to be in converting each UIImage into an MTLTexture that I can use to populate an MTKView's currentDrawable buffer.
The drawing loop looks like this:
for strokeDataCurrent in strokeDataArray {
let strokeImage = UIImage(data: strokeDataCurrent.image) // brushstroke
let strokeBbox = strokeDataCurrent.bbox // brush bounding box
let strokeType = strokeDataCurrent.strokeType // used to define comp mode
// convert strokeImage to a MTLTexture and composite
drawStrokeImage(paintingViewMetal: self.canvasMetalViewPainting, strokeImage: strokeImage!, strokeBbox: strokeBbox, strokeType: strokeType)
} // end of for strokeDataCurrent in strokeDataArray
Inside of drawStrokeImage, I convert each stroke to an MTLTexture like this:
guard let stampTexture = device!.makeTexture(descriptor: texDescriptor) else { return }
let destCGImage = strokeImage.cgImage!
let dstData: CFData = (destCGImage!.dataProvider!.data)!
let pixelData = CFDataGetBytePtr(dstData)
let region = MTLRegionMake2D(0, 0, Int(width), Int(height))
stampTexture.replace(region: region, mipmapLevel: 0, withBytes: pixelData!, bytesPerRow: Int(rowBytes))
with all this in place, I define a vertex buffer, set a commandEncoder:
defineCommandEncoder(renderCommandEncoder: renderCommandEncoder, vertexArrayStamps: vertexArrayStamps, metalTexture: stampTexture)
and call setNeedsDisplay() to render. This is happening for each stroke in the above for loop.
While I get ok performance in this approach, I'm wondering if I can squeeze more performance somewhere along the way? Like I said, I think the current bottleneck is in going from CGImage -> MTLTexture.
Note that I am rendering to a defined MTLTexture metalDrawableTextureComposite which I am blitting to the currentDrawable for each stroke:
copyTexture(buffer: commandBuffer!, from: metalDrawableTextureComposite, to: self.currentDrawable!.texture)
Hopefully this is enough detail to provide context for my question. Also, if anyone has ideas for other (GPU/Metal-based hopefully) faster compositing approaches that would be awesome. Any thoughts would be appreciated.
I've just ran a couple of tests comparing the performance of different ways of loading/creating a scene to see the performance impact. The test was simply rendering a 32x32 grid of cubes and eyeballing the CPU usage, memory, energy and rendering times. Not very scientific but there were some clear results. The four tests consisted of...
Load a .dae, e.g. SCNScene(named: "grid.dae")
Converting a .dae to .scn file in XCode and loadinf that
Building a grid in the Scnenekit editor manually using a reference node
Building a grid programatically using an SCNReference node (see code at bottom of question)
I expected 1 & 2 to be broadly the same and they were.
I expected test 3 to have much better performance than tests 1 & 2, and it did. The CPU load and energy usage was very low. It had half the memory foootprint and the rendering time was a fraction of the rendering times for test 1&2.
I was hoping test 4 would match test 3, but it didn't. It appeared to be the same or worse than tests 1&2.
// Code for test 4
let boxPath = Bundle.main.path(forResource: "box", ofType: "scn")
let boxUrl = URL(fileURLWithPath: boxPath!)
let offset: Int = 16
for xIndex:Int in 0...32 {
for yIndex:Int in 0...32 {
let boxReference = SCNReferenceNode(url: boxUrl)
scene.rootNode.addChildNode(boxReference!)
boxReference?.position.x = Float(xIndex - offset)
boxReference?.position.y = Float(yIndex - offset)
boxReference?.load()
}
}
Is the performance advantage that SceneKit's level editor provides available to developers and I'm just going about it wrong, or is Scenekit/XCode doing something bespoke under the hood?
UPDATE
In response to Confused's comment, I tried using the flattenedCone method on SCNNode. Here is a variation on the original code using that technique...
let boxPath = Bundle.main.path(forResource: "box", ofType: "scn")
let boxUrl = URL(fileURLWithPath: boxPath!)
let offset: Int = 16
let testNode = SCNNode()
for xIndex:Int in 0...32 {
for yIndex:Int in 0...32 {
let boxReference = SCNReferenceNode(url: boxUrl)
testNode.addChildNode(boxReference!)
boxReference?.position.x = Float(xIndex - offset)
boxReference?.position.y = Float(yIndex - offset)
boxReference?.load()
}
}
let optimizedNode = testNode.flattenedClone()
scene.rootNode.addChildNode(optimizedNode)
I'm trying to create a procedural game using Metal, and I'm using an octree based chunk approach for a Level of Detail implementation.
The method I'm using involves the CPU creating the octree nodes for the terrain, which then has its mesh created on the GPU using a compute shader. This mesh is stored in a vertex buffer and index buffer in the chunk object for rendering.
All of this seems to work fairly well, however when it comes to rendering chunks I'm hitting performance issues early on. Currently I gather an array of chunks to draw, then submit that to my renderer, that will create an MTLParallelRenderCommandEncoder to then create an MTLRenderCommandEncoder for each chunk, which is then submitted to the GPU.
By the looks of it around 50% of the CPU time is spent on creating the MTLRenderCommandEncoder for each chunk. Currently I'm just creating a simple 8 vertex cube mesh for each chunk, and I have an 4x4x4 array of chunks and I'm dropping to around 50fps in these early stages. (In reality it seems that there can only be up to 63 MTLRenderCommandEncoder in each MTLParallelRenderCommandEncoder so it's not fully 4x4x4)
I've read that the point of the MTLParallelRenderCommandEncoder is to create each MTLRenderCommandEncoder in a separate thread, yet I've not had much luck with getting this to work. Also multithreading it wouldn't get around the cap of 63 chunks being rendered as a max.
I feel that somehow consolidating the vertex and index buffers for each chunk into one or two larger buffers for submission would help, but I'm not sure how to do this without copious memcpy() calls and whether or not this would even improve efficiency.
Here's my code that takes in the array of nodes and draws them:
func drawNodes(nodes: [OctreeNode], inView view: AHMetalView){
// For control of several rotating buffers
dispatch_semaphore_wait(displaySemaphore, DISPATCH_TIME_FOREVER)
makeDepthTexture()
updateUniformsForView(view, duration: view.frameDuration)
let commandBuffer = commandQueue.commandBuffer()
let optDrawable = layer.nextDrawable()
guard let drawable = optDrawable else{
return
}
let passDescriptor = MTLRenderPassDescriptor()
passDescriptor.colorAttachments[0].texture = drawable.texture
passDescriptor.colorAttachments[0].clearColor = MTLClearColorMake(0.2, 0.2, 0.2, 1)
passDescriptor.colorAttachments[0].storeAction = .Store
passDescriptor.colorAttachments[0].loadAction = .Clear
passDescriptor.depthAttachment.texture = depthTexture
passDescriptor.depthAttachment.clearDepth = 1
passDescriptor.depthAttachment.loadAction = .Clear
passDescriptor.depthAttachment.storeAction = .Store
let parallelRenderPass = commandBuffer.parallelRenderCommandEncoderWithDescriptor(passDescriptor)
// Currently 63 nodes as a maximum
for node in nodes{
// This line is taking up around 50% of the CPU time
let renderPass = parallelRenderPass.renderCommandEncoder()
renderPass.setRenderPipelineState(renderPipelineState)
renderPass.setDepthStencilState(depthStencilState)
renderPass.setFrontFacingWinding(.CounterClockwise)
renderPass.setCullMode(.Back)
let uniformBufferOffset = sizeof(AHUniforms) * uniformBufferIndex
renderPass.setVertexBuffer(node.vertexBuffer, offset: 0, atIndex: 0)
renderPass.setVertexBuffer(uniformBuffer, offset: uniformBufferOffset, atIndex: 1)
renderPass.setTriangleFillMode(.Lines)
renderPass.drawIndexedPrimitives(.Triangle, indexCount: AHMaxIndicesPerChunk, indexType: AHIndexType, indexBuffer: node.indexBuffer, indexBufferOffset: 0)
renderPass.endEncoding()
}
parallelRenderPass.endEncoding()
commandBuffer.presentDrawable(drawable)
commandBuffer.addCompletedHandler { (commandBuffer) -> Void in
self.uniformBufferIndex = (self.uniformBufferIndex + 1) % AHInFlightBufferCount
dispatch_semaphore_signal(self.displaySemaphore)
}
commandBuffer.commit()
}
You note:
I've read that the point of the MTLParallelRenderCommandEncoder is to create each MTLRenderCommandEncoder in a separate thread...
And you're correct. What you're doing is sequentially creating, encoding with, and ending command encoders — there's nothing parallel going on here, so MTLParallelRenderCommandEncoder is doing nothing for you. You'd have roughly the same performance if you eliminated the parallel encoder and just created encoders with renderCommandEncoderWithDescriptor(_:) on each pass through your for loop... which is to say, you'd still have the same performance problem due to the overhead of creating all those encoders.
So, if you're going to encode sequentially, just reuse the same encoder. Also, you should reuse as much of your other shared state as possible. Here's a quick pass at a possible refactoring (untested):
let passDescriptor = MTLRenderPassDescriptor()
// call this once before your render loop
func setup() {
makeDepthTexture()
passDescriptor.colorAttachments[0].clearColor = MTLClearColorMake(0.2, 0.2, 0.2, 1)
passDescriptor.colorAttachments[0].storeAction = .Store
passDescriptor.colorAttachments[0].loadAction = .Clear
passDescriptor.depthAttachment.texture = depthTexture
passDescriptor.depthAttachment.clearDepth = 1
passDescriptor.depthAttachment.loadAction = .Clear
passDescriptor.depthAttachment.storeAction = .Store
// set up render pipeline state and depthStencil state
}
func drawNodes(nodes: [OctreeNode], inView view: AHMetalView) {
updateUniformsForView(view, duration: view.frameDuration)
// Set up completed handler ahead of time
let commandBuffer = commandQueue.commandBuffer()
commandBuffer.addCompletedHandler { _ in // unused parameter
self.uniformBufferIndex = (self.uniformBufferIndex + 1) % AHInFlightBufferCount
dispatch_semaphore_signal(self.displaySemaphore)
}
// Semaphore should be tied to drawable acquisition
dispatch_semaphore_wait(displaySemaphore, DISPATCH_TIME_FOREVER)
guard let drawable = layer.nextDrawable()
else { return }
// Set up the one part of the pass descriptor that changes per-frame
passDescriptor.colorAttachments[0].texture = drawable.texture
// Get one render pass descriptor and reuse it
let renderPass = commandBuffer.renderCommandEncoderWithDescriptor(passDescriptor)
renderPass.setTriangleFillMode(.Lines)
renderPass.setRenderPipelineState(renderPipelineState)
renderPass.setDepthStencilState(depthStencilState)
for node in nodes {
// Update offsets and draw
let uniformBufferOffset = sizeof(AHUniforms) * uniformBufferIndex
renderPass.setVertexBuffer(node.vertexBuffer, offset: 0, atIndex: 0)
renderPass.setVertexBuffer(uniformBuffer, offset: uniformBufferOffset, atIndex: 1)
renderPass.drawIndexedPrimitives(.Triangle, indexCount: AHMaxIndicesPerChunk, indexType: AHIndexType, indexBuffer: node.indexBuffer, indexBufferOffset: 0)
}
renderPass.endEncoding()
commandBuffer.presentDrawable(drawable)
commandBuffer.commit()
}
Then, profile with Instruments to see what, if any, further performance issues you might have. There's a great WWDC 2015 session about that showing several of the common "gotchas", how to diagnose them in profiling, and how to fix them.