this is my first post on stack, so pardon any etiquette problems I'm not aware of
I am creating a simple drawing engine for one of my apps. It is mostly based on Apple's Touch Canvas demo for the apple pencil
https://developer.apple.com/documentation/uikit/touches_presses_and_gestures/illustrating_the_force_altitude_and_azimuth_properties_of_touch_input
That engine works by stroking paths, since I need to vary the width smoothly I decided to do it by filling paths, the logic to create that path is very similar to this cocos2d project
https://github.com/krzysztofzablocki/LineDrawing
I’m very happy with the performance of the engine at the moment, so I decided to be more ambitious: introduce variable opacity according to force. To do this, instead of drawing the whole path from the stroke, I subdivide it into segments, a segment looks like this using a stroke line

I draw each segment path one by one, in filled out form, it looks like this
So no issues right? Well, if you have an Ipad (or zoom), you notice a problem

The paths are actually stitched together! There is no smooth transition. Seems to be an antialiasing problem. What I noticed is that, if I very slightly offset the beginning of a new segment into the previous segment (right now, the end of the previous segment is the start of the new one), it seems to blend better , sometimes disappearing the stitches completely, but this is very tricky, while it works with colors at full opacity, when you reduce the opacity the same constant might not work, and sometimes there are variations in results when using the simulator vs a real device
I’m kinda lost here, is there any way kinda smooth the transition between segments?
This is my drawing routine, right now I'm using gradients, but I already tried to fill the path with a solid color with no clipping and got the same results
for (index,subpath) in pathResults.subPaths.enumerated(){
//Si no es inicio de linea el primer segmento es historico para suavizar, no dibujar
if !startOfLine && index == 0{
continue
}
let colors = [properties.color.cgColor.copy(alpha: opacityForPoint(subpath.startPoint)),
properties.color.cgColor.copy(alpha: opacityForPoint(subpath.endPoint))]
as CFArray
if let gradient = CGGradient(colorsSpace: context.colorSpace,
colors: colors,
locations: [0.0,1.0]){
context.addPath(subpath.subpath)
context.clip()
context.drawLinearGradient(gradient,
start: subpath.startPoint.properties.location,
end: subpath.endPoint.properties.location,
options: gradientOptions)
context.resetClip()
}
}
this is the code for the context I draw in:
private lazy var frozenContext: CGContext = {
let scale = self.properties.scale
var size = self.properties.size
size.width *= scale
size.height *= scale
let colorSpace = CGColorSpaceCreateDeviceRGB()
let context: CGContext = CGContext(data: nil,
width: Int(size.width),
height: Int(size.height),
bitsPerComponent: 8,
bytesPerRow: 0,
space: colorSpace,
bitmapInfo: CGImageAlphaInfo.premultipliedLast.rawValue)!
let transform = CGAffineTransform(scaleX: scale, y: scale)
context.concatenate(transform)
return context
}()
Related
I have an app where the user can draw numbers, I'm currently trying to differentiate between digits. My idea was to group overlapping lines by checking if they intersect. Currently it takes all the points and draws a SwiftUI Path.
The problem is that the paths contain a lot of points. The arm of the 4 contains 49 points, the stalk of the 4 has 30, and the 2 has 82 points. This makes comparing all the line segments for an intersection very expensive.
I have two questions:
Are there any swift functions to reduce the number of points while retaining the overall shape?
Are there any good methods for quickly determining whether complex paths intersect?
For curve simplification, start with Ramer–Douglas–Peucker. For this problem, that might get you most of the way there all by itself.
Next, consider the bounding boxes of the curves. For a Path, this is the boundingRect property. If two bounding boxes do not overlap, then the paths cannot intersect. If all your curves are composed of straight lines, this boundingRect should work well. If you also use quad or cubic curves, you may want to investigate path.cgPath.boundingBoxOfPath, which will be a tighter box (it doesn't include the control points).
I expect those approaches will be the best way, but another approach is to draw the paths and then look for the intersections. For example, you can create a small context and draw one curve in red and another in blue with a .screen blend mode. Then scan for any purple. (You can easily make this work for three curves simultaneously. With some linear algebra, it may be possible to scale to more simultaneous curves.)
This approach is an approximation, and its accuracy can be tuned by changing the size of the context. I expect an approximation is going to be fine (and even preferable) for your problem.
Here is a slapped together implementation just to show how it can work. I haven't given any thought to making this clean; the point is just the technique:
let width = 64
let height = 64
let context = CGContext(data: nil,
width: width,
height: height,
bitsPerComponent: 8,
bytesPerRow: width * 4,
space: CGColorSpaceCreateDeviceRGB(),
bitmapInfo: CGImageAlphaInfo.noneSkipFirst.rawValue)!
context.setShouldAntialias(false)
context.setBlendMode(.screen)
let path1 = UIBezierPath(rect: CGRect(x: 10, y: 20, width: 50, height: 8))
context.addPath(path1.cgPath)
context.setFillColor(UIColor.blue.cgColor)
context.drawPath(using: .fill)
let path2 = UIBezierPath(rect: CGRect(x: 40, y: 0, width: 8, height: 50))
context.addPath(path2.cgPath)
context.setFillColor(UIColor.red.cgColor)
context.drawPath(using: .fill)
let data = context.data!.bindMemory(to: UInt8.self, capacity: width * height * 4)
for i in stride(from: 0, to: width * height * 4, by: 4) {
if data[i + 1] == 255 && data[i + 3] == 255 {
print("Found overlap!")
break
}
}
I've turned off anti-aliasing here for consistency (and speed). With anti-aliasing, you may get partial colors. On the other hand, turning on anti-aliasing and adjusting what range of colors you treat as "overlap" may lead to more accurate results.
I do have a Metal rendering pipeline setup which encodes render commands and operates on a texture: MTLTexture object to load and store the output. This texture is rather large and and each render command just operates on a small fraction of the whole texture. The basic setup is roughly the following:
// texture: MTLTexture, pipelineState: MTLRenderPipelineState, commandBuffer: MTLCommandBuffer
// create and setup MTLRenderPassDescriptor with loadAction = .load
let renderPassDescriptor = MTLRenderPassDescriptor()
if let attachment = self.renderPassDescriptor?.colorAttachments[0] {
attachment.clearColor = MTLClearColorMake(0.0, 0.0, 0.0, 0.0)
attachment.texture = texture // texture size is rather large
attachment.loadAction = .load
attachment.storeAction = .store
}
// create MTLRenderCommandEncoder
guard let renderCommandEncoder = commandBuffer.makeRenderCommandEncoder(descriptor: renderPassDescriptor) else { return }
// limit rendering to small fraction of texture
let scissorRect = CGRect(origin: CGPoint.zero, size: 0.1 * CGSize(width: CGFloat(texture.width), height: CGFloat(texture.height))) // create rect begin small fraction of texture rect
let metalScissorRect = MTLScissorRect(x: Int(scissorRect.origin.x), y: Int(scissorRect.origin.y), width: Int(scissorRect.width), height: Int(scissorRect.height))
renderCommandEncoder.setScissorRect(metalScissorRect)
renderCommandEncoder.setRenderPipelineState(pipelineState)
renderCommandEncoder.setScissorRect(metalScissorRect)
// encode some commands here
renderCommandEncoder.endEncoding()
In practice many renderCommandEncoder objects are created, each time just operating on a small fraction of the texture. Unfortunately, each time a renderCommandEncoder is commited the whole texture is loaded and stored at the end, which is specified by the renderPassDescriptor due to the corresponding setting of its colorAttachment loadAction and storeAction.
My Question is:
Is it possible to limit the load and store process to a region of texture? (in order to avoid wasting computation time for loading and storing huge parts of the texture when only a small part is needed)
One approach, to avoid loading and storing the entire texture into the render pipeline, could be the following, assuming your scissor rectangle is constant between drawcalls:
Blit (MTLBlitCommandEncoder) the region of interest from the large texture to a smaller(e.g. the size of your scissor rectangle) intermediate texture.
Load and store, and draw/operate only on the smaller intermediate texture.
When done encoding, blit back the result to the original source region of the larger texture.
This way you load and store only the region of interest in your pipeline, with only the added constant memory cost of maintaining a smaller intermediate texture(assuming region of interest is constant between drawcalls).
Blitting is a fast operation and the above method should thus optimize your current pipeline.
Blocks just crumble apart.
How can this problem be solved?
Initializing blocks:
var boxNode = SCNNode(geometry: SCNBox(width: 0.75, height: 0.15, length: 0.25, chamferRadius: 0))
boxNode.position = SCNVector3(x: x1, y: y, z: z1)
boxNode.geometry?.firstMaterial = SCNMaterial()
boxNode.geometry?.firstMaterial?.diffuse.contents = UIImage(named: "wood.jpg")
boxNode.physicsBody = SCNPhysicsBody(type: .dynamic, shape: nil)
boxNode.eulerAngles.y = Float(Double.pi / 2) * rotation
boxNode.physicsBody?.friction = 1
boxNode.physicsBody?.mass = 0.5
boxNode.physicsBody?.angularDamping = 1.0
boxNode.physicsBody?.damping = 1
picture
video
full code
I won't be able to tell you how to fix it as I have the exact same problem which I wasn't able to solve. However, as I played around I figured out a couple of things (which you may find useful):
The same problem hasn't happened to me in pure SceneKit, hence I think it's a bug in ARKit
Node with physics has to be added to the rootNode of the scene, otherwise odd stuff happens (elements passing through each other, gravity behaving in an inconsistent way)
If you pass nil as shape parameter, SceneKit will figure bounding box based on the geometry of the node. This hasn't worked properly for me so what I've done (using SceneKit editor) was to duplicate the geometry and then set it as a custom shape for the bounding box (have a look at the attached image)
Overall I've found physics simulation in SceneKit when used with ARKit to be extremely buggy and I spent a lot of time "tricking" it into working more-or-less how I wanted it to work.
Apologies in advance as I'm not sure exactly what the right question is. The problems that I'm ultimately trying to address are:
1) Game gets laggy at times
2) CPU % can get high, as much as 50-60% at times, but is also sometimes relatively low (<20%)
3) Device (iPhone 6s) can get slightly warm
I believe what's driving the lagginess is that I'm constantly creating and removing circles in the SKScene. It's pretty much unavoidable because the circles are a critical element to the game and I have to constantly change their size and physicsBody properties so there's not much I can do in terms of reusing nodes. Additionally, I'm moving another node almost constantly.
func addCircle() {
let attributes = getTargetAttributes() //sets size, position, and color of the circle
let target = /*SKShapeNode()*/SKShapeNode(circleOfRadius: attributes.size.width)
let outerPathRect = CGRect(x: 0, y: 0, width: attributes.size.width * 2, height: attributes.size.width * 2)
target.position = attributes.position
target.fillColor = attributes.color
target.strokeColor = attributes.stroke
target.lineWidth = 8 * attributes.size.width / 35
target.physicsBody = SKPhysicsBody(circleOfRadius: attributes.size.width)
addStandardProperties(node: target, name: "circle", z: 5, contactTest: ContactCategory, category: CircleCategory) //Sets physicsBody properties
addChild(target)
}
The getAttributes() function is not too costly. It does have a while loop to set the circle position, but it doesn't usually get used when the function is called. Otherwise, it's simple math.
Some other details:
1) The app runs at a constant 120 fps. I've tried setting the scene/view lower by adding view.preferredFramesPerSecond = 60 in GameScene.swift and gameScene.preferredFramesPerSecond = 60 in GameViewController. Neither one of these does anything to change the fps. Normally when I've had performance issues in other apps, the fps dipped, however, that isn't happening here.
2) I’ve tried switching the SKShapeNode initializer to use a path versus circleOfRadius and then resetting the path. I’ve also tried images, however, because I have to reset the physicsBody, there doesn’t appear to be a performance gain.
3) I tried changing the physicsWorld speed, but this also had little effect.
4) I've also used Instruments to try to identify the issue. There are big chunks of resources being used by SKRenderer, however, I can't find much information on this.
Creating SKShapeNodes are inefficient, try to use it as few times as you can. instead, create a template shape, and convert it to an SKSpriteNode.
If you need to change the size, then use xScale and yScale, if you need to change the color, then use color with colorBlendFactor of 1
If you need to have a varying color stroke, then change the below code to have 2 SKSpriteNodes, 1 SKSpriteNode that handles only the fill, and 1 SKSpriteNode that handles only the stroke. Have the stroke sprite be a child of the fill sprite with a zPosition of 0 and set the stroke color to white. You can then apply the color and colorBlendFactor to the child node of the circle to change the color.
lazy var circle =
{
let target = SKShapeNode(circleOfRadius: 1000)
target.fillColor = .white
//target.strokeColor = .black //if stroke is anything other than black, you may need to do 2 SKSpriteNodes that layer each other
target.lineWidth = 8 * 1000 / 35
let texture = SKView().texture(from:target)
let spr = SKSpriteNode(texture:texture)
spr.physicsBody = SKPhysicsBody(circleOfRadius: 1000)
addStandardProperties(node: spr, name: "circle", z: 5, contactTest:ContactCategory, category: CircleCategory) //Sets physicsBody properties
return spr
}()
func createCircle(of radius:CGFloat,color:UIColor) -> SKSpriteNode
{
let spr = circle.copy()
let scale = radius/1000.0
spr.xScale = scale
spr.yScale = scale
spr.color = color
spr.colorBlendFactor = 1.0
return spr
}
func addCircle() {
let attributes = getTargetAttributes() //sets size, position, and color of the circle
let spr = createCircle(of:attribute.width,color:attributes.color)
spr.position = attributes.position
addChild(str)
}
In learning 3d graphics programming for games I decided to start off simple by using the Scene Kit 3D API. My first gaming goal was to build a very simplified mimic of MineCraft. A game of just cubes - how hard can it be.
Below is a loop I wrote to place a ride of 100 x 100 cubes (10,000) and the FPS performance was abysmal (~20 FPS). Is my initial gaming goal too much for Scene Kit or is there a better way to approach this?
I have read other topics on StackExchange but don't feel they answer my question. Converting the exposed surface blocks to a single mesh won't work as the SCNGeometry is immutable.
func createBoxArray(scene : SCNScene, lengthCount: Int, depthCount: Int) {
let startX : CGFloat = -(CGFloat(lengthCount) * CUBE_SIZE) + (CGFloat(lengthCount) * CUBE_MARGIN) / 2.0
let startY : CGFloat = 0.0
let startZ : CGFloat = -(CGFloat(lengthCount) * CUBE_SIZE) + (CGFloat(lengthCount) * CUBE_MARGIN) / 2.0
var currentZ : CGFloat = startZ
for z in 0 ..< depthCount {
currentZ += CUBE_SIZE + CUBE_MARGIN
var currentX = startX
for x in 0 ..< lengthCount {
currentX += CUBE_SIZE + CUBE_MARGIN
createBox(scene, x: currentX, y: startY, z: currentZ)
}
}
}
func createBox(scene : SCNScene, x: CGFloat, y: CGFloat, z: CGFloat) {
var box = SCNBox(width: CUBE_SIZE, height: CUBE_SIZE, length: CUBE_SIZE, chamferRadius: 0.0)
box.firstMaterial?.diffuse.contents = NSColor.purpleColor()
var boxNode = SCNNode(geometry: box)
boxNode.position = SCNVector3Make(x, y, z)
scene.rootNode.addChildNode(boxNode)
}
UPDATE 12-30-2014:
I modified the code so the SCNBoxNode is created once and then each additional box in the array of 100 x 100 is created via:
var newBoxNode = firstBoxNode.clone()
newBoxNode.position = SCNVector3Make(x, y, z)
This change appears to have increased FPS to ~30fps. The other statistics are as follows (from the statistics displayed in the SCNView):
10K (I assume this is draw calls?)
120K (I assume this is faces)
360K (Assuming this is the vertex count)
The bulk of the run loop is in Rendering (I'm guesstimating 98%). The total loop time is 26.7ms (ouch). I'm running on a Mac Pro Late 2013 (6-core w/Dual D500 GPU).
Given that a MineCraft style game has a landscape that constantly changes based on the players actions I don't see how I can optimize this within the confines of Scene Kit. A big disappointment as I really like the framework. I'd love to hear someone's ideas on how I can address this issue - without that, I'm forced to go with OpenGL.
UPDATE 12-30-2014 # 2:00pm ET:
I am seeing a significant performance improvement when using flattenedClone(). The FPS is now a solid 60fps even with more boxes and TWO drawing calls. However, accommodating a dynamic environment (as MineCraft supports) is still proving problematic - see below.
Since the array would change composition over time I added a keyDown handler to add an even larger box array to the existing and timed the difference between adding the array of boxes resulting in far more calls versus adding as a flattenedClone. Here's what I found:
On keyDown I add another array of 120 x 120 boxes (14,400 boxes)
// This took .0070333 milliseconds
scene?.rootNode.addChildNode(boxArrayNode)
// This took .02896785 milliseconds
scene?.rootNode.addChildNode(boxArrayNode.flattenedClone())
Calling flattenedClone() again is 4x slower than adding the array.
This results in two drawing calls having 293K faces and 878K vertices. I'm still playing with this and will update if I find anything new. Bottom line, with my additional testing I still feel Scene Kit's immutable geometric constraints mean I can't leverage the framework.
As you mentionned Minecraft, I think it's worth looking at how it works.
I have no technical details or code sample for you, but everything should be pretty straightfoward:
Have you ever played minecraft online, and the terrain is not loading allowing you to see through? That's because there is no geometry inside.
let's assume I have a 2x2x2 array of cubes. That makes 2*2*2*6*2 = 96 triangles.
However, if you test and draw only the polygons on the visible from the camera point of view, maybe by testing the normals (easy since it's cubes), this number goes down to 48 triangles.
If you find a way to see which faces are occluded by other ones (which shouldn't be too hard either considering you're working with flat, quared, grid based faces) you can only draw these. that way, we're drawing between 8 and 24 triangleS. That's up to 90% optimisation.
If you want to get really deep, you can even combine faces, to make a single N-gon out of the visible, flat faces. You can do that if you create a new way to generate the geometry on the fly that combines the two previous methods and test for adgacent visible faces on the same plane.
If you succeed, we're talking 2 to 6 polygons instead of 96, to render 8 cubes.
Note that the last method only works if your blocks are touching each other.
There is probably a ton of Minecraft-like renderer papers, a few googles will help you figure it out!
Why does drop-frame occur?
September 04, 2022
Almost 8 years passed since you asked this question, but not much has changed...
1. Polygons' count
The number of polygons in SceneKit or RealityKit scene must not exceed 100,000 triangular polygons. An ideal SceneKit's scene, that is capable of rendering all the models faster, should contain less than 50,000 polygons. Your scene contains 120,000 polygons. Do not forget that SceneKit renders models using single thread (unlike multi-threaded RealityKit renderer).
2. Shaders
In Xcode 14.0+, SceneKit's default .lightingModel of any 3D library's primitive set in Material Inspector (UI version) is .physicallyBased material. This is the most computationally intensive shader. Programmatic version of the .lightingModel for any SCN procedural geometry is .blinn shading model. The least computationally intensive shader is .constant (it doesn't depend on lighting).
3. What's inside a frustum
If all 10,000 cubes are inside the SceneKit camera frustum, then the frame rate will be 20-30 fps. But if you dollied in the cubes' matrix and see no more than a ninth part of it, then the frame rate will be 60 fps. Thus, SceneKit does not render those objects that are outside the frustum's bounds.
4. Number of meshes in SCNScene
Each model mesh results in a draw call. To achieve 60 fps each draw call should be 16 milliseconds or less. For best performance, Apple engineers advise to limit the number of meshes in a .usdz file to around 50. Unfortunately, I did not find a value for .scn files in the official documentation.
5. Lighting and shadows
Lighting and shadowing (especially shadowing) are very computationally intensive tasks. The general advice is the following – avoid using .forward shadows and hi-rez textures with fake shadows.
Look at this post for details.
SwiftUI code for testing
Xcode 14.0+, SwiftUI 4.0+, Swift 5.7
import SwiftUI
import SceneKit
struct ContentView: View {
var scene = SCNScene()
var options: SceneView.Options = [.allowsCameraControl]
var body: some View {
ZStack {
ForEach(-50...49, id: \.self) { x in
ForEach(-50...49, id: \.self) { z in
let _ = DispatchQueue.global().async {
scene.rootNode.addChildNode(createCube(x, 0, z))
}
}
}
SceneView(scene: scene, options: options)
.ignoresSafeArea()
let _ = scene.background.contents = UIColor.black
}
}
func createCube(_ posX: Int, _ posY: Int, _ posZ: Int) -> SCNNode {
let geo = SCNBox(width: 0.5, height: 0.5, length: 0.5,
chamferRadius: 0.0)
geo.firstMaterial?.lightingModel = .constant
let boxNode = SCNNode(geometry: geo)
boxNode.position = SCNVector3(posX, posY, posZ)
return boxNode
}
}
Here, all cubes are within the viewing frustum, so there are obvious reasons for a drop-frame.
And here, just a part of a scene is within the viewing frustum, so there is no drop-frame.