Rough Metal rasterization - swift

I followed a tutorial to see how to draw a triangle in Metal. I am beginning metal, and what the issue is is that, the triangle has really rough edges. It's like the rasterizer is cutting corners. It looks pixelated, and the pixels of the triangle edges are much larger than my screen's pixels. How can I rasterize it smoother if I am using rasterization correctly?
import Cocoa
import MetalKit
class ViewController: NSViewController {
var MetalView: MTKView {
return view as! MTKView
}
var Device: MTLDevice!
var CommandQue: MTLCommandQueue!
var PipelineState: MTLRenderPipelineState?
var VertexBuffer: MTLBuffer?
override func viewDidLoad() {
super.viewDidLoad()
MetalView.device = MTLCreateSystemDefaultDevice()
Device = MetalView.device
MetalView.clearColor = MTLClearColorMake(0, 1, 1, 1)
CommandQue = Device.makeCommandQueue()
let CommandBuffer = CommandQue.makeCommandBuffer()
let CommandEncoder = CommandBuffer!.makeRenderCommandEncoder(descriptor: MetalView.currentRenderPassDescriptor!)
let Library = Device.makeDefaultLibrary()
let VertexFunction = Library!.makeFunction(name: "VertexShader")
let FragmentFunction = Library!.makeFunction(name: "FragmentShader")
let PipelineDescriptor = MTLRenderPipelineDescriptor()
PipelineDescriptor.vertexFunction = VertexFunction
PipelineDescriptor.fragmentFunction = FragmentFunction
PipelineDescriptor.colorAttachments[0].pixelFormat = .bgra8Unorm
do {
PipelineState = try Device.makeRenderPipelineState(descriptor: PipelineDescriptor)
} catch let Error as NSError {
print("Error: \(Error.localizedDescription)")
}
let Vertices: [Float] = [0, 1, 0, -1, -1, 0, 1, -1, 0]
VertexBuffer = Device.makeBuffer(bytes: Vertices, length: Vertices.count*MemoryLayout<Float>.size, options: [])
CommandEncoder!.setRenderPipelineState(PipelineState!)
CommandEncoder!.setVertexBuffer(VertexBuffer, offset: 0, index: 0)
CommandEncoder!.drawPrimitives(type: .triangle, vertexStart: 0, vertexCount: Vertices.count)
CommandEncoder!.endEncoding()
CommandBuffer!.present(MetalView.currentDrawable!)
CommandBuffer!.commit()
}
}

Not sure, but I think viewDidLoad is too early for rendering. The view might not have its final size here.
Instead, only perform the initialization code in viewDidLoad (pipeline state and buffers), register yourself as the delegate of the MTKView and then implement this callback function:
func draw(in view: MTKView) {
// your drawing code here
}
This is called every time you view gets drawn to the screen.

As Frank has suggested, viewDidLoad is not the place to issue draw calls. This way you are rendering only 1 frame.
About the aliasing, it depends on the resolution you are rendering in. But there are techniques to handle that, like smoothstep()ing edges in the fragment shader or post-processing anti-aliasing like rendering at higher resolution and then downsampling.
Also, in swift variables use camelCase, using PascalCase seems like you are calling statics on classes.

Related

Use MetalView with SwiftUI? How do I put something to display in there?

I'm stuck with SwiftUI and Metal up to the point of being about to give up.
I got this example from https://developer.apple.com/forums/thread/119112?answerId=654964022#654964022 :
import MetalKit
struct MetalView: NSViewRepresentable {
func makeCoordinator() -> Coordinator {
Coordinator(self)
}
func makeNSView(context: NSViewRepresentableContext<MetalView>) -> MTKView {
let mtkView = MTKView()
mtkView.delegate = context.coordinator
mtkView.preferredFramesPerSecond = 60
mtkView.enableSetNeedsDisplay = true
if let metalDevice = MTLCreateSystemDefaultDevice() {
mtkView.device = metalDevice
}
mtkView.framebufferOnly = false
mtkView.clearColor = MTLClearColor(red: 0, green: 0, blue: 0, alpha: 0)
mtkView.drawableSize = mtkView.frame.size
mtkView.enableSetNeedsDisplay = true
return mtkView
}
func updateNSView(_ nsView: MTKView, context: NSViewRepresentableContext<MetalView>) {
}
class Coordinator : NSObject, MTKViewDelegate {
var parent: MetalView
var metalDevice: MTLDevice!
var metalCommandQueue: MTLCommandQueue!
init(_ parent: MetalView) {
self.parent = parent
if let metalDevice = MTLCreateSystemDefaultDevice() {
self.metalDevice = metalDevice
}
self.metalCommandQueue = metalDevice.makeCommandQueue()!
super.init()
}
func mtkView(_ view: MTKView, drawableSizeWillChange size: CGSize) {
}
func draw(in view: MTKView) {
guard let drawable = view.currentDrawable else {
return
}
let commandBuffer = metalCommandQueue.makeCommandBuffer()
let rpd = view.currentRenderPassDescriptor
rpd?.colorAttachments[0].clearColor = MTLClearColorMake(0, 1, 0, 1)
rpd?.colorAttachments[0].loadAction = .clear
rpd?.colorAttachments[0].storeAction = .store
let re = commandBuffer?.makeRenderCommandEncoder(descriptor: rpd!)
re?.endEncoding()
commandBuffer?.present(drawable)
commandBuffer?.commit()
}
}
}
... but I can't get my head around how to use this MetalView(), which does seem to work when I call it from a SwiftUI view, to display data. I want to use it to display a CIImage which will be filtered and manipulated with CIFilters...
Can someone please point me in the right direction on how to tell this view how to display something? I think I need it to display the content of a texture but tried countless hours and ended up starting from scratch for more countless times...
This is how I run my image filters now but it results in very slow sliders, which is why I decided to try learning about Metal... but it's been really time-consuming and. frustrating due to the lack of documentation...
func ciExposure (inputImage: CIImage, inputEV: Double) -> CIImage {
let filter = CIFilter(name: "CIExposureAdjust")!
filter.setValue(inputImage, forKey: kCIInputImageKey)
filter.setValue(inputEV, forKey: kCIInputEVKey)
return filter.outputImage!
}
I think I need to take that filter.outputImage and pass it on to the MetalView somehow?
Any help is really, really appreciated...
Apple's WWDC 2022 contained a tutorial/video entitled "Display EDR Content with Core Image, Metal, and SwiftUI" which describes how to blend Core Image with Metal and SwiftUI. It points to some new sample code entitled "Generating an Animation with a Core Image Render Destination" (here).
This sample project is very CoreImage-centric (which should suit your purposes nicely), but I wish Apple would post more sample-code examples showing Metal integrated with SwiftUI.
I have a small Core Image + SwiftUI sample project on Github that might be a good starting point for you. It doesn't cover a lot yet, but it demonstrates how to display filtered camera frames already.
Especially check out the draw function of the view. It's used to render a CIImage into the MTKView (you can do the same in your delegate's draw function).
Ok so this does the trick for me:
func draw(in view: MTKView) {
guard let drawable = view.currentDrawable else {
return
}
let colorSpace = CGColorSpaceCreateDeviceRGB()
let commandBuffer = metalCommandQueue.makeCommandBuffer()
let rpd = view.currentRenderPassDescriptor
rpd?.colorAttachments[0].clearColor = MTLClearColorMake(0, 1, 0, 1)
rpd?.colorAttachments[0].loadAction = .clear
rpd?.colorAttachments[0].storeAction = .store
let re = commandBuffer?.makeRenderCommandEncoder(descriptor: rpd!)
re?.endEncoding()
context.render((AppState.shared.rawImage ?? AppState.shared.rawImageOriginal)!,
to: drawable.texture,
commandBuffer: commandBuffer,
bounds: AppState.shared.rawImageOriginal!.extent,
colorSpace: colorSpace)
commandBuffer?.present(drawable)
commandBuffer?.commit()
}
AppState.shared.rawImage is my CIImage texture I got from my filtering function.
The context is made somewhere else but should be:
context = CIContext(mtlDevice: metalDevice)
Next up is adding the centering part of the code provided by Frank Schlegel.

How do I update a CALayer with a CVPixelBuffer/IOSurface?

I have an IOSurface-backed CVPixelBuffer that is getting updated from an outside source at 30fps. I want to render a preview of the image data in an NSView -- what's the best way for me to do that?
I can directly set the .contents of a CALayer on the view, but that only updates the first time my view updates (or if, say, I resize the view). I've been poring over the docs but I can't figure out the correct invocation of needsDisplay on the layer or view to let the view infrastructure know to refresh itself, especially when updates are coming from outside the view.
Ideally I'd just bind the IOSurface to my layer and any changes I make to it would be propagated, but I'm not sure if that's possible.
class VideoPreviewController: NSViewController, VideoFeedConsumer {
let customLayer : CALayer = CALayer()
override func viewDidLoad() {
super.viewDidLoad()
// Do view setup here.
print("Loaded our video preview")
view.layer?.addSublayer(customLayer)
customLayer.frame = view.frame
// register our view with the browser service
VideoFeedBrowser.instance.registerConsumer(self)
}
override func viewWillDisappear() {
// deregister our view from the video feed
VideoFeedBrowser.instance.deregisterConsumer(self)
super.viewWillDisappear()
}
// This callback gets called at 30fps whenever the pixelbuffer is updated
#objc func updateFrame(pixelBuffer: CVPixelBuffer) {
guard let surface = CVPixelBufferGetIOSurface(pixelBuffer)?.takeUnretainedValue() else {
print("pixelbuffer isn't IOsurface backed! noooooo!")
return;
}
// Try and tell the view to redraw itself with new contents?
// These methods don't work
//self.view.setNeedsDisplay(self.view.visibleRect)
//self.customLayer.setNeedsDisplay()
self.customLayer.contents = surface
}
}
Here's my attempt of a scaling version that's NSView rather than NSViewController-based, that also doesn't update correctly (or scale correctly for that matter):
class VideoPreviewThumbnail: NSView, VideoFeedConsumer {
required init?(coder decoder: NSCoder) {
super.init(coder: decoder)
self.wantsLayer = true
// register our view with the browser service
VideoFeedBrowser.instance.registerConsumer(self)
}
override init(frame frameRect: NSRect) {
super.init(frame: frameRect)
self.wantsLayer = true
// register our view with the browser service
VideoFeedBrowser.instance.registerConsumer(self)
}
deinit{
VideoFeedBrowser.instance.deregisterConsumer(self)
}
override func updateLayer() {
// Do I need to put something here?
print("update layer")
}
#objc
func updateFrame(pixelBuffer: CVPixelBuffer) {
guard let surface = CVPixelBufferGetIOSurface(pixelBuffer)?.takeUnretainedValue() else {
print("pixelbuffer isn't IOsurface backed! noooooo!")
return;
}
self.layer?.contents = surface
self.layer?.transform = CATransform3DMakeScale(
self.frame.width / CGFloat(CVPixelBufferGetWidth(pixelBuffer)),
self.frame.height / CGFloat(CVPixelBufferGetHeight(pixelBuffer)),
CGFloat(1))
}
}
What am I missing?
Maybe I'm wrong, but I think you are you updating your NSView on a background thread. (I suppose that the callback to updateFrame is on a background thread)
If I'm right, when you want to update the NSView, convert your pixelBuffer to whatever you want (NSImage?), and then dispatch it on the main thread.
Pseudocode (I don't work often with CVPixelBuffer so I'm not sure this is the right way to convert to an NSImage)
let ciImage = CIImage(cvImageBuffer: pixelBuffer)
let context = CIContext(options: nil)
let width = CVPixelBufferGetWidth(pixelBuffer)
let height = CVPixelBufferGetHeight(pixelBuffer)
let cgImage = context.createCGImage(ciImage, from: CGRect(x: 0, y: 0, width: width, height: height))
let nsImage = NSImage(cgImage: cgImage, size: CGSize(width: width, height: height))
DispatchQueue.main.async {
// assign the NSImage to your NSView here
}
Another catch: I did some tests, and it seems that you cannot assign an IOSurface directly to the contents of a CALayer.
I tried with this:
let textureImageWidth = 1024
let textureImageHeight = 1024
let macPixelFormatString = "ARGB"
var macPixelFormat: UInt32 = 0
for c in macPixelFormatString.utf8.reversed() {
macPixelFormat *= 256
macPixelFormat += UInt32(c)
}
let ioSurface = IOSurfaceCreate([kIOSurfaceWidth: textureImageWidth,
kIOSurfaceHeight: textureImageHeight,
kIOSurfaceBytesPerElement: 4,
kIOSurfaceBytesPerRow: textureImageWidth * 4,
kIOSurfaceAllocSize: textureImageWidth * textureImageHeight * 4,
kIOSurfacePixelFormat: macPixelFormat] as CFDictionary)!
IOSurfaceLock(ioSurface, IOSurfaceLockOptions.readOnly, nil)
let test = CIImage(ioSurface: ioSurface)
IOSurfaceUnlock(ioSurface, IOSurfaceLockOptions.readOnly, nil)
v1?.layer?.contents = ioSurface
Where v1 is my view. No effect
Even with a CIImage no effect (just last few lines)
IOSurfaceLock(ioSurface, IOSurfaceLockOptions.readOnly, nil)
let test = CIImage(ioSurface: ioSurface)
IOSurfaceUnlock(ioSurface, IOSurfaceLockOptions.readOnly, nil)
v1?.layer?.contents = test
If I create a CGImage it works
IOSurfaceLock(ioSurface, IOSurfaceLockOptions.readOnly, nil)
let test = CIImage(ioSurface: ioSurface)
IOSurfaceUnlock(ioSurface, IOSurfaceLockOptions.readOnly, nil)
let context = CIContext.init()
let img = context.createCGImage(test, from: test.extent)
v1?.layer?.contents = img
I encountered this problem myself and the solution is to double buffer the IOSurface source: use two IOSurface objects instead of one and render to the current surface, set the surface to the layer contents and then on the next rendering pass use the alternate (back/front) surface and then swap.
It would appear that setting the CALayer.contents twice to the same CVPixelBufferRef has no effect. However, if you alternate between two IOSurfaceRef it works wonderfully.
It maybe also possible to invalidate the layer contents by setting it to nil and then reset. I did not try that case but am using the double buffer technique.
If you have some IBActions that update it then create an observed variable with the didSet block and whenever the IBAction is triggered, change its value. Also remember to write the code you want to run when updated in that block.
I'd suggest making the variable an Int, set its default value to 0 and add 1 to it every time it updates.
And you can cast the NSView into an NSImageView for the part where you ask about showing the IMAGE data on an NSView so that does the job.
You need to convert the pixel buffer to CGImage and convert it to a layer so that you can change the layer of the main view.
Please try this code
#objc
func updateFrame(pixelBuffer: CVPixelBuffer) {
guard let surface = CVPixelBufferGetIOSurface(pixelBuffer)?.takeUnretainedValue() else {
print("pixelbuffer isn't IOsurface backed! noooooo!")
return;
}
void *baseAddr = CVPixelBufferGetBaseAddress(pixelBuffer);
size_t width = CVPixelBufferGetWidth(pixelBuffer);
size_t height = CVPixelBufferGetHeight(pixelBuffer);
CGColorSpaceRef colorSpace = CGColorSpaceCreateDeviceRGB();
CGContextRef cgContext = CGBitmapContextCreate(baseAddr, width, height, 8, CVPixelBufferGetBytesPerRow(pixelBuffer), colorSpace, kCGImageAlphaNoneSkipLast);
CGImageRef cgImage = CGBitmapContextCreateImage(cgContext);
CGContextRelease(cgContext);
let outputImage = UIImage(cgImage: outputCGImage, scale: 1, orientation: img.imageOrientation)
let newLayer:CGLayer = CGLayer.init(cgImage: outputImage)
self.layer = newLayer
CVPixelBufferUnlockBaseAddress(pixelBuffer, kCVPixelBufferLock_ReadOnly);
CVPixelBufferRelease(pixelBuffer);
}

Orienting a directional light and adding to scene in ARKit

I have found a few great examples how to add a directional light to my code, but not how to change the orientation as well as add it to my scene. How do I do this with my code? Here is my light class:
class Lighting: Entity, HasDirectionalLight {
required init() {
super.init()
self.light = DirectionalLightComponent(color: .white,
intensity: 100000,
isRealWorldProxy: true)
}
}
And here is the function that calls it:
func addTableToPlane(arView: ARView) {
let tableAnchor = AnchorEntity(plane: .horizontal)
let table = try! Entity.load(named: "Table_1500")
tableAnchor.addChild(table)
let dirLight = Lighting().light
let shadow = Lighting().shadow
tableAnchor.components.set(shadow!)
tableAnchor.components.set(dirLight)
}
I'm a pretty new to ARKit, so I haven't figured out how to edit the orientation of the directional light as I have it.
Another unsuccessful method that I tried was to create a lighting function, but I haven't been able to figure out how to add it to the scene:
func addLights(arView: ARView) {
// 1
let directionalLight = SCNLight()
directionalLight.type = .directional
directionalLight.intensity = 500
// 2
directionalLight.castsShadow = true
directionalLight.shadowMode = .deferred
// 3
directionalLight.shadowColor = UIColor(red: 0, green: 0, blue: 0, alpha: 0.5)
// 4
let directionalLightNode = SCNNode()
directionalLightNode.light = directionalLight
directionalLightNode.rotation = SCNVector4Make(1, 0, 0, -Float.pi / 3)
sceneView.scene.rootNode.addChildNode(directionalLightNode)
}
I then added addLights(arView: uiView) to the addTableToPlane function. I tried to add the light with:
arView.scene.rootNode.addChildNode(ambientLightNode)
but this gives the error that I don't have a childNode and so on. I guess that I'm spoiled with decent docs for Python that supply examples interspersed to help figure out problems, unlike the overly concise docs for Xcode, such as, what the heck I do with "Use the light’s look(at:from:upVector:relativeTo:) method to aim the light". Where do I put this? Where might I find answers to these simple questions?
Chasing my tail for the past couple days just to rotate a light is frustrating.
Use the following code to control orientation of directional light:
Take into consideration that position of Directional Light is not important!
import ARKit
import RealityKit
class Lighting: Entity, HasDirectionalLight, HasAnchoring {
required init() {
super.init()
self.light = DirectionalLightComponent(color: .green,
intensity: 1000,
isRealWorldProxy: true)
}
}
class ViewController: UIViewController {
#IBOutlet var arView: ARView!
override func viewWillAppear(animated: Bool) {
super.viewWillAppear(animated)
let light = Lighting()
light.orientation = simd_quatf(angle: .pi/8,
axis: [0, 1, 0])
let boxAnchor = try! Experience.loadBox()
let directLightAnchor = AnchorEntity()
directLightAnchor.addChild(light)
boxAnchor.addChild(directLightAnchor)
boxAnchor.steelBox!.scale = [30,30,30]
boxAnchor.steelBox!.position.z = -3
arView.scene.anchors.append(boxAnchor)
}
}
If you want to know how implement directional light's orientation in SceneKit, read this post.

Why is an iPhone XS getting worse CPU performance when using the camera live than an iPhone 6S Plus?

I'm using live camera output to update a CIImage on a MTKView. My main issue is that I have a large, negative performance difference where an older iPhone gets better CPU performance than a newer one, despite all their settings I've come across are the same.
This is a lengthy post, but I decided to include these details since they could be important to the cause of this problem. Please let me know what else I can include.
Below, I have my captureOutput function with two debug bools that I can turn on and off while running. I used this to try to determine the cause of my issue.
applyLiveFilter - bool whether or not to manipulate the CIImage with a CIFilter.
updateMetalView - bool whether or not to update the MTKView's CIImage.
// live output from camera
func captureOutput(_ output: AVCaptureOutput, didOutput sampleBuffer: CMSampleBuffer, from connection: AVCaptureConnection){
/*
Create CIImage from camera.
Here I save a few percent of CPU by using a function
to convert a sampleBuffer to a Metal texture, but
whether I use this or the commented out code
(without captureOutputMTLOptions) does not have
significant impact.
*/
guard let texture:MTLTexture = convertToMTLTexture(sampleBuffer: sampleBuffer) else{
return
}
var cameraImage:CIImage = CIImage(mtlTexture: texture, options: captureOutputMTLOptions)!
var transform: CGAffineTransform = .identity
transform = transform.scaledBy(x: 1, y: -1)
transform = transform.translatedBy(x: 0, y: -cameraImage.extent.height)
cameraImage = cameraImage.transformed(by: transform)
/*
// old non-Metal way of getting the ciimage from the cvPixelBuffer
guard let pixelBuffer = CMSampleBufferGetImageBuffer(sampleBuffer) else
{
return
}
var cameraImage:CIImage = CIImage(cvPixelBuffer: pixelBuffer)
*/
var orientation = UIImage.Orientation.right
if(isFrontCamera){
orientation = UIImage.Orientation.leftMirrored
}
// apply filter to camera image
if debug_applyLiveFilter {
cameraImage = self.applyFilterAndReturnImage(ciImage: cameraImage, orientation: orientation, currentCameraRes:currentCameraRes!)
}
DispatchQueue.main.async(){
if debug_updateMetalView {
self.MTLCaptureView!.image = cameraImage
}
}
}
Below is a chart of results between both phones toggling the different combinations of bools discussed above:
Even without the Metal view's CIIMage updating and no filters being applied, the iPhone XS's CPU is 2% greater than iPhone 6S Plus's, which isn't a significant overhead, but makes me suspect that somehow how the camera is capturing is different between the devices.
My AVCaptureSession's preset is set identically between both phones
(AVCaptureSession.Preset.hd1280x720)
The CIImage created from captureOutput is the same size (extent)
between both phones.
Are there any settings I need to set manually between these two phones AVCaptureDevice's settings, including activeFormat properties, to make them the same between devices?
The settings I have now are:
if let captureDevice = AVCaptureDevice.default(for:AVMediaType.video) {
do {
try captureDevice.lockForConfiguration()
captureDevice.isSubjectAreaChangeMonitoringEnabled = true
captureDevice.focusMode = AVCaptureDevice.FocusMode.continuousAutoFocus
captureDevice.exposureMode = AVCaptureDevice.ExposureMode.continuousAutoExposure
captureDevice.unlockForConfiguration()
} catch {
// Handle errors here
print("There was an error focusing the device's camera")
}
}
My MTKView is based off code written by Simon Gladman, with some edits for performance and to scale the render before it is scaled up to the width of the screen using Core Animation suggested by Apple.
class MetalImageView: MTKView
{
let colorSpace = CGColorSpaceCreateDeviceRGB()
var textureCache: CVMetalTextureCache?
var sourceTexture: MTLTexture!
lazy var commandQueue: MTLCommandQueue =
{
[unowned self] in
return self.device!.makeCommandQueue()
}()!
lazy var ciContext: CIContext =
{
[unowned self] in
return CIContext(mtlDevice: self.device!)
}()
override init(frame frameRect: CGRect, device: MTLDevice?)
{
super.init(frame: frameRect,
device: device ?? MTLCreateSystemDefaultDevice())
if super.device == nil
{
fatalError("Device doesn't support Metal")
}
CVMetalTextureCacheCreate(kCFAllocatorDefault, nil, self.device!, nil, &textureCache)
framebufferOnly = false
enableSetNeedsDisplay = true
isPaused = true
preferredFramesPerSecond = 30
}
required init(coder: NSCoder)
{
fatalError("init(coder:) has not been implemented")
}
// The image to display
var image: CIImage?
{
didSet
{
setNeedsDisplay()
}
}
override func draw(_ rect: CGRect)
{
guard var
image = image,
let targetTexture:MTLTexture = currentDrawable?.texture else
{
return
}
let commandBuffer = commandQueue.makeCommandBuffer()
let customDrawableSize:CGSize = drawableSize
let bounds = CGRect(origin: CGPoint.zero, size: customDrawableSize)
let originX = image.extent.origin.x
let originY = image.extent.origin.y
let scaleX = customDrawableSize.width / image.extent.width
let scaleY = customDrawableSize.height / image.extent.height
let scale = min(scaleX*IVScaleFactor, scaleY*IVScaleFactor)
image = image
.transformed(by: CGAffineTransform(translationX: -originX, y: -originY))
.transformed(by: CGAffineTransform(scaleX: scale, y: scale))
ciContext.render(image,
to: targetTexture,
commandBuffer: commandBuffer,
bounds: bounds,
colorSpace: colorSpace)
commandBuffer?.present(currentDrawable!)
commandBuffer?.commit()
}
}
My AVCaptureSession (captureSession) and AVCaptureVideoDataOutput (videoOutput) are setup below:
func setupCameraAndMic(){
let backCamera = AVCaptureDevice.default(for:AVMediaType.video)
var error: NSError?
var videoInput: AVCaptureDeviceInput!
do {
videoInput = try AVCaptureDeviceInput(device: backCamera!)
} catch let error1 as NSError {
error = error1
videoInput = nil
print(error!.localizedDescription)
}
if error == nil &&
captureSession!.canAddInput(videoInput) {
guard CVMetalTextureCacheCreate(kCFAllocatorDefault, nil, MetalDevice, nil, &textureCache) == kCVReturnSuccess else {
print("Error: could not create a texture cache")
return
}
captureSession!.addInput(videoInput)
setDeviceFrameRateForCurrentFilter(device:backCamera)
stillImageOutput = AVCapturePhotoOutput()
if captureSession!.canAddOutput(stillImageOutput!) {
captureSession!.addOutput(stillImageOutput!)
let q = DispatchQueue(label: "sample buffer delegate", qos: .default)
videoOutput.setSampleBufferDelegate(self, queue: q)
videoOutput.videoSettings = [
kCVPixelBufferPixelFormatTypeKey as AnyHashable as! String: NSNumber(value: kCVPixelFormatType_32BGRA),
kCVPixelBufferMetalCompatibilityKey as String: true
]
videoOutput.alwaysDiscardsLateVideoFrames = true
if captureSession!.canAddOutput(videoOutput){
captureSession!.addOutput(videoOutput)
}
captureSession!.startRunning()
}
}
setDefaultFocusAndExposure()
}
The video and mic are recorded on two separate streams. Details on the microphone and recording video have been left out since my focus is performance of live camera output.
UPDATE - I have a simplified test project on GitHub that makes it a lot easier to test the problem I'm having: https://github.com/PunchyBass/Live-Filter-test-project
From the top of my mind, you are not comparing pears with pears, even if you are running with the 2.49 GHz of A12 against 1.85 GHz of A9, the differences between the cameras are also huge, even if you use them with the same parameters there are several features from XS's camera that require more CPU resources (dual camera, stabilization, smart HDR, etc).
Sorry for the sources, I tried to find metrics of the CPU cost of those features, but I couldn't find it, unfortunately for your needs, that information is not relevant for marketing, when they are selling it as the best camera ever for an smartphone.
They are selling it as the best processor as well, we don't know what would happen using the XS camera with an A9 processor, it would probably crash, we will never know...
PS.... Your metrics are for the whole processor or for the used core? For the whole processor, you also need to consider other tasks that the devices can be executing, for the single core, is 21% of 200% against 39% of 600%

Can't display texture with sampler

I am trying to display a texture loaded with MTKTextureLoader, I have a buffer that stores my vertices coordinates (I build two triangles to have a rectangle in which to display my image), then I have a buffer that stores the texture coordinates of each vertex.
I made a sampler to sample data from my texture, the problem is that I am getting nothing (black image).
I putted the Swift code just in case my error comes from there, but I think it comes form the Metal code. If you look at my fragment shader, you will see two comments, they show something that I can't understand :
If I give the coordinates directly to the sample function, it works (colours the triangles with the color that corresponds to the given coordinates).
If I give the coordinates I pass to the sampler as color components, it also displays something coherent (triangles coloured in function of the given coordinates).
So it doesn't seem to come from the sampler, nor from the coordinates, that's what I don't understand.
Here is my Swift code :
import Cocoa
import MetalKit
import Metal
class ViewController: NSViewController, MTKViewDelegate {
var device:MTLDevice!
var texture:MTLTexture!
var commandQueue:MTLCommandQueue!
var vertexBuffer:MTLBuffer!
var vertexCoordinates:[Float] = [
-1, 1, 0, 1,
-1, -1, 0, 1,
1, -1, 0, 1,
1,-1,0,1,
1,1,0,1,
-1,1,0,1,
]
var vertexUVBuffer:MTLBuffer!
var vertexUVCoordinates:[Float] = [
0,1,
0,0,
1,0,
1,0,
1,1,
0,1
]
var library:MTLLibrary!
var defaultPipelineState:MTLRenderPipelineState!
var samplerState:MTLSamplerState!
#IBOutlet var metalView: MTKView!
override func viewDidLoad() {
super.viewDidLoad()
device = MTLCreateSystemDefaultDevice()
let textureLoader = MTKTextureLoader(device: device)
metalView.device = device
metalView.delegate = self
metalView.preferredFramesPerSecond = 0
metalView.sampleCount = 4
texture = try! textureLoader.newTextureWithContentsOfURL(NSBundle.mainBundle().URLForResource("abeilles", withExtension: "jpg")!, options: [MTKTextureLoaderOptionAllocateMipmaps:NSNumber(bool: true)])
commandQueue = device.newCommandQueue()
library = device.newDefaultLibrary()
vertexBuffer = device.newBufferWithBytes(&vertexCoordinates, length: sizeof(Float)*vertexCoordinates.count, options: [])
vertexUVBuffer = device.newBufferWithBytes(&vertexUVCoordinates, length: sizeof(Float)*vertexUVCoordinates.count, options: [])
let renderPipelineDescriptor = MTLRenderPipelineDescriptor()
renderPipelineDescriptor.vertexFunction = library.newFunctionWithName("passTroughVertex")
renderPipelineDescriptor.fragmentFunction = library.newFunctionWithName("myFragmentShader")
renderPipelineDescriptor.sampleCount = metalView.sampleCount
renderPipelineDescriptor.colorAttachments[0].pixelFormat = metalView.colorPixelFormat
defaultPipelineState = try! device.newRenderPipelineStateWithDescriptor(renderPipelineDescriptor)
let samplerDescriptor = MTLSamplerDescriptor()
samplerDescriptor.minFilter = .Linear
samplerDescriptor.magFilter = .Linear
samplerDescriptor.mipFilter = .Linear
samplerDescriptor.sAddressMode = .ClampToEdge
samplerDescriptor.rAddressMode = .ClampToEdge
samplerDescriptor.tAddressMode = .ClampToEdge
samplerDescriptor.normalizedCoordinates = true
samplerState = device.newSamplerStateWithDescriptor(samplerDescriptor)
metalView.draw()
// Do any additional setup after loading the view.
}
func drawInMTKView(view: MTKView) {
let commandBuffer = commandQueue.commandBuffer()
let commandEncoder = commandBuffer.renderCommandEncoderWithDescriptor(metalView.currentRenderPassDescriptor!)
commandEncoder.setRenderPipelineState(defaultPipelineState)
commandEncoder.setVertexBuffer(vertexBuffer, offset: 0, atIndex: 0)
commandEncoder.setVertexBuffer(vertexUVBuffer, offset:0, atIndex:1)
commandEncoder.setFragmentSamplerState(samplerState, atIndex: 0)
commandEncoder.setFragmentTexture(texture, atIndex: 0)
commandEncoder.drawPrimitives(MTLPrimitiveType.Triangle, vertexStart: 0, vertexCount: 6, instanceCount: 1)
commandEncoder.endEncoding()
commandBuffer.presentDrawable(metalView.currentDrawable!)
commandBuffer.commit()
}
func mtkView(view: MTKView, drawableSizeWillChange size: CGSize) {
// view.draw()
}
override var representedObject: AnyObject? {
didSet {
// Update the view, if already loaded.
}
}
}
Here is my Metal code :
#include <metal_stdlib>
using namespace metal;
struct VertexOut {
float4 position [[position]];
float2 texCoord;
};
vertex VertexOut passTroughVertex(uint vid [[ vertex_id]],
constant float4 *vertexPosition [[ buffer(0) ]],
constant float2 *vertexUVPos [[ buffer(1)]]) {
VertexOut vertexOut;
vertexOut.position = vertexPosition[vid];
vertexOut.texCoord = vertexUVPos[vid];
return vertexOut;
}
fragment float4 myFragmentShader(VertexOut inFrag [[stage_in]],
texture2d<float> myTexture [[ texture(0)]],
sampler mySampler [[ sampler(0) ]]) {
float4 myColor = myTexture.sample(mySampler,inFrag.texCoord);
// myColor = myTexture.sample(mySampler,float2(1));
// myColor = float4(inFrag.texCoord.r,inFrag.texCoord.g,0,1);
return myColor;
}
You're allocating space for mipmaps but not actually generating them. The docs say that when specifying MTKTextureLoaderOptionAllocateMipmaps, "a full set of mipmap levels are allocated for the texture when the texture is loaded, and it is your responsibility to generate the mipmap contents."
Your sampler configuration causes the resulting texture to be sampled at the base mipmap level as long as the texture is small relative to the rect on the screen, but if you feed in a larger texture, it starts sampling the smaller levels of the mipmap stack, picking up all-black pixels, which are then blended together to either darken the image or cause the output to be entirely black.
You should use the -generateMipmapsForTexture: method on a MTLBlitCommandEncoder to generate a complete set of mipmaps once your texture is loaded.