I have a swift based iPhone application (this tutorial https://www.kodeco.com/1163620-face-detection-tutorial-using-the-vision-framework-for-ios). It takes the camera feed and processes each frame using the vision API to find the landmarks on a face, and then draws and overlay on the video of the landmarks. All I was trying to do was take the position of a landmark and crop a rectangle around that position from the original image (after that I was going to run it through an ML model to determine some things). However, I have an issue translating the vision API landmark position back to the original image location to do the cropping. Below is hopefully the relevant portions of the code that show how I attempted to do this and failed (I pulled code from a number of functions/classes just to focus it on the problem).
Capture video frame
let imageBuffer = CMSampleBufferGetImageBuffer(sampleBuffer)
let ciimage = CIImage(cvImageBuffer: imageBuffer)
Find the Face Landmarks
let detectFaceRequest = VNDetectFaceLandmarksRequest(completionHandler: detectedFace)
sequenceHandler.perform([detectFaceRequest],on: imageBuffer,orientation: .leftMirrored)
Get the left Eye Pupil Location
let point = result.landmarks?.leftPupil?.pointsInImage(imageSize: ciimage.extent.size)
Draw the cropped image around the leftPupil
let cropped = ciimage?.cropped(to: CGRect(x: point.x-100, y: point.y-100, width: 200, height: 200))
let uicropped = UIImage(ciImage: cropped!)
uicropped.draw(at: CGPoint(x:100,y:100))
The issue is the cropped image is not positioned over the left pupil.
Related
My goal is to render an SCNScene off screen with a transparent background, as a PNG. Full reproducing project here.
It works, but when I enable jittering, the resulting render is semitransparent. In this example I pasted in the resulting PNG on top of an image with black squares, and you will notice that the black squares are in fact visible:
As you can see, the black boxes are visible through the 3D objects.
But, when I disable jittering, I get this:
As you can see, the black boxes are not visible.
I'm on Monterrey 12.1 (21C52). I'm testing the images in Preview and in Figma.
I'm using standard SDK features only. Here's what I do:
scene.background.contents = NSColor.clear
let snapshotRenderer = SCNRenderer(device: MTLCreateSystemDefaultDevice())
snapshotRenderer.pointOfView = sceneView.pointOfView
snapshotRenderer.scene = scene
snapshotRenderer.scene!.background.contents = NSColor.clear
snapshotRenderer.autoenablesDefaultLighting = true
// setting this to false does not make the image semi-transparent
snapshotRenderer.isJitteringEnabled = true
let size = CGSize(width: 1000, height: 1000)
let image = snapshotRenderer.snapshot(atTime: .zero, with: size, antialiasingMode: .multisampling16X)
let imageRep = NSBitmapImageRep(data: image.tiffRepresentation!)
let pngData = imageRep?.representation(using: .png, properties: [:])
try! pngData!.write(to: destination)
The docs for jittering says
Jittering is a process that SceneKit uses to improve the visual quality of a rendered scene. While the scene’s content is still, SceneKit moves the pointOfView location very slightly (by less than a pixel in projected screen space). It then composites images rendered after several such moves to create the final rendered scene, creating an antialiasing effect that smooths the edges of rendered geometry.
To me, that doesn't sound like something that is expected to produce semi-transparency?
I am currently trying to render a bounding boxes inside a UIView, however currently I'm facing the issue that there is a misalignment in the X axis when trying to render the box as can be seen in the screenshot below.
When the object is on the left of the view the misalignment will be on the right like seen in the image. However when the object is on the right the misalignment will be to the left. The misalignment increases the further it gets to the edge of the screen.
Currently are use ARKit to capture the current frame as a pixel buffer.
let pixelBuffer = sceneView.session.currentFrame?.capturedImage
// Capture current device orientation
let orientation = CGImagePropertyOrientation(rawValue: UIDevice.current.exifOrientation)
let handler = VNImageRequestHandler(cvPixelBuffer: pixelBuffer, orientation: orientation)
Additionally additionally my CoroML vision request looks as follows
findObjectRequest = VNCoreMLRequest(model: visionModel, completionHandler: visionRequestDidComplete)
findObjectRequest?.imageCropAndScaleOption = .scaleFit
I then try to reschedule the normalised bounding box to image Space like this:
public func scaleImageForCameraOutput(predictionRect finderrItem: FinderrItem, viewRect: CGRect) -> FinderrItem {
let scale = CGAffineTransform.identity.scaledBy(x: viewRect.width, y: viewRect.height)
let transform = CGAffineTransform(scaleX: 1, y: -1).translatedBy(x: 0, y: -1)
let bgRect = finderrItem.box.applying(transform).applying(scale)
finderrItem.box = bgRect
return finderrItem
}
I also tried to follow the Apple developer documentation and using the API code to re-scale the banding boxes as follows
let newBox = VNImageRectForNormalizedRect(
boundingBox,
Int(self.sceneView.bounds.width),
Int(self.sceneView.bounds.height))
However this still has the same issue with another issue that the y-axis is now inverted.
Does anyone know why I'm having this problem I've been stuck on it for quite awhile now and can't seem to figure it out.
Not sure if I am asking this question correctly, but I have two components; a CIImage and a UIBezierPath. Ideally, I want to create a CGRect that encapsulates my UIBezierPath; everything inside of the path would be white, everything outside of the path would be black. This way, I can then render this CGRect to some sort of an image, which I could then use as a mask for other purposes.
I am struggling to figure out how to do this with a focus on performance. My tests, as noted below, leverage using UIGraphicsImageRenderer which is far too slow for my needs (I will be doing this on sample buffers from a camera). Therefore, I would like to stick within CoreImage. This is my attempt;
// Path
let path = UIBezierPath()
// ... define the path's shape and close it
// My source image
let image = CIImage(cgImage: UIImage(named: "test.jpg")!.cgImage!)
// Renderer
let renderer = UIGraphicsImageRenderer(size: image.extent.size)
// Render path as mask
let img = renderer.image { ctx in
ctx.cgContext.setFillColor(UIColor.black.cgColor)
ctx.cgContext.fill(CGRect(x: 0, y: 0, width: image.extent.size.width, height: image.extent.size.height))
ctx.cgContext.setFillColor(UIColor.white.cgColor)
ctx.cgContext.addPath(path.cgPath)
ctx.cgContext.drawPath(using: .fill)
}
// Put a filter on the image
let imageFiltered = image.applyingFilter("CIPhotoEffectNoir")
// Blend with mask
let maskFilter = CIFilter.blendWithMask()
maskFilter.inputImage = imageFiltered
maskFilter.backgroundImage = image
maskFilter.maskImage = CIImage(cgImage: img.cgImage!)
// Output
if let output = maskFilter.outputImage {
... use CIContext() to render back to CVPixelBuffer for preview on MTKView.
}
Overall, the goal is to have a defined portion of an image (which will not conform to a traditional shape like a square or circle) which will be filtered with a CIFilter, then composited back over the original. If there is a better approach (such as somehow taking the original image, filtering it, cropping it to the path (leaving everything outside of the path transparent) and composing, that would likely be better performant.
To note, the above sample code results in a crash as the UIGraphicsImageRenderer cannot render the mask fast enough.
Your approach looks good so far. I assume the slow part is the generation of the mask image with Core Graphics. Unfortunately, there is no direct way to do the same with Core Image directly (on the GPU). However, you can try the following:
(Assuming from your previous question that the path always has a certain shape,) you can generate a mask image containing the path once for a certain reference size of your choice. Make sure that the path doesn't "touch" the border.
Then, when you want to use it as a mask, move and scale the shape image to the correct place using transformations and let its edges extend infinitely (to cover the whole underlying image; that's why the shape shouldn't touch the edges). Something like this:
let pathImage = CIImage(cgImage: img.cgImage!)
// scale path to the size of the area you want to mask
var mask = pathImage.transformed(by: CGAffineTransform(scaleX: scaleX, y: scaleY))
// move path to the place you want to cover
mask = mask.transformed(by: CGAffineTransform(translationX: offsetX, y: offsetY))
// let mask fill the rest of the area
mask = mask.clampedToExtent()
// use mask as maskImage...
You should be able to recycle the pathImage for every frame and thereby avoiding Core Graphics and CPU-GPU-synchronization.
If the title isn't clear, open Photos app on your iPhone and edit a picture. Try the rotation tool. The frame don't rotate, only the image does and zoom in to stay in the frame.
Here is a picture of this:
I would like to copy this comportment in my Swift project. It means rotating an image (it can be UIImage, CIImage or CGImage) but crop it keeping the same width/height ratio.
Where should I begin ? Thanks for your time.
There's an app a filter for that. CIStraightenFilter rotates an image by any amount while stretching it and cropping it to maintain the original dimensions:
let image: CIImage = /* my image 480x320 */
let rotated = image.applyingFilter("CIStraightenFilter",
withInputParameters: [ kCIInputAngleKey: Double.pi / 6 ]) // 30 degrees
rotated.extent // still 480x320
Here's some examples (rotations of zero, 15°, 30°, and 45°):
I'm trying to draw a path on a high resolution image, that's nothing complicated for an iPhone but if I add shadow to my path everything lags. It lags only when I work on images with a certain resolution (2000 x 3000) even less.
The Storyboard vies are:
-Scroll View
-Image View
-Draw View
So I have the DrawingView on top of the ImageView when I need to draw.
So the ImageView and the DrawView (view.bounds.size) have the same resolution as the image (e.g. 2000 x 3000) (there's the problem).
I'm drawing on a view with a high resolution.
I'm not directly calling drawRect: but only calling setNeedsDisplay() inside touchesBegan() and touchesMoved() after doing some operations (path.moveToPoint, path.addCurveToPoint, array operations) and adding points to my array.
In drawRect: I essentially have:
override func drawRect(rect: CGRect) {
print(self.bounds.size)
UIColor.greenColor().setStroke()
path.lineCapStyle = .Round
path.lineJoinStyle = .Round
path.lineWidth = 60.0
context = UIGraphicsGetCurrentContext()!
CGContextAddPath(context, path.CGPath)
CGContextSetShadowWithColor(context, CGSizeZero, 14.0, UIColor.whiteColor().CGColor) // <-- with this shadow it lags a lot.
path.stroke()
}
My path is a UIBezierPath().
Any ideas to improve the speed?
Update:
I followed what #brimstone said. I now have ImageView with a lower resolution, but have to apply my drawn path to the high resolution image.
(I'm trying to hand crop an image with the path that the user draws)
In this code I already got my closed path:
let layer = CAShapeLayer()
layer.path = path.CGPath
self.imageToEditView.layer.mask = layer
UIGraphicsBeginImageContext(self.imageEdited.size)
self.imageToEditView.layer.renderInContext(UIGraphicsGetCurrentContext()!)
let image = UIGraphicsGetImageFromCurrentImageContext()
UIGraphicsEndImageContext()
let croppedCGImage = CGImageCreateWithImageInRect(image.CGImage!, CGPathGetBoundingBox(path.CGPath))
let croppedImage = UIImage(CGImage: croppedCGImage!)
self.imageToEditView.image = croppedImage
self.imageToEditView.layer.mask = nil
imageToEditView.bounds.size = low resolution
imageEdited.size = high resolution
I need to set the hight resolution (I think) when i renderInContext. But how can I change the resolution of the imageView now?
Try downsizing it for the user to draw over (doesn't make a huge difference on small iPhone screens for user experience), then apply the edits to the high-res image.
To downsize images, either use UIImagePNGRepresentation, which may make your image sufficiently smaller, or (if you're still having memory issues), try using techniques in this tutorial and this answer to make it even smaller.
Then, you can take the content of what they've drawn and apply it to the high-res image.
Alternatively, look at high-res optimisation techniques by Apple: https://developer.apple.com/library/ios/documentation/2DDrawing/Conceptual/DrawingPrintingiOS/SupportingHiResScreensInViews/SupportingHiResScreensInViews.html