How to force SKTextureAtlas created from a dictionary to not modify textures size? - swift

In my project, textures are procedurally generated from method provided by PaintCode (paint-code).
I then create a SKTextureAtlas from a dictionary filed with UIImage generated by these methods :
myAtlas = SKTextureAtlas(dictionary: myTextures)
At last, textures are retrieve from atlas using textureNamed:
var sprite1 = SKSpriteNode(texture:myAtlas.textureNamed("texture1"))
But displayed nodes are double sized on iPhone4S simulator. And triple sized on iPhone 6 Plus simulator.
It seems that at init, atlas compute images at the device resolution.
But generated images already have the correct size and do not need to be changed. See Drawing Method below.
Here is the description of the generated image:
<UIImage: 0x7f86cae56cd0>, {52, 52}
And the description of the corresponding texture in atlas:
<SKTexture> 'image1' (156 x 156)
This for iPhone 6 Plus, using #3x images, that's why size is x3.
And for iPhone 4S, using #2x images, as expected:
<UIImage: 0x7d55dde0>, {52, 52}
<SKTexture> 'image1' (156 x 156)
At last, the scaleproperty for generated UIImage is set to the right device resolution: 2.0 for #2x (iPhone 4S) and 3.0 for #3x (iPhone 6 Plus).
The Question
So what can I do to avoid atlas resizing the pictures?
Drawing method
PaintCode generate drawing methods as the following:
public class func imageOfCell(#frame: CGRect) -> UIImage {
UIGraphicsBeginImageContextWithOptions(frame.size, false, 0)
StyleKit.drawCell(frame: frame)
let imageOfCell = UIGraphicsGetImageFromCurrentImageContext()!
UIGraphicsEndImageContext()
return imageOfCell
}
Update 1
Comparing two approaches to generate SKTextureAtlas
// Some test image
let testImage:UIImage...
// Atlas creation
var myTextures = [String:UIImage]()
myTextures["texture1"] = testImage
myAtlas = SKTextureAtlas(dictionary: myTextures)
// Create two textures from the same image
let texture1 = myAtlas.textureNamed("texture1")
let texture2 = SKTexture(image:testImage)
// Wrong display : node is oversized
var sprite1 = SKSpriteNode(texture:texture1)
// Correct display
var sprite2 = SKSpriteNode(texture:texture2)
It seems that the problem lie on SKTextureAtlas from a dictionary as as SKSpriteNode initialization does not use scale property from UIImage to correctly size the node.
Here are descriptions on console:
- texture1: '' (84 x 84)
- texture2: 'texture1' (84 x 84)
texture2 miss some data! That could explain the lack of scale information to properly size the node as:
node's size = texture's size divide by texture's scale.
Update 2
The problem occur when the scale property of UIImage is different than one.
So you can use the following method to generate picture:
func imageOfCell(frame: CGRect, color:SKColor) -> UIImage {
UIGraphicsBeginImageContextWithOptions(frame.size, false, 0)
var bezierPath = UIBezierPath(rect: frame)
color.setFill()
bezierPath.fill()
let imageOfCell = UIGraphicsGetImageFromCurrentImageContext()!
UIGraphicsEndImageContext()
return imageOfCell
}

The problem come from the use of SKTextureAtlas(dictionary:) to initialize atlas.
SKTexture created using this method does not embed data related to image's scale property. So during the creation of SKSpriteNode by init(texture:) the lack of scale information in texture leads to choose texture's size in place of image's size.
One way to correct it is to provide node's size during SKSpriteNode creation: init(texture:size:)

From the documentation for the scale parameter for UIGraphicsBeginImageContextWithOptions,
The scale factor to apply to the bitmap. If you specify a value of
0.0, the scale factor is set to the scale factor of the device’s main screen.
Therefore, if you want the textures to be the same "size" across all devices, set this value to 1.0.
EDIT:
override func didMoveToView(view: SKView) {
let image = imageOfCell(CGRectMake(0, 0, 10, 10),scale:0)
let dict:[String:UIImage] = ["t1":image]
let texture = SKTextureAtlas(dictionary: dict)
let sprite1 = SKSpriteNode(texture: texture.textureNamed("t1"))
sprite1.position = CGPointMake (CGRectGetMidX(view.frame),CGRectGetMidY(view.frame))
addChild(sprite1)
println(sprite1.size)
// prints (30.0, 30.0) if scale = 0
// prints (10,0, 10,0) if scale = 1
}
func imageOfCell(frame: CGRect, scale:CGFloat) -> UIImage {
UIGraphicsBeginImageContextWithOptions(frame.size, false, scale)
var bezierPath = UIBezierPath(rect: frame)
UIColor.whiteColor().setFill()
bezierPath.fill()
let imageOfCell = UIGraphicsGetImageFromCurrentImageContext()!
UIGraphicsEndImageContext()
return imageOfCell
}

Related

Merge CAShapeLayer into CVPixelBuffer

I'm capturing the output of a playing video using AVPlayerItemVideoOutput.copyPixelBuffer
I'm able to convert the pixel buffer into a CIImage, then render it back into a pixel buffer again, and then an AVAssetWriter writes the buffer stream out to a new movie clip successfully.
The reason I'm converting to CIImage is I want to do some manipulation of each frame. (So far I don't understand how to manipulate pixel buffers directly).
In this case I want to overlay a "scribble" style drawing that the user does with their finger. While the video plays, they can draw over it. I'm capturing this drawing successfully into a CAShapeLayer.
The code below outputs just the overlay CAShapeLayer successfully. When I try to reincorporate the original frame by uncommenting the lines shown, the entire process bogs down drastically and drops from 60fps to an unstable 10fps or so on an iPhone 12. I get stable 60fps in all cases except when I uncomment that code.
What's the best way to incorporate the shape layer into this stream of pixel buffers in 60fps "real time"?
Note: some of this code is not finalized -- setting bounds correctly, etc. However this is not related to my question and I'm aware that has to be done. The rotation/translation are there to orient the shape layer -- this all works for now.
func addShapesToBuffer(buffer: CVPixelBuffer, shapeLayer: CAShapeLayer) -> CVPixelBuffer? {
let coreImage = CoreImage.CIImage.init(cvImageBuffer: buffer)
let newBuffer = getBuffer(from: coreImage)
CVPixelBufferLockBaseAddress(newBuffer!, [])
let rect = CGRect(origin: CGPoint.zero, size: CGSize(width: 800, height: 390))
shapeLayer.shouldRasterize = true
shapeLayer.rasterizationScale = UIScreen.main.scale
shapeLayer.backgroundColor = UIColor.clear.cgColor
let renderer = UIGraphicsImageRenderer(size: rect.size)
let uiImageDrawing = renderer.image {
context in
// let videoImage = UIImage(ciImage: coreImage)
// videoImage.draw(in: rect)
let cgContext = context.cgContext
cgContext.rotate(by: deg2rad(-90))
cgContext.translateBy(x: -390, y: 0)
return shapeLayer.render(in: cgContext)
}
let ciContext = CIContext()
let newImage = CIImage(cgImage: uiImageDrawing.cgImage!)
ciContext.render(_: newImage, to: newBuffer!)
CVPixelBufferUnlockBaseAddress(newBuffer!, [])
return newBuffer
}

Scaled up MTKView shows gaps when joining CIImages

I'm using a MTKView written by Simon Gladman that "exposes an image property type of 'CIImage' to simplify Metal based rendering of Core Image filters." It has been slightly altered for performance. I left out an additional scaling operation since it has nothing to do with the issue here.
Problem: When creating a composite of smaller CIImages into a larger one, they are aligned pixel perfect. MTKView's image property is set to this CIImage composite. However, there is a scale done to this image so it fits the entire MTKView which makes gaps between the joined images visible. This is done by dividing the drawableSize width/height by the CIImage's extent width/height.
This makes me wonder if something needs to be done the CIImage side to actually join those pixels. Saving that CIImage to the camera roll shows no separation between the joined images. It's only visible when the MTKView scales up. In addition, whatever needs to be done needs to have virtually no impact on performance since these image renders are being done in real time through the camera's output. (The MTKView is a preview of the effect being done)
Here is the MTKView that I'm using to render with:
class MetalImageView: MTKView
{
let colorSpace = CGColorSpaceCreateDeviceRGB()
var textureCache: CVMetalTextureCache?
var sourceTexture: MTLTexture!
lazy var commandQueue: MTLCommandQueue =
{
[unowned self] in
return self.device!.makeCommandQueue()
}()!
lazy var ciContext: CIContext =
{
[unowned self] in
//cacheIntermediates
return CIContext(mtlDevice: self.device!, options:[.cacheIntermediates:false])
//return CIContext(mtlDevice: self.device!)
}()
override init(frame frameRect: CGRect, device: MTLDevice?)
{
super.init(frame: frameRect,
device: device ?? MTLCreateSystemDefaultDevice())
if super.device == nil
{
fatalError("Device doesn't support Metal")
}
CVMetalTextureCacheCreate(kCFAllocatorDefault, nil, self.device!, nil, &textureCache)
framebufferOnly = false
enableSetNeedsDisplay = true
isPaused = true
preferredFramesPerSecond = 30
}
required init(coder: NSCoder)
{
fatalError("init(coder:) has not been implemented")
}
/// The image to display
var image: CIImage?
{
didSet
{
//renderImage()
//draw()
setNeedsDisplay()
}
}
override func draw(_ rect: CGRect)
{
guard let
image = image,
let targetTexture = currentDrawable?.texture else
{
return
}
let commandBuffer = commandQueue.makeCommandBuffer()
let bounds = CGRect(origin: CGPoint.zero, size: drawableSize)
let originX = image.extent.origin.x
let originY = image.extent.origin.y
let scaleX = drawableSize.width / image.extent.width
let scaleY = drawableSize.height / image.extent.height
let scale = min(scaleX, scaleY)
let scaledImage = image
.transformed(by: CGAffineTransform(translationX: -originX, y: -originY))
.transformed(by: CGAffineTransform(scaleX: scale, y: scale))
ciContext.render(scaledImage,
to: targetTexture,
commandBuffer: commandBuffer,
bounds: bounds,
colorSpace: colorSpace)
commandBuffer?.present(currentDrawable!)
commandBuffer?.commit()
}
}
When compositing the images, I have a full size camera image as the background just as a foundation for what the size should be, then I duplicate half of that halfway across the width or height of the image using the CISourceAtopCompositing CIFilter and translate it using a CGAffineTransform. I also give it a negative scale to add a mirror effect:
var scaledImageTransform = CGAffineTransform.identity
scaledImageTransform = scaledImageTransform.translatedBy(x:0, y:sourceCore.extent.height)
scaledImageTransform = scaledImageTransform.scaledBy(x:1.0, y:-1.0)
alphaMaskBlend2 = alphaMaskBlend2?.applyingFilter("CISourceAtopCompositing",
parameters: [kCIInputImageKey: alphaMaskBlend2!,
kCIInputBackgroundImageKey: sourceCore])
alphaMaskBlend2 = alphaMaskBlend2?.applyingFilter("CISourceAtopCompositing",
parameters: [kCIInputImageKey: (alphaMaskBlend2?.cropped(to: cropRect).transformed(by: scaledImageTransform))!,
kCIInputBackgroundImageKey: alphaMaskBlend2!])
sourceCore is the original image that came through the camera. alphaMaskBlend2 is the final CIImage that I assign the MTKView to. The cropRect correctly crops the mirrored part of the image. In the scaled up MTKView there is a visible gap between these two joined CIImages. What can be done to make this image display as continuous pixels no matter how scaled the MTKView is just like any other image does?

I'm having some trouble using x and y coordinates from touchesBegan as the center key in a CI filter

I'm trying to setup having the users tap a location in an image view and the X,Y of the tap becomes the center point (kCIInputCenterKey) of the current image filter in use.
These are my global variables:
var x: CGFloat = 0
var y: CGFloat = 0
var imgChecker = 0
This is my touchesBegan function that checks if the user is touching inside the image view or not, if not then sets the filter center key to the center of the image view:
override func touchesBegan(_ touches: Set<UITouch>, with event: UIEvent?) {
if let touch = touches.first {
let position = touch.location(in: self.imageView)
if (touch.view == imageView){
print("touchesBegan | This is an ImageView")
x = position.x * 4
y = position.y * 4
imgChecker = 1
}else{
print("touchesBegan | This is not an ImageView")
x = 0
y = 0
imgChecker = 0
}
print("x: \(x)")
print("y: \(y)")
}
}
As you can see I have the checker there to make the filter center appear in the middle of the image if inside the image view was not tapped. I'm also printing out the coordinates tapped to xCode's console and they appear without issue.
This is the part where i apply my filter:
currentFilter = CIFilter(name: "CIBumpDistortion")
currentFilter.setValue(200, forKey: kCIInputRadiusKey)
currentFilter.setValue(1, forKey: kCIInputScaleKey)
if imgChecker == 1 {
self.currentFilter.setValue(CIVector(x: self.x, y: self.y), forKey: kCIInputCenterKey)
}else{
self.currentFilter.setValue(CIVector(x: currentImage.size.width / 2, y: currentImage.size.height / 2), forKey: kCIInputCenterKey)
}
x = 0
y = 0
let beginImage = CIImage(image: currentImage)
currentFilter.setValue(beginImage, forKey: kCIInputImageKey)
let cgimg = context.createCGImage(currentFilter.outputImage!, from: currentFilter.outputImage!.extent)
currentImage = UIImage(cgImage: cgimg!)
self.imageView.image = currentImage
This is the CGRect I'm using, ignore the "frame" in there, its just a image view in front of the first one that allows me to save a "frame" over the current filtered image:
func drawImagesAndText() {
let renderer = UIGraphicsImageRenderer(size: CGSize(width: imageView.bounds.size.width, height: imageView.bounds.size.height))
img = renderer.image { ctx in
let bgImage = currentImage
bgImage?.draw(in: CGRect(x: 0, y: 0, width: imageView.bounds.size.width, height: imageView.bounds.size.height))
frames = UIImage(named: framesAr)
frames?.draw(in: CGRect(x: 0, y: 0, width: imageView.bounds.size.width, height: imageView.bounds.size.height))
}
}
When I do set the x,y by tapping inside the image view, the center of the filter in the image view keeps appearing in the lower left hand side of it regardless of where I tapped inside. If i keep tapping around the image view, the center does seem to move around a bit, but its no where near where I'm actually tapping.
any insight would be greatly appreciated, thank you.
Keep two things in mind.
First (and I think you probably know this), the CI origin (0,0) is lower left, not top left.
Second (and I think this is the issue) UIKit (meaning UIImage and potentially CGPoint coordinates) are not the same as CIVector coordinates. You need to take the UIKit touchesBegan coordinate and turn it into the CIImage.extent coordinate.
EDIT:
All coordinates that follow are X then Y, and Width then Height.
After posting my comment I thought I'd give an example of what I mean by scaling. Let's say you have a UIImageView sized at 250x250, using a content mode of AspectFit, displaying an image whose size is 1000x500.
Now, let's say the touchesBegan is CGPoint(200,100). (NOTE: If your UIImageView is part of a larger superview, it could be something more like 250,400 - I'm working on the point within the UIImageView.)
Scaling down the image size (remember, AspectFit) means the image is actually centered vertically (landscape appearing) within the UIImageView at CGRect(0, 62.5, 250, 125). So first off, good! The touch point not only began within the image view, it also began wishing the image. (You'll probably want to consider the not-so-edge case of touches beginning outside of the image.)
Dividing by 4 gives you the scaled down image view coordinates, and as you'd expect, multiplying up will give you the needed vector coordinates. So a touchesBegan CGPoint(200,100) turns into a CIVector(800,400).
I have some code written - not much in the way of comments, done in Swift 2 (I think) and very poorly written - that is part of a subclass (probably should have been an extension) of UIImageView that computes all this. Using the UIImageView's bounds and it's image's size is what you need. Keep in mind - images in AspectFit can also be scaled up!
One last note on CIImage - extent. Many times it's a UIImage's size. But many masks and generated output may have an infinite eatent.
SECOND EDIT:
I made a stupid mistake in my scaling example. Remember, the CIImage Origin is bottom left, not upper left. So in my example a CGPoint(200,100), scaled to CGPoint(800,400) would be CGVector(800,100).
THIRD EDIT:
Apologies for the multiple/running edits, but it seems important. (Besides, only the last was due my stupidity! Worthwhile, to note, but still.)
Now we're talking "near real time" updating using a Core Image filter. I'm planning to eventually have some blog posts on this, but the real source you want is Simon Gladman (he's moved on, look back to his posts in 2015-16), and his eBook Core Image for Swift (uses Swift 2 but most is automatically upgraded to Swift 3). Just giving credit where it is due.
If you want "near real time" usage of Core Image, you need to use the GPU. UIView, and all it's subclasses (meaning UIKit) uses the CPU. That's okay, using the GPU means using a Core Graphics, and specifically using a GLKView. It's the CG equivalent of a UIImage.
Here's my subclass of it:
open class GLKViewDFD: GLKView {
var renderContext: CIContext
var myClearColor:UIColor!
var rgb:(Int?,Int?,Int?)!
open var image: CIImage! {
didSet {
setNeedsDisplay()
}
}
public var clearColor: UIColor! {
didSet {
myClearColor = clearColor
}
}
public init() {
let eaglContext = EAGLContext(api: .openGLES2)
renderContext = CIContext(eaglContext: eaglContext!)
super.init(frame: CGRect.zero)
context = eaglContext!
}
override public init(frame: CGRect, context: EAGLContext) {
renderContext = CIContext(eaglContext: context)
super.init(frame: frame, context: context)
enableSetNeedsDisplay = true
}
public required init?(coder aDecoder: NSCoder) {
let eaglContext = EAGLContext(api: .openGLES2)
renderContext = CIContext(eaglContext: eaglContext!)
super.init(coder: aDecoder)
context = eaglContext!
enableSetNeedsDisplay = true
}
override open func draw(_ rect: CGRect) {
if let image = image {
let imageSize = image.extent.size
var drawFrame = CGRect(x: 0, y: 0, width: CGFloat(drawableWidth), height: CGFloat(drawableHeight))
let imageAR = imageSize.width / imageSize.height
let viewAR = drawFrame.width / drawFrame.height
if imageAR > viewAR {
drawFrame.origin.y += (drawFrame.height - drawFrame.width / imageAR) / 2.0
drawFrame.size.height = drawFrame.width / imageAR
} else {
drawFrame.origin.x += (drawFrame.width - drawFrame.height * imageAR) / 2.0
drawFrame.size.width = drawFrame.height * imageAR
}
rgb = (0,0,0)
rgb = myClearColor.rgb()
glClearColor(Float(rgb.0!)/256.0, Float(rgb.1!)/256.0, Float(rgb.2!)/256.0, 0.0);
glClear(0x00004000)
// set the blend mode to "source over" so that CI will use that
glEnable(0x0BE2);
glBlendFunc(1, 0x0303);
renderContext.draw(image, in: drawFrame, from: image.extent)
}
}
}
A few notes.
I absolutely need to credit Objc.io for much of this. This is also a great resource for Swift and UIKit coding.
I wanted AspectFit content mode with the potential to change the "backgroundColor" of the GLKView, which is why I subclassed and and called if clearColor.
Between the two resources I linked to, you should have what you need to have a good performing, near real time use of Core Image, using the GPU. One reason my afore-mentioned code to use scaling after getting the output of a filter was never updated? It didn't need it.
Lots here to process, I know. But I've found this side of things (Core Image effects) to be the most fun side (and pretty cool too) of iOS.

Rounding corners of images in an array xcode

I'm attempting to round the corners of images that are stored within an array, but i'm not entirely sure if its possible?
var holeImages = [UIImage(named:"1.png"),UIImage(named:"2.png"),UIImage(named:"3.png")]
self.holeImages1.layer.cornerRadius = 10.0f;
No, it's not possible. If you wish apply some method of object for all objects in array, you should do this in cycle, since Array type most likely don't have this method:
for image in holeImages {
image.performSomeMethod()
}
Also you can write Array extension to teach Array of objects (UIImage for example) for this method:
extension Array where Element: UIImage {
func performSomeMethod() {
for element in self {
element.performSomeMethod()
}
}
}
and then you can do
holeImages.performSomeMethod()
But let's return to your case. UIImage type don't have property called layer; moreover, idea rounding corners of image looks strange without context. Usually you need to round corners when you present image on screen, and you usually use UIImageView container for this. So, you probably better round corners of this container instead images:
let imageView = UIImageView()
imageView.layer.cornerRadius = 10
imageView.contentMode = .ScaleAspectFill
imageView.clipsToBounds = true
imageView.image = holeImages.first
You probably want to just apply a corner radius to the layer of the UIImageView in which you present the images, rather than rounding the images themselves, e.g.
imageView.layer.cornerRadius = 10
But if you really want to round the images, themselves, rather than rounding the UIImageView in which you present the images, you could also build a new array of rounded images from your holeImages array:
let roundedHoleImages = holeImages.map { return $0?.rounded(cornerRadius: 10) }
Where you could round the images with something like:
extension UIImage {
/// Round the corners of an image
///
/// - parameter cornerRadius: The `CGFloat` corner radius to apply to the images.
///
/// - returns: The rounded image.
func rounded(cornerRadius cornerRadius: CGFloat) -> UIImage? {
let path = UIBezierPath(roundedRect: CGRect(x: 0, y: 0, width: size.width, height: size.height), cornerRadius: cornerRadius)
UIGraphicsBeginImageContextWithOptions(size, false, scale)
let context = UIGraphicsGetCurrentContext()
CGContextAddPath(context, path.CGPath)
CGContextClip(context)
drawAtPoint(CGPointZero)
let outputImage = UIGraphicsGetImageFromCurrentImageContext()
UIGraphicsEndImageContext()
return outputImage
}
}
You might use this rounding of the actual images if you were, for example, uploading the images to some web service and you wanted to upload images with rounded corners. But, if not, rounding the image views is not only easier, but avoids problems resulting from images of different scales (especially if those scales are different from the display scale) as well as minimizing the memory impact of creating a separate array of images.

SKCropNode Strange Behaviour

When using SKCropNode, I wanted the image I add to the cropNode to adjust each individual pixel alpha value in accordance to the corresponding mask pixel alpha value.
After a lot of research, I came to the conclusion that the image pixel alpha values were not going to adjust to the mask, however after just continuing with my project, I notice that one specific cropNode image's pixels were in fact fading to the mask pixel alpha value??? Which was great! However after reproducing this, I don't know why it is doing it?
import SpriteKit
var textureArray: [SKTexture] = []
var display: SKSpriteNode!
class GameScene: SKScene {
override func didMoveToView(view: SKView) {
anchorPoint = CGPointMake(0.5, 0.5)
backgroundColor = UIColor.greenColor()
fetchTexures()
display = SKSpriteNode()
let image = SKSpriteNode(texture: textureArray[0])
display.addChild(image)
let randomCropNode = SKCropNode()
display.addChild(randomCropNode)
let cropNode = SKCropNode()
cropNode.maskNode = display
let fill = SKSpriteNode(color: UIColor.whiteColor(), size: frame.size)
cropNode.addChild(fill)
cropNode.zPosition = 10
addChild(cropNode)
}
func fetchTexures() {
var x: Int = 0
while x < 1 {
let texture: SKTexture = SKTextureAtlas(named: "texture").textureNamed("\(x)")
textureArray.append(texture)
x += 1
}
}
}
The above code gives me my desired effect, however if you remove the below, the image pixel alpha values no longer adjust in accordance with the mask?? The below code is not actually using in my project, but it's the only way I can make the pixel alpha value's adjust.
let randomCropNode = SKCropNode()
display.addChild(randomCropNode)
Can anybody see what is causing this behaviour, or if there a better way of getting my desired effect?
Mask:
Result:
If remove:
let randomCropNode = SKCropNode()
display.addChild(randomCropNode)
Result:
Crop node will only turn on and off pixels if the alpha varies between <.5 (off) and >=.5(on)
However to apply a fade, if your alpha mask is just black(with various alpha levels) and transparent, you apply the mask as a regular texture to your crop node, and you let alpha blending take care of the fade effect.
As for your issues with the code, are you sure your crop node is cropping, and not just rendering the texture? I do not know what the texture looks like to try and reproduce this.
The node supplied to the crop node must not be a child of another
node; however, it may have children of its own.
When the crop node’s contents are rendered, the crop node first draws
its mask into a private buffer. Then, it renders its children. When
rendering its children, each pixel is verified against the
corresponding pixel in the mask. If the pixel in the mask has an alpha
value of less than 0.05, the image pixel is masked out. Any pixel not
rendered by the mask node is automatically masked out.
https://developer.apple.com/library/ios/documentation/SpriteKit/Reference/SKCropNode_Ref/#//apple_ref/occ/instp/SKCropNode/maskNode