How to merge several CGImage into a bigger CGImage? - swift

Question

I have created an app that applies a neural network (convnet) to an input image followed by a post-processing. This convnet is basically a filter that inputs an image (plus one parameter) and outputs an image of similar size. Since the convnet cannot process a large image in one pass due to memory issue, the image must be split in tiles which are then sticked (or unified) together after the model is applied. My problem is about image manipulation. It will be more clear after I present you what is done in details:
Input image is a UIImage
Split the input image into as a list of UIImage called listInput
Create an empty listOutput
For each tile in listInput:
Convert the UIImage into a CGImage
Convert the CGImage into a CVPixelBuffer
Apply a CoreML model to the CVPixelBuffer which returns a CVPixelBuffer of the same size
Convert the CVPixelBuffer into a CIImage
Convert the CIImage into a CGImage
Convert the CGImage into a UIImage
Append the UIImage into listOutput
Unify all the tiles in listOutput into an output UIImage
Fuse input and output UIImage (post-processing):
Convert input UIImage into CGImage then to CIImage
Convert output UIImage into CGImage then to CIImage
Fuse the 2 CIImage using a CIFilter
Convert the resulting CIImage into a CGImage
Convert the CGImage into a UIImage
I can post the code corresponding to any of the part listed above if needed.
The general problem I have is all the conversions between UIImage to CGImage to CIImage and conversly. I'm trying to get rid of UIImage completely (except for loading the image). I want indeed to manipulate CGImage from the start until the end. This will already simplify the code.
I've modified my code to manipulate list of CGImage instead of list of UIImage. The cropping part is in fact simpler with CGImage than with UIImage. But I cannot figure out the other way around: unify CGImage together into a bigger image. This is my specific problem. Bellow is the function I've created to unify the UIImage.
func unifyTiles(listTiles: [UIImage], listRect: [CGRect]) -> UIImage? {
guard let input = input else {
return nil
}
let outputSize = CGSize(width : Int(input.size.width), height: Int(input.size.height))
UIGraphicsBeginImageContextWithOptions(outputSize, true, 1.0)
for i in 0..<listTiles.count {
listTiles[i].draw(at: listRect[i].origin)
}
guard let output = UIGraphicsGetImageFromCurrentImageContext() else {
return nil
}
UIGraphicsEndImageContext()
return output
}
So my question is:
Is it possible to do the same just manipulating CGImage?
Is it even a good idea?
Some notes:
The post-procesing must be separated from the previous part because the user wants to modify the post-processing parameters without reapplying the convnet. The application of the convnet is indeed very long and can take up to a minute to compute on large images while the post-processing is near real-time.
In the post-processing part, it was suggested to me to convert directly UIImage <-> CIImage without going through CGImage. For some reason that I don't know, this doesn't work as far as I remember.
I'm aware that using Vision I could feed directly a CGImage into the network instead of a CVPixelBuffer, but I don't know if Vision can output a CGImage as well. This will be investigated soon hopefully.
Thanks for any information you could give me.

Answer 1

UIImage to CIImage
This step:
UIImage into CGImage then to CIImage
is overblown, as CIImage has an init(image:) initializer that goes directly from UIImage to CIImage.
Cropping
You seem to think that cropping a CGImage is easier than cropping a UIImage, but it isn't. To crop a UIImage, just draw it into a smaller graphics context, offset so as to place the desired point at the top left of the crop.
Graphics Contexts
You can only draw in a graphics context, and that's going to be an image context whether you like it or not. There's no such thing as drawing into a "CGImage context" if that's what you're thinking. You can draw a CGImage directly into an image context, but the results are usually disastrous (as I shall explain in the next paragraph).
Final thoughts
In general I would like to set your mind at rest about UIImage. A UIImage is a good thing. It is (usually) a very lightweight wrapper around a CGImage. The CGImage is the bitmap data; the wrapper adds information like scale and orientation. Losing that information can cause your drawing to come out very badly; trying to draw a CGImage can cause the drawing to be flipped and incorrectly scaled. Don't do it! Use UIImage and be happy.