I'm making a video with transparent background using HEVC with alpha, based on a bunch of NSImages, a CVPixelBuffer pool and AVAssetWriter. The NSImages are rendered by an SCNRenderer's [snapshot][1] method.
To turn the NSImages into CVPixelbuffers so I can append them to the pool, I first get a new pixel buffer from the pool, then use this code
CVPixelBufferLockBaseAddress(pixelBuffer, CVPixelBufferLockFlags(rawValue: 0))
let pixelData = CVPixelBufferGetBaseAddress(pixelBuffer)
let colorSpace = CGColorSpaceCreateDeviceRGB()
guard let context = CGContext(
data: pixelData,
width: Int(image.size.width),
height: Int(image.size.height),
bitsPerComponent: 8,
bytesPerRow: CVPixelBufferGetBytesPerRow(pixelBuffer),
space: colorSpace,
bitmapInfo: CGImageAlphaInfo.premultipliedFirst.rawValue
)
else { return }
let graphicsContext = NSGraphicsContext(cgContext: context, flipped: false)
NSGraphicsContext.saveGraphicsState()
NSGraphicsContext.current = graphicsContext
image.draw(in: CGRect(x: 0, y: 0, width: image.size.width, height: image.size.height))
NSGraphicsContext.restoreGraphicsState()
CVPixelBufferUnlockBaseAddress(pixelBuffer, CVPixelBufferLockFlags(rawValue: 0))
In the exported movie, the colors look good, and the transparency kicks in fine.
But all the frames are added on top of each other, as in the example below.
SIDE NOTE: The weird thing is that if I obtain the pixel buffer with Metal (using the SCNRenderer's render method), but leave everything else as is, it works fine. That's why I'm suspecting that it might be my CGContext that is acting up. Why don't I just use Metal, then? Would love to, but I can't get antialiasing.
Related
I need to get pixel data as byte array from a CGImage that can be RGB8, RGB16, GRAYSCALE8 or GRAYSCALE16. Previous solutions such as this one produce a dark or distorted image.
Per the link provided in your question, you can get the pixelData by doing
extension UIImage {
func pixelData() -> [UInt8]? {
let size = self.size
let dataSize = size.width * size.height * 4
var pixelData = [UInt8](repeating: 0, count: Int(dataSize))
let colorSpace = CGColorSpaceCreateDeviceRGB()
let context = CGContext(data: &pixelData,
width: Int(size.width),
height: Int(size.height),
bitsPerComponent: 8,
bytesPerRow: 4 * Int(size.width),
space: colorSpace,
bitmapInfo: CGImageAlphaInfo.noneSkipLast.rawValue)
guard let cgImage = self.cgImage else { return nil }
context?.draw(cgImage, in: CGRect(x: 0, y: 0, width: size.width, height: size.height))
return pixelData
}
}
However, as a developer, the concerning objects here are the bitmapInfo and colorSpace. Your image may be getting distorted or colored differently depending on the information provided. The exact solution will be dependent upon how you obtained the image and what color schemes were provided from the image. You may just need to play with the variables.
I've never had an issue using CGColorSpaceCreateDeviceRGB() as my colorSpace but I have had to alter my bitmapInfo many times as my images were coming in as a different value.
Here is the location to reference the different types of bitmaps. More than likely though, you only need a variation of the CGImageAlphaInfo which can be located here.
If necessary, you can change the colorSpace. The default CGcolorSpace webpage can be found here. However, you could probably get away with one of the default ones located here
I have a UIImage extension that can change the color of it's image that I pulled off somewhere. The problem is that it downgrades it's resolution after it colors the image. I've seen other answers based on this, but I'm not sure how to adapt this to rendering a retina image in this instance:
extension UIImage {
func maskWithColor(color: UIColor) -> UIImage? {
let maskImage = cgImage!
let width = size.width
let height = size.height
let bounds = CGRect(x: 0, y: 0, width: width, height: height)
let colorSpace = CGColorSpaceCreateDeviceRGB()
let bitmapInfo = CGBitmapInfo(rawValue: CGImageAlphaInfo.premultipliedLast.rawValue)
let context = CGContext(data: nil, width: Int(width), height: Int(height), bitsPerComponent: 8, bytesPerRow: 0, space: colorSpace, bitmapInfo: bitmapInfo.rawValue)!
context.clip(to: bounds, mask: maskImage)
context.setFillColor(color.cgColor)
context.fill(bounds)
if let cgImage = context.makeImage() {
let coloredImage = UIImage(cgImage: cgImage)
return coloredImage
} else {
return nil
}
}
}
I've seen people using UIGraphicsBeginImageContextWithOptions and setting it's scale to the main screen, but I don't think it works if I'm using the CGContext function.
I think you want:
let width = size.width * scale
let height = size.height * scale
and:
let coloredImage = UIImage(cgImage: cgImage, scale:scale, orientation:.up)
(You may need to use imageOrientation instead of .up.)
I have a gray scale image of depth data that has been upsampled from its original resolution. I'm stuck as to how I can convert the pixel values of the upscaled depth image (r,g,b) to a float.
Is there a way to convert the level of whiteness of a pixel a float value?
Is there anyway I could convert the CVPixelBufferFormatTypes of the CVPixelBuffer associated with the image?
Said another way, is there a way to convert the pixel buffer of a gray scale image into CVpixelbuffer containing disparity floats?
I use the following code to extract the cvpixelbuffer from a cgimage representation of the upsampled depth data:-
func pixelBuffer() -> CVPixelBuffer? {
let frameSize = CGSize(width: self.width, height: self.height)
//COLOR IS BGRA
var pixelBuffer:CVPixelBuffer? = nil
let status = CVPixelBufferCreate(kCFAllocatorDefault, Int(frameSize.width), Int(frameSize.height), kCVPixelFormatType_32BGRA , nil, &pixelBuffer)
if status != kCVReturnSuccess {
return nil
}
CVPixelBufferLockBaseAddress(pixelBuffer!, CVPixelBufferLockFlags.init(rawValue: 0))
let data = CVPixelBufferGetBaseAddress(pixelBuffer!)
let rgbColorSpace = CGColorSpaceCreateDeviceRGB()
let bitmapInfo = CGBitmapInfo(rawValue: CGBitmapInfo.byteOrder32Big.rawValue | CGImageAlphaInfo.premultipliedFirst.rawValue)
let context = CGContext(data: data, width: Int(frameSize.width), height: Int(frameSize.height), bitsPerComponent: 8, bytesPerRow: CVPixelBufferGetBytesPerRow(pixelBuffer!), space: rgbColorSpace, bitmapInfo: bitmapInfo.rawValue)
context?.draw(self, in: CGRect(x: 0, y: 0, width: self.width, height: self.height))
CVPixelBufferUnlockBaseAddress(pixelBuffer!, CVPixelBufferLockFlags(rawValue: 0))
return pixelBuffer
}
In Objective-C I was able to use CGBitmapContextCreate to create an empty context. I am trying to to the same in Swift 3, but for some reason it is nil. What am I missing?
let inImage: UIImage = ...
let width = Int(inImage.size.width)
let height = Int(inImage.size.height)
let bitmapBytesPerRow = width * 4
let bitmapByteCount = bitmapBytesPerRow * height
let pixelData = UnsafeMutablePointer<UInt8>.allocate(capacity: bitmapByteCount)
let context = CGContext(data: pixelData,
width: width,
height: height,
bitsPerComponent: 8,
bytesPerRow: bitmapBytesPerRow,
space: CGColorSpaceCreateDeviceRGB(),
bitmapInfo: CGImageAlphaInfo.alphaOnly.rawValue)
I'm not sure what the code in the liked article would do, but two things are different with your Swift code.
bytesPerRow: width // width * 4 (== bitmapBytesPerRow)
space : NULL // CGColorSpaceCreateDeviceRGB()
The documentation of CGBitmapContextCreate does not say anything about supplying NULL for colorspace, but the header doc says The number of components for each pixel is specified by space, so, at least, CGColorSpaceCreateDeviceRGB() is not appropriate for alphaOnly (which should have only 1 component per pixel).
As far as I tested, this code returns non-nil CGContext:
let bitmapBytesPerRow = width //<-
let bitmapByteCount = bitmapBytesPerRow * height
let pixelData = UnsafeMutablePointer<UInt8>.allocate(capacity: bitmapByteCount)
let context = CGContext(data: pixelData,
width: width,
height: height,
bitsPerComponent: 8,
bytesPerRow: bitmapBytesPerRow,
space: CGColorSpaceCreateDeviceGray(), //<-
bitmapInfo: CGImageAlphaInfo.alphaOnly.rawValue)
But, not sure if this works for your purpose or not.
I was working on this thing and faced same issue.
The solution I found is to use
var colorSpace = CGColorSpace.init(name: CGColorSpace.sRGB)!
let context = CGContext(data: nil,
width: Int(outputSize.width),
height: Int(outputSize.height),
bitsPerComponent: self.bitsPerComponent,
bytesPerRow: bitmapBytesPerRow,
space: colorSpace,
bitmapInfo: CGImageAlphaInfo.premultipliedLast.rawValue)
Actually my image's color space was Indexed Which can't be used to make a context.
So instead of using image's own colorSpace I made my own by using
var colorSpace = CGColorSpace.init(name: CGColorSpace.sRGB)!
and passed it to the context.
it resolved my error (nil context issue).
I would like to get the RGB (actually, the image provided has been grey-scaled, so grey-scale information would be sufficient) values of the individual pixels of a CIImage.
I currently have the following code:
//
// Conversion of CIImage to CGImage
//
func convertCIImageToCGImage(inputImage: CIImage) -> CGImage! {
return CIContext(options: nil).createCGImage(inputImage, fromRect: inputImage.extent)
}
process(image: CGImage) {
let img=convertCIImageToCGImage(image)
// Set image width, height
width = CGImageGetWidth(img)
height = CGImageGetHeight(img)
let colorSpace = CGColorSpaceCreateDeviceRGB()
let bitmapInfo = CGBitmapInfo(rawValue: CGImageAlphaInfo.PremultipliedFirst.rawValue)
// Create the bitmap context (ARGB)
context = CGBitmapContextCreate(nil, width, height, 8, bitmapBytesPerRow, colorSpace, bitmapInfo.rawValue)!
// draw the image onto the context
let rect = CGRect(x: 0, y: 0, width: width, height: height)
CGContextDrawImage(context, rect, img)
let uncasted_data = CGBitmapContextGetData(context)
data = UnsafePointer<UInt8>(uncasted_data)
...
}
I then access the individual Pixels using the data variable. I realise that most of the time of this function is spent in
CGContextDrawImage(context, rect, img)
Is there another way of accessing the bitmap of a CIImage or CGImage?