I need to get pixel data as byte array from a CGImage that can be RGB8, RGB16, GRAYSCALE8 or GRAYSCALE16. Previous solutions such as this one produce a dark or distorted image.
Per the link provided in your question, you can get the pixelData by doing
extension UIImage {
func pixelData() -> [UInt8]? {
let size = self.size
let dataSize = size.width * size.height * 4
var pixelData = [UInt8](repeating: 0, count: Int(dataSize))
let colorSpace = CGColorSpaceCreateDeviceRGB()
let context = CGContext(data: &pixelData,
width: Int(size.width),
height: Int(size.height),
bitsPerComponent: 8,
bytesPerRow: 4 * Int(size.width),
space: colorSpace,
bitmapInfo: CGImageAlphaInfo.noneSkipLast.rawValue)
guard let cgImage = self.cgImage else { return nil }
context?.draw(cgImage, in: CGRect(x: 0, y: 0, width: size.width, height: size.height))
return pixelData
}
}
However, as a developer, the concerning objects here are the bitmapInfo and colorSpace. Your image may be getting distorted or colored differently depending on the information provided. The exact solution will be dependent upon how you obtained the image and what color schemes were provided from the image. You may just need to play with the variables.
I've never had an issue using CGColorSpaceCreateDeviceRGB() as my colorSpace but I have had to alter my bitmapInfo many times as my images were coming in as a different value.
Here is the location to reference the different types of bitmaps. More than likely though, you only need a variation of the CGImageAlphaInfo which can be located here.
If necessary, you can change the colorSpace. The default CGcolorSpace webpage can be found here. However, you could probably get away with one of the default ones located here
Related
I'm making a video with transparent background using HEVC with alpha, based on a bunch of NSImages, a CVPixelBuffer pool and AVAssetWriter. The NSImages are rendered by an SCNRenderer's [snapshot][1] method.
To turn the NSImages into CVPixelbuffers so I can append them to the pool, I first get a new pixel buffer from the pool, then use this code
CVPixelBufferLockBaseAddress(pixelBuffer, CVPixelBufferLockFlags(rawValue: 0))
let pixelData = CVPixelBufferGetBaseAddress(pixelBuffer)
let colorSpace = CGColorSpaceCreateDeviceRGB()
guard let context = CGContext(
data: pixelData,
width: Int(image.size.width),
height: Int(image.size.height),
bitsPerComponent: 8,
bytesPerRow: CVPixelBufferGetBytesPerRow(pixelBuffer),
space: colorSpace,
bitmapInfo: CGImageAlphaInfo.premultipliedFirst.rawValue
)
else { return }
let graphicsContext = NSGraphicsContext(cgContext: context, flipped: false)
NSGraphicsContext.saveGraphicsState()
NSGraphicsContext.current = graphicsContext
image.draw(in: CGRect(x: 0, y: 0, width: image.size.width, height: image.size.height))
NSGraphicsContext.restoreGraphicsState()
CVPixelBufferUnlockBaseAddress(pixelBuffer, CVPixelBufferLockFlags(rawValue: 0))
In the exported movie, the colors look good, and the transparency kicks in fine.
But all the frames are added on top of each other, as in the example below.
SIDE NOTE: The weird thing is that if I obtain the pixel buffer with Metal (using the SCNRenderer's render method), but leave everything else as is, it works fine. That's why I'm suspecting that it might be my CGContext that is acting up. Why don't I just use Metal, then? Would love to, but I can't get antialiasing.
I have a Metal app and I'm trying to export frames to a quicktime movie. I am rendering frames in super hi-res and then scaling them down before writing, in order to antialias the scene.
To scale it, I'm taking the hi-res texture and converting it to a CGImage, then I resize the image and write out the smaller version. I have this extension I found online for converting an MTLTexture to a CGImage:
extension MTLTexture {
func bytes() -> UnsafeMutableRawPointer {
let width = self.width
let height = self.height
let rowBytes = self.width * 4
let p = malloc(width * height * 4)
self.getBytes(p!, bytesPerRow: rowBytes, from: MTLRegionMake2D(0, 0, width, height), mipmapLevel: 0)
return p!
}
func toImage() -> CGImage? {
let p = bytes()
let pColorSpace = CGColorSpaceCreateDeviceRGB()
let rawBitmapInfo = CGImageAlphaInfo.premultipliedFirst.rawValue | CGBitmapInfo.byteOrder32Little.rawValue // noneSkipFirst
let bitmapInfo:CGBitmapInfo = CGBitmapInfo(rawValue: rawBitmapInfo)
let size = self.width * self.height * 4
let rowBytes = self.width * 4
let releaseMaskImagePixelData: CGDataProviderReleaseDataCallback = { (info: UnsafeMutableRawPointer?, data: UnsafeRawPointer, size: Int) -> () in
// https://developer.apple.com/reference/coregraphics/cgdataproviderreleasedatacallback
// N.B. 'CGDataProviderRelease' is unavailable: Core Foundation objects are automatically memory managed
return
}
if let provider = CGDataProvider(dataInfo: nil, data: p, size: size, releaseData: releaseMaskImagePixelData) {
let cgImageRef = CGImage(width: self.width, height: self.height, bitsPerComponent: 8, bitsPerPixel: 32, bytesPerRow: rowBytes, space: pColorSpace, bitmapInfo: bitmapInfo, provider: provider, decode: nil, shouldInterpolate: true, intent: CGColorRenderingIntent.defaultIntent)!
p.deallocate() //this fixes the memory leak
return cgImageRef
}
p.deallocate() //this fixes the memory leak
return nil
}
} // end extension
I'm not positive, but it seems like something in this function is resulting in the memory leak -- with every frame it is holding on to the amount of memory in the giant texture / cgimage and not releasing it.
The CGDataProvider initialization takes that 'releaseData' callback argument, but I was under the impression that it was no longer needed.
I also have a resizing extention to CGImage -- this might also cause a leak, I don't know. However, I can comment out the resizing and writing of the frame, and the memory leak still builds up, so it seems to me that the conversion to CGImage is the main problem.
extension CGImage {
func resize(_ scale:Float) -> CGImage? {
let imageWidth = Float(width)
let imageHeight = Float(height)
let w = Int(imageWidth * scale)
let h = Int(imageHeight * scale)
guard let colorSpace = colorSpace else { return nil }
guard let context = CGContext(data: nil, width: w, height: h, bitsPerComponent: bitsPerComponent, bytesPerRow: Int(Float(bytesPerRow)*scale), space: colorSpace, bitmapInfo: alphaInfo.rawValue) else { return nil }
// draw image to context (resizing it)
context.interpolationQuality = .high
let r = CGRect(x: 0, y: 0, width: w, height: h)
context.clear(r)
context.draw(self, in:r)
// extract resulting image from context
return context.makeImage()
}
}
Finally, here is the big function that I call every frame when exporting. I'm sorry for the length but it is probably better to provide too much information than too little. So, basically at the start of rendering I allocate a giant MTL texture ('exportTextureBig'), the size of my normal screen multiplied by 'zoom_subvisions' in each direction. I render the scene in chunks, one for each spot on the grid, and assemble the large frame by using blitCommandEncoder.copy() to copy each small chunk onto the large texture. Once the entire frame is filled in, then I try to make a CGImage from it, scale it down to another CGImage, and write that out.
I'm calling commandBuffer.waitUntilCompleted() every frame while exporting -- hoping to avoid having the renderer hold on to textures that it is still using.
func exportFrame2(_ commandBuffer:MTLCommandBuffer, _ texture:MTLTexture) { // texture is the offscreen render target for the screen-size chunks
if zoom_index < zoom_subdivisions*zoom_subdivisions { // copy screen-size chunk to large texture
if let blitCommandEncoder = commandBuffer.makeBlitCommandEncoder() {
let dx = Int(BigRender.globals_L.displaySize.x) * (zoom_index%zoom_subdivisions)
let dy = Int(BigRender.globals_L.displaySize.y) * (zoom_index/zoom_subdivisions)
blitCommandEncoder.copy(from:texture,
sourceSlice: 0,
sourceLevel: 0,
sourceOrigin: MTLOrigin(x:0,y:0,z:0),
sourceSize: MTLSize(width:Int(BigRender.globals_L.displaySize.x),height:Int(BigRender.globals_L.displaySize.y), depth:1),
to:BigVideoWriter!.exportTextureBig!,
destinationSlice: 0,
destinationLevel: 0,
destinationOrigin: MTLOrigin(x:dx,y:dy,z:0))
blitCommandEncoder.synchronize(resource: BigVideoWriter!.exportTextureBig!)
blitCommandEncoder.endEncoding()
}
}
commandBuffer.commit()
commandBuffer.waitUntilCompleted() // do this instead
// is big frame complete?
if (zoom_index == zoom_subdivisions*zoom_subdivisions-1) {
// shrink the big texture here
if let cgImage = self.exportTextureBig!.toImage() { // memory leak here?
// this can be commented out and memory leak still happens
if let smallImage = cgImage.resize(1.0/Float(zoom_subdivisions)) {
writeFrame(nil, smallImage)
}
}
}
}
This all works, except for the huge memory leak. Is there something I can do to make it release the cgImage data each frame? Why is it holding onto it?
Thanks very much for any suggestions!
I think you've misunderstood the issue with CGDataProviderReleaseDataCallback and CGDataProviderRelease() being unavailable.
CGDataProviderRelease() is (in C) used to release the CGDataProvider object itself. But that's not the same thing as the byte buffer that you've provided to the CGDataProvider when you created it.
In Swift, the lifetime of the CGDataProvider object is managed for you, but that doesn't help deallocate the byte buffer.
Ideally, CGDataProvider would be able to automatically manage the lifetime of the byte buffer, but it can't. CGDataProvider doesn't know how to release that byte buffer because it doesn't know how it was allocated. That's why you have to provide a callback that it can use to release it. You are essentially providing the knowledge of how to release the byte buffer.
Since you're using malloc() to allocate the byte buffer, your callback needs to free() it.
That said, you'd be much better off using CFMutableData rather than UnsafeMutableRawPointer. Then, create the data provider using CGDataProvider(data:). In this case, all of the memory is managed for you.
I'm using very similar code, once I added code to deallocate P, the issue got solved:
func toImage() -> CGImage? {
let p = bytes()
let pColorSpace = CGColorSpaceCreateDeviceRGB()
let rawBitmapInfo = CGImageAlphaInfo.premultipliedFirst.rawValue | CGBitmapInfo.byteOrder32Little.rawValue // noneSkipFirst
let bitmapInfo:CGBitmapInfo = CGBitmapInfo(rawValue: rawBitmapInfo)
let size = self.width * self.height * 4
let rowBytes = self.width * 4
let releaseMaskImagePixelData: CGDataProviderReleaseDataCallback = { (info: UnsafeMutableRawPointer?, data: UnsafeRawPointer, size: Int) -> () in
// https://developer.apple.com/reference/coregraphics/cgdataproviderreleasedatacallback
// N.B. 'CGDataProviderRelease' is unavailable: Core Foundation objects are automatically memory managed
return
}
if let provider = CGDataProvider(dataInfo: nil, data: p, size: size, releaseData: releaseMaskImagePixelData) {
let cgImageRef = CGImage(width: self.width, height: self.height, bitsPerComponent: 8, bitsPerPixel: 32, bytesPerRow: rowBytes, space: pColorSpace, bitmapInfo: bitmapInfo, provider: provider, decode: nil, shouldInterpolate: true, intent: CGColorRenderingIntent.defaultIntent)!
p.deallocate() //this fixes the memory leak
return cgImageRef
}
p.deallocate() //this fixes the memory leak, but the data provider is no longer available (you just deallocated it's backing store)
return nil
}
anywhere you need to rapidly use CGImage
autoreleasepool {
let lastDrawableDisplayed = self.metalView?.currentDrawable?.texture
let cgImage = lastDrawableDisplayed?.toImage() // your code to convert drawable to CGImage
// do work with cgImage
}
I have a gray scale image of depth data that has been upsampled from its original resolution. I'm stuck as to how I can convert the pixel values of the upscaled depth image (r,g,b) to a float.
Is there a way to convert the level of whiteness of a pixel a float value?
Is there anyway I could convert the CVPixelBufferFormatTypes of the CVPixelBuffer associated with the image?
Said another way, is there a way to convert the pixel buffer of a gray scale image into CVpixelbuffer containing disparity floats?
I use the following code to extract the cvpixelbuffer from a cgimage representation of the upsampled depth data:-
func pixelBuffer() -> CVPixelBuffer? {
let frameSize = CGSize(width: self.width, height: self.height)
//COLOR IS BGRA
var pixelBuffer:CVPixelBuffer? = nil
let status = CVPixelBufferCreate(kCFAllocatorDefault, Int(frameSize.width), Int(frameSize.height), kCVPixelFormatType_32BGRA , nil, &pixelBuffer)
if status != kCVReturnSuccess {
return nil
}
CVPixelBufferLockBaseAddress(pixelBuffer!, CVPixelBufferLockFlags.init(rawValue: 0))
let data = CVPixelBufferGetBaseAddress(pixelBuffer!)
let rgbColorSpace = CGColorSpaceCreateDeviceRGB()
let bitmapInfo = CGBitmapInfo(rawValue: CGBitmapInfo.byteOrder32Big.rawValue | CGImageAlphaInfo.premultipliedFirst.rawValue)
let context = CGContext(data: data, width: Int(frameSize.width), height: Int(frameSize.height), bitsPerComponent: 8, bytesPerRow: CVPixelBufferGetBytesPerRow(pixelBuffer!), space: rgbColorSpace, bitmapInfo: bitmapInfo.rawValue)
context?.draw(self, in: CGRect(x: 0, y: 0, width: self.width, height: self.height))
CVPixelBufferUnlockBaseAddress(pixelBuffer!, CVPixelBufferLockFlags(rawValue: 0))
return pixelBuffer
}
In Objective-C I was able to use CGBitmapContextCreate to create an empty context. I am trying to to the same in Swift 3, but for some reason it is nil. What am I missing?
let inImage: UIImage = ...
let width = Int(inImage.size.width)
let height = Int(inImage.size.height)
let bitmapBytesPerRow = width * 4
let bitmapByteCount = bitmapBytesPerRow * height
let pixelData = UnsafeMutablePointer<UInt8>.allocate(capacity: bitmapByteCount)
let context = CGContext(data: pixelData,
width: width,
height: height,
bitsPerComponent: 8,
bytesPerRow: bitmapBytesPerRow,
space: CGColorSpaceCreateDeviceRGB(),
bitmapInfo: CGImageAlphaInfo.alphaOnly.rawValue)
I'm not sure what the code in the liked article would do, but two things are different with your Swift code.
bytesPerRow: width // width * 4 (== bitmapBytesPerRow)
space : NULL // CGColorSpaceCreateDeviceRGB()
The documentation of CGBitmapContextCreate does not say anything about supplying NULL for colorspace, but the header doc says The number of components for each pixel is specified by space, so, at least, CGColorSpaceCreateDeviceRGB() is not appropriate for alphaOnly (which should have only 1 component per pixel).
As far as I tested, this code returns non-nil CGContext:
let bitmapBytesPerRow = width //<-
let bitmapByteCount = bitmapBytesPerRow * height
let pixelData = UnsafeMutablePointer<UInt8>.allocate(capacity: bitmapByteCount)
let context = CGContext(data: pixelData,
width: width,
height: height,
bitsPerComponent: 8,
bytesPerRow: bitmapBytesPerRow,
space: CGColorSpaceCreateDeviceGray(), //<-
bitmapInfo: CGImageAlphaInfo.alphaOnly.rawValue)
But, not sure if this works for your purpose or not.
I was working on this thing and faced same issue.
The solution I found is to use
var colorSpace = CGColorSpace.init(name: CGColorSpace.sRGB)!
let context = CGContext(data: nil,
width: Int(outputSize.width),
height: Int(outputSize.height),
bitsPerComponent: self.bitsPerComponent,
bytesPerRow: bitmapBytesPerRow,
space: colorSpace,
bitmapInfo: CGImageAlphaInfo.premultipliedLast.rawValue)
Actually my image's color space was Indexed Which can't be used to make a context.
So instead of using image's own colorSpace I made my own by using
var colorSpace = CGColorSpace.init(name: CGColorSpace.sRGB)!
and passed it to the context.
it resolved my error (nil context issue).
I would like to get the RGB (actually, the image provided has been grey-scaled, so grey-scale information would be sufficient) values of the individual pixels of a CIImage.
I currently have the following code:
//
// Conversion of CIImage to CGImage
//
func convertCIImageToCGImage(inputImage: CIImage) -> CGImage! {
return CIContext(options: nil).createCGImage(inputImage, fromRect: inputImage.extent)
}
process(image: CGImage) {
let img=convertCIImageToCGImage(image)
// Set image width, height
width = CGImageGetWidth(img)
height = CGImageGetHeight(img)
let colorSpace = CGColorSpaceCreateDeviceRGB()
let bitmapInfo = CGBitmapInfo(rawValue: CGImageAlphaInfo.PremultipliedFirst.rawValue)
// Create the bitmap context (ARGB)
context = CGBitmapContextCreate(nil, width, height, 8, bitmapBytesPerRow, colorSpace, bitmapInfo.rawValue)!
// draw the image onto the context
let rect = CGRect(x: 0, y: 0, width: width, height: height)
CGContextDrawImage(context, rect, img)
let uncasted_data = CGBitmapContextGetData(context)
data = UnsafePointer<UInt8>(uncasted_data)
...
}
I then access the individual Pixels using the data variable. I realise that most of the time of this function is spent in
CGContextDrawImage(context, rect, img)
Is there another way of accessing the bitmap of a CIImage or CGImage?