Memory leak when making CGImage from MTLTexture (Swift, macOS) - swift

Question

I have a Metal app and I'm trying to export frames to a quicktime movie. I am rendering frames in super hi-res and then scaling them down before writing, in order to antialias the scene.
To scale it, I'm taking the hi-res texture and converting it to a CGImage, then I resize the image and write out the smaller version. I have this extension I found online for converting an MTLTexture to a CGImage:
extension MTLTexture {
func bytes() -> UnsafeMutableRawPointer {
let width = self.width
let height = self.height
let rowBytes = self.width * 4
let p = malloc(width * height * 4)
self.getBytes(p!, bytesPerRow: rowBytes, from: MTLRegionMake2D(0, 0, width, height), mipmapLevel: 0)
return p!
}
func toImage() -> CGImage? {
let p = bytes()
let pColorSpace = CGColorSpaceCreateDeviceRGB()
let rawBitmapInfo = CGImageAlphaInfo.premultipliedFirst.rawValue | CGBitmapInfo.byteOrder32Little.rawValue // noneSkipFirst
let bitmapInfo:CGBitmapInfo = CGBitmapInfo(rawValue: rawBitmapInfo)
let size = self.width * self.height * 4
let rowBytes = self.width * 4
let releaseMaskImagePixelData: CGDataProviderReleaseDataCallback = { (info: UnsafeMutableRawPointer?, data: UnsafeRawPointer, size: Int) -> () in
// https://developer.apple.com/reference/coregraphics/cgdataproviderreleasedatacallback
// N.B. 'CGDataProviderRelease' is unavailable: Core Foundation objects are automatically memory managed
return
}
if let provider = CGDataProvider(dataInfo: nil, data: p, size: size, releaseData: releaseMaskImagePixelData) {
let cgImageRef = CGImage(width: self.width, height: self.height, bitsPerComponent: 8, bitsPerPixel: 32, bytesPerRow: rowBytes, space: pColorSpace, bitmapInfo: bitmapInfo, provider: provider, decode: nil, shouldInterpolate: true, intent: CGColorRenderingIntent.defaultIntent)!
p.deallocate() //this fixes the memory leak
return cgImageRef
}
p.deallocate() //this fixes the memory leak
return nil
}
} // end extension
I'm not positive, but it seems like something in this function is resulting in the memory leak -- with every frame it is holding on to the amount of memory in the giant texture / cgimage and not releasing it.
The CGDataProvider initialization takes that 'releaseData' callback argument, but I was under the impression that it was no longer needed.
I also have a resizing extention to CGImage -- this might also cause a leak, I don't know. However, I can comment out the resizing and writing of the frame, and the memory leak still builds up, so it seems to me that the conversion to CGImage is the main problem.
extension CGImage {
func resize(_ scale:Float) -> CGImage? {
let imageWidth = Float(width)
let imageHeight = Float(height)
let w = Int(imageWidth * scale)
let h = Int(imageHeight * scale)
guard let colorSpace = colorSpace else { return nil }
guard let context = CGContext(data: nil, width: w, height: h, bitsPerComponent: bitsPerComponent, bytesPerRow: Int(Float(bytesPerRow)*scale), space: colorSpace, bitmapInfo: alphaInfo.rawValue) else { return nil }
// draw image to context (resizing it)
context.interpolationQuality = .high
let r = CGRect(x: 0, y: 0, width: w, height: h)
context.clear(r)
context.draw(self, in:r)
// extract resulting image from context
return context.makeImage()
}
}
Finally, here is the big function that I call every frame when exporting. I'm sorry for the length but it is probably better to provide too much information than too little. So, basically at the start of rendering I allocate a giant MTL texture ('exportTextureBig'), the size of my normal screen multiplied by 'zoom_subvisions' in each direction. I render the scene in chunks, one for each spot on the grid, and assemble the large frame by using blitCommandEncoder.copy() to copy each small chunk onto the large texture. Once the entire frame is filled in, then I try to make a CGImage from it, scale it down to another CGImage, and write that out.
I'm calling commandBuffer.waitUntilCompleted() every frame while exporting -- hoping to avoid having the renderer hold on to textures that it is still using.
func exportFrame2(_ commandBuffer:MTLCommandBuffer, _ texture:MTLTexture) { // texture is the offscreen render target for the screen-size chunks
if zoom_index < zoom_subdivisions*zoom_subdivisions { // copy screen-size chunk to large texture
if let blitCommandEncoder = commandBuffer.makeBlitCommandEncoder() {
let dx = Int(BigRender.globals_L.displaySize.x) * (zoom_index%zoom_subdivisions)
let dy = Int(BigRender.globals_L.displaySize.y) * (zoom_index/zoom_subdivisions)
blitCommandEncoder.copy(from:texture,
sourceSlice: 0,
sourceLevel: 0,
sourceOrigin: MTLOrigin(x:0,y:0,z:0),
sourceSize: MTLSize(width:Int(BigRender.globals_L.displaySize.x),height:Int(BigRender.globals_L.displaySize.y), depth:1),
to:BigVideoWriter!.exportTextureBig!,
destinationSlice: 0,
destinationLevel: 0,
destinationOrigin: MTLOrigin(x:dx,y:dy,z:0))
blitCommandEncoder.synchronize(resource: BigVideoWriter!.exportTextureBig!)
blitCommandEncoder.endEncoding()
}
}
commandBuffer.commit()
commandBuffer.waitUntilCompleted() // do this instead
// is big frame complete?
if (zoom_index == zoom_subdivisions*zoom_subdivisions-1) {
// shrink the big texture here
if let cgImage = self.exportTextureBig!.toImage() { // memory leak here?
// this can be commented out and memory leak still happens
if let smallImage = cgImage.resize(1.0/Float(zoom_subdivisions)) {
writeFrame(nil, smallImage)
}
}
}
}
This all works, except for the huge memory leak. Is there something I can do to make it release the cgImage data each frame? Why is it holding onto it?
Thanks very much for any suggestions!

Answer 1

I think you've misunderstood the issue with CGDataProviderReleaseDataCallback and CGDataProviderRelease() being unavailable.
CGDataProviderRelease() is (in C) used to release the CGDataProvider object itself. But that's not the same thing as the byte buffer that you've provided to the CGDataProvider when you created it.
In Swift, the lifetime of the CGDataProvider object is managed for you, but that doesn't help deallocate the byte buffer.
Ideally, CGDataProvider would be able to automatically manage the lifetime of the byte buffer, but it can't. CGDataProvider doesn't know how to release that byte buffer because it doesn't know how it was allocated. That's why you have to provide a callback that it can use to release it. You are essentially providing the knowledge of how to release the byte buffer.
Since you're using malloc() to allocate the byte buffer, your callback needs to free() it.
That said, you'd be much better off using CFMutableData rather than UnsafeMutableRawPointer. Then, create the data provider using CGDataProvider(data:). In this case, all of the memory is managed for you.

Answer 2

I'm using very similar code, once I added code to deallocate P, the issue got solved:
func toImage() -> CGImage? {
let p = bytes()
let pColorSpace = CGColorSpaceCreateDeviceRGB()
let rawBitmapInfo = CGImageAlphaInfo.premultipliedFirst.rawValue | CGBitmapInfo.byteOrder32Little.rawValue // noneSkipFirst
let bitmapInfo:CGBitmapInfo = CGBitmapInfo(rawValue: rawBitmapInfo)
let size = self.width * self.height * 4
let rowBytes = self.width * 4
let releaseMaskImagePixelData: CGDataProviderReleaseDataCallback = { (info: UnsafeMutableRawPointer?, data: UnsafeRawPointer, size: Int) -> () in
// https://developer.apple.com/reference/coregraphics/cgdataproviderreleasedatacallback
// N.B. 'CGDataProviderRelease' is unavailable: Core Foundation objects are automatically memory managed
return
}
if let provider = CGDataProvider(dataInfo: nil, data: p, size: size, releaseData: releaseMaskImagePixelData) {
let cgImageRef = CGImage(width: self.width, height: self.height, bitsPerComponent: 8, bitsPerPixel: 32, bytesPerRow: rowBytes, space: pColorSpace, bitmapInfo: bitmapInfo, provider: provider, decode: nil, shouldInterpolate: true, intent: CGColorRenderingIntent.defaultIntent)!
p.deallocate() //this fixes the memory leak
return cgImageRef
}
p.deallocate() //this fixes the memory leak, but the data provider is no longer available (you just deallocated it's backing store)
return nil
}
anywhere you need to rapidly use CGImage
autoreleasepool {
let lastDrawableDisplayed = self.metalView?.currentDrawable?.texture
let cgImage = lastDrawableDisplayed?.toImage() // your code to convert drawable to CGImage
// do work with cgImage
}