I am trying to get the average RGB value for my "AVCaptureVideoDataOutput" feed. I found the following solution on StackOverflow:
let pixelBuffer = CMSampleBufferGetImageBuffer(sampleBuffer)
let cameraImage = CIImage(CVPixelBuffer: pixelBuffer!)
let filter = CIFilter(name: "CIAreaAverage")
filter!.setValue(cameraImage, forKey: kCIInputImageKey)
let outputImage = filter!.valueForKey(kCIOutputImageKey) as! CIImage!
let ctx = CIContext(options:nil)
let cgImage = ctx.createCGImage(outputImage, fromRect:outputImage.extent)
let rawData:NSData = CGDataProviderCopyData(CGImageGetDataProvider(cgImage))!
let pixels = UnsafePointer<UInt8>(rawData.bytes)
let bytes = UnsafeBufferPointer<UInt8>(start:pixels, count:rawData.length)
var BGRA_index = 0
for pixel in UnsafeBufferPointer(start: bytes.baseAddress, count: bytes.count) {
switch BGRA_index {
case 0:
bluemean = CGFloat (pixel)
case 1:
greenmean = CGFloat (pixel)
case 2:
redmean = CGFloat (pixel)
case 3:
break
default:
break
}
BGRA_index++
}
But this produces the average as an Int but I need it in a Float format with the precision kept. The rounding is quite problematic in the problem domain I'm working with. Is there a way to a Float average efficiently?
Thanks a lot!
May I recommend using our library CoreImageExtensions for reading the value? We added methods for reading pixel values from CIImages in different formats. For your case it would look like this:
import CoreImageExtensions
let pixelBuffer = CMSampleBufferGetImageBuffer(sampleBuffer)
let cameraImage = CIImage(cvPixelBuffer: pixelBuffer!)
let filter = CIFilter(name: "CIAreaAverage")!
filter.setValue(cameraImage, forKey: kCIInputImageKey)
filter.setValue(CIVector(cgRect: cameraImage.extent), forKey: kCIInputExtentKey)
let outputImage = filter.outputImage!
let context = CIContext()
// get the value of a specific pixel as a `SIMD4<Float32>`
let average = context.readFloat32PixelValue(from: outputImage, at: CGPoint.zero)
Also keep in mind, if you want to compute the average regularly (not just once), to only create a single instance of CIContext and reuse it for every camera frame. Creating it is expensive and it actually increases performance to use the same instance since it caches internal resources.
Related
What I Have:
Referencing Apple's Chroma Key Code, it states that we can create a Chroma Key Filter Cube via
func chromaKeyFilter(fromHue: CGFloat, toHue: CGFloat) -> CIFilter?
{
// 1
let size = 64
var cubeRGB = [Float]()
// 2
for z in 0 ..< size {
let blue = CGFloat(z) / CGFloat(size-1)
for y in 0 ..< size {
let green = CGFloat(y) / CGFloat(size-1)
for x in 0 ..< size {
let red = CGFloat(x) / CGFloat(size-1)
// 3
let hue = getHue(red: red, green: green, blue: blue)
let alpha: CGFloat = (hue >= fromHue && hue <= toHue) ? 0: 1
// 4
cubeRGB.append(Float(red * alpha))
cubeRGB.append(Float(green * alpha))
cubeRGB.append(Float(blue * alpha))
cubeRGB.append(Float(alpha))
}
}
}
let data = Data(buffer: UnsafeBufferPointer(start: &cubeRGB, count: cubeRGB.count))
// 5
let colorCubeFilter = CIFilter(name: "CIColorCube", withInputParameters: ["inputCubeDimension": size, "inputCubeData": data])
return colorCubeFilter
}
I then created a function to be able to insert any image into this filter and return the filtered image.
public func filteredImage(ciimage: CIImage) -> CIImage? {
let filter = chromaKeyFilter(fromHue: 110/360, toHue: 130/360)! //green screen effect colors
filter.setValue(ciimage, forKey: kCIInputImageKey)
return RealtimeDepthMaskViewController.filter.outputImage
}
I can then execute this function on any image and obtain a chroma key'd image.
if let maskedImage = filteredImage(ciimage: ciimage) {
//Do something
}
else {
print("Not filtered image")
}
Update Issues:
let data = Data(buffer: UnsafeBufferPointer(start: &cubeRGB, count: cubeRGB.count))
However, once I updated Xcode to v11.6, I obtain the warning Initialization of 'UnsafeBufferPointer<Float>' results in a dangling buffer pointer as well as a runtime error Thread 1: EXC_BAD_ACCESS (code=1, address=0x13c600020) on the line of code above.
I tried addressing this issue with this answer to correct Swift's new UnsafeBufferPointer warning. The warning is then corrected and I no longer have a runtime error.
Problem
Now, although the warning doesn't appear and I don't experience a runtime error, I still get the print statement Not filtered image. I assume that the issue stems from the way the data is being handled, or deleted, not entirely sure how to correctly handle UnsafeBufferPointers alongside Data.
What is the appropriate way to correctly obtain the Data for the Chroma Key?
I wasn't sure what RealtimeDepthMaskViewController was in this context, so just returned the filter output instead. Apologies if this was meant to be left as-is. Also added a guard statement with the possibility of returning nil - which matches your optional return type for the function.
public func filteredImage(ciImage: CIImage) -> CIImage? {
guard let filter = chromaKeyFilter(fromHue: 110/360, toHue: 130/360) else { return nil }
filter.setValue(ciImage, forKey: "inputImage")
return filter.outputImage // instead of RealtimeDepthMaskViewController.filter.outputImage
}
For the dangling pointer compiler warning, I found a couple approaches:
// approach #1
var data = Data()
cubeRGB.withUnsafeBufferPointer { ptr in
data = Data(buffer: ptr)
}
// approach #2
let byteCount = MemoryLayout<Float>.size * cubeRGB.count
let data = Data(bytes: &cubeRGB, count: byteCount)
One caveat: looked at this with Xcode 11.6 rather than 11.5
I have followed this tutorial (https://medium.com/#dominicfholmes/generating-qr-codes-in-swift-4-b5dacc75727c) to generate qr, but I am trying to generate customized qr and one of the requirements is that instead of being squares they are circles in the corners. This is possible?
func generateQR(fromString : String) -> UIImage? {
let data = fromString.data(using: String.Encoding.ascii)
// Get a QR CIFilter
guard let qrFilter = CIFilter(name: "CIQRCodeGenerator") else { return nil}
// Input the data
qrFilter.setValue(data, forKey: "inputMessage")
// Get the output image
guard let qrImage = qrFilter.outputImage else { return nil}
// Scale the image
let transform = CGAffineTransform(scaleX: 10, y: 10)
let scaledQrImage = qrImage.transformed(by: transform)
// Invert the colors
guard let colorInvertFilter = CIFilter(name: "CIColorInvert") else { return nil}
colorInvertFilter.setValue(scaledQrImage, forKey: "inputImage")
guard let outputInvertedImage = colorInvertFilter.outputImage else { return nil}
// Replace the black with transparency
guard let maskToAlphaFilter = CIFilter(name: "CIMaskToAlpha") else { return nil}
maskToAlphaFilter.setValue(outputInvertedImage, forKey: "inputImage")
guard let outputCIImage = maskToAlphaFilter.outputImage else { return nil}
// Do some processing to get the UIImage
let context = CIContext()
guard let cgImage = context.createCGImage(outputCIImage, from: outputCIImage.extent) else { return nil}
let processedImage = UIImage(cgImage: cgImage)
return processedImage
}
There is an example of expected result
https://www.qrcode-monkey.com/img/qrcode-logo.png
It's been a while since I've used the Core Image QR code generator filter, CIQRCodeGenerator. Looking at the docs, it only takes a couple of parameters, inputMessage and inputCorrectionLevel. There's no facility other than those parameters to customize the QR code it generates.
I guess you could do image processing on the resulting image to find the "bullseye" corner squares to change them to rounded rectangles, but that would be a fair challenge.
Conversely you could always write your own QR code rendering library. The image processing part isn't that complicated. It's figuring out the QR code standard and how to generate the dot pattern that would be hard. I haven't looked up the specs for QR codes but it's public.
You might take an existing open source QR code library and modify it to create the rounded rectangle corner squares you are after. I think this is the option I would pursue if it was my task. With any luck you can find a well-written library that first generates the QR code as a grid of booleans, and then uses a separate function to render that grid of boons into an image.
Using CIFIlter I want to apply same filter to multiple images
I have multiple animationImages of UIImageView
let sepiaFilter = CIFilter(name:"CIColorControls")
let brightness = 0.8
for image in imageView.animationImages {
guard let ciimage = CIImage(image: image) else { return }
if let newimage = self.sepiaFilter(ciimage, filter: filter, intensity:brightness )
{
let cgImage:CGImage = ciImageCtx!.createCGImage(newimage, from: newimage.extent)!
let image:UIImage = UIImage.init(cgImage: cgImage)
newImages.append(image)
}
}
}
func sepiaFilter(_ input: CIImage,filter: CIFilter?, intensity: Double) -> CIImage?
{
filter?.setValue(input, forKey: kCIInputImageKey)
filter?.setValue(intensity, forKey: kCIInputBrightnessKey)
return filter?.outputImage
}
So let me know what is best solution to apply CIFilter to multiple images ?
Using above for loop CPU Usage increased more than 100% so it is totally wrong way.
Is it possible animations in GLKit View ?
If yes let me provide deatils about it or Give best solution
**let cgImage:CGImage = ciImageCtx!.createCGImage(newimage, from: newimage.extent)!**
This line taking more CPU usage and time
Thanks.
I've been beating my head against a wall for hours now. I am trying to modify a texture inside my app using a CIFilter and then use that new texture as a part of a new SKTileDefinition to recolor tiles on my map.
The function bellow finds tiles that players "own" and attempts to recolor them by changing the SKTileDefinition to the coloredDefinition.
func updateMapTileColoration(for players: Array<Player>){
for player in players {
for row in 0..<mainBoardMap!.numberOfRows {
for col in 0..<mainBoardMap!.numberOfColumns {
let rowReal = mainBoardMap!.numberOfRows - 1 - row
if player.crownLocations!.contains(CGPoint(x: row, y: col)) {
if let tile = mainBoardMap!.tileDefinition(atColumn: col, row: rowReal) {
let coloredDefinition = colorizeTile(tile: tile, into: player.color!)
print(coloredDefinition.name)
mainBoardMap!.tileSet.tileGroups[4].rules[0].tileDefinitions.append(coloredDefinition)
mainBoardMap!.setTileGroup(crownGroup!, andTileDefinition: crownGroup!.rules[0].tileDefinitions[1], forColumn: col, row: rowReal)
}
}
}
}
}
And here is the function that actulaly applies the CIFilter: colorizeTile
func colorizeTile(tile: SKTileDefinition, into color: UIColor) -> SKTileDefinition{
let texture = tile.textures[0]
let colorationFilter = CIFilter(name: "CIColorMonochrome")
colorationFilter!.setValue(CIImage(cgImage: texture.cgImage()), forKey: kCIInputImageKey)
colorationFilter!.setValue(CIColor(cgColor: color.cgColor), forKey: "inputColor")
colorationFilter!.setValue(0.25, forKey: "inputIntensity")
let coloredTexture = texture.applying(colorationFilter!)
let newDefinition = SKTileDefinition(texture: texture)
newDefinition.textures[0] = coloredTexture
newDefinition.name = "meow"
return newDefinition
}
I would love any help in figuring out why I cannot change the tileDefinition like I am trying to do. It seems intuitively correct to be able to define a new TileDefinition and add it to the tileGroup and then set the tile group to the specific tile definition. However, this is leading to blank tiles...
Any pointers?
After trying a bunch of things I finally figured out what is wrong. The tile definition wasn't being created correctly because I never actually drew a new texture. As I learned, a CIImage is not the drawn texture its just a recipe and we need a context to draw the texture. After this change, the SKTileDefinition is properly created. The problem wasn't where I thought it was so I am sort-of second hand anwering the question. My method for creating a SKTileDefinition was correct.
if drawContext == nil{
drawContext = CIContext()
}
let texture = tile.textures[0]
let colorationFilter = CIFilter(name: "CIColorMonochrome")
colorationFilter!.setValue(CIImage(cgImage: texture.cgImage()), forKey: kCIInputImageKey)
colorationFilter!.setValue(CIColor(cgColor: color.cgColor), forKey: "inputColor")
colorationFilter!.setValue(0.75, forKey: "inputIntensity")
let result = colorationFilter!.outputImage!
let output = drawContext!.createCGImage(result, from: result.extent)
let coloredTexture = SKTexture(cgImage: output!)
let newDefinition = SKTileDefinition(texture: texture)
newDefinition.textures[0] = coloredTexture
newDefinition.name = "meow"
I was wondering if it is possible to binarize an image (convert to black and white only) with Core Image?
I made it with OpenCV and GPUImage, but would prefer it to use Apple Core Image, if that's possible
You can use MetalPerformanceShaders for that. And the CIImageProcessingKernel.
https://developer.apple.com/documentation/coreimage/ciimageprocessorkernel
Here is the code of the class needed.
class ThresholdImageProcessorKernel: CIImageProcessorKernel {
static let device = MTLCreateSystemDefaultDevice()
override class func process(with inputs: [CIImageProcessorInput]?, arguments: [String : Any]?, output: CIImageProcessorOutput) throws {
guard
let device = device,
let commandBuffer = output.metalCommandBuffer,
let input = inputs?.first,
let sourceTexture = input.metalTexture,
let destinationTexture = output.metalTexture,
let thresholdValue = arguments?["thresholdValue"] as? Float else {
return
}
let threshold = MPSImageThresholdBinary(
device: device,
thresholdValue: thresholdValue,
maximumValue: 1.0,
linearGrayColorTransform: nil)
threshold.encode(
commandBuffer: commandBuffer,
sourceTexture: sourceTexture,
destinationTexture: destinationTexture)
}
}
And this is how you can use it:
let context = CIContext(options: nil)
if let binaryCIImage = try? ThresholdImageProcessorKernel.apply(
withExtent: croppedCIImage.extent,
inputs: [croppedCIImage],
arguments: ["thresholdValue": Float(0.2)]) {
if let cgImage = context.createCGImage(binaryCIImage, from: binary.extent) {
DispatchQueue.main.async {
let resultingImage = UIImage(cgImage: cgImage)
if resultingImage.size.width > 100 {
print("Received an image \(resultingImage.size)")
}
}
}
}
Yes. You have at least two options, CIPhotoEffectMono or a small custom CIColorKernel.
CIPhotoEffectMono:
func createMonoImage(image:UIImage) -> UIImage {
let filter = CIFilter(name: "CIPhotoEffectMono")
filter!.setValue(CIImage(image: image), forKey: "inputImage")
let outputImage = filter!.outputImage
let cgimg = ciCtx.createCGImage(outputImage!, from: (outputImage?.extent)!)
return UIImage(cgImage: cgimg!)
}
Note, I'm writing this quickly, you may need to tighten up things for nil returns.
CIColorKernel:
The FadeToBW GLSL (0.0 factor full color, 1.0 factor is no color):
kernel vec4 fadeToBW(__sample s, float factor) {
vec3 lum = vec3(0.299,0.587,0.114);
vec3 bw = vec3(dot(s.rgb,lum));
vec3 pixel = s.rgb + (bw - s.rgb) * factor;
return vec4(pixel,s.a);
}
The code below opens this as a file called FadeToBW.cikernel. You can also post this as a String directly into the openKernelFile call.
The Swift code:
func createMonoImage(image:UIImage, inputColorFade:NSNumber) -> UIImage {
let ciKernel = CIColorKernel(string: openKernelFile("FadeToBW"))
let extent = image.extent
let arguments = [image, inputColorFade]
let outputImage = ciKernel.applyWithExtent(extent, arguments: arguments)
let cgimg = ciCtx.createCGImage(outputImage!, from: (outputImage?.extent)!)
return UIImage(cgImage: cgimg!)
}
Again, add some guards, etc.
I have had success by converting it to greyscale using CIPhotoEffectMono or equivalent, and then using CIColorControls with a ridiculously high inputContrast number (I used 10000). This effectively makes it black and white and thus binarized. Useful for those who don't want to mess with custom kernels.
Also, you can use an example like Apple's "Chroma Key" filter which uses Hue to filter, but instead of looking at Hue you just give the rules for binarizing the data (ie: when to set RGB all to 1.0 and when to set to 0.0).
https://developer.apple.com/documentation/coreimage/applying_a_chroma_key_effect
Found this thread from a Google search, and thought I'd mention that as of iOS 14 and OSX 11.0, CoreImage includes CIColorThreshold and CIColorThresholdOtsu filters (the latter using Otsu's method to calculate the threshold value from the image histogram)
See:
https://cifilter.io/CIColorThreshold/
https://cifilter.io/CIColorThresholdOtsu/
let outputImage = inputImage.applyingFilter("CIColorMonochrome",
parameters: [kCIInputColorKey: CIColor.white])
In you want to play with every out of 250 CIFilters please check this app out: https://apps.apple.com/us/app/filter-magic/id1594986951