I have this model.
In Mac, when an image is input, it detects objects and represents what it is.
https://i.stack.imgur.com/qdFBr.png
However, this model has four outputs.
I think the first output is the result.
So I converted the results into an image in Python as follows and saved the model as mlmodel.
import coremltools.proto.FeatureTypes_pb2 as ft
output = spec.description.output[0]
output.type.imageType.colorSpace = ft.ImageFeatureType.GRAYSCALE
output.type.imageType.height = 416
output.type.imageType.width = 416
In Swift, the converted results are stored in this form.
lazy var var_944: CVPixelBuffer = {
[unowned self] in return self.provider.featureValue(for: "var_944")!.imageBufferValue
}()!
After putting an image in the input, convert the output CVPixelBuffer to UIImage, and put UIImage in ImageView, no image appears.
Does anyone know the solution?
(Please understand that I used Papago because I am not good at English.)
Related
I'm writing a macOS app which performs a complicated chain of CIFilter operations on an image to greatly change the appearance of high resolution photographs, often 24 megapixels or larger.
Some of these effects included are gaussian blurs, unsharp masks, bloom, gloom, as well as a custom "grain" fragment shader I wrote in Metal using a custom CIKernel. The CIContext is using a Metal device to render it. Essentially, it's a long chain of initialImage -> CIFilter -> outputImage -> CIFilter -> outputImage -> CIFilter -> ...
Not only must all of these CIFilters be run in sequence for a final output, they must also be run on the full resolution for effects to be correctly scaled.
The problem I face is that executing entire process results in a massive usage of memory. With a 6000x4000 input image, the memory usage jumps to 6.6GiB while rendering.
I used the Metal instrumentation in Xcode to diagnose the problem and it seems that CoreImage is recursively allocating memory for each filter, so that memory just piles up and up until it can finally let go of all of the resources at the end.
I would have hoped this was sequential, releasing each source buffer before the next operation. I'm not sure exactly how to help the situation. Is there some way to pass each output image to the next filter, forcefully cleaning up each input CIImage rendering first?
if let device = self.device,
let texture = device.makeTexture(descriptor: descriptor),
let queue = device.makeCommandQueue(),
let buffer = queue.makeCommandBuffer() {
let destination = CIRenderDestination(
width: descriptor.width,
height: descriptor.height,
pixelFormat: self.colorPixelFormat,
commandBuffer: buffer) {
return texture
}
try! metalImage.context.context.startTask(toRender: metalImage.image, to: destination)
self.renderedImage = texture
buffer.commit()
}
This seems to work well with only a few hundred MB of memory usage when displaying a 24 megapixel CIImage to an NSImageView without any CIContext.
let image = // Create CIImage from complex CIFilter pipeline
let rep = NSBitmapImageRep(ciImage: image)
let previewImage = NSImage(size: rep.size)
previewImage.addRepresentation(rep)
previewImageView?.image = previewImage
On a 5K iMac, it will render pretty efficiently for previewing (less than half a second to update). No CIContexts are needed until exporting the image using a CGImageDestination.
This may be an obscure question, but I see lots of very cool samples online of how people are using the new ARKit people occlusion technology in ARKit 3 to effectively "separate" the people from the background, and apply some sort of filtering to the "people" (see here).
In looking at Apple's provided source code and documentation, I see that I can retrieve the segmentationBuffer from an ARFrame, which I've done, like so;
func session(_ session: ARSession, didUpdate frame: ARFrame) {
let image = frame.capturedImage
if let segementationBuffer = frame.segmentationBuffer {
// Get the segmentation's width
let segmentedWidth = CVPixelBufferGetWidth(segementationBuffer)
// Create the mask from that pixel buffer.
let sementationMaskImage = CIImage(cvPixelBuffer: segementationBuffer, options: [:])
// Smooth edges to create an alpha matte, then upscale it to the RGB resolution.
let alphaUpscaleFactor = Float(CVPixelBufferGetWidth(image)) / Float(segmentedWidth)
let alphaMatte = sementationMaskImage.clampedToExtent()
.applyingFilter("CIGaussianBlur", parameters: ["inputRadius": 2.0)
.cropped(to: sementationMaskImage.extent)
.applyingFilter("CIBicubicScaleTransform", parameters: ["inputScale": alphaUpscaleFactor])
// Unknown...
}
}
In the "unknown" section, I am trying to determine how I would render my new "blurred" person on top of the original camera feed. There does not seem to be any methods to draw the new CIImage on "top" of the original camera feed, as the ARView has no way of being manually updated.
In the following code snippet we see personSegmentationWithDepth type property for depth compositing (there are RGB, Alpha and Depth channels):
// Automatically segmenting and then compositing foreground (people),
// middle-ground (3D model) and background.
let session = ARSession()
if let configuration = session.configuration as? ARWorldTrackingConfiguration {
configuration.frameSemantics.insert(.personSegmentationWithDepth)
session.run(configuration)
}
You can manually access a Depth Data of World Tracking in CVPixelBuffer (depth values for a performed segmentation):
let image = frame.estimatedDepthData
And you can manually access a Depth Data of Face Tracking in CVPixelBuffer (from TrueDepth camera):
let image = session.currentFrame?.capturedDepthData?.depthDataMap
Also, there's a generateDilatedDepth instance method in ARKit 3.0:
func generateDilatedDepth(from frame: ARFrame,
commandBuffer: MTLCommandBuffer) -> MTLTexture
In your case you have to use estimatedDepthData because Apple documentation says:
It's a buffer that represents the estimated depth values from the camera feed that you use to occlude virtual content.
var estimatedDepthData: CVPixelBuffer? { get }
If you multiply DEPTH data from this buffer (at first you have to convert Depth channel to RGB) by RGB or ALPHA using compositing techniques and you'll get awesome effects.
Look at these 6 images: the lower row represents three RGB-images corrected with Depth channel: depth grading, depth blurring, depth point position pass.
the Bringing People into AR WWDC session has some information, especially about ARMatteGenerator. The session also comes with a sample code.
https://github.com/mateo951/ISBN-Vista-Jera-rquica- Github Link
The structure I have is supposed to be appending values after an internet search. The internet search is called within a function and returns two strings and an image. When I try to append the returned values in the structure, the image is saved but strings are nil.
var datosLibros = [bookData]()
#IBAction func Search(sender: UITextField) {
let (title1, author1, cover1) = (internetSearch(sender.text!))
let libro = bookData(title: title1, author: author1,image:cover1)
datosLibros.append(libro)
print(datosLibros)
}
The saved structured that is printed to the console is the following:
bookData(title: "", author: "", image: <UIImage: 0x7f851a57fbf0>, {0, 0})
Structure:
struct bookData {
var title: String
var author: String
var image: UIImage
init(title: String, author: String, image: UIImage) {
self.title = title
self.author = author
self.image = image
}
}
Thanks in advanced for any advice of help provided. I'm new to swift so there are a lot of stuff uncovered.
The problem is not with the code you posted but with internetSearch.
But before I explain what is going on there, just a quick note about Swift structs. Structs come with one free initializer that takes as its parameters one value for each stored property defined on the struct. Argument labels correspond to the variable labels.
So for your struct bookData (which really should be BookData since types should be capitalized), you do not need to include that initializer you wrote because it will be automatically provided for you as long as you do not create any additional BookData initializers.
Now for the reason your results are not what you expect. Your Strings are not coming back as nil. Instead, they are coming back as empty Strings, or "". In Swift, "" is very different from nil, which means a complete absence of a value. So your Strings are indeed there, they are just empty.
Okay, our Strings are coming back empty. How about our image? No, our image is not coming back either. You thought it was because you saw a UIImage reference printed in the console, but if you look closer you will notice it is a bogus image. Notice "{0, 0}" after the memory address for the instance. As far as I'm aware, this means the image has a size of 0 x 0. How many useful images do you know that have a size of 0 x 0?
So now we have discovered that our Strings are coming back empty and effectively so is our image. What is going on here?
Well, in your implementation of internetSearch I found on GitHub, this is the first thing you do:
var bookTitle = String()
var bookAuthor = String()
var bookCover = UIImage()
Naturally, you do this so that you have some variables of the correct types ready to plop in some actual results if you find them. Just for fun, let's see what the result of the code above would be if there were no results.
Well, the initializer for String that accepts no parameters results in an empty String being created.
Okay, how about our image. While the documentation for UIImage does not even mention an initializer that takes no parameters, it does inherit one from NSObject and it turns out that it will just create an empty image object.
So we now have discovered that what internetSearch is returning is actually the same as what it would be if there were no results. Assuming you are searching for something that you know exists, there must be a problem with the search logic, right? Not necessarily. I noticed that your implementation of the rest of internetSearch relies on an NSURLSession that you use like so:
var bookTitle = String()
var bookAuthor = String()
var bookCover = UIImage()
let session = NSURLSession.sharedSession()
let task = session.dataTaskWithURL(url) { (data, response, error) -> Void in
// Lots of code that eventually sets the three variables above to a found result
}
task.resume()
return (bookTitle, bookAuthor, bookCover)
That seems fine and dandy, except for the fact that NSURLSession performs its tasks asynchronously! Yes, in parts you even dispatch back to the main queue to perform some tasks, but the closure as a whole is asynchronous. This means that as soon as you call task.resume(), NSURLSession executes that task on its own thread/queue/network and as soon as that task is set up it returns way before it completes. So task.resume() returns almost immediately, before any of your search code in the task actually runs, and especially before it completes.
The runtime then goes to the next line and returns those three variables, just like you told it to. This, of course, is the problem because your internetSearch function is returning those initial empty variables before task has a chance to run asynchronously and set them to helpful values.
Suggesting a fully-functional solution is probably beyond the scope of this already-long answer, but it will require a big change in your implementation detail and you should search around for using data returned by NSURLSession.
One possible solution, without me posting any code, is to have your internetSearch function not return anything, but on completion of the task call a function that would then append the result to an array and print it out, like you show. Please research this concept.
Also, I recommend changing your implementation of internetSearch further by declaring your initial values not as:
var bookTitle = String()
var bookAuthor = String()
var bookCover = UIImage()
…but as:
var bookTitle: String?
var bookAuthor: String?
var bookCover: UIImage?
This way, if you find a result than you can represent it wrapped in an Optional and if not you can represent that as nil, which will automatically be the default value of the variables in the code directly above.
rowDescriptor is a AnyObject? type, I get convert self.rowDescriptor?.value to images. I want to add image to the self.rowDescriptor?.value. I failed to do it as below. images has one value. but self.rowDescriptor?.value still empty. I can't find any document about it. what's the reason about it.
var images :[UIImage] = self.rowDescriptor?.value as! [UIImage]
if let image = editedImage {
images.append(image)
} else {
images.append(originalImage!)
}
The array type in Swift is a struct. Structs in swift are value types not reference types. To clarify:
images contains a copy of self.rowDescriptor?.value
images.append( changes the copy images, not the original value in your rowDescriptor.value
You want to change self.rowDescriptor?.value so just set that to your newly changed images array.
self.rowDescriptor?.value = images
td;dr
To fix, add self.rowDescriptor?.value = images.
After already having parsed my data through swiftyjson I am looking to display the parsed information in a uitableview. Let's say I have a list of names that I parsed out of the json file. I would like those names to display each in their own respective cells to allow the user to view them all.
I would also assume that if they selected the selected cells which are being displayed strings now after being parsed that I can actually have those stored into a core data object storing them as returnable strings? The thing is I used the .append function onto an array that I created we'll call names = [String]() to actually allow the values to actually come in order for the cells.
But when doing this I noticed a huge spike in cpu usage doing this as it keeps appending them in order lets say first name is matt second name is john and so forth then it creates a new situation of the array every time. so the first time it reads just matt then the next println would print out matt, john then the next one prints out matt, john, tom and so forth until the list of json names is completed.
If I have 500 names or more it takes the cpu awhile to process all those names into the array. I know there has to be another way but all references I see all use the .append method. So my code would be names.append(Name). Name is a parsed string from swiftyjson. From there I am actually able to populate the tableview which is fine it displays correctly in the end but it takes the device awhile to populate as it first needs to process adding all 500 names each time the parse is called. I am hoping to allow the 500 names be rechecked every time in case new ones have been added. This all stores currently as a local json file already.
The code I am using is below I can figure out the UItableview part but it's more a question of whether appending an array is the best and most efficient way of displaying the parsed data from swiftyjson.
let path = NSBundle.mainBundle().pathForResource("List", ofType: "json")
let jsonData = NSData(contentsOfFile: path!, options: NSDataReadingOptions.DataReadingMappedIfSafe, error: nil)
let json = JSON(data: jsonData!)
for (key, subJson) in json["array"] {
if let Name = subJson["Name"].string {
NameList.append(Name)
println(NameList)
}
}
maybe the Swift map function is more effective
…
let json = JSON(data: jsonData!)
var nameList = [String]()
if let subJson = json["array"] as [Dictionary<String,NSObject>]? {
nameList = subJson.map { $0["name"] as! String }
}