Videos Appearing as Black Boxes in UIImage Snapshots of WKWebView - swift

I am captruing the content of WKWebView into a UIImage via its CALayer by using the following extenstion to UIImage:
extension UIImage {
class func imageFromLayer(layer: CALayer) -> UIImage {
UIGraphicsBeginImageContextWithOptions(layer.bounds.size, layer.isOpaque, 0.0)
layer.render(in: UIGraphicsGetCurrentContext()!)
let img = UIGraphicsGetImageFromCurrentImageContext()
UIGraphicsEndImageContext()
return img!
}
}
When I render the captured image to a UIImageView, I see everything from the web view in the image view; everything, except videos, whose content instead appears as a black box. I encounter the same problem if I use WKWebView's takeSnapshot function or the drawHierarchy function of the UIView class to capture UIImage of the web view. Does anyone know why videos are displayed as black boxes and how one can fix this?
Update 1: Additionally, all GIFs in Google Images are frozen until clicked. A GIF can be unfrozen by clicking on it, at which point it gets resized to take up the whole screen and starts looping.
Update 2: I have read that the recording software OBS sometimes experiences the same problem with capturing videos when the browser that it is capturing uses hardware acceleration. I also read that WKWebView uses hardware acceleration. Is there a way to disable hardware acceleration in WKWebView to try this out? Alternatively, does UIWebView use hardware acceleration?
Update 3: There is another post that reports a similar issue. The poster reports that all GPU rendered content appears black in his screenshots and he believes that hardware acceleration is causing the issue.
Update 4: By using Xcode's "Quick Look" feature I have observed that the layer that is passed to imageFromLayer does not contain the videos (i.e. videos are already represents as black boxes in the passed in layer). This makes me think that WKWebView does not render videos to the same layer as it renders its other content. It also makes me think that if I could access this hypothetical video layer I would be able to combine the two snapshots to form a snapshot of the entire web view.
Update 5: I tried modifying my imageFromLayer function to take a UIView instead of a CALayer so that I could pass WKWebView into it directly. The modified version is shown below, but it did not solve the issue of videos not being displayed. Nevertheless, through the use of Xcode's "Quick Look" feature I was able to observe that the videos are already missing in the UIView of the passed in WKWebView object.
class func createImage(view: UIView) -> UIImage {
UIGraphicsBeginImageContextWithOptions(view.frame.size, view.isOpaque, 0.0)
view.drawHierarchy(in: view.bounds, afterScreenUpdates: true)
let rasterizedView = UIGraphicsGetImageFromCurrentImageContext()!
UIGraphicsEndImageContext()
return rasterizedView
}
Update 6: Trying to capture the web view via the UIWindow that contains it also results in videos not being captured. I used the below code (note that keyWindow is deprecated) and Xcode's Quick Look feature to test this idea.
let window = UIApplication.shared.keyWindow
let windowLayer = window!.layer

Related

Webkit vs SFSafariViewController and Core Data

Which is the most preferred method for an in-app web browser? My app has the need to have a toolbar at the bottom, and I need to be able to take screenshots of visited web pages by hitting a button on the toolbar.
Here is what I'm running into. I want to be able to click a link in my app, open an in-app browser, take a screenshot, and save to core data along with other information about the site.
I'm able to save a screenshot to the camera roll with .takesnapshot() method for Webkit. I've been unable to save it to core data. As of last night, I've found a few functions on SO that show how to take a screenshot of the UIView and return a UIImage, but I've been unable to cast this back to Data since this is what Core Data expects for binary data. Does anyone have a good resource to save Webkit snapshots to Core Data?
In one of the functions I attempted last night, I was able to return the UIImage object, but I was unable to convert it back to Data. I can save all other data about the site to Core Data, but I'm unable to save the image - in fact, when I attempted to save the data directly with Webkit's .takesnapshot() method, the result was nil.
I'm going to answer your question in two parts.
WebKit vs SFSafariViewController
If you want to have a custom toolbar with a button for taking screenshots, I would use a UIViewController with a custom tab bar where you can add whatever buttons you want, then I would also embed a WebKit view in the view controller too. That should be pretty straightforward.
Saving screenshot to Core Data
By the sounds of it, you may be getting a screenshot the wrong way, you want to do so with graphics begin and end image context as follows:
let layer = UIApplication.shared.keyWindow!.layer
let scale = UIScreen.main.scale
UIGraphicsBeginImageContextWithOptions(layer.frame.size, false, scale)
layer.render(in: UIGraphicsGetCurrentContext()!)
let screenshot = UIGraphicsGetImageFromCurrentImageContext()
Now you have a screenshot saved as a UIImage? in your screenshot variable. Next up you should convert this to data with:
let data = UIImagePNGRepresentation(screenshot!)
Which should give you a data representation of your image that you can then save to Core Data.

How to fix blurry images on NSButton in cocoa application?

I have a couple of NSButtons which act like player controls (play, pause etc.)
When I assign image to them, that image looks blurry.
let playPauseButton: NSButton = {
let btn = NSButton(frame: .zero)
let image = NSImage(named: "pauseIcon")
btn.imageScaling = NSImageScaling.scaleNone
btn.image = image
btn.setAccessibilityLabel("Pause")
btn.isBordered = false
return btn
}()
The image can be viewed on:
https://www.dropbox.com/s/38tvh49ikc2m4x7/Play.pdf?dl=0
This image is .pdf, but I tried .png with 1x,2x and 3x, and it still looked blurry.
I also checked this question:
Images in NSButton and NSImageView Blurred
But it did not help me either.
The solution was very simple.
Open the asset catalog.
Select all your assets that are blurry.
Open the attribute inspector by pressing: command + option + 4.
Check the box that says "Preserve vector data".
Under that checkbox, look for the "Scale" and set it to "Single scale"
Please note that this only works if your designer exported icons as a PDF file. Also, it wont work if you just convert .png or .jpg to .pdf, the actual vector image must be exported to .pdf.
Here are a number of to try that have fixed blurry edges for me in the past.
Assuming you're running in Mojave and/or using layer backed views, be sure that the NSButton draws its own background color ( setDrawsBackground YES ). Otherwise what can happen with CoreAnimation CALayer is that the blending operation of the NSImage background alpha is substandard/wrong.
This shouldn't apply to an NSImage, but maybe its worth a shot: http://stackoverflow.com/questions/5722085/nsbezierpath-rounded-rectangle-has-bad-corners
[nsImage setScalesWhenResized:NO];
try different NSCompositingOperation NSCompositeCopy vs. NSCompositeSourceOver

Thumbnail view of images

I want to show some images as a thumbnail in View controller but i dont know how to do this. Can anyone please give me some ideas or sample code if possible.
Are you asking how to create thumbnails from larger images, or about how to build a view controller which displays them nicely?
Building a view controller:
You could use TTPhotoViewController from the Three20 project (description), which acts similarly to the iPhone's built in camera roll to view images.
You can look at the Scrolling sample code from apple, referred to in this question about using it for thumbnails.
If you want to build one yourself, you might consider using a GridView from the moriarty library, either as a large grid in a UIScrollView, or on a more efficient row-by-row basis in a UITableView. There's a previous question on optimized image loading in a UIScrollView.
Creating thumbnails from larger images:
The code-easiest way to scale down an image is to simply display it in a UIImageView with a frame set to the smaller size that you want - it's scaled for you.
If you want to save a thumbnail, or care about memory usage, you can create a hard-scaled version. The sample code below is taken from this blog post, and can be added to UIImage as a category method:
- (UIImage*) imageScaledToSize: (CGSize) newSize {
UIGraphicsBeginImageContext(newSize);
[self drawInRect:CGRectMake(0, 0, newSize.width, newSize.height)];
UIImage* newImage = UIGraphicsGetImageFromCurrentImageContext();
UIGraphicsEndImageContext();
return newImage;
}
Finally, here's a previous question on masking out round corners on an image, similar to the app icons used on the home screen.
Added
Using an image's built-in thumbnail:
There's a nice function called CGImageSourceCreateThumbnailAtIndex that understands built-in thumbnails in certain image data formats. You can see some useful sample code for this under Creating a Thumbnail Image from an Image Source in the Image I/O Programming Guide from Apple.
There are multiple issues with thumbnails; perhaps you could clarify which ones you are most concerned about?
How to display a smaller version of an existing image
How to speed up paging by caching the thumbnails (instead of just dynamically shrinking the originals)
How to allow the user to page through the thumbnails
How to synchronize the thumbnails with the originals, in the event that your images are editable or the user can add to them

Getting an image representation of the camera preview in UIImagePickerController

When I use the standard [view.layer renderInContext:UIGraphicsGetCurrentContext()]; method for turning views into images on the iPhone camera modal view, all I get is the controls with black where the camera preview was.
Anyone have a solution for this? Speed is not an issue.
Not possible from a documented api, but possible. Find the "PLCameraView" view in the camera's subviews, then call
CGImageRef img = (CGImageRef)[foundCameraView imageRef];
This will return a reference to the image that camera holds.
You can do this with the new AVFoundation stuff in iOS 4.0.
You should be able to call UIGetScreenImage() (returns a UIImage) to get a current capture of the whole screen, including the preview. That's how all the barcode apps worked before. But supposedly Apple is disallowing that now and only allow the AVFoundation technique - which only works under 4.0.
The whole reason there's even an issue is because UIGetScreenImage() is not part of the documented API, but Apple made a specific exception for using it. It's not like they are pulling current apps, but they are not allowing new submissions (or updates) that use the older technique.
There is some lobbying on behalf on a number of people to convince Apple to let app developers use the old technique for iOS 3.x only, so send an email to developer relations if you want to use it.
This isn't possible unfortunately. For performance the iPhone uses some form of direct rendering to draw the camera preview directly onto the screen instead of as part of a UIView surface. As such when you "capture" the camera view you will only get the UIView elements, not the preview image.
(FWIW this is similar to the reasons why its difficult to screengrab some movie software on Windows/Mac)
You could try AVCaptureVideoPreviewLayer

Rendering a UIWebView into an ImageContext

I am trying to capture the contents of a UIWebView including that which is not visible to the user. i.e. The whole web page even though the user is only looking at the top.
Looking around I found the best way to capture a UIView is to retrieve its layer and use renderInContext.
However, UIWebView seems to be using its own CALayer implementation which is behaving a lot more like CATiledLayer, although it is still claiming to be a standard CALayer. When I call renderInContext I only get one portion of the web page, up to 940px down, as opposed to the whole page.
Has anyone got any ideas on how to either: force the UIWebView to scroll down another 940px (obviously that is far from ideal) or tell whatever breed of CALayer it is backing the WebView to render all of its content when I ask it to.
Cheers
EDIT: I should add that currently change the frame of the webview to fit the size of the page retrieved through javascript.
I've released an app (Web2Pic) doing that, and please trust me that UIGraphicsBeginImageContext(webView.frame.size);
can do nothing except getting a small image from the visible area in our UIWebView ;-(
The right way is a bit complex but it just works:
1.Use JavaScript in our UIWebView to get these float values:
//Whole page size in HTML coordinate
document.body.scrollWidth
document.body.scrollHeight
//UIWebView visible size in HTML coordinate
window.innerWidth
window.innerHeight
2.Now we can 'cut' the whole page into dozens of UIWebView-sized small pieces. Then we can capture every small pieces individually and save them into our Cache. I implemented this by calculating page-offsets and use UIGraphicsBeginImageContext(webView.frame.size); to get a array of images. In addition, you should cache the image array into the file system, or the app will eventually crash!
3.When we finally got all the small pieces, we can start a full-resolution context: UIGraphicsBeginImageContext(CGSizeMake(document.body.scrollWidth,document.body.scrollHeight));
4.Render every small images into the big context based on the coordinates. And be careful to the corners, the last image in every line/row may not be a full image.
5.There is still one step left: saving the big image. Do not save it into the PhotoAlbum, because iOS will automatically cut down the resolution of images in the album. Instead, we can save it into the file system and enable the app's iTunes File Share support, or even write a simple in-app photo manager.
Hope these can help ;-)
Yichao Peak Ji
It looks to me like UIWebView renders on demand (witness the checkerboard as you scroll downwards rapidly on a large page), so there won't be anything in the part of the layer below what you can reach until the user scrolls down there. The layer won't be able to write out what it doesn't have, so I don't think you'll be able to capture the whole area.
As far as scrolling, there aren't any obvious exposed API methods that I can think of to move it down. The web view seems to host a UIScrollView or something similar, which you could try to access by traversing the view hierarchy and use its scrollRectToVisible:animated:, but that doesn't sound like a good long-term solution.
Just to answer one of my questions, you can scroll the view by using javascript.
So you use stringByEvaluatingJavaScriptFromString: with a javascript of window.scrollTo(0,%i);, where %i is where you want to scroll to. You can also use scrollY to retrieve the current scroll value.
Resizing the Webview is the way I solved it. If you want a bigger image with more detail, you just have to zoom in before calling this method. Beware that large dimensions could infulence the performance.
- (UIImage*) imageFromWebview:(UIWebView*) webview{
//store the original framesize to put it back after the snapshot
CGRect originalFrame = webview.frame;
//get the width and height of webpage using js (you might need to use another call, this doesn't work always)
int webViewHeight = [[webview stringByEvaluatingJavaScriptFromString:#"document.body.scrollHeight;"] integerValue];
int webViewWidth = [[webview stringByEvaluatingJavaScriptFromString:#"document.body.scrollWidth;"] integerValue];
//set the webview's frames to match the size of the page
[webview setFrame:CGRectMake(0, 0, webViewWidth, webViewHeight)];
//make the snapshot
UIGraphicsBeginImageContextWithOptions(webview.frame.size, false, 0.0);
[webview.layer renderInContext:UIGraphicsGetCurrentContext()];
UIImage *image = UIGraphicsGetImageFromCurrentImageContext();
UIGraphicsEndImageContext();
//set the webview's frame to the original size
[webview setFrame:originalFrame];
//and VOILA :)
return image;
}
All UIViews have a size limit of 1024x1024, so you will probably have programmatically scroll, capture the page in chunks, and stitch them together somehow.
Set the height of webview equal to the contentsize of scrollview of webview.
-(UIImage *)captureImage:(UIWebView *)webvw
{
webvw.frame=CGRectMake(webvw.frame.origin.x, webvw.frame.origin.y, webvw.frame.size.width, webvw.scrollView.contentSize.height);
UIGraphicsBeginImageContextWithOptions(webvw.frame.size, NO, 0.0f);
[webvw.layer renderInContext:UIGraphicsGetCurrentContext()];
UIImage *screenshot = UIGraphicsGetImageFromCurrentImageContext();
return screenshot;
}