Stream remote framebuffer into NSView - swift

Question

I'm trying to implement a simple VNC client in Swift using LibvnccClient.
I have created a subclass of NSView. I create a CGContext and pass the data pointer to the library for use as the framebuffer. libvncclient updates the framebuffer as contents of the screen changes and calls my provided callback.
Here is the relevant code
class VNCClient {
var localClient: rfbClient
var view: NSView
init(view: NSView) {
guard let client_ptr = rfbGetClient(8, 3, 4) else {
fatalError("Trouble")
}
self.view = view
self.localClient = client_ptr.pointee
localClient.MallocFrameBuffer = resize
localClient.GotFrameBufferUpdate = update
let fbPointer = UnsafeMutablePointer<UInt8>(OpaquePointer((view as! RFBView).buffer))
localClient.frameBuffer = fbPointer
rfbClientSetClientData(&localClient, &viewTag, &self.view)
var argc: Int32 = 0
let b = rfbInitClient(&localClient, &argc, nil)
if b == RFB_TRUE {
print("Connected!")
}
Timer.scheduledTimer(withTimeInterval: 0.001, repeats: true) {_ in
if WaitForMessage(&self.localClient, 1) > 0 {
HandleRFBServerMessage(&self.localClient)
}
}
}
func update(client: UnsafeMutablePointer<rfbClient>?, x: Int32, y: Int32, w: Int32, h: Int32) -> Void {
let cl = client!.pointee
let view_ptr = rfbClientGetClientData(client, &viewTag)
let view = view_ptr?.assumingMemoryBound(to: RFBView.self).pointee
view?.setNeedsDisplay(NSRect(x: Int(x), y: Int(y), width: Int(w), height: Int(h)))
}
class RFBView: NSView {
let ctx = CGContext(data: nil, width: 800, height: 600, bitsPerComponent: 8, bytesPerRow: 4 * 800, space: CGColorSpaceCreateDeviceRGB(), bitmapInfo: CGImageAlphaInfo.noneSkipLast.rawValue)!
var buffer: UnsafeMutableRawPointer? {
return ctx.data
}
override func draw(_ dirtyRect: NSRect) {
let image = ctx.makeImage()
NSGraphicsContext.current()?.cgContext.draw(image!, in: frame)
}
It works but the display is not smooth. The server is running on the same machine in a VM so no network issues.
I'm redrawing the whole image for every update which I assume is the cause of the problem. So how can I redraw only the part of the framebuffer that is updated?
Is CoreGraphics fast enough for this or should I use NSOpenGLView?

Answer 1

I assumed that HandleRFBServerMessage would call update function once per iteration if there was an update. But that is not the case.
I replaced update with a stub that printed to console, and noticed it was being called more than thousand times when there was a lot of activity on the screen.
However, with the setNeedsDisplay call inside update, it was only being called a few hundred times. Since it was spending too much time rendering it was missing out on a few frames in between.
I moved the drawing code inside the Timer and it works perfectly now.
localClient.GotFrameBufferUpdate = { _, _, _, _, _ in needs_update = true }
Timer.scheduledTimer(withTimeInterval: 0.016, repeats: true) {_ in
if needs_update {
view.display()
needs_update = false
}
if WaitForMessage(&self.localClient, 1) > 0 {
HandleRFBServerMessage(&self.localClient)
}
}