How can I use the screen as a video input to darkflow - python-imaging-library

Question

I've trained darkflow on my data set and have good result! I can feed it a pre recorded image or video and it draws the bounding boxes around the right things, win!
Now I'd like to run it live as has been done with camera feeds, except I'd like my feed to be from the screen, not the camera. I have a specific window, which is launched from a specific process, or I can just take a section of the screen (from coords) either is fine for my application.
Currently I use PILs image grab and then feed the images into darkflow, but this feels quite slow (maybe a few frames per second) nothing like the 30 ish fps you can get with video files!

Answer 1

I get more than 25 fps with Python MSS on my slow laptop under Ubuntu.
Here is an example:
from mss import mss
from PIL import Image
import time
def capture_screenshot():
with mss() as sct:
monitor = sct.monitors[1]
sct_img = sct.grab(monitor)
# Convert to PIL/Pillow Image
return Image.frombytes('RGB', sct_img.size, sct_img.bgra, 'raw', 'BGRX')
N = 100
t = time.time()
for _ in range(N):
capture_screenshot()
print ("Frame rate = %.2f fps" % (N/(time.time()-t)))
Output:
Frame rate = 27.55 fps

Answer 2

I got over 40 fps with this script (on i5-7500 3.4GHz, GTX 1060, 48GB RAM).
There are a lot of APIs used to capture the screen. Among them, mss runs much faster and is not difficult to use. Here is an implementation of mss with darkflow(YOLOv2), in which 'mon' defines the area you want apply prediction on the screen.
options is passed to the darkflow, that specifies which config file and checkpoint we want to use, threshold for detection, and how much this process occupies the GPU. Before we run this script, we have to have at least one trained model (or Tensorflow checkpoint). Here, load is the checkpoint number.
If you think that the network detects too many bounding boxes, I recommend you to lower the threshold.
import numpy as np
import cv2
import glob
from moviepy.editor import VideoFileClip
from mss import mss
from PIL import Image
from darkflow.net.build import TFNet
import time
options = {
'model' : 'cfg/tiny-yolo-voc-1c.cfg' ,
'load' : 5500,
'threshold' : 0.1,
'gpu' : 0.7 }
tfnet = TFNet( options )
color = (0, 255, 0) # bounding box color.
# This defines the area on the screen.
mon = {'top' : 10, 'left' : 10, 'width' : 1000, 'height' : 800}
sct = mss()
previous_time = 0
while True :
sct.get_pixels(mon)
frame = Image.frombytes( 'RGB', (sct.width, sct.height), sct.image )
frame = np.array(frame)
# image = image[ ::2, ::2, : ] # can be used to downgrade the input
frame = cv2.cvtColor(frame, cv2.COLOR_BGR2RGB)
results = tfnet.return_predict( frame )
for result in results :
tl = ( result['topleft']['x'], result['topleft']['y'] )
br = ( result['bottomright']['x'], result['bottomright']['y'] )
label = result['label']
confidence = result['confidence']
text = '{} : {:.0f}%'.format( label, confidence * 100 )
frame = cv2.rectangle( frame, tl, br, color, 5 )
frame = cv2.putText( frame, text, tl, cv2.FONT_HERSHEY_COMPLEX, 1, (0, 0, 0), 2 )
cv2.imshow ( 'frame', frame )
if cv2.waitKey ( 1 ) & 0xff == ord( 'q' ) :
cv2.destroyAllWindows()
txt1 = 'fps: %.1f' % ( 1./( time.time() - previous_time ))
previous_time = time.time()
print txt1