ipython notebook and leaking file descriptors - sockets

Question

I'm having problems with leaking file descriptors in code I have running in ipython notebook. I'm downloading lots of files with urllib2 and saving them locally. Apparently, urllib2 has a history of leaking file descriptors, which I suspect is causing problem. In the end, I get an IoError: Too many open files.
As a workaround, I periodically close a bunch of sockets using os.close. Unfortunately, ipython notebook has lots of sockets running which I don't want to close.
Is there a way that I can identify which file descriptors/sockets/etc.. belong to ipython?

Answer 1

This isn't really an answer, but a couple of workarounds in case others find themselves here with leaking file descriptor problems.
The first workaround, which is probably better, is to use subprocess.call() to download the files I want with wget. It's been about 4 times as fast as the method below.
The second workaround is to use a couple of handy functions I found on SO (which I can't find at the moment - if you find it, edit this or let me know and I'll link):
import resource
import fcntl
import os
def get_open_fds():
fds = []
soft, hard = resource.getrlimit(resource.RLIMIT_NOFILE)
for fd in range(0, soft):
try:
flags = fcntl.fcntl(fd, fcntl.F_GETFD)
except IOError:
continue
fds.append(fd)
return fds
def get_file_names_from_file_number(fds):
names = []
for fd in fds:
names.append(os.readlink('/proc/self/fd/%d' % fd))
return names
With these, I store the active file descriptors and corresponding names before I start downloading files. I then periodically test the number of open file descriptors, and if it's getting dangerously large, use os.close() on all the ones that aren't in the original list (I check the names too - descriptors themselves get recycled).
It's ugly, and occasionally ipython notebook complains with things like "can't save history" (presumably I've clobbered something it was using), but it is working pretty well otherwise.