How to debug session leaking or close all sessions in python MongoDB? - mongodb

Question

It is my first time to use MongoDB to manage an image dataset(~10 million images).
My environment is MongoDB 5.0.6 with PymongoDB 4.0.2 and Python 3.9.6 on Ubuntu 18.04.
My dataset is accessing PyMongoDB and then it is used to train a DNN in Pytorch. My code warns at the begining:
UserWarning: MongoClient opened before fork.
Create MongoClient only after forking.
See PyMongo's documentation for details: https://pymongo.readthedocs.io/en/stable/faq.html#is-pymongo-fork-safe
(I check this url and get nothing. I think I indeed recreate the client each time when the class is instantiaed)
After running for a while, my code crashs, and exit
Unable to add session ID ffd152cf-97d3-454a-882a-c6fc693e2985 - 47DEQpj8HBSa+/TImW+5JCeuQeRkm5NMpJWZG3hSuFU=
into the cache because the number of active sessions is too high,
full error:
{'ok': 0.0,
'errmsg': 'Unable to add session ID ffd152cf-97d3-454a-882a-c6fc693e2985 - 47DEQpj8HBSa+/TImW+5JCeuQeRkm5NMpJWZG3hSuFU= into the cache because the number of active sessions is too high',
'code': 261,
'codeName': 'TooManyLogicalSessions'}Unable to add session ID 85b35e6c-fc83-41d1-915b-83e6841c5467 - 47DEQpj8HBSa+/TImW+5JCeuQeRkm5NMpJWZG3hSuFU= into the cache because the number of active sessions is too high, full error: {'ok': 0.0, 'errmsg': 'Unable to add session ID 85b35e6c-fc83-41d1-915b-83e6841c5467 - 47DEQpj8HBSa+/TImW+5JCeuQeRkm5NMpJWZG3hSuFU= into the cache because the number of active sessions is too high',
'code': 261,
'codeName': 'TooManyLogicalSessions'}
As the error message, It seems I just opened too many sessions.
How can I check current active sessions in PyMongoDB? How can I close all sessions? Or how can I further debug this problem?
Thank you very much! My active sessions' number and my dataset code sample is offered as shown below:
1. Check the active sessions
My active sessions (db.serverStatus().connections) are indeed increasing:
{
current: 6,
available: 51194,
totalCreated: 199,
active: 2,
threaded: 6,
exhaustIsMaster: 0,
exhaustHello: 1,
awaitingTopologyChanges: 1
}
to
{
current: 156,
available: 51044,
totalCreated: 361,
active: 54,
threaded: 156,
exhaustIsMaster: 0,
exhaustHello: 51,
awaitingTopologyChanges: 51
}
and 5 minutes later, my program broke. (I don't know how to get this result from PymongoDB instead of mongosh. So I can only check it manually periodly. At this moment It seems there are still 51044 availiable, and should not be consumed up so quickly only in 5 minutes.)
2. my sample dataset code
like:
from torch.utils.data import Dataset
from pymongo import MongoClient
class MongoDataset(Dataset):
def __init__(self, dbName):
client = MongoClient(host = '127.0.0.1', port = 27017, connect=False)
db = client[dbName]
self.dataTable = db["dataTable"]
def getData(self, _id):
return self.dataTable.find_one({"_id" : _id})
def __len__(self):
return self.dataTable.estimated_document_count()
This class will be automatically forked, and recreated.