GridFS put command hangs from pymongo when used within a transaction - mongodb

Question

I am using GridFS to store some video files in my database. I have updated to MongoDB 4.0 and trying to use the multi collection transaction model. The problem that I am facing is that the put() command gridfs hangs the system. The way I am using it as follows:
client = pymongo.MongoClient(mongo_url)
db = client[db_name]
fs = gridfs.GridFS(db)
Now I try to use the transaction model as follows:
with db.client.start_session() as session:
try:
file_path = "video.mp4"
session.start_transaction()
with open(file_path, 'rb') as f:
fid = self.fs.put(f, metadata={'sequence_id': '0001'})
session.commit_transaction()
except Exception as e:
raise
finally:
session.end_session()
The issue is that the put command hangs for about a minute. It then returns but the commit fails. I have a feeling this is because the session object is not passed to the put command but I do not see any parameter in the help that takes the session as an input. After the hang, the test fails with the following stack:
Traceback (most recent call last):
session.commit_transaction()
File "/Users/xargon/anaconda/envs/deep/lib/python3.6/site-packages/pymongo/client_session.py", line 393, in commit_transaction
self._finish_transaction_with_retry("commitTransaction")
File "/Users/xargon/anaconda/envs/deep/lib/python3.6/site-packages/pymongo/client_session.py", line 457, in _finish_transaction_with_retry
return self._finish_transaction(command_name)
File "/Users/xargon/anaconda/envs/deep/lib/python3.6/site-packages/pymongo/client_session.py", line 452, in _finish_transaction
parse_write_concern_error=True)
File "/Users/xargon/anaconda/envs/deep/lib/python3.6/site-packages/pymongo/database.py", line 514, in _command
client=self.__client)
File "/Users/xargon/anaconda/envs/deep/lib/python3.6/site-packages/pymongo/pool.py", line 579, in command
unacknowledged=unacknowledged)
File "/Users/xargon/anaconda/envs/deep/lib/python3.6/site-packages/pymongo/network.py", line 150, in command
parse_write_concern_error=parse_write_concern_error)
File "/Users/xargon/anaconda/envs/deep/lib/python3.6/site-packages/pymongo/helpers.py", line 155, in _check_command_response
raise OperationFailure(msg % errmsg, code, response)
pymongo.errors.OperationFailure: Transaction 1 has been aborted.
EDIT
I tried replacing the put block as:
try:
gf = self.fs.new_file(metadata={'sequence_id': '0000'})
gf.write(f)
finally:
gf.close()
However, the hang happens again at gf.close()
I also tried instantiating the GridIn directly so that I could provide the session object and this fails as:
gin = gridfs.GridIn(root_collection=self.db["fs.files"], session=session)
gin.write(f)
gin.close()
This fails with the error message:
It is illegal to provide a txnNumber for command createIndexes

Answer 1

The issue is that the put command hangs for about a minute
The first attempt with self.fs.put() does not actually use transactions, it just took a while to upload the file.
Then after the upload was completed while attempting to commit the (empty) transaction, unfortunately the transaction reached the maximum lifetime limit due to the time taken to upload. See transactionLifetimeLimitSeconds. The default limit has been set to 60 seconds to set the maximum transaction runtime.
If you are considering raising this limit, keep in mind that as write volume enters MongoDB after a transactions snapshot has been created, WiredTiger cache pressure builds up. This cache pressure can only be released once a transaction commits. This is the reason behind the 60 second default limit.
It is illegal to provide a txnNumber for command createIndexes
First, operations that affect the database catalog such as creating or dropping a collection or an index, are not allowed in multi-document transactions.
The code for PyMongo GridFS is attempting to create indexes for the GridFS collection, which is prohibited on the server when used with transactional session (You may use session but not transactions).
I have updated to MongoDB 4.0 and trying to use the multi collection transaction model
I'd recommend to use a normal database operations with GridFS. MongoDB multi-document transaction is designed to be used for multi-documents atomicity. Which I don't think it's necessary in the file uploading case.