I've a simple data structure (the Transaction referenced below) to be inserted into mongodb:
{"amount":111,"debitAcc":"588188286231743e7d5c923d","type":"CHARGE"}
An I got the following error stackļ¼
com.mongodb.MongoSocketReadException: Prematurely reached end of stream
at com.mongodb.connection.SocketStream.read(SocketStream.java:88)
at com.mongodb.connection.InternalStreamConnection.receiveResponseBuffers(InternalStreamConnection.java:494)
at com.mongodb.connection.InternalStreamConnection.receiveMessage(InternalStreamConnection.java:224)
at com.mongodb.connection.UsageTrackingInternalConnection.receiveMessage(UsageTrackingInternalConnection.java:96)
at com.mongodb.connection.DefaultConnectionPool$PooledConnection.receiveMessage(DefaultConnectionPool.java:440)
at com.mongodb.connection.WriteCommandProtocol.receiveMessage(WriteCommandProtocol.java:262)
at com.mongodb.connection.WriteCommandProtocol.execute(WriteCommandProtocol.java:104)
at com.mongodb.connection.InsertCommandProtocol.execute(InsertCommandProtocol.java:67)
at com.mongodb.connection.InsertCommandProtocol.execute(InsertCommandProtocol.java:37)
at com.mongodb.connection.DefaultServer$DefaultServerProtocolExecutor.execute(DefaultServer.java:168)
at com.mongodb.connection.DefaultServerConnection.executeProtocol(DefaultServerConnection.java:289)
at com.mongodb.connection.DefaultServerConnection.insertCommand(DefaultServerConnection.java:118)
at com.mongodb.operation.InsertOperation.executeCommandProtocol(InsertOperation.java:76)
at com.mongodb.operation.BaseWriteOperation$1.call(BaseWriteOperation.java:139)
at com.mongodb.operation.BaseWriteOperation$1.call(BaseWriteOperation.java:133)
at com.mongodb.operation.OperationHelper.withConnectionSource(OperationHelper.java:422)
at com.mongodb.operation.OperationHelper.withConnection(OperationHelper.java:413)
at com.mongodb.operation.BaseWriteOperation.execute(BaseWriteOperation.java:133)
at com.mongodb.operation.BaseWriteOperation.execute(BaseWriteOperation.java:60)
at com.mongodb.Mongo.execute(Mongo.java:845)
at com.mongodb.Mongo$2.execute(Mongo.java:828)
at com.mongodb.DBCollection.executeWriteOperation(DBCollection.java:342)
at com.mongodb.DBCollection.insert(DBCollection.java:337)
at com.mongodb.DBCollection.insert(DBCollection.java:328)
at org.mongodb.morphia.DatastoreImpl.saveDocument(DatastoreImpl.java:1297)
at org.mongodb.morphia.DatastoreImpl.tryVersionedUpdate(DatastoreImpl.java:1340)
at org.mongodb.morphia.DatastoreImpl.save(DatastoreImpl.java:1286)
at org.mongodb.morphia.DatastoreImpl.save(DatastoreImpl.java:775)
at org.mongodb.morphia.DatastoreImpl.save(DatastoreImpl.java:758)
My mongodb version is 3.4.2.
Intereting is that I don't have the issue on my local dev environment (mint linux 18.1). But it just can't work on my SIT environment, which is a ubuntu 16.04
Any idea?
Updates with code to insert the document
Where transactionDao.save(...) implementation could be found at:
https://github.com/actframework/act-morphia/blob/master/src/main/java/act/db/morphia/MorphiaDaoBase.java#L206
update 2
The system works with other writes (even with much bigger records)
Most of the time this is a result of timeouts with long reads\writes.
try to increase the timeouts or remove them completely:
MongoClientOptions.Builder options_builder = new MongoClientOptions.Builder();
options_builder.maxConnectionIdleTime(<some_long_time>);
MongoClientOptions options = options_builder.build();
MongoClient mongo_db = new MongoClient ("your.db.address", options);
I don't have the root cause, but in the end I get the issue fixed by changing a field type from BigDecimal to double.
The issue is found in our SIT environmet and one developer's windows environment which has a mongodb cluser setup.
The most likely reason is the compatibility version is set too low.
Try db.adminCommand( { setFeatureCompatibilityVersion: "3.4" } )
https://docs.mongodb.com/manual/reference/command/setFeatureCompatibilityVersion/
Related
I have a simple query loop that gets a MongoCursorNotFoundException after processing about 44,000 of 96,945 documents in around 93 minutes.
MongoIterable<MasterDocument> query = masterCollection.find().noCursorTimeout(true);
for (MasterDocument masterDocument : query) { ... do some stuff ... }
The "do some stuff" part takes a while, which is why the entire loop takes so long.
My problem is that I get this exception after handling maybe half of the documents in the collection.
I am running both the client application and the mongod server locally on my Windows 10 laptop, accessing the server via localhost.
The server log shows lots of messages like this:
{"t":{"$date":"2021-01-04T20:21:35.510-08:00"},"s":"I", "c":"COMMAND", "id":51803, "ctx":"conn27","msg":"Slow query","attr":{"type":"command","ns":"master_database.MasterCollection","command":{"find":"MasterCollection","filter":{"hashCode":1753339282},"$db":"master_database","lsid":{"id":{"$uuid":"6a252f51-2c6e-4c01-ae03-1a80aab109e0"}}},"planSummary":"COLLSCAN","keysExamined":0,"docsExamined":96944,"cursorExhausted":true,"numYields":96,"nreturned":0,"queryHash":"DBC59907","planCacheKey":"DBC59907","reslen":121,"locks":{"ReplicationStateTransition":{"acquireCount":{"w":97}},"Global":{"acquireCount":{"r":97}},"Database":{"acquireCount":{"r":97}},"Collection":{"acquireCount":{"r":97}},"Mutex":{"acquireCount":{"r":1}}},"storage":{},"protocol":"op_msg","durationMillis":147}}
The last of these messages is followed by:
{"t":{"$date":"2021-01-04T20:21:35.521-08:00"},"s":"I", "c":"NETWORK", "id":22944, "ctx":"conn27","msg":"Connection ended","attr":{"remote":"127.0.0.1:58990","connectionId":27,"connectionCount":14}}
{"t":{"$date":"2021-01-04T20:21:35.522-08:00"},"s":"I", "c":"NETWORK", "id":22944, "ctx":"conn26","msg":"Connection ended","attr":{"remote":"127.0.0.1:58989","connectionId":26,"connectionCount":13}}
{"t":{"$date":"2021-01-04T20:21:35.922-08:00"},"s":"I", "c":"-", "id":20883, "ctx":"conn25","msg":"Interrupted operation as its client disconnected","attr":{"opId":310196}}
I have tried:
Using "noCursorTimeout(true)" on the query cursor (as shown above)
Starting the server with "mongod --setParameter localLogicalSessionTimeoutMinutes=240". This last seems to have caused additional log messages that say "error":"Location13111: wrong type for field (expireAfterSeconds) long != int"
I am using mongod 4.4 and the latest mongo java api.
You may need to increase the default cursor idle timeout to bigger value in all shards and mongos:
check the parameter(default is 10 min = 600000 ms ):
use admin
db.runCommand({getParameter:1, cursorTimeoutMillis: 1})
and update to bigger value:
use admin
db.runCommand({setParameter:1, cursorTimeoutMillis: 600000000 })
also the COLSCAN in your logs indicate that you dont use indexes in your query , maybe you need to create one on "hashCode" ...
Thanks for the response.
It turned out that my application ran to completion once I started mongod with "--setParameter localLogicalSessionTimeoutMinutes=240, despite the error message that I saw in the console log.
You are absolutely right that I should have an index on "hashCode". (I had one before but forgot to recreate it after recreating the collection.)
I'm currently working on a simple python CRUD script to check MongoDB out. Turns out I'm liking it a lot, but I have found myself being unable to work with MongoDB transactions. Everytime I try to start a transaction an exception is thrown saying:
This MongoDB deployment does not support retryable writes. Please add retryWrites=false to your connection string.
And, eventhough I've already added that option to my connection string:
self._client = MongoClient('mongodb://localhost/?retryWrites=false')
self._db = self._client.workouts
self._collection = self._db.workouts
That error is still popping up when running the following lines of code:
with self._client.start_session() as s:
with s.start_transaction():
self._collection.delete_one({'_id': id}, session=s)
next = self._collection.find_one({'_id': next_id}, session=s)
return next
What can I do?
I'm running python 3.7.3, pymongo 3.9.0 and MongoDB 4.0.12.
I followed this tutorial to deploy a sample application to Heroku. I just added the below method in MyResource class and returned the result from it instead of "Hello World" from getIt() method. I'm connecting to an atlas free tier cluster:
static String getMessage() {
MongoClient mongoClient = new MongoClient(new MongoClientURI("mongodb://<USER>:<PASSWORD>#cluster0-shard-00-00-2lbue.mongodb.net:27017,cluster0-shard-00-01-2lbue.mongodb.net:27017,cluster0-shard-00-02-2lbue.mongodb.net:27017/test?ssl=true&replicaSet=Cluster0-shard-0&authSource=admin"));
DB database = mongoClient.getDB("mastery");
DBCollection collection = database.getCollection("summary");
DBObject query = new BasicDBObject("_id", new ObjectId("5c563fa2645d6b444c018dcb"));
DBCursor cursor = collection.find(query);
return (String)cursor.one().get("message");
}
This is the driver I'm using:
<dependency>
<groupId>org.mongodb</groupId>
<artifactId>mongodb-driver</artifactId>
<version>3.9.1</version>
</dependency>
This is my import:
import com.mongodb.*;
The application works fine from my local system. But I face the below error when I deploy the application to Heroku and hit the service:
INFO: Exception in monitor thread while connecting to server cluster0-shard-00-01-2lbue.mongodb.net:27017
com.mongodb.MongoCommandException: Command failed with error 8000 (AtlasError): 'no SNI name sent, make sure using a MongoDB 3.4+ driver/shell.' on server cluster0-shard-00-01-2lbue.mongodb.net:27017. The full response is { "ok" : 0, "errmsg" : "no SNI name sent, make sure using a MongoDB 3.4+ driver/shell.", "code" : 8000, "codeName" : "AtlasError" }
at com.mongodb.internal.connection.ProtocolHelper.getCommandFailureException(ProtocolHelper.java:179)
at com.mongodb.internal.connection.InternalStreamConnection.receiveCommandMessageResponse(InternalStreamConnection.java:299)
at com.mongodb.internal.connection.InternalStreamConnection.sendAndReceive(InternalStreamConnection.java:255)
at com.mongodb.internal.connection.CommandHelper.sendAndReceive(CommandHelper.java:83)
at com.mongodb.internal.connection.CommandHelper.executeCommand(CommandHelper.java:33)
at com.mongodb.internal.connection.InternalStreamConnectionInitializer.initializeConnectionDescription(InternalStreamConnectionInitializer.java:106)
at com.mongodb.internal.connection.InternalStreamConnectionInitializer.initialize(InternalStreamConnectionInitializer.java:63)
at com.mongodb.internal.connection.InternalStreamConnection.open(InternalStreamConnection.java:127)
at com.mongodb.internal.connection.DefaultServerMonitor$ServerMonitorRunnable.run(DefaultServerMonitor.java:117)
at java.lang.Thread.run(Thread.java:748)
What is this SNI name? I can understand that the drivers are able to pick it from my machine, but not from Heroku machine. But I'm clueless on how to go about solving this! Is there a way to configure Heroku to reveal the SNI name when the driver asks for it? Can we get this value manually from somewhere in Heroku and directly feed it to the MongoDB drivers? Any help is appreciated.
EDIT:
It turned out that the client mentions the SNI name of the server it wishes to connect to as part of TLS security. And there seems to be a way to manually indicate the name in python driver. Is there any way to do this from java? Still puzzled why this is not an issue when running the app locally.
The code I was using to connect to the cluster turned out to be wrong. I followed the directions from the docs and it mentioned this:
To connect to an Atlas M0 (Free Tier) cluster, you must use Java
version 8 or greater and use a Java driver version that supports
MongoDB 3.4.
So I changed the java version to 1.8 in system.properties file:
java.runtime.version=1.8
It was earlier set to 1.7. I was also getting a deprecation warning on one of the methods I had used. So I again followed the docs to use the latest code and it worked like a charm.
The real takeaway here is to refer to the official docs everytime :)
1. {ok,P}= mongoc:connect({rs, <<"dev_mongodb">>, [ "dev_mongodb001:27017", "dev_mongodb002:27017"]}, [{name, mongopool}, {register, mongotopology}, { rp_mode, primary},{ rp_tags, [{tag,1}]}], [{login, <<"root">>}, {password, <<"mongoadmin">>}, {database, <<"admin">>}]).
2. {ok, Pool} = mc_topology:get_pool(P, []).
3. mongoc:find(Pool, {<<"DoctorLBS">>, <<"mongoMessage">>}, #{<<"type">> => <<"5">>}).
I used latest version in github, and got an error at step 3.
It seems my selector is not valid, is there any example of how to use mongodb-erlang ?
My mongodb version is 3.2.6, auth type is SCRAM-SHA1.
mongoc:find(Pool, <<"mongoMessage">>, #{<<"type">> => <<"5">>}).
I tried this in rs and single mode, still got this error.
Is there any other simple way to connect and read?
I just need to read some data once from mongo when my erlang program start, no other actions.
Todays version of mongo does not support tuple colldb due to new query api introduced in mongo 2.6
You should connect to DoctorLBS database instead, and than use
mongoc:find(Pool, <<"mongoMessage">>, #{<<"type">> => <<"5">>}).
I'm having an issue with mongodb 2.6.5 and grails 2.4.4 that I can't resolve. For the sake of isolating the problem I created a simple 2.4.4 grails app, installed the grails mongodb plugin (compile ":mongodb:3.0.2"), commented out the hibernate dependencies, added my mongodb datasource, and set up a simple domain class (com.nerds.Nerd). When I generate-all and then start the app and navigate to the NerdController CRUD page I get the following error every time:
MongoTimeoutException occurred when processing request: [GET] /MONGO/nerd/index
Timed out while waiting to connect after 10000 ms. Stacktrace follows:
com.mongodb.MongoTimeoutException: Timed out while waiting to connect after 10000 ms
I can access mongo via http using http://localhost:28017/
I have also tested manually adding data and querying from mongo. This all works fine.
In the debug log prior to the timeout it looks like GORM aquired a mongo session and then tried rolling back a transaction.
DatastoreTransactionManager:128 - Found thread-bound Session [org.grails.datastore.mapping.mongo.MongoSession#e47ee6] for Datastore transaction
DatastoreTransactionManager:128 - Creating new transaction with name [null]: PROPAGATION_REQUIRED,ISOLATION_DEFAULT,readOnly
DatastoreTransactionManager:128 - Initiating transaction rollback
DatastoreTransactionManager:128 - Rolling back Datastore transaction on Session [org.grails.datastore.mapping.mongo.MongoSession#e47ee6]
DatastoreTransactionManager:128 - Resuming suspended transaction after completion of inner transaction
Any insight would be helpful. Thanks
edit: The mongo datasource is pretty simple. I'm using the correct port.
From the mongo log:
014-11-18T13:10:13.388-0900 [initandlisten] MongoDB starting : pid=17275 port=27017 dbpath=/var/lib/mongodb 32-bit host=enterprise
from DataSource.groovy
grails { mongo { host = 'localhost' port = 27017 databaseName = 'mydb' } }
I'm fairly certain the issue was on the mongod side. I stopped the mongo daemon, put it into high verbose debug mode (using mongod -vvvv command), and when I tried to replicate the issue while watching the console output, the issue did not happen. I'm not entirely sure what the exact cause of the timeout was, but its not happening now. Thanks for the responses.