I'm using Grails 2.4.4, mongo plugin 3.0.2, MongoDB 2.4.10 using a remote database connection.
grails {
mongo {
host = "11.12.13.14" // A remote server IP
port = 27017
databaseName = "blogger"
username = "blog"
password = "xyz"
options {
autoConnectRetry = true
connectTimeout = 3000
connectionsPerHost = 40
socketTimeout = 120000
threadsAllowedToBlockForConnectionMultiplier = 5
maxAutoConnectRetryTime=5
maxWaitTime=120000
}
}
}
In a part of our application, a service method iterates over a 20,000 user's and sends them an email:
Person.withCriteria { // Line 323
eq("active", true)
order("dateJoined", "asc")
}.each { personInstance ->
// Code to send an email which takes an average of 1 second
}
After executing this for some 6000 user, I'm getting a MongoDB cursor exception:
2015-04-11 07:31:14,218 [quartzScheduler_Worker-1] ERROR listeners.ExceptionPrinterJobListener - Exception occurred in job: Grails Job
org.quartz.JobExecutionException: com.mongodb.MongoException$CursorNotFound: Cursor 1337814790631604331 not found on server 11.12.13.14:27017 [See nested exception: com.mongodb.MongoException$CursorNotFound: Cursor 1337814790631604331 not found on server 11.12.13.14:27017]
at grails.plugins.quartz.GrailsJobFactory$GrailsJob.execute(GrailsJobFactory.java:111)
at org.quartz.core.JobRunShell.run(JobRunShell.java:202)
at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:573)
Caused by: com.mongodb.MongoException$CursorNotFound: Cursor 1337814790631604331 not found on server 11.12.13.14:27017
at com.mongodb.QueryResultIterator.throwOnQueryFailure(QueryResultIterator.java:218)
at com.mongodb.QueryResultIterator.init(QueryResultIterator.java:198)
at com.mongodb.QueryResultIterator.initFromQueryResponse(QueryResultIterator.java:176)
at com.mongodb.QueryResultIterator.getMore(QueryResultIterator.java:141)
at com.mongodb.QueryResultIterator.hasNext(QueryResultIterator.java:127)
at com.mongodb.DBCursor._hasNext(DBCursor.java:551)
at com.mongodb.DBCursor.hasNext(DBCursor.java:571)
at org.grails.datastore.mapping.mongo.query.MongoQuery$MongoResultList$1.hasNext(MongoQuery.java:1893)
at com.test.person.PersonService.sendWeeklyEmail(PersonService.groovy:323)
at com.test.WeeklyJob.execute(WeeklyJob.groovy:41)
at grails.plugins.quartz.GrailsJobFactory$GrailsJob.execute(GrailsJobFactory.java:104)
... 2 more
I looked for documentation and found that cursor automatically get closed in 20 minutes and when I confirmed it with the logs, this exception came exactly after 20 minutes.
But this behaviour of auto-close in 20 minutes is applicable for inactive cursor but here the cursor is active.
UPDATE:
I read some articles and found that, this could be an issue of TCP keepalive timeout. So we changed the TCP keepalive timeout to 2 minutes from default 2 hours, but it still doesn't solve the problem.
Looks like a compatibility issue with MonoDB on this server. Read more on the Jira for details https://jira.mongodb.org/browse/SERVER-18439
Hope this helps to someone else!
Related
I have a simple query loop that gets a MongoCursorNotFoundException after processing about 44,000 of 96,945 documents in around 93 minutes.
MongoIterable<MasterDocument> query = masterCollection.find().noCursorTimeout(true);
for (MasterDocument masterDocument : query) { ... do some stuff ... }
The "do some stuff" part takes a while, which is why the entire loop takes so long.
My problem is that I get this exception after handling maybe half of the documents in the collection.
I am running both the client application and the mongod server locally on my Windows 10 laptop, accessing the server via localhost.
The server log shows lots of messages like this:
{"t":{"$date":"2021-01-04T20:21:35.510-08:00"},"s":"I", "c":"COMMAND", "id":51803, "ctx":"conn27","msg":"Slow query","attr":{"type":"command","ns":"master_database.MasterCollection","command":{"find":"MasterCollection","filter":{"hashCode":1753339282},"$db":"master_database","lsid":{"id":{"$uuid":"6a252f51-2c6e-4c01-ae03-1a80aab109e0"}}},"planSummary":"COLLSCAN","keysExamined":0,"docsExamined":96944,"cursorExhausted":true,"numYields":96,"nreturned":0,"queryHash":"DBC59907","planCacheKey":"DBC59907","reslen":121,"locks":{"ReplicationStateTransition":{"acquireCount":{"w":97}},"Global":{"acquireCount":{"r":97}},"Database":{"acquireCount":{"r":97}},"Collection":{"acquireCount":{"r":97}},"Mutex":{"acquireCount":{"r":1}}},"storage":{},"protocol":"op_msg","durationMillis":147}}
The last of these messages is followed by:
{"t":{"$date":"2021-01-04T20:21:35.521-08:00"},"s":"I", "c":"NETWORK", "id":22944, "ctx":"conn27","msg":"Connection ended","attr":{"remote":"127.0.0.1:58990","connectionId":27,"connectionCount":14}}
{"t":{"$date":"2021-01-04T20:21:35.522-08:00"},"s":"I", "c":"NETWORK", "id":22944, "ctx":"conn26","msg":"Connection ended","attr":{"remote":"127.0.0.1:58989","connectionId":26,"connectionCount":13}}
{"t":{"$date":"2021-01-04T20:21:35.922-08:00"},"s":"I", "c":"-", "id":20883, "ctx":"conn25","msg":"Interrupted operation as its client disconnected","attr":{"opId":310196}}
I have tried:
Using "noCursorTimeout(true)" on the query cursor (as shown above)
Starting the server with "mongod --setParameter localLogicalSessionTimeoutMinutes=240". This last seems to have caused additional log messages that say "error":"Location13111: wrong type for field (expireAfterSeconds) long != int"
I am using mongod 4.4 and the latest mongo java api.
You may need to increase the default cursor idle timeout to bigger value in all shards and mongos:
check the parameter(default is 10 min = 600000 ms ):
use admin
db.runCommand({getParameter:1, cursorTimeoutMillis: 1})
and update to bigger value:
use admin
db.runCommand({setParameter:1, cursorTimeoutMillis: 600000000 })
also the COLSCAN in your logs indicate that you dont use indexes in your query , maybe you need to create one on "hashCode" ...
Thanks for the response.
It turned out that my application ran to completion once I started mongod with "--setParameter localLogicalSessionTimeoutMinutes=240, despite the error message that I saw in the console log.
You are absolutely right that I should have an index on "hashCode". (I had one before but forgot to recreate it after recreating the collection.)
I'm trying to create an alarm system for my application, that will trigger when one of the services (e.g. MongoDB) is not working.
What I'm doing is, once the application is started, I shut down my MongoDB server and try to connect to it, but instead of receiving an error my application just gets stuck into the execution of the method. The server console looks like something is in execution.
My current code (coffeescript) is:
checkMongoService: ()->
mongo = Npm.require 'mongodb'
assert = Npm.require 'assert'
url = 'mongodb://....'
mongo.connect url, (err, db) ->
assert.equal null, err
console.log 'Connected correctly to server'
db.close()
return
I've also been trying by doing a simple
Meteor.users.find().count();
or using MongoInternals with
testConnection = new MongoInternals.RemoteCollectionDriver("mongodb://...);
but still same issue, when mongo is not running no error is thrown and the console stops to work. If then I start Mongo again, it will just return the result (in this case the log 'Connected correctly to server')
Something that I've noticed is if I try with meteor shell to execute testConnection = new MongoInternals.RemoteCollectionDriver("mongodb://...); I get an error "Error: failed to connect to [127.0.0.1:27017]"
TL;DR
Do you might have an idea on how I can check if mongo is reachable or do you know if I'm doing something wrong with the code above?
Try setting the timeouts to be a bit shorter than the default 30 seconds:
mongo.connect(url, {
connectTimeoutMS: 1000,
socketTimeoutMS: 1000,
reconnectTries: 1
}, function(err, db) {...}
(Full set of connection params are here)
Meteor.status().status
from the docs
This method returns the status of the connection between the client and the server. The return value is an object with the following fields:
connected (Boolean)
True if currently connected to the server. If false, changes and method invocations will be queued up until the connection is reestablished.
status (String)
Blockquote
Describes the current reconnection status. The possible values are connected (the connection is up and running), connecting (disconnected and trying to open a new connection), failed (permanently failed to connect; e.g., the client and server support different versions of DDP), waiting (failed to connect and waiting to try to reconnect) and offline (user has disconnected the connection).
https://docs.meteor.com/api/connections.html
How to set the use connectionstring which usess mongodb in production.
development:
grails:
mongodb:
connectionString: "mongodb://localhost:27017/fanfest"
production:
dataSource:
dbCreate: update
url: jdbc:h2:./prodDb;MVCC=TRUE;LOCK_TIMEOUT=10000;DB_CLOSE_ON_EXIT=FALSE
properties:
jmxEnabled: true
initialSize: 5
maxActive: 50
minIdle: 5
maxIdle: 25
maxWait: 10000
maxAge: 600000
timeBetweenEvictionRunsMillis: 5000
minEvictableIdleTimeMillis: 60000
validationQuery: SELECT 1
validationQueryTimeout: 3
validationInterval: 15000
testOnBorrow: true
testWhileIdle: true
testOnReturn: false
jdbcInterceptors: ConnectionState
defaultTransactionIsolation: 2 # TRANSACTION_READ_COMMITTED
Working with Grails 3 version. Able to connect with mongodb in development environment. Kindly provide some suggestions to set the mongodb in production environment.
You currently have development pointing at a mongo instance. The production configuration is pointed at the in memory H2 database. If you would like to configure a mongo database for your production environment may I suggest you take a look at Getting Started guide for Mongo and GORM.
In the production section of your configuration file you can use the connection string parameter as follows:
production {
grails {
mongodb {
connectionString = "mongodb://localhost:27017/PROD_fanfest"
}
}
}
Please note I have used your development URL but changed the table name since I recommend you keep the development database and production database separate. Configuring production datasource to use Mongo is that easy. There are more configuration options described in the Getting Started documentation.
Relevant information on configuring options below:
options {
connectionsPerHost = 10 // The maximum number of connections allowed per host
threadsAllowedToBlockForConnectionMultiplier = 5
maxWaitTime = 120000 // Max wait time of a blocking thread for a connection.
connectTimeout = 0 // The connect timeout in milliseconds. 0 == infinite
socketTimeout = 0 // The socket timeout. 0 == infinite
socketKeepAlive = false // Whether or not to have socket keep alive turned on
writeConcern = new com.mongodb.WriteConcern(0, 0, false) // Specifies the number of servers to wait for on the write operation, and exception raising behavior
sslEnabled = false // Specifies if the driver should use an SSL connection to Mongo
socketFactory = … // Specifies the SocketFactory to use for creating connections
}
I'm having an issue with mongodb 2.6.5 and grails 2.4.4 that I can't resolve. For the sake of isolating the problem I created a simple 2.4.4 grails app, installed the grails mongodb plugin (compile ":mongodb:3.0.2"), commented out the hibernate dependencies, added my mongodb datasource, and set up a simple domain class (com.nerds.Nerd). When I generate-all and then start the app and navigate to the NerdController CRUD page I get the following error every time:
MongoTimeoutException occurred when processing request: [GET] /MONGO/nerd/index
Timed out while waiting to connect after 10000 ms. Stacktrace follows:
com.mongodb.MongoTimeoutException: Timed out while waiting to connect after 10000 ms
I can access mongo via http using http://localhost:28017/
I have also tested manually adding data and querying from mongo. This all works fine.
In the debug log prior to the timeout it looks like GORM aquired a mongo session and then tried rolling back a transaction.
DatastoreTransactionManager:128 - Found thread-bound Session [org.grails.datastore.mapping.mongo.MongoSession#e47ee6] for Datastore transaction
DatastoreTransactionManager:128 - Creating new transaction with name [null]: PROPAGATION_REQUIRED,ISOLATION_DEFAULT,readOnly
DatastoreTransactionManager:128 - Initiating transaction rollback
DatastoreTransactionManager:128 - Rolling back Datastore transaction on Session [org.grails.datastore.mapping.mongo.MongoSession#e47ee6]
DatastoreTransactionManager:128 - Resuming suspended transaction after completion of inner transaction
Any insight would be helpful. Thanks
edit: The mongo datasource is pretty simple. I'm using the correct port.
From the mongo log:
014-11-18T13:10:13.388-0900 [initandlisten] MongoDB starting : pid=17275 port=27017 dbpath=/var/lib/mongodb 32-bit host=enterprise
from DataSource.groovy
grails { mongo { host = 'localhost' port = 27017 databaseName = 'mydb' } }
I'm fairly certain the issue was on the mongod side. I stopped the mongo daemon, put it into high verbose debug mode (using mongod -vvvv command), and when I tried to replicate the issue while watching the console output, the issue did not happen. I'm not entirely sure what the exact cause of the timeout was, but its not happening now. Thanks for the responses.
I just set up a replica set in Mongo (prod environment). I'm now getting a lot of exceptions like below (clipped).
I went into mongo and ran a serverStatus command on my primary mongo node and only have about 300 connections going, so it's hardly working.
Below are my connection option settings in my server code:
auto_connect_retry = false
connections_per_host = 10
threads_multiplier = 10
max_wait_time = 120000
connect_timeout = 10000
socket_timeout = 0
Do I have something mis-configured?
Sep 9, 2013 8:31:26 PM com.mongodb.DBPortPool gotError
WARNING: emptying DBPortPool to /10.0.8.10:27017 b/c of error
java.net.SocketException: Connection timed out
at java.net.SocketInputStream.socketRead0(Native Method)
at java.net.SocketInputStream.read(SocketInputStream.java:146)
at java.io.BufferedInputStream.fill(BufferedInputStream.java:235)
at java.io.BufferedInputStream.read1(BufferedInputStream.java:275)
at java.io.BufferedInputStream.read(BufferedInputStream.java:334)
at org.bson.io.Bits.readFully(Bits.java:46)
at org.bson.io.Bits.readFully(Bits.java:33)
at org.bson.io.Bits.readFully(Bits.java:28)
at com.mongodb.Response.<init>(Response.java:40)
at com.mongodb.DBPort.go(DBPort.java:142)
at com.mongodb.DBPort.call(DBPort.java:92)
at com.mongodb.DBTCPConnector.innerCall(DBTCPConnector.java:244)
at com.mongodb.DBTCPConnector.call(DBTCPConnector.java:216)
at com.mongodb.DBApiLayer$MyCollection.__find(DBApiLayer.java:288)
at com.mongodb.DBApiLayer$MyCollection.__find(DBApiLayer.java:273)
at com.mongodb.DBCollection.findOne(DBCollection.java:347)
at com.mongodb.DBCollection.findOne(DBCollection.java:332)
at com.mongodb.casbah.MongoCollectionBase$class.findOneByID(MongoCollection.scala:232)
at com.mongodb.casbah.MongoCollection.findOneByID(MongoCollection.scala:866)
at com.novus.salat.dao.SalatDAO.findOneById(SalatDAO.scala:353)
at com.novus.salat.dao.ModelCompanion$class.findOneById(ModelCompanion.scala:173)
Generally a connection timeout occurs from one of the following in a replica set
1) All members are not able to communicate with each other
2) A program is connecting to replica for update and it is unable to send it to primary due to overload or 1st as well
3) All relicas are not in sync and one is lagging behind too much
4) Leader election is going on but not completed due to some reason
Please check if your relica set is consistent and all nodes are working by issuing rs.status() on primary node , also as earlier suggested check primary logs for more information