mongodb java driver not finding master after rs.stepDown() - mongodb

Question

Grails 2.2.1
MongoDB GORM plugin 1.2
When running with a replica set I am finding that stepping down the primary causes the following infinitely repeated errors in the java driver.
2013-09-09 16:00:19,655 [SimpleAsyncTaskExecutor-1] ERROR grails.app.services.plover.UserStreamAnalyzerService - Exception while handling status update event: org.springframework.data.mongodb.UncategorizedMongoDbException: not talking to master and retries used up; nested exception is com.mongodb.MongoException: not talking to master and retries used up
...
Caused by: org.springframework.data.mongodb.UncategorizedMongoDbException: not talking to master and retries used up; nested exception is com.mongodb.MongoException: not talking to master and retries used up
The stacktrace is here:
Caused by: com.mongodb.MongoException: not talking to master and retries used up
at com.mongodb.DBTCPConnector.innerCall(DBTCPConnector.java:314)
at com.mongodb.DBTCPConnector.innerCall(DBTCPConnector.java:316)
at com.mongodb.DBTCPConnector.call(DBTCPConnector.java:257)
at com.mongodb.DBApiLayer$MyCollection.__find(DBApiLayer.java:310)
at com.mongodb.DBApiLayer$MyCollection.__find(DBApiLayer.java:295)
at com.mongodb.DBCursor._check(DBCursor.java:368)
at com.mongodb.DBCursor._hasNext(DBCursor.java:459)
at com.mongodb.DBCursor._fill(DBCursor.java:518)
at com.mongodb.DBCursor.toArray(DBCursor.java:553)
at com.mongodb.DBCursor.toArray(DBCursor.java:542)
at org.grails.datastore.mapping.mongo.query.MongoQuery$MongoResultList.<init>(MongoQuery.java:908)
at org.grails.datastore.mapping.mongo.query.MongoQuery$36.doInDB(MongoQuery.java:536)
at org.grails.datastore.mapping.mongo.query.MongoQuery$36.doInDB(MongoQuery.java:508)
I have setup a local test environment to replicate this problem. Here is the config output:
{
"set" : "rsMesh",
"date" : ISODate("2013-09-10T01:08:20Z"),
"myState" : 2,
"syncingTo" : "macbookpro.local:27018",
"members" : [
{
"_id" : 1,
"name" : "macbookpro.local:27018",
"health" : 1,
"state" : 1,
"stateStr" : "PRIMARY",
"uptime" : 9940,
"optime" : {
"t" : 1378767619,
"i" : 5
},
"optimeDate" : ISODate("2013-09-09T23:00:19Z"),
"lastHeartbeat" : ISODate("2013-09-10T01:08:19Z"),
"lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"),
"pingMs" : 0
},
{
"_id" : 2,
"name" : "macbookpro.local:27019",
"health" : 1,
"state" : 7,
"stateStr" : "ARBITER",
"uptime" : 9914,
"lastHeartbeat" : ISODate("2013-09-10T01:08:19Z"),
"lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"),
"pingMs" : 0
},
{
"_id" : 3,
"name" : "macbookpro.local:27017",
"health" : 1,
"state" : 2,
"stateStr" : "SECONDARY",
"uptime" : 10392,
"optime" : {
"t" : 1378767619,
"i" : 5
},
"optimeDate" : ISODate("2013-09-09T23:00:19Z"),
"self" : true
}
],
"ok" : 1
}
Replica set configuration has been set in Datasource.groovy as per documentation:
grails {
mongo {
replicaSet = ["macbookpro.local:27017", "macbookpro.local:27018", "macbookpro.local:27019"]
}
}
So I am not running in standalone, the replica set servers are synched properly, and all servers are running properly. But if I force a new server to become the primary then all access appears to fail as if the driver was not redirecting queries to the new primary.
What am I missing?

Answer 1

No one ever answered this so I decided that the recovery from replica sets was non-functional. Instead I went to sharding and hoped that layering a mongos between the app server and the clusters themselves would provide enough protection.
The answer was a definitive "sort of". Before, when I stepped a primary down (or simulated a crash) the app server would hang indefinitely. Now I just get a few errors about not finding a primary in a given cluster and then the system recovers. Not a desirable solution but at least it is better than a permanent failure.