Replica set read preference nearest is still slow - mongodb

Question

NOTICE: I've also posted this to dba.stackexchange.com. I'm not sure where this question belongs. If it's not here, tell me, and i'll delete it.
I'm testing my replica set, in particular the read preference, and i'm still getting slow reads even with a nearest read preference set.
For the purpose of this question, we can just assume there are 2 mongodb instances (there are in fact 3). PRIMARY is in Amsterdam (AMS). SECONDARY is in Singapore (SG).
I also have 2 application servers in those 2 locations where I am running my test scripts (node+mongoose).
From the AMS app server (so low latency with PRIMARY), if I run a
simple find query, I get a response in under a second.
However, If I run the same query from my app server in SG, I get response times of ~4-7 seconds.
If I just connect to the SG SECONDARY from SG app server, my query times drop to <1s, similar to (1).
Going back to a standard rep set setting (with nearest), and if I look at the logs, I've noticed that if I send a query to SG using 'nearest', i can see the query in there, but I also see an entry for that same query (but fewer lines) in the PRIMARY log. But it is interesting that there is always an entry in the PRIMARY log even when querying the SECONDARY. I'm not sure if that is somehow related.
So, if I connect directly to the nearest machine, I get a response <1s, but when using the replica set, unless i'm next to the PRIMARY, responses times are >4s.
My question is then, why? Have I set up my replica server incorrectly. Is it a problem on the client side (mongoose/mongodb), or is it in fact working as it is mean to, and i've misunderstood how it works under the hood?
Here are my files (apologies for the wall of text):
test.js
mongoose.connect(configDB.url);
var start = new Date().getTime();
Model.find({})
.exec(function(err, betas){
var end = new Date().getTime();
var time = end - start;
console.log(time/1000);
console.log('finished');
console.log(betas.length);
});
config, also tried with server and replSet options
module.exports = {
'url' : 'user:pwd#ip-primary/db,user:pwd#ip-secondary/db,user:pwd#ip-secondary/db'
}
Betas model
var betaSchema = mongoose.Schema({
.. some fields
}, { read: 'n' });
And the log output from doing a read query as above from the SG app server:
LOG OF PRIMARY:
2015-09-16T07:49:23.120-0400 D COMMAND [conn12520] run command db.$cmd { listIndexes: "betas", cursor: {} }
2015-09-16T07:49:23.120-0400 I COMMAND [conn12520] command db.$cmd command: listIndexes { listIndexes: "betas", cursor: {} } keyUpdates:0 writeConflicts:0 numYields:0 reslen:296 locks:{ Global: { acquireC
ount: { r: 2 } }, MMAPV1Journal: { acquireCount: { r: 1 } }, Database: { acquireCount: { r: 1 } }, Collection: { acquireCount: { R: 1 } } } 0ms
LOG OF SECONDARY
2015-09-16T07:49:19.368-0400 D QUERY [conn11831] [QLOG] Running query:
ns=db.betas limit=1000 skip=0
Tree: $and
Sort: {}
Proj: {}
2015-09-16T07:49:19.368-0400 D QUERY [conn11831] Running query: query: {} sort: {} projection: {} skip: 0 limit: 1000
2015-09-16T07:49:19.368-0400 D QUERY [conn11831] [QLOG] Beginning planning...
=============================
Options = INDEX_INTERSECTION KEEP_MUTATIONS
Canonical query:
ns=db.betas limit=1000 skip=0
Tree: $and
Sort: {}
Proj: {}
=============================
2015-09-16T07:49:19.368-0400 D QUERY [conn11831] [QLOG] Index 0 is kp: { _id: 1 } io: { v: 1, key: { _id: 1 }, name: "_id_", ns: "db.betas" }
2015-09-16T07:49:19.368-0400 D QUERY [conn11831] [QLOG] Index 1 is kp: { email: 1 } io: { v: 1, unique: true, key: { email: 1 }, name: "email_1", ns: "db.betas", background: true, safe: null }
2015-09-16T07:49:19.368-0400 D QUERY [conn11831] [QLOG] Rated tree:
$and
2015-09-16T07:49:19.368-0400 D QUERY [conn11831] [QLOG] Planner: outputted 0 indexed solutions.
2015-09-16T07:49:19.368-0400 D QUERY [conn11831] [QLOG] Planner: outputting a collscan:
COLLSCAN
---ns = db.betas
---filter = $and
---fetched = 1
---sortedByDiskLoc = 0
---getSort = []
2015-09-16T07:49:19.368-0400 D QUERY [conn11831] Only one plan is available; it will be run but will not be cached. query: {} sort: {} projection: {} skip: 0 limit: 1000, planSummary: COLLSCAN
2015-09-16T07:49:19.368-0400 D QUERY [conn11831] [QLOG] Not caching executor but returning 109 results.
2015-09-16T07:49:19.368-0400 I QUERY [conn11831] query db.betas planSummary: COLLSCAN ntoreturn:1000 ntoskip:0 nscanned:0 nscannedObjects:109 keyUpdates:0 writeConflicts:0 numYields:0 nreturned:109 resl
en:17481 locks:{ Global: { acquireCount: { r: 2 } }, MMAPV1Journal: { acquireCount: { r: 1 } }, Database: { acquireCount: { r: 1 } }, Collection: { acquireCount: { R: 1 } } } 0ms

Answer 1

The information in your output states that the database server is processing the query quickly. So this issue will likely lie outside of the database itself, probably in the client.
Are you running the same query multiple times and timing each execution?
I suspect that this may be due to some initial discovery on your MongoDB client's part - how is it to know what is the nearest before responding if it doesn't initially hit every node and time the responses?