How to log newly instantiated member status changes in a replica set - mongodb

Question

I'm trying to benchmark when a new mongo replica member
first communicates to my replica set.
change state from STARTUP2 -> RECOVERY
change state from RECOVERING -> SECONDARY
I'm specifically looking for exact duration on how long it took for the replica to run it's initial sync or the timestamps to parse such actions by a new member.
Is there a log that will give me this information?
I'm currently using the following script in mongo shell.
MAXSCRIPT_RUN_ITERATIONS = 1800 // approximately 30 minutes
for (var i = 0; i < MAXSCRIPT_RUN_ITERATIONS; i++) {
sleep(1000);
var currentdate = new Date();
var datetime = new Date().toLocaleString();
var rsStatus = rs.status()
var members = rsStatus.members
// Change this index depending on the # of members in your set
var resyncmember = members[0]
var resyncMemberState = resyncmember.stateStr
print("--------------")
print("Member Count: " + members.length + " " + datetime)
for (var n = 0; n < members.length; n++){
var member = members[n]
print("HOST: " + member.name + " State: " + member.stateStr)
}
if (resyncMemberState == "SECONDARY"){
print("########################")
var currentdate = new Date();
print("resyncMember finished: " + datetime)
print("########################")
break
}
}

Answer 1

You don't state what version you're running on, so I'll give information about 3.0 (3.2) should be the same.
Look through your mongodb log for entries like the following:
2016-02-25T14:59:43.684+0000 I REPL [rsSync] initial sync drop all databases
2016-02-25T14:59:43.684+0000 I REPL [rsSync] initial sync clone all databases
2016-02-25T14:59:43.688+0000 I REPL [rsSync] initial sync cloning db: admin
2016-02-25T14:59:43.833+0000 I REPL [rsSync] initial sync cloning db: db1
2016-02-26T10:31:33.763+0000 I REPL [rsSync] initial sync cloning db: test
2016-02-26T11:27:48.480+0000 I REPL [rsSync] initial sync data copy, starting syncup
2016-02-26T11:27:48.481+0000 I REPL [rsSync] oplog sync 1 of 3
2016-02-26T11:27:49.043+0000 I REPL [ReplicationExecutor] syncing from: xxxxxxxxxx:27017
2016-02-26T11:27:49.059+0000 I REPL [SyncSourceFeedback] replset setting syncSourceFeedback to xxxxxxxxxx:27017
2016-02-26T11:30:05.649+0000 I REPL [rsSync] oplog sync 2 of 3
2016-02-26T11:30:05.657+0000 I REPL [rsSync] initial sync building indexes
2016-02-26T11:30:05.657+0000 I REPL [rsSync] initial sync cloning indexes for : admin
2016-02-26T11:30:05.760+0000 I REPL [rsSync] initial sync cloning indexes for : db1
2016-02-26T11:43:37.262+0000 I REPL [rsSync] initial sync cloning indexes for : test
2016-02-26T11:43:48.271+0000 I REPL [rsSync] oplog sync 3 of 3
2016-02-26T11:43:48.319+0000 I REPL [rsSync] initial sync finishing up
2016-02-26T11:43:48.319+0000 I REPL [rsSync] replSet set minValid=56d03a74:1
2016-02-26T11:43:48.321+0000 I REPL [rsSync] initial sync done
2016-02-26T11:43:48.332+0000 I REPL [ReplicationExecutor] transition to RECOVERING
2016-02-26T11:43:48.348+0000 I REPL [ReplicationExecutor] transition to SECONDARY
As you can see, you see information for each stage of the initial sync process, and when the transition to a SECONDARY node occurs. You will see more lines than this, but I've cut down to show what you should look for.
You can view this realtime with a command similar to the following if you're on a Linux or Mac machine (make sure to run this before adding the new node or you might miss some lines)
tail -f /path/to/mongodb.log | grep REPL
This will watch your log file as entries are being added and only display the lines with the string REPL in it. Remember to kill the tail command once you've gotten the information you need.