mongodb replicaset failed syncing - mongodb

Question

Ever since I have added a new database into the mongodb, it stopped syncing replicaSet secondary instances, i.e. database name appear when running show dbs yet appear as (empty)
There is a repeating error in the log file at the secondary which also appear in
"errmsg" : "syncTail: ...
below is the output of rs.Status() on primary
PRIMARY> rs.status()
{
"set" : "contoso_db_set",
"date" : ISODate("2012-11-01T13:05:22Z"),
"myState" : 1,
"syncingTo" : "dbuse1d.int.contoso.com:27017",
"members" : [
{
"_id" : 0,
"name" : "dbuse1a.int.contoso.com:27017",
"health" : 1,
"state" : 1,
"stateStr" : "PRIMARY",
"optime" : {
"t" : 1351775119000,
"i" : 2
},
"optimeDate" : ISODate("2012-11-01T13:05:19Z"),
"self" : true
},
{
"_id" : 1,
"name" : "dbuse1d.int.contoso.com:27017",
"health" : 1,
"state" : 2,
"stateStr" : "SECONDARY",
"uptime" : 4108139,
"optime" : {
"t" : 1351405977000,
"i" : 12
},
"optimeDate" : ISODate("2012-10-28T06:32:57Z"),
"lastHeartbeat" : ISODate("2012-11-01T13:05:21Z"),
"pingMs" : 1,
"errmsg" : "syncTail: 10068 invalid operator: $oid, syncing: { ts: Timestamp 1351576230000|1, h: -2878874165043062831, op: \"i\", ns: \"new_contoso_db.accounts\", o: { _id: { $oid: \"4f79a1d1d4941d3755000000\" }, delegation: [ \"nE/UhsnmZ1BCCB+tiiS8fjjNwkxbND5PwESsaXeuaJw=\""
},
{
"_id" : 2,
"name" : "dbuse1a.int.contoso.com:8083",
"health" : 1,
"state" : 7,
"stateStr" : "ARBITER",
"uptime" : 10671267,
"optime" : {
"t" : 0,
"i" : 0
},
"optimeDate" : ISODate("1970-01-01T00:00:00Z"),
"lastHeartbeat" : ISODate("2012-11-01T13:05:21Z"),
"pingMs" : 0
}
],
"ok" : 1
}
PRIMARY>

Answer 1

The solution I have found was to delete the entire database from secondary
# rm -rf /data/db
# mkdir -p /data/db
And then restarting mongo and setting up the the replicaSet.
See more at Mongo's doc
What to do on a RS102 sync error
If one of your members has been offline and is now too far behind to
catch up, you will need to resync. There are a number of ways to do
this.
Perform a full resync. If you stop the failed mongod, delete all data in the dbpath (including subdirectories), and restart it, it will
automatically resynchronize itself. Obviously it would be better/safer
to back up the data first. If disk space is adequate, simply move it
to a backup location on the machine if appropriate. Resyncing may take
a long time if the database is huge or the network slow – even
idealized one terabyte of data would require three hours to transmit
over gigabit ethernet.*