MongoDB unexpected shutdown and data loss - mongodb

Question

The weirdest thing happened. We have a mongodb instance with both journal and logs on separate hard drives running on a single amazon ec2 backed by ebs. The server unexpectedly shutdown this morning and when I looked at it, I noticed that mongo didn't start on startup.
I manually started mongo using the command
sudo mongod --fork --logpath /var/log/mongodb.log
It ran fine but when I looked at the database, all our our data had gone missing. I tried exiting out with the --repair option but it did nothing. This is what the logs from the initial mongod looks like:
2015-10-05T16:39:17.881+0000 [initandlisten] MongoDB starting :
pid=2757 port=27017 dbpath=/data/db 64-bit host=ip-172-31-59-166
2015-10-05T16:39:17.882+0000 [initandlisten] db version v2.6.8
2015-10-05T16:39:17.882+0000 [initandlisten] git version: 3abc04d6d4f71de00b57378e3277def8fd7a6700
2015-10-05T16:39:17.882+0000 [initandlisten] build info: Linux build5.nj1.10gen.cc 2.6.32-431.3.1.el6.x86_64 #1 SMP Fri Jan 3 21:39:27 UTC 2014 x86_64 BOOST_LIB_VERSION=1_49
2015-10-05T16:39:17.882+0000 [initandlisten] allocator: tcmalloc
2015-10-05T16:39:17.882+0000 [initandlisten] options: { processManagement: { fork: true }, systemLog: { destination: "file", path: "/var/log/mongodb.log" } }
2015-10-05T16:39:17.890+0000 [initandlisten] journal dir=/data/db/journal
2015-10-05T16:39:17.893+0000 [initandlisten] recover : no journal files present, no recovery needed
2015-10-05T16:39:17.893+0000 [initandlisten] preallocating a journal file /data/db/journal/prealloc.1
2015-10-05T16:39:28.081+0000 [initandlisten] preallocating a journal file /data/db/journal/prealloc.2
2015-10-05T16:39:40.496+0000 [initandlisten] allocating new ns file /data/db/local.ns, filling with zeroes...
2015-10-05T16:39:40.654+0000 [FileAllocator] allocating new datafile /data/db/local.0, filling with zeroes...
2015-10-05T16:39:40.654+0000 [FileAllocator] creating directory /data/db/_tmp
2015-10-05T16:39:40.659+0000 [FileAllocator] done allocating datafile /data/db/local.0, size: 64MB, took 0.002 secs
2015-10-05T16:39:40.664+0000 [initandlisten] build index on: local.startup_log properties: { v: 1, key: { _id: 1 }, name: "_id_", ns: "local.startup_log" }
2015-10-05T16:39:40.664+0000 [initandlisten] added index to empty collection
2015-10-05T16:39:40.665+0000 [initandlisten] command local.$cmd command: create { create: "startup_log", size: 10485760, capped: true } ntoreturn:1 keyUpdates:0 numYields:0 reslen:37 169ms
2015-10-05T16:39:40.669+0000 [initandlisten] waiting for connections on port 27017
This looks like there are no problems with the MongoDB initial start. But it also says that it's creating new ns file though old ones existed. What is happening here and how can I recover the data?

Answer 1

There's no indication anything was corrupted. The log indicates this instance was started on an empty volume.
Aside from the logPath, your MongoDB process is starting up with the default parameters. So, it's going to use /data/db as the location of the data directory. Based on how on your initial description of the problem, you may have been running the separate journal and data directories an EBS mount, i.e. your MongoDB config used --dbpath.
Check that you're using the same MongoDB configuration as was used
before the server went down
Check that your EBS volumes are mounted expected. You may use the mount option to confirm that the
path /data/db is pointing to the proper EBS volume.
And just to be sure... if the instance had used ephemeral drives the data would be lost when the instance shutdown. Sometime people confuse ephemeral drives with elastic block store, which is persistent.