The process of adding a mongoDb (2.4) cartridge to my OpenShift application seems to work fine but ends up with an error and no cartridge is added. It looks like a disk space problem (I have already mysql in the same application), but I freed plenty of space and, strangely enough, the problem only appears at the very end of setup. Here is the log (hiding login details):
Starting MongoDB cartridge
note: noprealloc may hurt performance in many applications
Sat May 3 19:38:54.847 [initandlisten] MongoDB starting : pid=389973 port=2701
dbpath=/var/lib/openshift/5c0013917b4d45c68fddbb75e082a35a/mongodb/data/ 64-bit host=ex-std-node94.prod.rhcloud.com
Sat May 3 19:38:54.848 [initandlisten] db version v2.4.6
Sat May 3 19:38:54.848 [initandlisten] git version: nogitversion
Sat May 3 19:38:54.848 [initandlisten] build info: Linux x86-023.build.eng.bos.redhat.com 2.6.18-371.el5 #1 SMP Thu Sep 5 21:21:44 EDT 2013 x86_64 BOOST_LIB_VERSION=1_41
Sat May 3 19:38:54.849 [initandlisten] allocator: tcmalloc
Sat May 3 19:38:54.849 [initandlisten] options: { auth: true, bind_ip: "127.2.148.131", config: "/tmp/mongodb.repair.conf", dbpath: "/var/lib/openshift/5c0013917b4d45c68fddbb75e082a35a/mongodb/data/", nohttpinterface: "true", noprealloc: "true", pidfilepath: "/var/lib/openshift/5c0013917b4d45c68fddbb75e082a35a/mongodb/pid/mongodb.pid", quiet: "true", repair: true, smallfiles: "true" }
**************
You specified --repair but there are dirty journal files. Please
restart without --repair to allow the journal files to be replayed.
If you wish to repair all databases, please shutdown cleanly and
run with --repair again.
**************
Sat May 3 19:38:54.865 [initandlisten] exception in initAndListen: 12596 old lock file, terminating
Sat May 3 19:38:54.866 dbexit:
Sat May 3 19:38:54.866 [initandlisten] shutdown: going to close listening sockets...
Sat May 3 19:38:54.866 [initandlisten] shutdown: going to flush diaglog...
Sat May 3 19:38:54.867 [initandlisten] shutdown: going to close sockets...
Sat May 3 19:38:54.867 [initandlisten] shutdown: waiting for fs preallocator...
Sat May 3 19:38:54.867 [initandlisten] shutdown: closing all files...
Sat May 3 19:38:54.867 [initandlisten] closeAllFiles() finished
Sat May 3 19:38:54.868 dbexit: really exiting now
Warning: Gear 5c0013917b4d45c68fddbb75e082a35a is using 98.9% of disk quota
Warning: Gear 5c0013917b4d45c68fddbb75e082a35a is using 97.3% of disk quota
Attempting to repair MongoDB ...
MongoDB 2.4 database added. Please make note of these credentials:
Root User: ------------
Root Password: ------------
Database Name: ------------
Connection URL: mongodb://$OPENSHIFT_MONGODB_DB_HOST:$OPENSHIFT_MONGODB_DB_PORT/
Failed to execute: 'control start' for /var/lib/openshift/5c0013917b4d45c68fddbb75e082a35a/mongodb
Any idea on how to solve it? Thank you
Yes, as per the logs it looks like you have consumed most of your 1GB disk space. The 1 GB disk space is consumed by all your application cartridges and storage.Which web cartridge are you using? Can you check the disk space usage using quota -s command. SSH into the application gear and run quota -s command. Or if you have rhc command-line installed then you can use rhc ssh --app <app_name> --command 'quota -s'. You can clean up disk space using rhc tidy --app <app_name> command. After cleaning up, try running rhc cartridge command again. You can create a scalable application and that would allow every cartridge to be installed on a different gear. This would allow each cartridge more disk space.
Related
The mongod command gets stuck after the following line. I haven't changed any setting. I also encountered the problem saying "/data/db", solved it by creating the directory. Restarting doesn't have any changes.
**mongod --help for help and startup options
32-bit servers don't have journaling enabled by default. Please use --journal if you want durability.
pid=5517 port=27017 dbpath=/data/db/ 32-bit host=nxtgplus-test
[initandlisten]
[initandlisten] ** NOTE: This is a 32 bit MongoDB binary.
[initandlisten] ** 32 bit builds are limited to less than 2GB of data (or less with --journal).
[initandlisten] ** Note that journaling defaults to off for 32 bit and is currently off.
[initandlisten] ** See http://dochub.mongodb.org/core/32bit
[initandlisten]
[initandlisten] db version v2.4.14
[initandlisten] git version: nogitversion
[initandlisten] build info: Linux buildvm-03.phx2.fedoraproject.org 3.19.5-200.fc21.x86_64 #1 SMP Mon
Apr 20 19:51:56 UTC 2015 i686 BOOST_LIB_VERSION=1_41
[initandlisten] allocator: system
[initandlisten] options: {}
[initandlisten] waiting for connections on port 27017
[websvr] admin web console waiting for connections on port
I am working on CentOS release 6.7 (Final) server. Please help me solve the above issue.
Yesterday I have upgraded my Ubuntu 14.10 to 15.04. But which lead me to stop my work. I was working on chicagoboss and mongodb. i saw error that mongodb has crashed.
By typing just mongo got following error:
$ mongo
MongoDB shell version: 2.4.10
connecting to: test
Mon May 11 11:26:32.985 Error: couldn't connect to server 127.0.0.1:27017 at src/mongo/shell/mongo.js:145
exception: connect failed
Followed stackoverflow query and removed .lock file. Then tried to repair using
$sudo mongod --repair
Then got an error:
*********************************************************************
ERROR: dbpath (/data/db/) does not exist.
Create this directory or give existing directory in --dbpath.
See http://dochub.mongodb.org/core/startingandstoppingmongo
*********************************************************************
I observed that in my mongodb.conf file dbpath is /var/lib/mongodb. Which is correct where I found all my mongodb databases.So I tried to set dbpath to the same location but getting error:
$ mongod --dbpath "/var/lib/mongodb"
Mon May 11 11:33:18.607 [initandlisten] MongoDB starting : pid=4450 port=27017 dbpath=/var/lib/mongodb 64-bit host=laxmikant-alphonso
Mon May 11 11:33:18.607 [initandlisten] db version v2.4.10
Mon May 11 11:33:18.607 [initandlisten] git version: e3d78955d181e475345ebd60053a4738a4c5268a
Mon May 11 11:33:18.607 [initandlisten] build info: Linux ip-10-2-29-40 2.6.21.7-2.ec2.v1.2.fc8xen #1 SMP Fri Nov 20 17:48:28 EST 2009 x86_64 BOOST_LIB_VERSION=1_49
Mon May 11 11:33:18.607 [initandlisten] allocator: tcmalloc
Mon May 11 11:33:18.607 [initandlisten] options: { dbpath: "/var/lib/mongodb" }
Mon May 11 11:33:18.607 [initandlisten] exception in initAndListen: 10309 Unable to create/open lock file: /var/lib/mongodb/mongod.lock errno:13 Permission denied Is a mongod instance already running?, terminating
Mon May 11 11:33:18.607 dbexit:
Mon May 11 11:33:18.607 [initandlisten] shutdown: going to close listening sockets...
Mon May 11 11:33:18.607 [initandlisten] shutdown: going to flush diaglog...
Mon May 11 11:33:18.607 [initandlisten] shutdown: going to close sockets...
Mon May 11 11:33:18.607 [initandlisten] shutdown: waiting for fs preallocator...
Mon May 11 11:33:18.607 [initandlisten] shutdown: lock for final commit...
Mon May 11 11:33:18.607 [initandlisten] shutdown: final commit...
Mon May 11 11:33:18.607 [initandlisten] shutdown: closing all files...
Mon May 11 11:33:18.608 [initandlisten] closeAllFiles() finished
Mon May 11 11:33:18.608 [initandlisten] shutdown: removing fs lock...
Mon May 11 11:33:18.608 [initandlisten] couldn't remove fs lock errno:9 Bad file descriptor
Mon May 11 11:33:18.608 dbexit: really exiting now
How can I resolve this issue. I want all my databases back.
Also I checked that I'm getting following error for service start:
$ sudo service mongodb start
Failed to start mongodb.service: Unit mongodb.service failed to load: No such file or directory.
$ mongod --dbpath "/var/lib/mongodb"
In that case, you start mongodb as yourself. And you probably don't have R/W permission on the data directory.
$ sudo service mongodb start
Now you start mongodb as a service and it will probably change its identity from root to "mongo" or something like that. On debian-like system, the configuration for mongodb is in /etc/init/mongodb.conf or preferably in /etc/mongodb.conf. See http://packages.ubuntu.com/utopic/amd64/mongodb-server/filelist. There you can change the database path.
BTW, check the permissions for your data directory. The user "mongo"-or-whatever should have write access there:
$ ls -l /var/lib/mongodb
I have started learning MongoDB and for which trying to install it on Ubuntu (which I recently shifted from windows). Facing issues to start it with sudo service mongodb start. Following are the logs:
Sun Aug 4 20:25:36.774 [initandlisten] options: { config: "/etc/mongodb.conf", dbpath: "/var/lib/mongodb", logappend: "true", logpath: "/var/log/mongodb/mongodb.log" }
**************
Error: journal files are present in journal directory, yet starting without journaling enabled.
It is recommended that you start with journaling enabled so that recovery may occur.
**************
Sun Aug 4 20:25:36.774 [initandlisten] exception in initAndListen: 13597 can't start without --journal enabled when journal/ files are present, terminating
Sun Aug 4 20:25:36.774 dbexit:
Sun Aug 4 20:25:36.774 [initandlisten] shutdown: going to close listening sockets...
Remove journal files : everything under /var/lib/mongodb/journal if you want to disable journaling (not recommended) or use journaling: journal=true in config file or --journal from command line.
If MongoDB has been shutdown forcefully then the journal files are not cleaned up. The warning is here so that you can decide whether you want to recover from a failure (recommended). For recovery to work, you need to start MongoDB with --journal. Journalling is turned on by default though so I expect your /etc.mongodb.conf file has a nojournal=true line. You can remove that one instead as well.
If you really don't care about recovering, then you can simply remove all the files under /var/lib/mongodb/journal—but realise that you might end up with broken data files.
I have difficulty updating mongodb from version 2.0.6 to the current 2.4. Please help
Below is the log file that i get:
Tue May 14 08:02:03.339 [initandlisten] MongoDB starting : pid=1906 port=27017 dbpath=/mnt2/var/lib/mongodb/ 64-bit host=ip-10-131-65-73
Tue May 14 08:02:03.339 [initandlisten] db version v2.4.3
Tue May 14 08:02:03.339 [initandlisten] git version: fe1743177a5ea03e91e0052fb5e2cb2945f6d95f
Tue May 14 08:02:03.340 [initandlisten] build info: Linux ip-10-2-29-40 2.6.21.7-2.ec2.v1.2.fc8xen #1 SMP Fri Nov 20 17:48:28 EST 2009 x86_64 BOOST_LIB_VERSION=1_49
Tue May 14 08:02:03.340 [initandlisten] allocator: tcmalloc
Tue May 14 08:02:03.340 [initandlisten] options: { dbpath: "/mnt2/var/lib/mongodb/" }
Tue May 14 08:02:03.347 [initandlisten] journal dir=/mnt2/var/lib/mongodb/journal
Tue May 14 08:02:03.347 [initandlisten] recover : no journal files present, no recovery needed
Tue May 14 08:02:03.554 [initandlisten] build index backendtemp.system.users { user: 1, userSource: 1 }
Tue May 14 08:02:03.557 [initandlisten] Duplicate key exception while trying to build unique index on backendtemp.system.users. You most likely have user documents with duplicate "user" fields. To resolve this, start up with a version of MongoDB prior to 2.4, drop the duplicate user documents, then start up again with the current version.
Tue May 14 08:02:03.557 [initandlisten] exception in initAndListen: 11000 E11000 duplicate key error index: backendtemp.system.users.$user_1_userSource_1 dup key: { : "ad", : null }, terminating
Tue May 14 08:02:03.557 dbexit:
Tue May 14 08:02:03.557 [initandlisten] shutdown: going to close listening sockets...
Tue May 14 08:02:03.557 [initandlisten] shutdown: going to flush diaglog...
Tue May 14 08:02:03.557 [initandlisten] shutdown: going to close sockets...
Tue May 14 08:02:03.558 [initandlisten] shutdown: waiting for fs preallocator...
Tue May 14 08:02:03.558 [initandlisten] shutdown: lock for final commit...
Tue May 14 08:02:03.558 [initandlisten] shutdown: final commit...
Tue May 14 08:02:03.572 [initandlisten] shutdown: closing all files...
Tue May 14 08:02:03.573 [initandlisten] closeAllFiles() finished
Tue May 14 08:02:03.573 [initandlisten] journalCleanup...
Tue May 14 08:02:03.573 [initandlisten] removeJournalFiles
Tue May 14 08:02:03.575 [initandlisten] shutdown: removing fs lock...
Tue May 14 08:02:03.575 dbexit: really exiting now
In version 2.4 MongoDB introduced a role-based access control which requires uniqueness of the user in user privilege documents per database. Previous versions of MongoDB didn't have this requirement, so that your database may have duplicate user entries in system.users.
You can either:
Downgrade MongoDB to an older version, remove the duplicate
entries and upgrade MongoDB again, or:
If you are in a
development environment and/or don't care about your data (i.e.
loosing your entire database), you can remove
the database files manually through your filesystem. On Linux they
are usually located at "/var/lib/mongodb/name-of-the-database".
Try starting MongoDB after that and you are cool.
See: MongoDB 2.4 Upgrade Recommendations and Checklist
Small script pasties regarding this (ubuntu):
NOTE: replace CAPS by you vars...
If its a sharded cluster, always make sure balancer is off before this kinda things.
Run sudo apt-get install mongodb-10gen=2.2.7 <- reduce version, probably want 2.2.7 its the latest pre-2.4 (as of May 2014)
Remember to restart mongod config server
connect using mongoCONFIG_SERVER_HOST:CONFIG_SERVER_PORT/admin # <- connect to mongo config server,
db.system.users.find().sort({_id:1}) <- would display the admin users sorted by id which is like sorting ascending by time here, coz its bson.
db.system.users.remove({_id : ObjectId('ID_TO_BE_REMOVED')}) <- remove the culprit, USE WITH CARE. if you not sure, start by creating another admin user for backup.
Quit the mongo console
Run sudo apt-get install mongodb-10gen <- will upgrade back to latest 2.4.X. not it will not upgrade to 2.6 (which is good, coz you want to ensure 2.4 is working first) because its a different apt-get package for 2.6 (mongodb-org).
HTH
If you can't or won't downgrade mongodb, you can also use mongodump and then mongorestore. You will loose every mongo user but get your data back.
mongodump --dbpath OLD_DB_PATH -d DATABASENAME -o /tmp/dumps
mongorestore --dbpath NEW_DB_PATH -d DATABASENAME /tmp/dumps/DATABASENAME/
As always with mongo, be careful about the permissions of the restored files if you do not run the above commands as the mongodb user.
This won't work if your current mongodb version is at least 3.0 because the dbpath option has been removed, forcing you to have a running mongod on the dbpath to run those commands, which is precisely what's not working right now...
In case you have problems, you may try to run mongod --dbpath NEW_DB_PATH --repair
I just had to do an emergency reboot of my linux server. I did it gracefully, not a hard reboot. I have been gathering data through a node.js app for a month, but when the server started up again, all the data was gone!
I have a long log file: http://clintberry.com/mongod.log
I also ran a repair to see if it would help but to no avail.
New Development
It looks like I was running mongo with the command 'service mongod start' and when I try to run that command again, I get this:
**************
old lock file: /var/lib/mongo/mongod.lock. probably means unclean shutdown recommend
removing file and running --repair see: http://dochub.mongodb.org/core/repair
for more information
*************
So that means the repair I was trying was not working since I wasn't using the correct config file.
When I run repair with the correct config file it doesn't seem to run repair, just spits out this:
root#es2 [/var/log/mongo]# mongod --dbpath /var/lib/mongo/ --repair
Wed Nov 30 12:09:40 [initandlisten] MongoDB starting : pid=26916 port=27017 dbpath=/var/lib/mongo/ 64-bit
Wed Nov 30 12:09:40 [initandlisten] db version v1.8.4, pdfile version 4.5
Wed Nov 30 12:09:40 [initandlisten] git version: 81f12749a15e3d158b1b16bab6bc3faea538e166
Wed Nov 30 12:09:40 [initandlisten] build sys info: Linux bs-linux64.10gen.cc 2.6.21.7-2.ec2.v1.2.fc8xen #1 SMP Fri Nov 20 17:48:28 EST 2009 x86_64 BOOST_LIB_VERSION=1_41
**************
old lock file: /var/lib/mongo/mongod.lock. probably means unclean shutdown
recommend removing file and running --repair
see: http://dochub.mongodb.org/core/repair for more information
*************
Wed Nov 30 12:09:40 [initandlisten] exception in initAndListen std::exception: old lock file, terminating
Wed Nov 30 12:09:40 dbexit:
Wed Nov 30 12:09:40 [initandlisten] shutdown: going to close listening sockets...
Wed Nov 30 12:09:40 [initandlisten] shutdown: going to flush diaglog...
Wed Nov 30 12:09:40 [initandlisten] shutdown: going to close sockets...
Wed Nov 30 12:09:40 [initandlisten] shutdown: waiting for fs preallocator...
Wed Nov 30 12:09:40 [initandlisten] shutdown: closing all files...
Wed Nov 30 12:09:40 closeAllFiles() finished
Wed Nov 30 12:09:40 dbexit: really exiting now
As mpobrien's comment suggested, the issue was that when starting mongo it was using a different dbpath (location for the data). For those that have mongo installed on CentOS, if you use the service command to start mongo,
service mongod start
then mongo will use the configuration file /etc/mongo.conf when starting, which puts your dbpath as /var/lib/mongo
but if you start mongo with the regular mongod command, it will not use that folder and will use the mongo default.
Thanks for the help.