I'm trying to better understand the Meteor/MongoDB data model. When you create a new meteor project I'd like to know where the data in a collection is stored when you create a new collection or add data to a collection. I understand that it is supposed to be under the .meteor/local/db directory but thus far I have not found it. I've both created new collections and added data to preexisting collection to both the basic project and to the Meteor demo projects (like Leaderboard) and I can't find where this data is stored. Could someone please guide me on this matter?
I imagine that I would at least see a JSON type list somewhere or a GUI similar to something like a MYSQL work bench (is there anything out there like this for Meteor - I've looked high and low but I haven't found it; Houston is insufficient).
In addition to scouring Stack Overflow for the answer to this question I've looked through a number of APIs (like Meteor's and Mongo's) and tutorials like http://meteortips.com/book/databases-part-1/
Again all I want to know is how can I see the data in Mongo as it is added to a collection. Thank you.
The data files are in the mongodb format; and are not human readable.
If you want to query mongo directly --
while meteor is running (from your app's directory)
meteor mongo
If meteor isn't running, and you want to launch just the database, you can try:
mongod --smallfiles --dbpath /path/to/my/app/.meteor/local/db --port 3001
Then connect with the regular mongo shell.
To access database in nice GUI form I use Robomongo.
What is nice you can connect to local (on port 3001) or production mongodb from it (see how to do that).
Update:
Remember to run meteor command before connecting to local mongodb.
Thanks #iAmME
I have been using MONOVUE (http://www.mongovue.com/downloads/) for viewing the collections and it has been very handy in checking the data.
The different kinds of views : Table View, Tree View and Text View make it easier to understand how the data is inserted especially for anyone(like me) jumping from RDBMS to NOSQL.
Related
Is there a way to switch off the ability of mongo to sporadically create dbs and collections as soon as it sees one in a query. I run queries on the mongo console all the time and mistype a db or collection name, causing mongo to just create one. There should be a switch to have mongo only explicitly create dbs and collections. I can't find one on the docs.
To be clear, MongoDB does not auto create collections or databases on queries. For collections, they are auto created when you actually save data to them. You can test this yourself, run a query on a previously unknown collection in a database like this:
use unknowndb
db.unknowncollection.find()
show collections
No collection named "unknowncollection" shows up until you insert or save into it.
Databases are a bit more complex. A simple "use unknowndb" will not auto create the database. However, if after you do that you run something like "show collections" it will create the empty database.
I agree, an option to control this behavior would be great. Happy to vote for it if you open a Jira ticket at mongoDB.
No, implicit creation of collections and DBs is a feature of the console and may not be disabled. You might take a look at the security/authorization/role features of 2.6 and see if anything might help (although there's not something that exactly matches your request as far as I know).
I'd suggest looking through the MongoDB issues/bug/requests database system here to and optionally add the feature request if it doesn't already exist.
For people who are using Mongoose, a new database will get created automatically if your Mongoose Schema contains any form of index. This is because Mongo needs to create a database before it can insert said index.
I have two mongo collections. One we can call a template and second is instance. Every time new instance is created, rather large data field is copied from template to instance. Currently the field is retrieved from mongo db template collection in application and then sent back to db as a part of instance collection insert.
Would it be possible to somehow perform this copy on insert directly in mongo db, to avoid sending several megabytes over the network back and forth?
Kadira is reporting 3 seconds lag due to this. And documents are only going to get bigger.
I am using Meteor, but I gather that that should not influence the answer much.
I have done some searching and I can't really find an elegant solution for you. The two ways I can think of doing it are:
1.) Fork a process to run a mongo command to copy your template as your new instance via db.collection.copyTo().
http://eureka.ykyuen.info/2015/02/26/meteor-run-shell-command-at-server-side/
https://docs.mongodb.org/manual/reference/method/db.collection.copyTo/
Or
2.) Attempt to access the raw mongo collection rather than the minimongo collection meteor provides you with so you can use the db.collection.copyTo() functionality supplied by Mongo.
var rawCollection = Collection.rawCollection();
rawCollection.copyTo(newCollection);
Can meteor mongo driver handle $each and $position operators?
I haven't tried accessing the rawCollection to see if copyTo is available, and I also don't know if it will bring it into meteor before writing out the new collection. I'm just throwing this out here as an idea for you; hopefully someone else has a better one.
Is there a way to switch off the ability of mongo to sporadically create dbs and collections as soon as it sees one in a query. I run queries on the mongo console all the time and mistype a db or collection name, causing mongo to just create one. There should be a switch to have mongo only explicitly create dbs and collections. I can't find one on the docs.
To be clear, MongoDB does not auto create collections or databases on queries. For collections, they are auto created when you actually save data to them. You can test this yourself, run a query on a previously unknown collection in a database like this:
use unknowndb
db.unknowncollection.find()
show collections
No collection named "unknowncollection" shows up until you insert or save into it.
Databases are a bit more complex. A simple "use unknowndb" will not auto create the database. However, if after you do that you run something like "show collections" it will create the empty database.
I agree, an option to control this behavior would be great. Happy to vote for it if you open a Jira ticket at mongoDB.
No, implicit creation of collections and DBs is a feature of the console and may not be disabled. You might take a look at the security/authorization/role features of 2.6 and see if anything might help (although there's not something that exactly matches your request as far as I know).
I'd suggest looking through the MongoDB issues/bug/requests database system here to and optionally add the feature request if it doesn't already exist.
For people who are using Mongoose, a new database will get created automatically if your Mongoose Schema contains any form of index. This is because Mongo needs to create a database before it can insert said index.
I have a Mongo database that I did not create or architect, is there a good way to introspect the db or print out what the structure is to start to get a handle on what types of data are being stored, how the data types are nested, etc?
Just query the database by running the following commands in the mongo shell:
use mydb //this switches to the database you want to query
show collections //this command will list all collections in the database
db.collectionName.find().pretty() //this will show all documents in the database in a readable format; do the same for each collection in the database
You should then be able to examine the document structure.
There is actually a tool to help you out here called Variety:
http://blog.mongodb.org/post/21923016898/meet-variety-a-schema-analyzer-for-mongodb
You can view the Github repo for it here: https://github.com/variety/variety
I should probably warn you that:
It uses MR to accomplish its tasks
It uses certain other queries that could bring a production set-up to a near halt in terms of performance.
As such I recommend you run this on a development server or a hidden node of a replica or something.
Depending on the size and depth of your documents it may take a very long time to understand the rough structure of your database through this but it will eventually give one.
This will print name and its type
var schematodo = db.collection_name.findOne()
for (var key in schematodo) { print (key, typeof key) ; }
I would recommend limiting the result set rather than issuing an unrestricted find command.
use mydb
db.collectionName.find().limit(10)
var z = db.collectionName.find().limit(10)
Object.keys(z[0])
Object.keys(z[1])
This will help you being to understand your database structure or lack thereof.
This is an open-source tool that I, along with my friend, have created - https://pypi.python.org/pypi/mongoschema/
It is a Python library with a pretty simple usage. You can try it out (even contribute).
One option is to use the Mongoeye. It is open-source tool similar to the Variety.
The difference is that Mongoeye is a stand-alone program (Mongo Shell is not required) and has more features (histograms, most frequent values, etc.).
https://github.com/mongoeye/mongoeye
Few days ago I found GUI client MongoDB Compass with some nice visualizations. See the product overview. It comes directly from the mongodb people and according to their doc:
MongoDB Compass is designed to allow users to easily analyze and understand the contents of their data collections within MongoDB...
You may've asked about validation schema. Here's the answer how to get it:
How to retrieve MongoDb collection validator rules?
Use Mongo Compass
which does a sample as explained here
Which does a random sample of 1000 documents to get you the schema - it could miss something but it's the only rational option if you database is several GBs.
Visualisation
The schema then can be exported as JSON
Documentation
You can use MongoDB's tool mongodump. On running it, a dump folder is created in the directory from which you executed mongodump. In that folder, there are multiple folders that correspond to the databases in MongDB, and there are subfolders that correspond to the collections, and files that correspond to the documents.
This method is the best I know of, as you can also make out the schema of empty collections.
I just installed mongodb from Synaptic (Using ubuntu 11.10).
I'm following this tutorial: http://howtonode.org/express-mongodb
to build a simple blog using node.js, express.js and mongodb.
I reached the part where you can actually store post in the mongodb.
It worked for me but I'm not sure how. I didn't even start the mongodb (just installed it).
The second thing that has me puzzled is: where is that data going?
Does anyone have any clue?
EDIT:
I think I found where is the data going:
var/lib/
Where are some files that are apparently written in binary code.
MongoDB stores all your data as BSON within the directory you found. The internal Mongo processes handle the sharding of the data when necessary following your specific setup, accomodating you if you have setup distribution across several servers.
One of the most confusing things about coming over to Mongo from RDBMS is the nature of collections vs tables.
http://www.mongodb.org/display/DOCS/MongoDB%2C+CouchDB%2C+MySQL+Compare+Grid
Of Note: A Mongo collection does not need to have any schema designed, it is assumed that the client application will validate and enforce any particular scheme. The way the data comes in and is serialized is the way it will be saved in its BSON representation. It is entirely possible to have missing keys and and totally different docs in the same collection, which is why robust data checking in your app becomes so important.
Mongo collections also do not need to be saved. There isn't anything that approximates the CREATE TABLE syntax from MySQL, since there's no schema there isn't much such a command would have to describe. Your collection will be created with the first document inserted, and every document inserted will be given a unique hashed '_id' key (assuming you don't define this key in your objects using your own methodology.'
Here's a helpful resource with the basic commands.
http://cheat.errtheblog.com/s/mongo
Mongo, and NoSQL in general, is a developer lead movement trying to bridge the gap between the object-oriented application layer and the persistency layer in an app, which would traditionally have been represented by an RDBMS. Developers just would much rather work within one paradigm, and OOP has become predominant especially in the web environment.
If you're coming over from LAMP and used PhpMyAdmin in the past, I really must suggest RockMongo. This tool makes it so much easier to observe and understand the actual structure of the BSON documents stored in your server.
http://code.google.com/p/rock-php/wiki/rock_mongo
Best of luck!