I have two mongo collections. One we can call a template and second is instance. Every time new instance is created, rather large data field is copied from template to instance. Currently the field is retrieved from mongo db template collection in application and then sent back to db as a part of instance collection insert.
Would it be possible to somehow perform this copy on insert directly in mongo db, to avoid sending several megabytes over the network back and forth?
Kadira is reporting 3 seconds lag due to this. And documents are only going to get bigger.
I am using Meteor, but I gather that that should not influence the answer much.
I have done some searching and I can't really find an elegant solution for you. The two ways I can think of doing it are:
1.) Fork a process to run a mongo command to copy your template as your new instance via db.collection.copyTo().
http://eureka.ykyuen.info/2015/02/26/meteor-run-shell-command-at-server-side/
https://docs.mongodb.org/manual/reference/method/db.collection.copyTo/
Or
2.) Attempt to access the raw mongo collection rather than the minimongo collection meteor provides you with so you can use the db.collection.copyTo() functionality supplied by Mongo.
var rawCollection = Collection.rawCollection();
rawCollection.copyTo(newCollection);
Can meteor mongo driver handle $each and $position operators?
I haven't tried accessing the rawCollection to see if copyTo is available, and I also don't know if it will bring it into meteor before writing out the new collection. I'm just throwing this out here as an idea for you; hopefully someone else has a better one.
Related
Is there a way to switch off the ability of mongo to sporadically create dbs and collections as soon as it sees one in a query. I run queries on the mongo console all the time and mistype a db or collection name, causing mongo to just create one. There should be a switch to have mongo only explicitly create dbs and collections. I can't find one on the docs.
To be clear, MongoDB does not auto create collections or databases on queries. For collections, they are auto created when you actually save data to them. You can test this yourself, run a query on a previously unknown collection in a database like this:
use unknowndb
db.unknowncollection.find()
show collections
No collection named "unknowncollection" shows up until you insert or save into it.
Databases are a bit more complex. A simple "use unknowndb" will not auto create the database. However, if after you do that you run something like "show collections" it will create the empty database.
I agree, an option to control this behavior would be great. Happy to vote for it if you open a Jira ticket at mongoDB.
No, implicit creation of collections and DBs is a feature of the console and may not be disabled. You might take a look at the security/authorization/role features of 2.6 and see if anything might help (although there's not something that exactly matches your request as far as I know).
I'd suggest looking through the MongoDB issues/bug/requests database system here to and optionally add the feature request if it doesn't already exist.
For people who are using Mongoose, a new database will get created automatically if your Mongoose Schema contains any form of index. This is because Mongo needs to create a database before it can insert said index.
In the Meteor documents it says
On the server (if you do not specify a connection), a collection with that name is created on a backend Mongo server.
If there already is a collection with that name on the server, I assume then it merely opens it? Or in Mongo is collection != database, with the latter being the persistent entity on the server?
Typically MongoDB only creates collections when you perform writes on a collection that does not exist, but you can create a collection explicitly. Mongo.collection('name') will create the name collection if it does not exist. Otherwise it does nothing, but if you perform write operations on the collection they will still persist.
In other words this behavior shouldn't have a tremendous impact on you. You will be able to use the collection even if it does not exist yet because Meteor will create it for you.
On the server, new Mongo.Collection() loads the Mongo collection.
On the client, the same loads the in-browser Mini-Mongo collection.
Unless you have the autopublish package as part of your project, you must use Meteor.publish(), connection.allow(), and Meteor.subscribe() methods.
If you create a collection on the client without a name: new Mongo.Collection(null), it will not be synchronized with the server at all.
If a client side collection doesn't have a corresponding server collection, it will cause an error Method Not Found.
Is there a way to switch off the ability of mongo to sporadically create dbs and collections as soon as it sees one in a query. I run queries on the mongo console all the time and mistype a db or collection name, causing mongo to just create one. There should be a switch to have mongo only explicitly create dbs and collections. I can't find one on the docs.
To be clear, MongoDB does not auto create collections or databases on queries. For collections, they are auto created when you actually save data to them. You can test this yourself, run a query on a previously unknown collection in a database like this:
use unknowndb
db.unknowncollection.find()
show collections
No collection named "unknowncollection" shows up until you insert or save into it.
Databases are a bit more complex. A simple "use unknowndb" will not auto create the database. However, if after you do that you run something like "show collections" it will create the empty database.
I agree, an option to control this behavior would be great. Happy to vote for it if you open a Jira ticket at mongoDB.
No, implicit creation of collections and DBs is a feature of the console and may not be disabled. You might take a look at the security/authorization/role features of 2.6 and see if anything might help (although there's not something that exactly matches your request as far as I know).
I'd suggest looking through the MongoDB issues/bug/requests database system here to and optionally add the feature request if it doesn't already exist.
For people who are using Mongoose, a new database will get created automatically if your Mongoose Schema contains any form of index. This is because Mongo needs to create a database before it can insert said index.
I'm doing my first project with MongoDB and from what I've seen the implicit creation of collections is the recommended practice (ie. db.myCollection.insert() will create the collection the first time an insert is made)
I'm using PHP and using different collections this way, but the problem is that I don't know where I should create the indexes I'll need for that collection. As I wouldn't know when a collection is created, the naive approach would be calling ensureIndex() just before every operation on that collection (which doesn't sound very good). Or whenever a connection to the database is made, make sure the indexes exist (what happens if I create an index on a collection that wasn't created? Is that defined?)
Any best practice advice for this?
Not sure if it's the best practice, but I tend to not put the ensureIndex in the app. I usually put the ones I am sure I will need using the db shell. Then I keep an eye during load testing(or when things start to slow down in production) and add any I missed again in the shell. You can build indexes in the background by doing ensureIndex({a : 1}, {background : true}), so building them later isn't as terrible as some other dbs.
MongoDB has a good profiler to find what is going slow: http://www.mongodb.org/display/DOCS/Database+Profiler.
10gen(MongoDB's commercial counterpart) has a free monitoring service that is talked about a lot although I haven't used it yet: http://www.10gen.com/mongodb-monitoring-service.
But as far as what happens when you call db.collection.ensureIndex() before collection is created, it will create the collection and put the index on it.
If you definitely want it in the app, I would go with the second option you put forth(ensure indexes right after db connect) instead of before each operation. I would probably save something in the db when I did so they don't run each time if there is more than a couple. Don't know php but here is pseudo code:
var test = db.systemChecks.findOne({indexes : true})
if (test == null) //item doesn't exist
{
//do all the ensureIndex() commands
db.systemChecks.insert({indexes : true})
}
Just remember to delete the systemCheck item if you find you need more indexes later to run through the indexes
Actually, ensureIndex() is exactly what you will need to do. I would do it in each Model's constructor that uses a specific connection - if you have one of those models). ensureIndex() will make sure that an index is only created when it doesn't already exists. You can alternatively do it when you create your database connection. If you run ensureIndex() on a non-existing collection, it would just create that collection and makes empty indexes (as there are no documents yet).
In the future, the PHP driver will also cache whether ensureIndex() was already run in the same request, basically making it a no-op: https://jira.mongodb.org/browse/PHP-581
I'm using MongoDB for our web application. Assume there will be a 'find()' on MongoDB for incoming requests. What is the recommended way/place to add index on a MongoDB collection ?
Couple of options I can think of:-
1) 'ensureIndex' on the collection while initializing the application. [But how will I 'ensureindex' at the very first time application initialize ? since there won't be any data in place]
2 'ensureIndex' before every 'find' operation (on web request) ? but isn't this an overhead even if 'ensureIndex' wouldn't create index if it is already created ?
Any other options ?
Thanks in advance.
I would put it when you initialize the application. If the collection does not exist when you call ensureIndex, the index (and collection) will be created at that time.
I am assuming that you know a priori what kinds of queries you will be running on the data, and what kind of data you will be putting into the index, of course.
Clearly run the ensureIndex when you start the application. It doesn't matter if you call the ensureIndex on an empty collection (as it will add the data afterwards to the index).
Furthermore it depends on which property your queries are based. If you for example query based on the logged in user, then you should add the index on the user id.
You could create a initialisation script for the collection to be run from the command line of your server (the Mongo docs discuss how to write Mongo scripts and run them from the CLI). Then have it run whenever mongod starts? The script could just call ensureIndex() on the collection object.