My initial thought was to have MongoDB automatically create the _id field. However, the frontend of my app is PHP and I submit a job into Beanstalkd. The backend is picked up by a Python daemon that pulls the necessary info from Beanstalk. What I'm trying to accomplish is so that the user gets redirected to a status page, but I need the ObjectId that was used. The only way I can think of accomplishing this is to generate the ObjectId as part of the job info. Hence, I have this information handy and I can use it to display the status page.
So my question is, could I run into issues? I figure a timestamp is part of the ObjectId therefore it should be unique even if I pre-generate it before inserting into Mongo.
Pretty sure that it won't be a problem to create and attach _id to db by your code rather than mongoDB driver code. If you write your code to generate ID correctly you won't dive in any issue, because it would be quite same or similar algorithm as for mongoDB driver.
Take a look at MongoID class.
Related
I have a MEAN database application with a number of Mongo collections with hierarchical relationships via ObjectId. A copy of the application works locally offline, and another copy runs on the production server.
The data contain collectively describe rules and content that drive a complex process. These data need to be entered offline so that these processes can be tested before the data go into the production environment.
What I assumed I would be able to easily do is to export selected documents as JSON, then relatively simply import them into the production database. So, the system would have a big "Export" button that would take the current document and all subdocuments and related documents, and export them as a single JSON file. Then, my "Import" button would parse that JSON file on the production server.
So, exporting is no problem. Did that in a couple of hours.
But, I quickly found that when I import a document, its _id field value is not preserved. This breaks relationships, obviously.
I have considered writing parsing routines that preserved these relationships by programmatically setting ObjectIds in parent documents after the child documents have been saved. This will be a huge headache though.
I'm hoping there is either:
a) ... and easy way to import a JSON document with _id fields intact, or ...
b) ... another way to accomplish this entirely that is easier than I am making it.
I appreciate any advice.
There's always got to be someone that doesn't know the answer who complains about the question. The question is clear and the problem is familiar.
Indeed, Mongoose will overwrite any value you provide for _id when you create a document either via the create() method or using the constructor (var thing = new Thing()).
Also, mongoexport/mongoimport will not fill the need to do this programmatically, at least not easily.
If I'm understanding correctly, you want to export a subset of documents, along with any related documents, keeping references intact. Then, you want to import this data into a remote system, again, keeping references intact.
The approach you took would work just fine except it will destroy all references, as you found out.
I've worked on a similar problem and I believe that the best way to do this is to do what it sounds like you wanted to avoid. That is, you'll iterate over your collections and let Mongo generate its _ids as it will. Add your child documents first, then set the references correctly in your parent documents. I really don't think there is a better way that still gives you granular control.
In current version of mongodb you can use db.copyDatabase(). Start current instance of mongodb where you want to copy database and run following command:
db.copyDatabase(fromDB, toDB).
For more options and details refer to db.copyDatabase()
I have two mongo collections. One we can call a template and second is instance. Every time new instance is created, rather large data field is copied from template to instance. Currently the field is retrieved from mongo db template collection in application and then sent back to db as a part of instance collection insert.
Would it be possible to somehow perform this copy on insert directly in mongo db, to avoid sending several megabytes over the network back and forth?
Kadira is reporting 3 seconds lag due to this. And documents are only going to get bigger.
I am using Meteor, but I gather that that should not influence the answer much.
I have done some searching and I can't really find an elegant solution for you. The two ways I can think of doing it are:
1.) Fork a process to run a mongo command to copy your template as your new instance via db.collection.copyTo().
http://eureka.ykyuen.info/2015/02/26/meteor-run-shell-command-at-server-side/
https://docs.mongodb.org/manual/reference/method/db.collection.copyTo/
Or
2.) Attempt to access the raw mongo collection rather than the minimongo collection meteor provides you with so you can use the db.collection.copyTo() functionality supplied by Mongo.
var rawCollection = Collection.rawCollection();
rawCollection.copyTo(newCollection);
Can meteor mongo driver handle $each and $position operators?
I haven't tried accessing the rawCollection to see if copyTo is available, and I also don't know if it will bring it into meteor before writing out the new collection. I'm just throwing this out here as an idea for you; hopefully someone else has a better one.
How can I determine what collection a Mongo document belongs to?
I'm using MeteorJS, but even if you don't use this, I'm pretty sure I have access to MongoDB's internal commands so please answer anyways.
Happy to close if this is a duplicate, but I didn't find anything close to this.
There isn't a real short command but to cycle through each collection and query for the field, or fields that uniquely identifies this document. Don't know MeteorJS but if you are looking for just a quick way to get the data to find out and not add a program function you can download RoboMongo. Think of it as SQL Management Studio for MongoDB. If you are looking for a program function to do this then I suggest you make it a jscript function you create inside MongoDB (similar to stored procedures) to do the work and return the results to you when done.
Are you querying the document now and wondering where it is coming from? If so, it should be in the code.
I planned to create a document with an _id before inserting it into the DB.
I wanted to generate this _id using Meteor.uuid() (which theoretically always return a unique id) but I felt on this following git issue
Thanks, good catch. The reason that this wasn't documented is that
we'd eventually like to move away from string _ids to native binary
Mongo _ids. Since it's used in an example though, I think we should go
ahead and document it and cross that bridge later. I'll do this
It seems to be a difference between a string id and a binary mongo one. Back to my question, is there a good reason I should then avoid using my custom _id ?
When you create a Mongo.Collection you have the option to select the way Meteor handles creating _id for documents which do not have and _id already. You can read more about it here.
The key point to take away is if it does not already have an _id. You are free to use whatever custom _id field you want. At this point this is no longer a Meteor question, but a Mongo question. Read up on the pros and cons of manually setting the _id field in Mongo.
Back to your question, with respect to Meteor there is no reason to avoid creating your own _id fields. If the custom _id you want to use uniquely identify's that document, then you are good to go.
And don't worry about Meteor.uuid(). It's no longer documented, so I'd imagine it will eventually disappear.
MongoDB ObjectID are guaranteed unique by the algorithm, so are totally conflict-safe. You could have the same safeness only actually having somewhere a sort of incremental counter shared by all your application servers and persisted, so actually reimplementing what already on you db server.
From my POV you should choose between accepting a little risk and generating a random large ID, or using MongoDB to bring them to you in advance.
With the latter I mean you could implement your custom ID generation as an empty save on your collection, that you could later user within the actual document save: you'll pay the price of an additional database roundtrip but if your function involve moving files I'm sure it would be negligible.
My personal advice is the latter solution.
I deployed a test app (http://meteortipslinda2.meteor.com/) and added accounts-facebook. Facebook login does not work, however, because I probably input the wrong API key.
I know I can access the shell with meteor mongo meteortipslinda2.meteor.com, but I'm at a loss about how to see everything in these collections:
meteor_accounts_loginServiceConfiguration
meteor_oauth_pendingCredentials
meteor_oauth_pendingRequestTokens
I'm assuming that I can update the value to the correct one in the mongoDB shell, but first I need to figure out how to see the contents of the collections.
To list all the documents in a collection, just call find without any parameters.
db.myCollection.find()
If there are a lot of documents, it will batch them up. You will then be able to show the next batch by typing it.