I'm trying to migrate data from Parse to a new project that uses Mongo as its database (without Parse/Parse Server). Since the schemas are different between the two projects, I'm manually writing a migration script to achieve this.
As I understand it, Parse appears to use 10-character-long IDs for their objects (combinations of digits, lower-case letters, and upper-case letters), while Mongo uses 24-character-long IDs (12 bytes represented as hex).
Right now, when migrating data for a document from the old project to the new one, I'm using a function that converts the Parse ID to a unique Mongo ObjectId (it converts each character to a 2-digit hex value, then pads the 20-character string with 4 zeroes).
Is this a good approach? I'm avoiding using Mongo's automatic ObjectId generation in case I ever need to re-migrate any of the old Parse documents and find the matching document in the new database. I know automatically generated ObjectIds in Mongo also embed some other information like creation dates, but I don't think this would be important and I can just use my custom ObjectId generator? However, I'm not sure about the implications for performance/if I'm just going about this migration the wrong way.
The approach i recommend is letting Mongo auto-generate the ids and then storing Parse's ids in a new field called parseID for future reference if needed.
For example:
PARSE DATA:
"_id": ObjectId(1234567890),
"title": "Mongo Migrate",
"description": "Migrating from Parse to Mongo"
MONGO DATA:
"_id": ObjectId(1ad83e4k2ab8e0daa8ebde7), //mongo generated
"parseId":ObjectId(1234567890),
"title": "Mongo Migrate",
"description": "Migrating from Parse to Mongo"
Then if you need to match a document between the two databases later, you can write a script that goes along the lines of Parse.find({"_id": Mongo.parseId}).....
MongoDB uses _id as primary key by default. _id has to be unique to avoid collision. The way you are generating unique ObjectId to _id is fine. As long as they are unique, you could even reduce the 20-character pad to save space.
Related
I have imported my database entries from a csv file into mongodb and in the process of doing so the imported entries's '_id's aren't in the typical randomised format. e.g instead of looking like
"_id": "R6AnePqKecNNe7dkr"
they look like
"_id": 28
For some reason this means that when I try to refer to the database entry with an _id of 28 it can't find it. I thought it might be an issue in that they _id is not a string because they're not in quotes but I've tried to convert them to strings and that doesn't make a difference. I've tried making a function that randomly generates a new _id but it can't find the old _id to put the new _id into.
Is there any way to do a bulk "generate new _ids for everything in this collection"? I'm having trouble finding anything equivalent
Using mgo, it seems that best practice is to set object ids to be bson.ObjectId.
This is not very convenient, as the result is that instead of a plain string id the id is stored as binary in the DB. Googling this seems to yield tons of questions like "how do I get a string out of the bson id?", and indeed in golang there is the Hex() method of the ObjectId to allow you to get the string.
The bson becomes even more annoying to work with when exporting data from mongo to another DB platform (this is the case when dealing with big data that is collected and you want to merge it with some properties from the back office mongo DB), this means a lot of pain (you need to transform the binary ObjectId to a string in order to join with the id in different platforms that do not use bson representation).
My question is: what are the benefits of using bson.ObjectId vs string id? Will I lose anything significant if I store my mongo entities with a plain string id?
As was already mentioned in the comments, storing the ObjectId as a hex string would double the space needed for it and in case you want to extract one of its values, you'd first need to construct an ObjectId from that string.
But you have a misconception. There is absolutely no need to use an ObjectId for the mandatory _id field. Quite often, I advice against that. Here is why.
Take the simple example of a book, relations and some other considerations set aside for simplicty:
{
_id: ObjectId("56b0d36c23da2af0363abe37"),
isbn: "978-3453056657",
title: "Neuromancer",
author: "William Gibson",
language: "German"
}
Now, what use would have the ObjectId here? Actually none. It would be an index with hardly any use, since you would never search your book databases by an artificial key like that. It holds no semantic value. It would be a unique ID for an object which already has a globally unique ID – the ISBN.
So we simplify our book document like this:
{
_id: "978-3453056657",
title: "Neuromancer",
author: "William Gibson",
language: "German"
}
We have reduced the size of the document, make use of a preexisting globally unique ID and do not have a basically unused index.
Back to your basic question wether you loose something by not using ObjectIds: Quite often, not using the ObjectId is the better choice. But if you use it, use the binary form.
I'm developing an application that create permalinks. I'm not sure how save the documents in MondoDB. Two strategies:
ObjectId autogeneration
MongoDB autogenerates the _id. I need to create an index on the permalink field because I get the information by the permalink. Also I can access to the creation time of the ObjectId, using the getTimestamp() method, so datetime fields seems to be redundant but if I delete this field I need two calls to MongoDB one to take the information and another to take the timestamp.
{
"_id": ObjectId("5210a64f846cb004b5000001"),
"permalink": "ca8W7mc0ZUx43bxTuSGN",
"data": "a lot of stuff",
"datetime": ISODate("2013-08-18T11:47:43.460+-100")
}
Generate _id
I generate the _id with the permalink.
{
"_id": "ca8W7mc0ZUx43bxTuSGN",
"data": "a lot of stuff",
"datetime": ISODate("2013-08-18T11:47:43.460+-100")
}
I not see any advantage to use ObjectIds. Am I missing something?
ObjectIds are there for situations where you don't have a unique key for every document in a collection. They're unique, so you don't have to worry about conflicts and they shard reasonably well in large deployments without too much worry (they have they're pros and cons, read more here).
The ObjectId also contains the timestamp of the client where the ObjectId was generated (unless the DB server is configured to generate all keys). With that, as you noticed, you can use the time stamp to perform some date operations. However, if you plan on using the Aggregation Framework, you'll find that you can't use an ObjectId in any date operations currently (issue). If you want to use the AF, you'll need a second field that contains the date, unfortunately doubly storing it with the ObjectId's internal value.
If you can be assured that the _id you're generating is unique, then there's not much reason to use an ObjectId in your data structure.
in SQL world I could do something to the effect of:
SELECT name FROM table WHERE UPPER(name) = UPPER('Smith');
and this would match a search for "Smith", "SMITH", "SmiTH", etc... because it forces the query and the value to be the same case.
However, MongoDB doesn't seem to have this capability without using a RegEx, which won't use indexes and would be slow for a large amount of data.
Is there a way to convert a stored value to a particular case before doing a search against it in MongoDB?
I've come across the $toUpper aggregate, but I can't figure out how that would be used in this particular case.
If there's not way to convert stored values before searching, is it possible to have MongoDB convert a value when it's created in Mongo? So when I add a document to the collection it would force the "name" attribute to a particular case? Something like a callback in the Rails world.
It looks like there's the ability to create stored JS for MongoDB as well, similar to a Stored Procedure. Would that be a feasible solution as well?
Mostly looking for a push in the right direction; I can figure out the particular code once I know what I'm looking for, but so far I'm not even sure if my desired functionality is doable.
You have to normalize your data before storing them. There is no support for performing normalization as part of a query at runtime.
The simplest thing to do is probably to save both a case-normalized (i.e. all-uppercase) and display version of the field you want to search by. Suppose you are storing users and want to do a case-insensitive search on last name. You might store:
{
_id: ObjectId(...),
first_name: "Dan",
last_name: "Crosta",
last_name_upper: "CROSTA"
}
You can then create an index on last_name_upper, and query like:
> db.users.find({last_name_upper: "CROSTA"})
{ "_id" : ObjectId("4c2209f9f3924d31102bd84a"), "name" : "mongo" }
what exactly is the purpose of objectId? It's a big number that is generated using a timestamp.
If I see any nosql which is key-value, I query with key the value.
Here we use key and value in the as the data and use find () function.
So, I am trying to understand when we really need the objectid?
What are the reasons behind that giving access to the user to view the value of the object ID?
After reading the docs, one basic question is mongo DB as hash table type implementation?
After readying doc..one basic question is mongo DB as hash table type implementation?
MongoDB used BSON, a binary form of JSON. A JSON object is basically just a "hashtable" or a set of key / value pairs.
what exactly is the use of object id? that is a big number that is generated with time.
In MongoDB, each document you store must have an _id. If you do not set a value for _id, then MongoDB will automatically generate one for you. If you have a unique key when you are inserting the object, you can use that instead. For details on the ObjectId see here.
If I see any nosql which is key-value, I query with key the value.
MongoDB is not just key-value. MongoDB supports multiple indexes on a single collection, you can query on many different fields, not just the "key" or "id".
Object ID is similar to primary key in RDBMS
Whenever u insert a new document, mongodb will generate object ID.
Object ID is a 12 byte BSON Type.
First 4 Byte represents timestamp
next 3 byte unique machine identifier
next 2 byte process id
next 3 byte random increment counter
Returns the equivalent 16 digit hex