Good morning,
I'm currently trying to build a REST-API project for learning purposes and want to use the repository pattern as practice.
I have a model module where I define structs to use. Like user and posts (for both I have set the ID to a string). MongoDB uses ObjectIDs as IDs though. So my current solution is converting that in the MongoDB repository awkwardly to a user/post-struct using that ObjectID and then back again to the user/post-model that the rest of the program is using.
Is there a better way to achieve that same thing without awkward struct conversions?
Here is the code for that:
https://github.com/schattenbrot/mini-blog-api
Thanks in advance already for any ideas and tips I can try out.
Mongo uses the field _id as an object's id. When you insert a document without this field, mongo auto generates its own object id which is what is happening in your case. The solution is to insert your struct with the ID field marked as _id.
This can be done by adding a bson tag on your struct with _id. Mongo will use your id instead of generating one on its own. When you marshal to json, the bson tag is ignored so everything else in your application stays the same.
type Post struct {
ID string `json:"id,omitempty" bson:"_id"`
Title string `json:"title,omitempty" validate:"omitempty,min=3,max=40"`
Text string `json:"text,omitempty" validate:"omitempty,min=5,max=700"`
Creator string `json:"user,omitempty" validate:"omitempty"`
CreatedAt time.Time `json:"created_at,omitempty"`
UpdatedAt time.Time `json:"updated_at,omitempty"`
}
Related
I'm trying to put a prefix before the auto generated _id, to identify from which collection came an id, but I still want to use the mongo unique id generator.
So I can know that this id model_5e1a51821c9d44000089e3e0 came from the Model collection.
Is there a solution for that without messing with random string ?
Edit
The _id need to be string castable, since I use it as id in a graphQL object. I need to differentiate ids because I use an union in my schema and resolver need to know in which table to find the data.
The _id can be generated within an application with the constructor ObjectId(). If you want to add a prefix for the generated field, you can use an embedded document as a field for the _id, like this:
_id: {
idPrefix: "Model",
_id: ObjectId("5e1bd112b7f18a490a4bafb5")
}
Other way of identifying if a document is from another collection is use a separate boolean field:
{
_id: ObjectId("5e1bd112b7f18a490a4bafb5"),
isFromModel: <boolean true or false>,
...
}
There are some options available to do this, I'm just trying to tell you the way how would I do if I need this.
Step 1: You can generate the document and it will return you ObjectId (_id) .
Step 2: Take that value and prefix it with model like this.
let _id=5e1a51821c9d44000089e3e0;let new_idValue="model_"+_id;
Step3: Now update your document by _id and push new value in place of if as
this.db.document.findByIdAndUpdate(_id,{$set:{{_id:new_idValue}})
This is what you can do. If you find some best solution than mine, let me know as well. I will highly appreciate.
Using mgo, it seems that best practice is to set object ids to be bson.ObjectId.
This is not very convenient, as the result is that instead of a plain string id the id is stored as binary in the DB. Googling this seems to yield tons of questions like "how do I get a string out of the bson id?", and indeed in golang there is the Hex() method of the ObjectId to allow you to get the string.
The bson becomes even more annoying to work with when exporting data from mongo to another DB platform (this is the case when dealing with big data that is collected and you want to merge it with some properties from the back office mongo DB), this means a lot of pain (you need to transform the binary ObjectId to a string in order to join with the id in different platforms that do not use bson representation).
My question is: what are the benefits of using bson.ObjectId vs string id? Will I lose anything significant if I store my mongo entities with a plain string id?
As was already mentioned in the comments, storing the ObjectId as a hex string would double the space needed for it and in case you want to extract one of its values, you'd first need to construct an ObjectId from that string.
But you have a misconception. There is absolutely no need to use an ObjectId for the mandatory _id field. Quite often, I advice against that. Here is why.
Take the simple example of a book, relations and some other considerations set aside for simplicty:
{
_id: ObjectId("56b0d36c23da2af0363abe37"),
isbn: "978-3453056657",
title: "Neuromancer",
author: "William Gibson",
language: "German"
}
Now, what use would have the ObjectId here? Actually none. It would be an index with hardly any use, since you would never search your book databases by an artificial key like that. It holds no semantic value. It would be a unique ID for an object which already has a globally unique ID – the ISBN.
So we simplify our book document like this:
{
_id: "978-3453056657",
title: "Neuromancer",
author: "William Gibson",
language: "German"
}
We have reduced the size of the document, make use of a preexisting globally unique ID and do not have a basically unused index.
Back to your basic question wether you loose something by not using ObjectIds: Quite often, not using the ObjectId is the better choice. But if you use it, use the binary form.
If I generate a new object id for a document in mgo:
obId := bson.NewObjectId()
and then insert it, it ends up in mongo (looking via the cli) as
"_id" : "U�`�\u0006#�\rU\u0000\u0000\u0001"
When it should be
"_id" : ObjectId("559a47643d9827f0d9405420")
Same goes if I try and update an existing document where I generate the id by
obId := bson.ObjectIdHex(stringId)
It still gets serialized to the corrupted format.
My struct which I'm trying to insert looks like this:
type MyStruct struct {
Id bson.ObjectId `bson:"_id,omitempty" json:"id"`
...
}
The representation "U�`�\u0006#�\rU\u0000\u0000\u0001" is clearly indicating that an ObjectId got sent to the database as a string rather than as a properly typed object id. Every such case before was a code path in the application side delivering the string explicitly as such by mistake. I recommend investigating every code path that inserts objects in that collection, and if you can find no case that is sending it as an actual string, then try to create a reproducer and report it upstream to the mgo driver.
Update: Per your comment below, the issue is being caused because some part of the application is using an ObjectId type from a package that is not the one actually used during communication with the database. This has the effect described above: the ObjectId type coming from the wrong package is just a normal string, as far as the correct bson package is concerned.
I have a Meteor application whereby I initially use the _id field from each record in my collection when naming list items in my template.
When get the _id field, I convert it to a string to use in the template.
Now I want to update these records in Mongo and am passing the _id back to a Meteor.method, but these are still in string format and Mongo is expecting an ObjectID(). Is there a simple way to convert this string to the ObjectID()? If not, what alternatives do I have?
Ok, found it! On the /server, within your Meteor method function do this to convert it:
var mid = new Mongo.ObjectID(str_id_sent_to_server);
I'm using pymongo to seed a database with old information from a different system, and I have a lot of queries like this:
studentId = studentsRemote.insert({'price': price})
In the actual python script, that studentId prints as a string, but in the javascript Meteor application I'm using this data in, it shows up everywhere as ObjectId(...).
I want to configure pymongo to generate the _id as a string and not bother with ObjectId's
Any objects I create with the Meteor specification will use the string format, and not the ObjectId format. I don't want to have mixing of id types in my application, because it's causing me interoperability headaches.
I'm aware I can create ObjectId's from Meteor but frankly I'd much rather use the string format. It's the Meteor default, it's much simpler, and I can't find any good reason to use ObjectId's in my particular app.
The valueOf() mongo function or something similar could parse the _id and be used to update the document once it's in the database, but it would be nice to have something more direct.
in .py files:
from bson.objectid import ObjectId
......
kvdict['_id'] = str(ObjectId())
......
mongoCollection.insert(kvdict)
it's ok!
It ended up being fairly simple.
The son_manipulator module can be used to change incoming documents to a different form. Most of the time this is used to encode custom objects, but it worked for this as well.
With the manipulator in place, it was just a matter of calling the str() function on the ObjectId to make the transformation.
from pymongo.son_manipulator import SONManipulator
class ObjectIdManipulator(SONManipulator):
def transform_incoming(self, son, collection):
son[u'_id'] = str(son[u'_id'])
return son
db.add_son_manipulator(ObjectIdManipulator())