Well, I have used MongoDB in a while but I don't know how to handle this situation.
The scenario is: I have data inserted in MongoDB. I have exported the data in JSON format, and it is something like:
[
{
"_id": { "$oid": "60ff324f41c4d5b96054390d" },
"field": {
"due_date": { "$date": "2021-11-03T00:09:18.271Z" }
}
}
]
You can see that :
_id is { "$oid": "60ff324f41c4d5b96054390d" } and
date is { "$date": "2021-11-03T00:09:18.271Z" }.
So, the problem is trying to insert in another DB using Mongoose. I want to test some funcionality isolated so I wan't these values in other environment, so I have used insertMany() with the JSON previously exported.
(Maybe there is a more elegant way to import a JSON file but is only for testing purposes)
await model.insertMany(JSON.parse(fs.readFileSync('./data.json').toString()))
But the problem is here: It throws an error because my schema says that _id is an ObjectId and due_date is a Date object but they are actually read as objects: {$oid: ""} and {$date: ""}
ValidationError: model validation failed: _id: Cast to ObjectId failed for value "{ '$oid': '60ff324f41c4d5b96054390d' }" (type Object) at path "_id", field.due_date: Cast to date failed for value "{ '$date': '2021-11-03T00:09:18.271Z' }" (type Object) at path "field.due_date"
So the question is: Is there any way to cast the $oid and $date objects using mongoose while inserting?
Also, I have managed to insert the values using a very ugly script iterating over each value and checking if the object is $oid or $date and modifying the values... but it works.
So the question is not: "Is there a workaround to do this?" but "Is there a way to do it directly with mongoose functions as insertMany and casted automatically?"
Thanks.
The {"$oid": ...} and {"$date":...} constructs are MongoDB extended JSON notation, since JSON does not have any type for Date or ObjectId.
In order to insert those values, you will need to use a JSON parser that knows about this extended format.
One possibility is the json_util that is included with bson.
Or you can use mongoimport to read the JSON file and do the inserts for you.
How to handle if document structure after production changes.
Suppose I had 500 documents like this:
{
name: ‘n1’
height: ‘h1’
}
Later if I decide to add all the documents in below format:
{
name: ‘n501’
height: ‘h501’
weight: ‘w501’
}
I am using cursor.All(&userDetails) to decode(deserialize) in Go to get the output of the query in struct userDetails. If I modify the structure of further documents and userDetails accordingly, it will fail for the first 500 documents?
How to handle this change?
If you add a new field to your struct, querying old documents will not fail. Since the old documents do not have the new field saved in MongoDB, querying them will give you struct values where the new field will be its zero value. E.g. if its type is string, it will be the empty string "", if it's an int field, it will be 0.
If it bothers you that the old documents do not have this new field, you may extend them in the mongo console like this:
db.mycoll.updateMany({ "weight": {$exists:false} }, { $set: {"weight": ""} } )
This command adds a new weight field to old documents where this field did not exist, setting them to the empty string.
There's a MongoDB collection that was populated with documents like this one:
{
"_id" : ObjectId("5b7f83b591fae49715443590"),
"content" : "{\n\t\"email\":\"username#domain.com\",\n\t\"country_code\": \"US\"\n}"
}
As you can see "content"has a JSON string
I was able to parse the string using JSON.parse() like:
db.getCollection('my_collection').find({}).map(function(doc){ return {'_id':doc._id, 'content': JSON.parse(doc.content)}});
Giving me an array of objects with _id and content as an object.
Is there a way to execute queries (filtering, aggregate functions) over content preferably without getting an array from map?
like you said In terms of the database content is a string. So you need to save the entire collection where content will be a object so you will be able to filtering, aggregate etc...
I have the following model:
type UserModel struct {
Id string `bson:"_id,omitempty"`
CreatedAt *time.Time `bson:"createdAt,omitempty"`
BasicInfo *UserBasicInfoModel `bson:"basicInfo,omitempty"`
}
// *Embedded document*
type UserBasicInfoModel struct {
FirstName *string `bson:"firstName,omitempty"`
LastName *string `bson:"lastName,omitempty"`
}
I am using pointers, in order to be able to distinguish between a missing value (nil) and default value (eg empty strings, false values etc). I also use omitempty to be able to do partial updates.
When I create a user I get the following (correct) response back:
"id": "aba19b45-5e84-55e0-84f8-90fad41712f6",
"createdAt": "2018-05-26T15:08:56.764453386+03:00",
"basicInfo": {
"firstName": "Initial first name",
"lastName": "Initial last name"
}
When I try to update the document though I have an issue.
I send the changes as a new UserModel, to only change the FirstName field in the embedded document like this:
newFirstName := "New Value"
UserModel{
BasicInfo: &UserBasicInfoModel{
FirstName: &newFirstName,
},
}
The code I use to do the update is the following:
UpdateId(id, bson.M{"$set": changes})
The response I get back is the following:
"id": "aba19b45-5e84-55e0-84f8-90fad41712f6",
"createdAt": "2018-05-26T12:08:56.764Z",
"basicInfo": {
"firstName": "New Value",
"lastName": null
}
The createdAt value is not null (as I expected) however the lastName value is null (which is not what I expected)
I would have expected to get back the following:
"id": "aba19b45-5e84-55e0-84f8-90fad41712f6",
"createdAt": "2018-05-26T12:08:56.764Z",
"basicInfo": {
"firstName": "New Value",
"lastName": "Initial last name"
}
What can I do to achieve a partial update in a subdocument using mgo?
First let's quickly explain your createdAt field. This is the value you save: 2018-05-26T15:08:56.764453386+03:00. Know that MongoDB stores dates with millisecond precision, and in UTC timezone. So this date when saved and retrieved from MongoDB becomes 2018-05-26T12:08:56.764Z, this is the "same" time instant, just in UTC zone and precision truncated to milliseconds.
Now on to updating embedded documents:
The short and unfortunate answer is that we can't do this directly with the mgo library and Go models.
Why?
When we use the ,omitempty option, and we leave some pointer fields at their zero value (that is, being nil), it's like if we were using a value whose type didn't even have those fields.
So in your example, if you only change the BasicInfo.FirstName field, and you use this value to update, it is equivalent to using these structures:
type UserModel struct {
Id string `bson:"_id,omitempty"`
BasicInfo *UserBasicInfoModel `bson:"basicInfo,omitempty"`
}
type UserBasicInfoModel struct {
FirstName *string `bson:"firstName,omitempty"`
}
So the effect of your issued update command will be the following:
db.users.update({_id: "aba19b45-5e84-55e0-84f8-90fad41712f6"},
{$set:{
"_id": "aba19b45-5e84-55e0-84f8-90fad41712f6",
"basicInfo": {
"firstName": "New Value"
}
}}
)
What does this mean? To set the _id to the same value (it won't change), and to set the basicInfo field to an embedded document which only has a single firstName property. This will erase the lastName field of the embedded basicInfo document. So when you unmarshal the document after the update into a value of your UserModel type, the LastName field will remain nil (because it is not present in MongoDB anymore).
What can we do?
Flatten the embedded document
One trivial solution is to not use an embedded document, but add fields of UserBasicInfoModel to UserModel:
type UserModel struct {
Id string `bson:"_id,omitempty"`
CreatedAt *time.Time `bson:"createdAt,omitempty"`
FirstName *string `bson:"firstName,omitempty"`
LastName *string `bson:"lastName,omitempty"`
}
Hybrid with ,inline option
This solution keeps the separate Go struct, but in MongoDB it will not be an embedded document (BasicInfo will be flattened just like in the previous example):
type UserModel struct {
Id string `bson:"_id,omitempty"`
CreatedAt *time.Time `bson:"createdAt,omitempty"`
BasicInfo UserBasicInfoModel `bson:"basicInfo,omitempty,inline"`
}
Note that BasicInfo needs to be a non-pointer if ,inline is used. This is not a problem, as we can leave it being an empty struct if its fields are not to be changed, since its fields are pointers, so leaving them nil will not change them.
Doing "manual" update
If you do need to use embedded document, the mgo library allows you to update specific fields of embedded documents, but then you have to "manually" construct the update document, like in this example:
c.UpdateId(Id, bson.M{"$set": bson.M{
"basicInfo.firstName": newFirstName,
}})
Yes, this isn't convenient at all. If you do need this many times with different types, you may create a utility function that uses reflection, iterates over the fields recursively, and assemble the update document from fields which are not nil. Then you could pass that dynamically generated update doc to UpdateId() for example.
I have a collection called posts, initially they only had "text" and "author" field. Later I added a "date" field for later posts.
How I can add a date field for initial posts, for those who does not have date/created-at field.
I guess I need to use following two commands, but cant figure out how to make the connection (i.e. loop)?
db.posts.find({"date":{"$exists":false}}
ObjectId(_id).getTimestamp()
db.posts
.find({"date":{"$exists":false}) // looking for records without `date` field
.forEach(function (record) { // iterating through each record
record.date = null; // if you want to define current date use `new Date()`
db.posts.save(record); // saving updated record
});
I guess there is a forEach() function that you could use. forEach
db.posts.find({"date":{"$exists":false}).forEach(<function - do something>)