Data not fully retrieved while using mgo pipe function - mongodb

I was using pipe function in mgo to retrieve data. My data structs are as follows.
type Company struct {
Id bson.ObjectId `bson:"_id,omitempty"`
CompanyName string
Slug string
CompanyUsers []CompanyUser
}
type CompanyUser struct {
UserName string
Email string
FullName string
}
I needed to check whether a given "UserName" is present under "CompanyUsers" in a "Company" with a given "Slug".
Both Slug and UserName are provided by the User.
Using Pipe function I successfully done the search, but the data is returned with an empty CompanyUsers array.
My query is as follows:
var companyResults []Company
pipeline := []bson.M{
{"$match": bson.M{"slug": slug}},
{"$unwind": "$companyusers"},
{"$match": bson.M{
"companyusers.username": username,
}},
}
err := c.Pipe(pipeline).All(&companyResults)
This provides me a search result as follows:
[{ObjectIdHex("573aa0fddd731711c94830ca") MyCompany companyslug [] }]
None of the data in CompanyUsers are retrieved. How can I solve this error?

Try making companyResults a []map[string]interface{} instead of a []Company to see what sort of a result you get. This helps to figure out what the structure of companyResults should be.
var companyResults []map[string]interface{}
You will see a result like,
[map[companyname:MyCompany slug:companyslug companyusers:map[username:username] _id:test]]
See that companyusers is infact a map not an array. This is because your usage of the $unwind stage. It deconstructs the array, outputting one document for each element in the array. See docs.
I agree with John Smith's answer, that you do not need pipelines at all. But following should give you the desired result.
type Result struct {
Id string `bson:"_id,omitempty"`
CompanyName string
Slug string
CompanyUsers CompanyUser
}
var companyResults []Result

I might be missing something, but why do you need Pipe for this?
You probably could do it easier:
query := bson.M{"slug": slug, "companyusers.username": username}
err := c.Find(query).All(&companyResults)
I'm fairly certain this would give you the same results.
However, the main problem is you didn't give the bson marshaller the fields name. Since the field in your database is companyusers and username you'll have to tell that to the marshaller.
type Company struct {
Id bson.ObjectId `bson:"_id,omitempty"`
CompanyName string
Slug string
CompanyUsers []CompanyUser `bson:"companyuser"`
}
type CompanyUser struct {
UserName string `bson:"username"`
Email string
FullName string
}
You probably want to fix that for the rest of the fields as well.

Related

MongoDB Golang: FindOne() with nested Bson.M failed for 10% attempts, Bson.M+Bson.D or a single Bson.M always work, why?

Need some help to figure out why nested bson.M doesn’t work occasionally in FindOne().
For the following Golang structs stored in a MongoDb collection for type A:
type A struct {
Id primitive.ObjectID
Random1 string
Parents []B
Random2 int
}
type B struct {
Id primitive.ObjectID
Random3 string
Children []C
Random4 int
}
type C struct {
Random5 string
Name Name
Random6 int
}
type Name struct {
FirstName string
LastName string
}
The following filter for FindOne(), which uses two bson.M, worked in most situations but failed to find a match in about 10% runs
filter1 := bson.M{
"parents.0.chilren.0.name": bson.M{
"first_name": "Mike",
"last_name": "Anderson",
},
}
The following two filters alway work, where filter 2 uses bson.D inside bson.M, and filter 3 just uses one bson.M
filter2 := bson.M{
"parents.0.chilren.0.name": bson.D{
{Key: "first_name", Value: "Mike"},
{Key: "last_name", Value: "Anderson"},
},
}
filter3 := bson.M{
"parents.0.chilren.0.name.first_name": "Mike",
"parents.0.chilren.0.name.last_name": "Anderson",
}
I found a similar question in https://jira.mongodb.org/browse/GODRIVER-877 but still don’t understand the differences or root cause. Thanks for the help!
bson.M is a map, thus, the order of its elements are not guaranteed. Since you are comparing parents.0.chilren.0.name, which is a structure, the search criteria has to match exactly, with the order of fields in the criteria matching the order fields in the database. That's why when you use bson.M, it sometimes matches and sometimes not. When the search criteria matches the database field order, you get results, otherwise you don't.
That's the reason why the criteria with bson.D always matches, because it is based on a slice, so the order of fields are preserved.
The last search criteria compares individual fields, not a structure, so the order of fields are immeterial.

Go Mongo Driver Retrieve Schemaless Documents

While working with Mongo Go Driver I want to retrieve Schemaless Documents.
I am able to retrieve documents using bson.M json:",inline" bson:",inline"
But this adds extra "M" key in json when i try to Decode to a struct
type Product struct {
ID primitive.ObjectID `bson:"_id"`
ProductId string `bson:"product_id" json:"product_id"`
bson.M `json:",inline" bson:",inline"`
}
Output:-
{
"id":"<ObjectId>",
"M":{
"some":""
}
}
But instead what i want how it is stored in Mongo.
{
"id":"<ObjectId>",
"some":""
}
I cant use directly something like this as I want to cast it to struct to work with some properties
var pr bson.M
err := p.FindOne(ctx, &p.options,query, &pr)
How can I remove that extra key which is getting added while converting schemaless Documents from Mongo?
Do I need to explicitly overwrite MarshalJSON() or is there something provided using Tags?
How can I remove that extra key which is getting added while converting schemaless Documents from Mongo?
You can just define a field mapping name, which will be flattened when marshalled. For example:
type Product struct {
ID primitive.ObjectID `bson:"_id"`
ProductId string `bson:"product_id"`
Others bson.M `bson:",inline"`
}
When you decode a document, you'll see that it will include other fields without the Others name. For example if you have a document:
{
"_id": ObjectId("5e8d330de85566f5a0557ea4"),
"product_id": "foo",
"some": "x",
"more": "y"
}
doc := Product{}
err = cur.Decode(&doc)
fmt.Println(doc)
// Outputs
// {ObjectID("5e8d330de85566f5a0557ea4") foo map[more:y some:x]}
I cant use directly something like this as I want to cast it to struct to work with some properties
You can use this directly for a query predicate. For example:
// After decoding 'doc' to product
var result bson.M
err := collection.FindOne(context.TODO(), doc).Decode(&result)
Tested using MongoDB Go driver v1.3.2
UPDATED:
If you would like to return JSON, you could use bson.MarshalExtJSON(). This should be easier as well in terms of dealing with objects that don't exist in JSON. i.e. ObjectId. For example:
// After decoding 'doc' to product
ejson, err := bson.MarshalExtJSON(doc, true, false)
fmt.Println(string(ejson))

Making a unique field in Mongo-go-driver

I am very new to both Go and Mongodb and was writing my first rest-api with Go and Mongo. I am using mongo-go-driver and have the following Modal struct in Go
type Modal struct {
Group []string `bson:"group" json:"group"`
Hostname string `bson:"hostname" json:"hostname"`
Overrides map[string]string `bson:"overrides" json:"overrides"`
Excludes []string `bson:"excludes" json:"excludes"`
}
I do not want to use the default ObjectId field provided by mongo-db as my primary key and instead would like to make the Hostname field as the primary key.
If I make the type of Hostname field as primitive.ObjectID, then the hostname would be unique but its value will be randomly generated string by mongodb and not the actual hostname string value.
So is there a way I can do this.
You may use a unique index to enforce / allow only distinct values of a given field, e.g.:
db.collectionname.createIndex( { "hostname": 1 }, { unique: true } )
If you want to create such index using the official MongoDB driver, this is how you can do that:
indexName, err := coll.Indexes().CreateOne(
context.Background(),
mongo.IndexModel{
Keys: bson.D{{Key: "hostname", Value: 1}},
Options: options.Index().SetUnique(true),
},
)
But know that in MongoDB each document must have an _id property, so doing the above, documents will have an auto-generated _id field (of ObjectId type). If this doesn't bother you, you're done.
Also note that you may map Modal.Hostname to the _id field with struct tags:
type Modal struct {
Group []string `bson:"group" json:"group"`
Hostname string `bson:"_id" json:"hostname"`
Overrides map[string]string `bson:"overrides" json:"overrides"`
Excludes []string `bson:"excludes" json:"excludes"`
}
And again, you're done. The downside of this solution is that the documents in MongoDB will not have a property named hostname, as it will be stored in _id.

Partial update of embedded document in mongoDB using mgo

I have the following model:
type UserModel struct {
Id string `bson:"_id,omitempty"`
CreatedAt *time.Time `bson:"createdAt,omitempty"`
BasicInfo *UserBasicInfoModel `bson:"basicInfo,omitempty"`
}
// *Embedded document*
type UserBasicInfoModel struct {
FirstName *string `bson:"firstName,omitempty"`
LastName *string `bson:"lastName,omitempty"`
}
I am using pointers, in order to be able to distinguish between a missing value (nil) and default value (eg empty strings, false values etc). I also use omitempty to be able to do partial updates.
When I create a user I get the following (correct) response back:
"id": "aba19b45-5e84-55e0-84f8-90fad41712f6",
"createdAt": "2018-05-26T15:08:56.764453386+03:00",
"basicInfo": {
"firstName": "Initial first name",
"lastName": "Initial last name"
}
When I try to update the document though I have an issue.
I send the changes as a new UserModel, to only change the FirstName field in the embedded document like this:
newFirstName := "New Value"
UserModel{
BasicInfo: &UserBasicInfoModel{
FirstName: &newFirstName,
},
}
The code I use to do the update is the following:
UpdateId(id, bson.M{"$set": changes})
The response I get back is the following:
"id": "aba19b45-5e84-55e0-84f8-90fad41712f6",
"createdAt": "2018-05-26T12:08:56.764Z",
"basicInfo": {
"firstName": "New Value",
"lastName": null
}
The createdAt value is not null (as I expected) however the lastName value is null (which is not what I expected)
I would have expected to get back the following:
"id": "aba19b45-5e84-55e0-84f8-90fad41712f6",
"createdAt": "2018-05-26T12:08:56.764Z",
"basicInfo": {
"firstName": "New Value",
"lastName": "Initial last name"
}
What can I do to achieve a partial update in a subdocument using mgo?
First let's quickly explain your createdAt field. This is the value you save: 2018-05-26T15:08:56.764453386+03:00. Know that MongoDB stores dates with millisecond precision, and in UTC timezone. So this date when saved and retrieved from MongoDB becomes 2018-05-26T12:08:56.764Z, this is the "same" time instant, just in UTC zone and precision truncated to milliseconds.
Now on to updating embedded documents:
The short and unfortunate answer is that we can't do this directly with the mgo library and Go models.
Why?
When we use the ,omitempty option, and we leave some pointer fields at their zero value (that is, being nil), it's like if we were using a value whose type didn't even have those fields.
So in your example, if you only change the BasicInfo.FirstName field, and you use this value to update, it is equivalent to using these structures:
type UserModel struct {
Id string `bson:"_id,omitempty"`
BasicInfo *UserBasicInfoModel `bson:"basicInfo,omitempty"`
}
type UserBasicInfoModel struct {
FirstName *string `bson:"firstName,omitempty"`
}
So the effect of your issued update command will be the following:
db.users.update({_id: "aba19b45-5e84-55e0-84f8-90fad41712f6"},
{$set:{
"_id": "aba19b45-5e84-55e0-84f8-90fad41712f6",
"basicInfo": {
"firstName": "New Value"
}
}}
)
What does this mean? To set the _id to the same value (it won't change), and to set the basicInfo field to an embedded document which only has a single firstName property. This will erase the lastName field of the embedded basicInfo document. So when you unmarshal the document after the update into a value of your UserModel type, the LastName field will remain nil (because it is not present in MongoDB anymore).
What can we do?
Flatten the embedded document
One trivial solution is to not use an embedded document, but add fields of UserBasicInfoModel to UserModel:
type UserModel struct {
Id string `bson:"_id,omitempty"`
CreatedAt *time.Time `bson:"createdAt,omitempty"`
FirstName *string `bson:"firstName,omitempty"`
LastName *string `bson:"lastName,omitempty"`
}
Hybrid with ,inline option
This solution keeps the separate Go struct, but in MongoDB it will not be an embedded document (BasicInfo will be flattened just like in the previous example):
type UserModel struct {
Id string `bson:"_id,omitempty"`
CreatedAt *time.Time `bson:"createdAt,omitempty"`
BasicInfo UserBasicInfoModel `bson:"basicInfo,omitempty,inline"`
}
Note that BasicInfo needs to be a non-pointer if ,inline is used. This is not a problem, as we can leave it being an empty struct if its fields are not to be changed, since its fields are pointers, so leaving them nil will not change them.
Doing "manual" update
If you do need to use embedded document, the mgo library allows you to update specific fields of embedded documents, but then you have to "manually" construct the update document, like in this example:
c.UpdateId(Id, bson.M{"$set": bson.M{
"basicInfo.firstName": newFirstName,
}})
Yes, this isn't convenient at all. If you do need this many times with different types, you may create a utility function that uses reflection, iterates over the fields recursively, and assemble the update document from fields which are not nil. Then you could pass that dynamically generated update doc to UpdateId() for example.

How do I find a document's array, and only get the array

Database MongoDB
Program Language Golang
Data Driven mgo
Document struct
type Organizations struct {
Id bson.ObjectId `json:"_id"`
Name string `json:"name"`
Area string `json:"area"`
Type string `json:"type"`
CustomerType string `json:"customertype"`
State string `json:"state"`
LastUpdated string `json:"lastupdated"`
Assigned string `json:"assigned"`
Address `json:"address"`
}
Array Struct which is belong to Organizations
type Sections struct {
Name string `json:"name" bson:"name"`
Upper string `json:"upper" bson:"upper"`
Master string `json:"master" bson:"master"`
Uppermaster string `json:"uppermaster" bson:"uppermaster"`
OrgID bson.ObjectId `json:"orgid"`
}
My query command
result := []Sections{}
collection.Find(bson.M{"_id": bson.ObjectIdHex(orgid),
"sections": bson.M{"$exists": true}}).All(&result)
Finally, I get a document with array.
But, I just want to get only the data of "Sections" array in the document.
What should I do ?
Thank you!
I don't understand Go language + mgo syntax but I guess you can come up with something similar
db.collection.find({ _id : orgid, sections : { $exists : 1 }}, { _id : 0, sections : 1 })
Check out how projections are done here
http://docs.mongodb.org/manual/reference/method/db.collection.find/#projections