Right way to pull data from mongodb in golang - mongodb

I have a document in mongodb in this format,
{
field1: string,
field2: float64,
field3: {...float64}
}
Ultimately I would like to always get field1 & field2 and pick/choose from the field3 object.
To do it, I decode the data into a struct like this,
type MongoScore struct {
field1 string `json:"field1"`
field2 float64 `json:"field2"`
field3 map[string]float64 `json:"field3"`
}
The part I am wondering is whether there is a more efficient approach to pull this data with different types.

Assuming you have the following data struct:
type Product struct {
ID primitive.ObjectID `bson:"_id"`
Title string `bson:"product"`
Description string `bson:"description"`
count int64 // <- this is not exported, so data won't be filled
}
In Golang, Only exported fields or methods are accessible from outside it's package.
Use struct field tags to tell Mongodb load the data from collection straight to the matched fields. Here mongodb matches bson tag with fields in collection.
func main() {
ctx, _ := context.WithTimeout(context.Background(), 5*time.Second)
client, err = mongo.Connect(ctx, options.Client().ApplyURI("mongodb://localhost:27017/"))
if err != nil {
log.Fatalf("can't connect to database: %v\n", err)
}
objID, _ := primitive.ObjectIDFromHex("619dd79acad38082f9ce16af")
db := client.Database("db")
col := db.Collection("products")
filter := bson.D{
{"_id", objID},
}
dest := &Project{}
err := col.FindOne(ctx, filter).Decode(dest)
if err != nil {
log.Fatalln(err)
}
The method Decode Unmarshals found data into dest.

Related

MongoDB returns object data as array of key-value pair in Go

I have written following query which returns me records where updated_at date is greater than synced_at date from all records using mongodb library in Golang.
pipeline := []bson.M{}
filter := []string{"$updated_at", "$synced_at"}
pipeline = append(pipeline, bson.M{"$match": bson.M{"$expr": bson.M{"$gte": filter}}})
opts := options.Aggregate().SetMaxTime(2 * time.Second)
cursor, err := collection.Aggregate(ctx, pipeline, opts)
for cursor.Next(context.Background()) {
records := model.DData{}
err = cursor.Decode(&records)
}
The structure of Data is:
type DData struct {
Name string `json:"name" bson:"name"`
Data interface{} `json:"data" bson:"data"`
UpdatedAt time.Time `json:"updated_at" bson:"updated_at"`
SyncedAt time.Time `json:"synced_at" bson:"synced_at"`
}
Data in collection is of the form:
{
"name":"Vallabh",
"data":{
"field1":"value1",
"field2":"value2",
"field3":"value3",
},
"updated_at":2021-08-17T09:43:27Z,
"synced_at":2021-08-07T09:43:27Z
}
But with above query I am getting data in the form:
{
"name":"Vallabh",
"data":[
{
"key":"field1",
"value":"value1"
},
{
"key":"field2",
"value":"value2"
},
{
"key":"field3",
"value":"value3"
}
}],
"updated_at":"2021-08-17T09:43:27Z",
"synced_at":"2021-08-07T09:43:27Z"
}
What am I doing wrong? Its happening only when field type is an interface in struct.
Try with this
type DData struct {
Name string `json:"name" bson:"name"`
Data map[string]string `json:"data" bson:"data"`
UpdatedAt time.Time `json:"updated_at" bson:"updated_at"`
SyncedAt time.Time `json:"synced_at" bson:"synced_at"`
}
Always use the type if you know it beforehand, with interface{} the burden is on the library to find out what the type is. If you expect the varied values in the map you can use map[string]interface{}
Found a very weird solution to this problem where I decoded it to bson.M{}, then marshalled it and unmarshalled it to my struct. I am sure there are better ways but this one worked for me.
for cursor.Next(context.Background()) {
tempResult := bson.M{}
err = decode(cursor, &tempResult)
if err != nil {
logger.LogErrorf(err, "error while decoding")
continue
}
obj, err := marshal(tempResult)
if err != nil {
logger.LogErrorf(err, "error while marshalling")
continue
}
var data model.DData
err = json.Unmarshal(obj, &data)
if err != nil {
tenant.LogError(err, "error while marshalling")
continue
}
}
#VallabhLakade I had similar concern like this and tried the below way that helped .
So basically what's the problem is that the mongo-driver defaults to unmarshalling as bson.D for structs of type interface{} where as mgo mgo-driver defaults to bson.M .
So we will have to add the below code while trying to establish connection with mongo-db , SetRegistry() options as clientOpts To map the old mgo behavior, so that mongo-driver defaults to bson.M while unmarshalling structs of type interface{} , and this should not display the values back as key-value pair
tM := reflect.TypeOf(bson.M{})
reg := bson.NewRegistryBuilder().RegisterTypeMapEntry(bsontype.EmbeddedDocument, tM).Build()
clientOpts := options.Client().ApplyURI(SOMEURI).SetAuth(authVal).SetRegistry(reg)
client, err := mongo.Connect(ctx, clientOpts)
Reference -> MongoDB document returns array of key value pair in go mongo-driver
#Vallabh you shall use mgocompat too something like below this should mimic the old mgo behavior
registry := mgocompat.NewRegistryBuilder().Build()
connectOpts := options.Client().ApplyURI(SOMEURI).SetAuth(info).SetRegistry(registry)

Golang and MongoDB: DeleteMany with filter

I try to read and write and delete data from a Go application with the official mongodb driver for go (go.mongodb.org/mongo-driver).
Here is my struct I want to use:
Contact struct {
ID xid.ID `json:"contact_id" bson:"contact_id"`
SurName string `json:"surname" bson:"surname"`
PreName string `json:"prename" bson:"prename"`
}
// xid is https://github.com/rs/xid
I omit code to add to the collection as this is working find.
I can get a list of contacts with a specific contact_id using the following code (abbreviated):
filter := bson.D{}
cursor, err := contactCollection.Find(nil, filter)
for cur.Next(context.TODO()) {
...
}
This works and returns the documents. I thought about doing the same for delete or a matched get:
// delete - abbreviated
filter := bson.M{"contact_id": id}
result, _ := contactCollection.DeleteMany(nil, filter)
// result.DeletedCount is always 0, err is nil
if err != nil {
sendError(c, err) // helper function
return
}
c.JSON(200, gin.H{
"ok": true,
"message": fmt.Sprintf("deleted %d patients", result.DeletedCount),
}) // will be called, it is part of a webservice done with gin
// get complete
func Get(c *gin.Context) {
defer c.Done()
id := c.Param("id")
filter := bson.M{"contact_id": id}
cur, err := contactCollection.Find(nil, filter)
if err != nil {
sendError(c, err) // helper function
return
} // no error
contacts := make([]types.Contact, 0)
for cur.Next(context.TODO()) { // nothing returned
// create a value into which the single document can be decoded
var elem types.Contact
err := cur.Decode(&elem)
if err != nil {
sendError(c, err) // helper function
return
}
contacts = append(contacts, elem)
}
c.JSON(200, contacts)
}
Why does the same filter does not work on delete?
Edit: Insert code looks like this:
_, _ = contactCollection.InsertOne(context.TODO(), Contact{
ID: "abcdefg",
SurName: "Demo",
PreName: "on stackoverflow",
})
Contact.ID is of type xid.ID, which is a byte array:
type ID [rawLen]byte
So the insert code you provided where you use a string literal to specify the value for the ID field would be a compile-time error:
_, _ = contactCollection.InsertOne(context.TODO(), Contact{
ID: "abcdefg",
SurName: "Demo",
PreName: "on stackoverflow",
})
Later in your comments you clarified that the above insert code was just an example, and not how you actually do it. In your real code you unmarshal the contact (or its ID field) from a request.
xid.ID has its own unmarshaling logic, which might interpret the input data differently, and might result in an ID representing a different string value than your input. ID.UnmarshalJSON() defines how the string ID will be converted to xid.ID:
func (id *ID) UnmarshalJSON(b []byte) error {
s := string(b)
if s == "null" {
*id = nilID
return nil
}
return id.UnmarshalText(b[1 : len(b)-1])
}
As you can see, the first byte is cut off, and ID.UnmarshalText() does even more "magic" on it (check the source if you're interested).
All-in-all, to avoid such "transformations" happen in the background without your knowledge, use a simple string type for your ID, and do necessary conversions yourself wherever you need to store / transmit your ID.
For the ID Field, you should use the primitive.ObjectID provided by the bson package.
"go.mongodb.org/mongo-driver/bson/primitive"
ID primitive.ObjectID `json:"_id" bson:"_id"`

Select column from Mongodb in golang using mgo

As I know, we can use
> db['twitter-3'].find({}, {"text": 1})
to select all texts in collection.
How can we use mgo to find specific field in golang?
I tried
var result []string
err = conn.Find(bson.M{}, bson.M{"text", 1}).All(&result)
But it is not correct.
Use the query Select method to specify the fields to return:
var result []struct{ Text string `bson:"text"` }
err := c.Find(nil).Select(bson.M{"text": 1}).All(&result)
if err != nil {
// handle error
}
for _, v := range result {
fmt.Println(v.Text)
}
In this example, I declared an anonymous type with the one selected field. It's OK to use a type with all document fields.
to select multiple fields:
var result []struct{
Text string `bson:"text"`
Otherfield string `bson:"otherfield"`
}
err := c.Find(nil).Select(bson.M{"text": 1, "otherfield": 1}).All(&result)
if err != nil {
// handle error
}
for _, v := range result {
fmt.Println(v.Text)
}
var result interface{}
err = c.Find(nil).Select(bson.M{"text": 1}).All(&result)

Querying mongodb from golang using the _id stored in an array

So here is my question. I have an array which are stored the _ids of mongodbs objects. Whats the right way to retrieve them all in one query using the mgo and bson package?
So if the array is like that: ids:=["543d171c5b2c12420dd016","543d171c5b2dd016"]
How we make the query ? I tried that but I know its wrong.
query := bson.M{"_id": bson.M{"$in": ids}}
c.Find(query).All()
Thanks in advance
If the documents are stored with string ids, then the code looks correct.
The ids look like hex encoded object ids. If the object identifiers are object ids, then you need to the convert the hex strings to object ids:
oids := make([]bson.ObjectId, len(ids))
for i := range ids {
oids[i] = bson.ObjectIdHex(ids[i])
}
query := bson.M{"_id": bson.M{"$in": oids}}
MongoDB syntax for go.mongodb.org/mongo-driver has been updated, this should work using the official driver.
oids := make([]primitive.ObjectID, len(ids))
for i := range ids {
objID, err := primitive.ObjectIDFromHex(ids[i])
if err == nil {
oids = append(oids, objID)
}
}
This is to convert back to a struct that can be used through out the app
type MongoUser struct {
ID *primitive.ObjectID `json:"id" bson:"_id"`
FirstName string `json:"first_name" bson:"firstName"`
LastName string `json:"last_name" bson:"lastName"`
Email string `json:"email" bson:"email"`
}
This is a helper method that takes your slice of ids and turns it into the object id type.
func formatObjectIdMultiple(hex []string) ([]primitive.ObjectID, error) {
var list []primitive.ObjectID
oids := make([]primitive.ObjectID, len(hex))
for _, i := range hex {
objectId, err := primitive.ObjectIDFromHex(i)
if err != nil {
return nil, err
}
oids = append(oids, objectId)
}
return list, nil
}
Here is my method for the db. Its important you use bson.M for some reason bson.D does not work with this. Also dont forget to close your cursor the defer function will close it at the end of your GetMultipleUser function.
func (mongo *Mongo) GetMultipleUser(ids []string) ([]*MongoUser, error) {
objectIDs, err := formatObjectIdMultiple(ids)
if err != nil {
return nil, err
}
query := bson.M{"_id": bson.M{"$in": objectIDs}}
coll := mongo.Con.Database("dbName").Collection("users")
cursor, err := coll.Find(context.Background(), query)
if err != nil {
return nil, err
}
defer func() {
cursor.Close(context.Background())
}()
var output []*MongoUser
for cursor.Next(context.Background()) {
var temp *MongoUser
cursor.Decode(&temp)
output = append(output, temp)
}
return output, nil
}

Golang mongodb mgo driver Upsert / UpsertId documentation

The mongodb documentation says:
The fields and values of both the and parameters if the parameter contains only update operator expressions. The update creates a base document from the equality clauses in the parameter, and then applies the update expressions from the parameter.
And the mgo documentation says:
Upsert finds a single document matching the provided selector document and modifies it according to the update document. If no document matching the selector is found, the update document is applied to the selector document and the result is inserted in the collection.
But if i do an upsert like this:
session.UpsertId(data.Code, data)
I end up with an entry which have an ObjectID generated automatically by mongodb, instead of data.Code.
this means that UpsertId expect data to be formated with update operators and you can't use a an arbitrary struct? Or what i'm missing here?
Pd. Mongo 2.4.9 mgo v2 golang go version devel +f613443bb13a
EDIT:
This is a sample of what i mean, using the sample code from Neil Lunn:
package main
import (
"fmt"
"gopkg.in/mgo.v2"
// "gopkg.in/mgo.v2/bson"
)
type Person struct {
Code string
Name string
}
func main() {
session, err := mgo.Dial("admin:admin#localhost");
if err != nil {
fmt.Println("Error: ", err)
return
// panic(err)
}
defer session.Close()
session.SetMode(mgo.Monotonic, true)
c := session.DB("test").C("people")
var p = Person{
Code: "1234",
Name: "Bill",
}
_, err = c.UpsertId( p.Code, &p )
result := Person{}
err = c.FindId(p.Code).One(&result)
if err != nil {
fmt.Println("FindId Error: ", err)
return
// panic(err)
}
fmt.Println("Person", result)
}
I found the documentation of the MongoDB was right. The correct way to do this is to wrap the struct to insert into an update operator.
The sample code provided by Neil Lunn, would look like:
package main
import (
"fmt"
"gopkg.in/mgo.v2"
"gopkg.in/mgo.v2/bson"
)
type Person struct {
Code string
Name string
}
func main() {
session, err := mgo.Dial("admin:admin#localhost");
if err != nil {
fmt.Println("Error: ", err)
return
}
defer session.Close()
session.SetMode(mgo.Monotonic, true)
c := session.DB("test").C("people")
var p = Person{
Code: "1234",
Name: "Bill",
}
upsertdata := bson.M{ "$set": p}
info , err2 := c.UpsertId( p.Code, upsertdata )
fmt.Println("UpsertId -> ", info, err2)
result := Person{}
err = c.FindId(p.Code).One(&result)
if err != nil {
fmt.Println("FindId Error: ", err)
return
}
fmt.Println("Person", result)
}
Thank you very much for your interest and help Neil.
You seem to be talking about assigning a struct with a custom _id field here. This really comes down to how you define your struct. Here is a quick example:
package main
import (
"fmt"
"gopkg.in/mgo.v2"
"gopkg.in/mgo.v2/bson"
)
type Person struct {
ID string `bson:"_id"`
Name string
}
func main() {
session, err := mgo.Dial("127.0.0.1");
if err != nil {
panic(err)
}
defer session.Close()
session.SetMode(mgo.Monotonic, true)
c := session.DB("test").C("people")
var p = Person{
ID: "1",
Name: "Bill",
}
_, err = c.UpsertId( p.ID, &p )
result := Person{}
err = c.Find(bson.M{"_id": p.ID}).One(&result)
if err != nil {
panic(err)
}
fmt.Println("Person", result)
}
So in the custom definition here I am mapping the ID field to bson _id and defining it's type as string. As shown in the example this is exactly what happens when serialized via UpsertId and then retrieved.
Now you have elaborated I'll point to the difference on the struct definition.
What I have produces this:
{ "_id": 1, "name": "Bill" }
What you have ( without the same mapping on the struct ) does this:
{ "_id": ObjectId("53cfa557e248860d16e1f7e0"), "code": 1, "name": "Bill" }
As you see, the _id given in the upsert will never match because none of your fields in the struct are mapped to _id. You need the same as I have:
type Person struct {
Code string `bson:"_id"`
Name string
}
That maps a field to the mandatory _id field, otherwise one is automatically produced for you.