Golang mongodb mgo driver Upsert / UpsertId documentation - mongodb

The mongodb documentation says:
The fields and values of both the and parameters if the parameter contains only update operator expressions. The update creates a base document from the equality clauses in the parameter, and then applies the update expressions from the parameter.
And the mgo documentation says:
Upsert finds a single document matching the provided selector document and modifies it according to the update document. If no document matching the selector is found, the update document is applied to the selector document and the result is inserted in the collection.
But if i do an upsert like this:
session.UpsertId(data.Code, data)
I end up with an entry which have an ObjectID generated automatically by mongodb, instead of data.Code.
this means that UpsertId expect data to be formated with update operators and you can't use a an arbitrary struct? Or what i'm missing here?
Pd. Mongo 2.4.9 mgo v2 golang go version devel +f613443bb13a
EDIT:
This is a sample of what i mean, using the sample code from Neil Lunn:
package main
import (
"fmt"
"gopkg.in/mgo.v2"
// "gopkg.in/mgo.v2/bson"
)
type Person struct {
Code string
Name string
}
func main() {
session, err := mgo.Dial("admin:admin#localhost");
if err != nil {
fmt.Println("Error: ", err)
return
// panic(err)
}
defer session.Close()
session.SetMode(mgo.Monotonic, true)
c := session.DB("test").C("people")
var p = Person{
Code: "1234",
Name: "Bill",
}
_, err = c.UpsertId( p.Code, &p )
result := Person{}
err = c.FindId(p.Code).One(&result)
if err != nil {
fmt.Println("FindId Error: ", err)
return
// panic(err)
}
fmt.Println("Person", result)
}

I found the documentation of the MongoDB was right. The correct way to do this is to wrap the struct to insert into an update operator.
The sample code provided by Neil Lunn, would look like:
package main
import (
"fmt"
"gopkg.in/mgo.v2"
"gopkg.in/mgo.v2/bson"
)
type Person struct {
Code string
Name string
}
func main() {
session, err := mgo.Dial("admin:admin#localhost");
if err != nil {
fmt.Println("Error: ", err)
return
}
defer session.Close()
session.SetMode(mgo.Monotonic, true)
c := session.DB("test").C("people")
var p = Person{
Code: "1234",
Name: "Bill",
}
upsertdata := bson.M{ "$set": p}
info , err2 := c.UpsertId( p.Code, upsertdata )
fmt.Println("UpsertId -> ", info, err2)
result := Person{}
err = c.FindId(p.Code).One(&result)
if err != nil {
fmt.Println("FindId Error: ", err)
return
}
fmt.Println("Person", result)
}
Thank you very much for your interest and help Neil.

You seem to be talking about assigning a struct with a custom _id field here. This really comes down to how you define your struct. Here is a quick example:
package main
import (
"fmt"
"gopkg.in/mgo.v2"
"gopkg.in/mgo.v2/bson"
)
type Person struct {
ID string `bson:"_id"`
Name string
}
func main() {
session, err := mgo.Dial("127.0.0.1");
if err != nil {
panic(err)
}
defer session.Close()
session.SetMode(mgo.Monotonic, true)
c := session.DB("test").C("people")
var p = Person{
ID: "1",
Name: "Bill",
}
_, err = c.UpsertId( p.ID, &p )
result := Person{}
err = c.Find(bson.M{"_id": p.ID}).One(&result)
if err != nil {
panic(err)
}
fmt.Println("Person", result)
}
So in the custom definition here I am mapping the ID field to bson _id and defining it's type as string. As shown in the example this is exactly what happens when serialized via UpsertId and then retrieved.
Now you have elaborated I'll point to the difference on the struct definition.
What I have produces this:
{ "_id": 1, "name": "Bill" }
What you have ( without the same mapping on the struct ) does this:
{ "_id": ObjectId("53cfa557e248860d16e1f7e0"), "code": 1, "name": "Bill" }
As you see, the _id given in the upsert will never match because none of your fields in the struct are mapped to _id. You need the same as I have:
type Person struct {
Code string `bson:"_id"`
Name string
}
That maps a field to the mandatory _id field, otherwise one is automatically produced for you.

Related

What is the bson syntax for $set in UpdateOne for the official mongo-go-driver

I am trying to get some familiarity with the official mongo-go-driver and the right syntax for UpdateOne.
My simplest full example follows:
(NOTE: in order to use this code you will need to substitute in your own user and server names as well as export the login password to the environment as MONGO_PW):
package main
import (
"context"
"fmt"
"os"
"go.mongodb.org/mongo-driver/bson"
"go.mongodb.org/mongo-driver/bson/primitive"
"go.mongodb.org/mongo-driver/mongo"
"go.mongodb.org/mongo-driver/mongo/options"
)
type DB struct {
User string
Server string
Database string
Collection string
Client *mongo.Client
Ctx context.Context
}
var db = DB{
User: <username>,
Server: <server_IP>,
Database: "test",
Collection: "movies",
Ctx: context.TODO(),
}
type Movie struct {
ID primitive.ObjectID `bson:"_id" json:"id"`
Name string `bson:"name" json:"name"`
Description string `bson:"description" json:"description"`
}
func main() {
if err := db.Connect(); err != nil {
fmt.Println("error: unable to connect")
os.Exit(1)
}
fmt.Println("connected")
// The code assumes the original entry for dunkirk is the following
// {"Name":"dunkirk", "Description":"a world war 2 movie"}
updatedMovie := Movie{
Name: "dunkirk",
Description: "movie about the british evacuation in WWII",
}
res, err := db.UpdateByName(updatedMovie)
if err != nil {
fmt.Println("error updating movie:", err)
os.Exit(1)
}
if res.MatchedCount < 1 {
fmt.Println("error: update did not match any documents")
os.Exit(1)
}
}
// UpdateByName changes the description for a movie identified by its name
func (db *DB) UpdateByName(movie Movie) (*mongo.UpdateResult, error) {
filter := bson.D{{"name", movie.Name}}
res, err := db.Client.Database(db.Database).Collection(db.Collection).UpdateOne(
db.Ctx,
filter,
movie,
)
if err != nil {
return nil, err
}
return res, nil
}
// Connect assumes that the database password is stored in the
// environment variable MONGO_PW
func (db *DB) Connect() error {
pw, ok := os.LookupEnv("MONGO_PW")
if !ok {
fmt.Println("error: unable to find MONGO_PW in the environment")
os.Exit(1)
}
mongoURI := fmt.Sprintf("mongodb+srv://%s:%s#%s", db.User, pw, db.Server)
// Set client options and verify connection
clientOptions := options.Client().ApplyURI(mongoURI)
client, err := mongo.Connect(db.Ctx, clientOptions)
if err != nil {
return err
}
err = client.Ping(db.Ctx, nil)
if err != nil {
return err
}
db.Client = client
return nil
}
The function signature for UpdateOne from the package docs is:
func (coll *Collection) UpdateOne(ctx context.Context, filter interface{},
update interface{}, opts ...*options.UpdateOptions) (*UpdateResult, error)
So I am clearly making some sort of mistake in creating the update interface{} argument to the function because I am presented with this error
error updating movie: update document must contain key beginning with '$'
The most popular answer here shows that I need to use a document sort of like this
{ $set: {"Name" : "The Matrix", "Decription" "Neo and Trinity kick butt" } }
but taken verbatim this will not compile in the mongo-go-driver.
I think I need some form of a bson document to comply with the Go syntax. What is the best and/or most efficient syntax to create this bson document for the update?
After playing around with this for a little while longer I was able to solve the problem after A LOT OF TRIAL AND ERROR using the mongodb bson package by changing the UpdateByName function in my code above as follows:
// UpdateByName changes the description for a movie identified by its name
func (db *DB) UpdateByName(movie Movie) (*mongo.UpdateResult, error) {
filter := bson.D{{"name", movie.Name}}
update := bson.D{{"$set",
bson.D{
{"description", movie.Description},
},
}}
res, err := db.Client.Database(db.Database).Collection(db.Collection).UpdateOne(
db.Ctx,
filter,
update,
)
if err != nil {
return nil, err
}
return res, nil
}
Note the use of bson.D{{$"set", .... It is unfortunate the way MongoDB has implemented the bson package this syntax still does not pass the go-vet. If anyone has a comment to fix the lint conflict below it would be appreciated.
go.mongodb.org/mongo-driver/bson/primitive.E composite literal uses unkeyed fields
In many cases you can replace construction
filter := bson.D{{"name", movie.Name}}
with
filter := bson.M{"name": movie.Name}
if arguments order dowsn't matter

Is there a way to get slice as result of Find()?

Now I'm doing:
sess := mongodb.DB("mybase").C("mycollection")
var users []struct {
Username string `bson:"username"`
}
err = sess.Find(nil).Select(bson.M{"username": 1, "_id": 0}).All(&users)
if err != nil {
fmt.Println(err)
}
var myUsers []string
for _, user := range users{
myUsers = append(myUsers, user.Username)
}
Is there a more effective way to get slice with usernames from Find (or another search function) directly, without struct and range loop?
The result of a MongoDB find() is always a list of documents. So if you want a list of values, you have to convert it manually just as you did.
Using a custom type (derived from string)
Also note that if you would create your own type (derived from string), you could override its unmarshaling logic, and "extract" just the username from the document.
This is how it could look like:
type Username string
func (u *Username) SetBSON(raw bson.Raw) (err error) {
doc := bson.M{}
if err = raw.Unmarshal(&doc); err != nil {
return
}
*u = Username(doc["username"].(string))
return
}
And then querying the usernames into a slice:
c := mongodb.DB("mybase").C("mycollection") // Obtain collection
var uns []Username
err = c.Find(nil).Select(bson.M{"username": 1, "_id": 0}).All(&uns)
if err != nil {
fmt.Println(err)
}
fmt.Println(uns)
Note that []Username is not the same as []string, so this may or may not be sufficient to you. Should you need a user name as a value of string instead of Username when processing the result, you can simply convert a Username to string.
Using Query.Iter()
Another way to avoid the slice copying would be to call Query.Iter(), iterate over the results and extract and store the username manually, similarly how the above custom unmarshaling logic does.
This is how it could look like:
var uns []string
it := c.Find(nil).Select(bson.M{"username": 1, "_id": 0}).Iter()
defer it.Close()
for doc := (bson.M{}); it.Next(&doc); {
uns = append(uns, doc["username"].(string))
}
if err := it.Err(); err != nil {
fmt.Println(err)
}
fmt.Println(uns)
I don't see what could be more effective than a simple range loop with appends. Without all the Mongo stuff your code basically is this and that's exactly how I would do this.
package main
import (
"fmt"
)
type User struct {
Username string
}
func main() {
var users []User
users = append(users, User{"John"}, User{"Jane"}, User{"Jim"}, User{"Jean"})
fmt.Println(users)
// Interesting part starts here.
var myUsers []string
for _, user := range users {
myUsers = append(myUsers, user.Username)
}
// Interesting part ends here.
fmt.Println(myUsers)
}
https://play.golang.com/p/qCwENmemn-R

MongoDB find query with combination of $or and $and in golang

I want to get rows where:
{repair field has "ac" OR {repair is "tv" and phone field in range 1091-1100}}
I am trying the following query:
type M map[string]interface{}
conditions := M{"name": M{"$regex": "me"},
"$or": []M{M{"repair": M{"$eq": "ac"}},
"$and": []M{M{"repair": M{"$eq": "tv"}}, M{"phone": M{"$gte": 1091, "$lte": 1100}}}}}
fmt.Println(conditions)
err = c.Find(conditions).Sort("phone").Limit(20).All(&j)
However, I am getting a compile error:
index must be non-negative integer constant
cannot use []M literal (type []M) as type M in array or slice literal.
You're missing one M{ before "$and" and after you add that don't forget to add another closing brace }.
the good and the bad for comparison.
I don't know what driver your using, but I would probably do something like this..
package main
import (
"log"
"time"
mgo "gopkg.in/mgo.v2"
"gopkg.in/mgo.v2/bson"
)
const (
databaseString = ""
)
var db DataStore
type DataStore struct {
Session *mgo.Session
}
// database
func (ds *DataStore) ConnectToDB() {
mongoDBDialInfo := &mgo.DialInfo{
Addrs: []string{"localhost:27017"},
Timeout: 1 * time.Hour,
}
sess, err := mgo.DialWithInfo(mongoDBDialInfo)
if err != nil {
sess.Refresh()
panic(err)
}
sess.SetMode(mgo.Monotonic, true)
db.Session = sess
}
// J is the expected mongo return object
type J struct {
ID bson.ObjectId `bson:"_id,omitempty" json:"_id"`
// example data below
Status string `bson:"status" json:"status"`
}
func init() {
db.ConnectToDB()
}
func main() {
colectionString := ""
// probably best if this was a mapped mgo struct see above
// var j bson.M
var j J
// your orignal code
// I don't know why you are using $eq couldn't you just do bson.M{"repair":"ac"}, and bson.M{"repair":"tv"}
conditions := bson.M{"name": bson.M{"$regex": "me"},
"$or": []bson.M{
bson.M{"repair": bson.M{"$eq": "ac"}},
},
"$and": []bson.M{
bson.M{"repair": bson.M{"$eq": "tv"}},
bson.M{"phone": bson.M{"$gte": 1091, "$lte": 1100}},
}}
err := db.Session.DB(databaseString).C(colectionString).Find(conditions).Sort("phone").Limit(20).All(&j)
if err != nil {
log.Fatal(err)
}
}
I would probably also end up creating a separate package for the mongo connection stuff so that I can write wrapper functions around calls,
func FindItem(db *mgo.Session, id string) (data, error) {
defer sess.Close()
var res data //some data type struct
err := sess.DB("my db").C("my collection").Find(bson.M{"user": someID}).One(&data)
return data, err
}
then I'd be able to do things like this, which allow for concurrency
res, err := packagemain.FindItem(sess.Copy(), someID)
and your original code your you where missing } and ,. I suggest you use go vet or an ide which vets the code for you. Also, mgo is the mongo driver you probably what you want to be using if you're not using it already.

Not understanding how mongoDB Update works in Go

I'm trying to implement a MongoDB update for a Go struct. Stripped down to essentials, it looks something like this:
type MyStruct struct {
Id bson.ObjectId `bson:"_id"`
Fruit string `bson:"fruit"`
}
func TestUpdate(t *testing.T) {
obj1 := MyStruct{Id: bson.NewObjectId(),Fruit: "apple"}
var obj2 MyStruct
session, _ := mgo.Dial("whatever")
col := session.DB("test").C("collection")
col.Insert(&obj1)
obj1.Fruit = "cherry"
if err := col.Update(obj1.Id, bson.M{"$set": &obj1}); err != nil {
t.Errorf(err.Error())
}
if err := col.Find(bson.M{"Id": obj1.Id}).One(&obj2); err != nil {
t.Errorf(err.Error())
}
if obj1.Fruit != obj2.Fruit {
t.Errorf("Expected %s, got %s", obj1.Fruit, obj2.Fruit)
}
}
This generates the error message, indicating that the value wasn't updated. What am I missing?
I understand that just updating one field is possible, but given that this is in a data layer, above which the code doesn't have any knowledge of MongoDB, that would be challenging to implement in a general way. I.e. I really need to make any updates to the Go object, and then update the copy of the object in the backing store. I suppose that I could retrieve the object and do a "diff" manually, constructing a "$set" document, but that doesn't seem like adding a retrieval every time I do an update would be very efficient.
Edit: Trying a map with "_id" deleted
I've tried amending the code to the following:
package testmgo
import (
mgo "gopkg.in/mgo.v2"
"gopkg.in/mgo.v2/bson"
"github.com/fatih/structs"
"testing"
)
type MyStruct struct {
Id bson.ObjectId `bson:"_id"`
Fruit string `bson:"fruit"`
}
func TestUpdate(t *testing.T) {
obj1 := MyStruct{Id: bson.NewObjectId(),Fruit: "apple"}
var obj2 MyStruct
session, _ := mgo.Dial("localhost")
col := session.DB("test").C("collection")
col.Insert(&obj1)
obj1.Fruit = "cherry"
omap := structs.Map(&obj1)
delete(omap, "_id")
if err := col.UpdateId(obj1.Id, bson.M{"$set": bson.M(omap)}); err != nil {
t.Errorf(err.Error())
}
if err := col.Find(bson.M{"Id": obj1.Id}).One(&obj2); err != nil {
t.Errorf(err.Error())
}
if obj1.Fruit != obj2.Fruit {
t.Errorf("Expected %s, got %s", obj1.Fruit, obj2.Fruit)
}
}
and am still receiving the same results (Expected cherry, got apple). Note that the call to UpdateId() is not returning an error.
The problem was that I was using the wrong field as the key. I had mapped "Id" to "_id", but was then asking MongoDB to find a record using the Go attribute name rather than the name. This works correctly:
package testmgo
import (
mgo "gopkg.in/mgo.v2"
"gopkg.in/mgo.v2/bson"
"testing"
)
type MyStruct struct {
Id bson.ObjectId `bson:"_id"`
Fruit string `bson:"fruit"`
}
func TestUpdate(t *testing.T) {
obj1 := MyStruct{Id: bson.NewObjectId(), Fruit: "apple"}
var obj2 MyStruct
session, _ := mgo.Dial("localhost")
col := session.DB("test").C("collection")
col.Insert(&obj1)
obj1.Fruit = "cherry"
if err := col.UpdateId(obj1.Id, bson.M{"$set": &obj1}); err != nil {
t.Errorf(err.Error())
}
if err := col.Find(bson.M{"_id": obj1.Id}).One(&obj2); err != nil {
t.Errorf(err.Error())
}
if obj1.Fruit != obj2.Fruit {
t.Errorf("Expected %s, got %s", obj1.Fruit, obj2.Fruit)
}
}

How to insert a json object array to mongodb in golang

My json looks like the following,
[
{
"key1": 1,
"key2": "val2"
},
{
"key1": 2,
"key2": "val2"
}
]
This json comes in string format and I want the objects in the json array to be inserted as individual records in mongodb. I referred to https://labix.org/mgo but wasn't able to find enough examples on the above use-case. Appreciate your thoughts in finding a solution.
Unmarshal the JSON to []interface{} and insert the result in the database. Assuming that c is an mgo.Collection and data is a []byte containing the JSON value, use the following code:
var v []interface{}
if err := json.Unmarshal(data, &v); err != nil {
// handle error
}
if err := c.Insert(v...); err != nil {
// handle error
}
In this example I will store the mixed array
test_string := '[[1,"a","b",2,"000000",[[1,2,3],[1,2,3]],"\"x","[y","'z",[[1,2,3],[1,2,3]]]]'
inside the mongodb as the json:
{datum: [[1,"a","b",2,"000000",[[1,2,3],[1,2,3]],"\"x","[y","'z",[[1,2,3],[1,2,3]]]]}
package main
import (
"strings"
"context"
"encoding/json"
"fmt"
"log"
"net/http"
"go.mongodb.org/mongo-driver/bson"
"go.mongodb.org/mongo-driver/bson/primitive"
)
type datum2 struct {
Datum interface{} `json:datum`
}
var userCollection = db().Database("goTest").Collection("users") // get collection "users" from db() which returns *mongo.Client
func typeinterface2mongo() {
var datum2 datum2_instance
var interfacevalue []interface{}
test_string := `[[1,"a","b",2,"000000",[[1,2,3],[1,2,3]],"\"x","[y","'z",[[1,2,3],[1,2,3]]]]`
if err := json.Unmarshal([]byte(test_string), &interfacevalue); err != nil {
fmt.Println(err)
return
}
fmt.Println(test_string)
fmt.Println(interfacevalue)
datum2_instance.Datum=interfacevalue
userCollection.InsertOne(context.TODO(), datum2_instance)
fmt.Println(datum2_instance)
fmt.Println(datum2_instance.Datum)
}
If you have json data already please follow from step 2.
If you have Xml data first you need to convert to the json format by using this package("github.com/basgys/goxml2json")
type JsonFileResponse struct {
JsonData string `bson:"JsonData " json:"JsonData"`
}
step 1: jsonData, err := xml2json.Convert(xml)
if err != nil {
panic("getting error while converting xml to json",err)
}
step 2: session need to open by using your mongodb credentials.
collection := session.DB("database name").C("Collection Name")
err = collection.Insert(JsonFileResponse{JsonData :json.String()})
if err != nil {
log.Fatal(err)
}