BSON JSON id vs _id - mongodb

I am creating simple REST API using MongoDB and golang as a driver.
I was able to create POST request which can be found here:
terminal output.
However, when creating GET request, i always need to get it by bson _id. Would someone be able to let me know how to retrieve from json id not bson _id from golang script. If this is not possible, I would appreciate if someone let me know how to convert id to _id.
models/user.go
package models
import (
"gopkg.in/mgo.v2/bson"
)
type User struct {
Id bson.ObjectId `json:"id" bson: "_id"`
Name string `json:"name" bson: "name"`
Gender string `json:"gender" bson: "gender"`
Age int `json:"age" bson: "age"`
}
controllers/user.go file
type UserController struct {
session *mgo.Session}
func httpResponse(w http.ResponseWriter, jsonOut []byte, code int) {
w.Header().Set("Content-Type", "application/json")
w.WriteHeader(code)
fmt.Fprintf(w, "%s", jsonOut)
}
func NewUserCOntroller(s *mgo.Session) *UserController {
// return the address of UserController
return &UserController{s}
}
func (uc UserController) GetUser(w http.ResponseWriter, r *http.Request, p httprouter.Params) {
id := p.ByName("id")
if !bson.IsObjectIdHex(id) {
w.WriteHeader(http.StatusNotFound)
}
// oid is something you use in mongo
oid := bson.ObjectIdHex(id)
u := models.User{}
if err := uc.session.DB("mongolang").C("users").FindId(oid).One(&u); err != nil {
w.WriteHeader(404)
return
}
uj, err := json.Marshal(u)
if err != nil {
fmt.Println(err)
}
httpResponse(w, uj, http.StatusOK)
}
func (uc UserController) CreateUser(w http.ResponseWriter, r *http.Request, _ httprouter.Params) {
// its empty for now
u := models.User{}
json.NewDecoder(r.Body).Decode(&u)
u.Id = bson.NewObjectId()
uc.session.DB("mongolang").C("users").Insert(u)
jsonOut, _ := json.Marshal(u)
httpResponse(w, jsonOut, http.StatusOK)
fmt.Println("Response:", string(jsonOut), " 201 OK")
}
main.go
package main
import (
"net/http"
"github.com/julienschmidt/httprouter"
"gopkg.in/mgo.v2"
"mongo-golang/controllers"
)
func main() {
// create new instance
r := httprouter.New()
// new session
uc := controllers.NewUserCOntroller(getSession())
r.GET("/user/:id", uc.GetUser)
r.POST("/user", uc.CreateUser)
r.DELETE("/user/:id", uc.DeleteUser)
http.ListenAndServe("localhost:9000", r)
}
func getSession() *mgo.Session {
// get session and connect with mongo
s, err := mgo.Dial("mongodb://localhost")
if err != nil {
panic(err)
}
return s
}

Would someone be able to let me know how to retrieve from json id not bson _id from golang script.
You can do that only if you save bson Id as a Hex string in the additional field in an object, but don't do that. That is unnecessary and you won't get anything.
If this is not possible, I would appreciate if someone let me know how to convert id to _id.
There are two functions that converting bson ObjectID to string and string to bson ObjectID
for id to _id, you already use that function in your code:
oid := bson.ObjectIdHex(id)
bson.ObjectIdHex() convert hex representation of objectID to bson.objectID. Hex representation is what you would see in JSON output.
After you invoke that function you get bson.ObjetID. There is a method .Hex() that can get you Hex (text/json) representation of that object.
oid := bson.ObjectIdHex(id)
json_represenation_of_bson_object_id = oid.Hex()
Also, you use the old mongo driver, a new driver written and maintained by MongoDB is what you should use:
https://github.com/mongodb/mongo-go-driver

Related

Golang and MongoDB: DeleteMany with filter

I try to read and write and delete data from a Go application with the official mongodb driver for go (go.mongodb.org/mongo-driver).
Here is my struct I want to use:
Contact struct {
ID xid.ID `json:"contact_id" bson:"contact_id"`
SurName string `json:"surname" bson:"surname"`
PreName string `json:"prename" bson:"prename"`
}
// xid is https://github.com/rs/xid
I omit code to add to the collection as this is working find.
I can get a list of contacts with a specific contact_id using the following code (abbreviated):
filter := bson.D{}
cursor, err := contactCollection.Find(nil, filter)
for cur.Next(context.TODO()) {
...
}
This works and returns the documents. I thought about doing the same for delete or a matched get:
// delete - abbreviated
filter := bson.M{"contact_id": id}
result, _ := contactCollection.DeleteMany(nil, filter)
// result.DeletedCount is always 0, err is nil
if err != nil {
sendError(c, err) // helper function
return
}
c.JSON(200, gin.H{
"ok": true,
"message": fmt.Sprintf("deleted %d patients", result.DeletedCount),
}) // will be called, it is part of a webservice done with gin
// get complete
func Get(c *gin.Context) {
defer c.Done()
id := c.Param("id")
filter := bson.M{"contact_id": id}
cur, err := contactCollection.Find(nil, filter)
if err != nil {
sendError(c, err) // helper function
return
} // no error
contacts := make([]types.Contact, 0)
for cur.Next(context.TODO()) { // nothing returned
// create a value into which the single document can be decoded
var elem types.Contact
err := cur.Decode(&elem)
if err != nil {
sendError(c, err) // helper function
return
}
contacts = append(contacts, elem)
}
c.JSON(200, contacts)
}
Why does the same filter does not work on delete?
Edit: Insert code looks like this:
_, _ = contactCollection.InsertOne(context.TODO(), Contact{
ID: "abcdefg",
SurName: "Demo",
PreName: "on stackoverflow",
})
Contact.ID is of type xid.ID, which is a byte array:
type ID [rawLen]byte
So the insert code you provided where you use a string literal to specify the value for the ID field would be a compile-time error:
_, _ = contactCollection.InsertOne(context.TODO(), Contact{
ID: "abcdefg",
SurName: "Demo",
PreName: "on stackoverflow",
})
Later in your comments you clarified that the above insert code was just an example, and not how you actually do it. In your real code you unmarshal the contact (or its ID field) from a request.
xid.ID has its own unmarshaling logic, which might interpret the input data differently, and might result in an ID representing a different string value than your input. ID.UnmarshalJSON() defines how the string ID will be converted to xid.ID:
func (id *ID) UnmarshalJSON(b []byte) error {
s := string(b)
if s == "null" {
*id = nilID
return nil
}
return id.UnmarshalText(b[1 : len(b)-1])
}
As you can see, the first byte is cut off, and ID.UnmarshalText() does even more "magic" on it (check the source if you're interested).
All-in-all, to avoid such "transformations" happen in the background without your knowledge, use a simple string type for your ID, and do necessary conversions yourself wherever you need to store / transmit your ID.
For the ID Field, you should use the primitive.ObjectID provided by the bson package.
"go.mongodb.org/mongo-driver/bson/primitive"
ID primitive.ObjectID `json:"_id" bson:"_id"`

Omitempty in struct not omitting

I'm trying to store data to MongoDB without sending null data. The Struct in question is Poll and Question. Incoming data can range from have 2 questions, to 5. So if a user only enters 2 questions I wont have a need to use the 3 other fields in Poll struct. Id rather have the fields not appear at all than send null data to the server.
package main
// Omit Empty not working
type Poll struct {
Id bson.ObjectId `bson:"_id"`
Quest0 *Question `json:"quest0,omitempty"`
Quest1 *Question `json:"quest1,omitempty"`
Quest2 *Question `json:"quest2,omitempty"`
Quest3 *Question `json:"quest3,omitempty"`
Quest4 *Question `json:"quest4,omitempty"`
Quest5 *Question `json:"quest5,omitempty"`
}
type Question struct {
Count *int `json:"count,omitempty"`
Question *string `json:"question,omitempty"`
}
type ReceivedPoll struct {
Quest0 string `db:"quest0"`
Quest1 string `db:"quest1"`
Quest2 string `db:"quest2"`
Quest3 string `db:"quest3"`
Quest4 string `db:"quest4"`
Quest5 string `db:"quest5"`
}
func main() {
fmt.Println("server running...")
router := httprouter.New()
router.POST("/api/create", api)
router.NotFound = http.FileServer(http.Dir("./public"))
log.Fatal(http.ListenAndServe(":5000", router))
}
func api(w http.ResponseWriter, r *http.Request, ps httprouter.Params) {
w.Header().Set("Content-type", "application/json")
session, err := mgo.Dial(mkey)
if err != nil {
panic(err)
}
defer session.Close()
fmt.Println("is this running?")
switch r.URL.String() {
case "/api/create":
// LOOK HERE
poll := &Poll{}
json.NewDecoder(r.Body).Decode(&poll)
poll.Id = bson.NewObjectId()
fmt.Println(*poll)
c := session.DB("abase").C("polls")
err = c.Insert(*poll)
if err != nil {
fmt.Println(err)
}
rz, _ := json.Marshal(poll.Id)
w.Write(rz)
}
}
Add the bson key used by the mgo BSON encoder. The encoder ignores the json key. See bson.Marshal documentation for the details.
type Poll struct {
Id bson.ObjectId `bson:"_id"`
Quest0 *Question `json:"quest0,omitempty" bson:"ques0:omitempty"`
...

Is there a way to get slice as result of Find()?

Now I'm doing:
sess := mongodb.DB("mybase").C("mycollection")
var users []struct {
Username string `bson:"username"`
}
err = sess.Find(nil).Select(bson.M{"username": 1, "_id": 0}).All(&users)
if err != nil {
fmt.Println(err)
}
var myUsers []string
for _, user := range users{
myUsers = append(myUsers, user.Username)
}
Is there a more effective way to get slice with usernames from Find (or another search function) directly, without struct and range loop?
The result of a MongoDB find() is always a list of documents. So if you want a list of values, you have to convert it manually just as you did.
Using a custom type (derived from string)
Also note that if you would create your own type (derived from string), you could override its unmarshaling logic, and "extract" just the username from the document.
This is how it could look like:
type Username string
func (u *Username) SetBSON(raw bson.Raw) (err error) {
doc := bson.M{}
if err = raw.Unmarshal(&doc); err != nil {
return
}
*u = Username(doc["username"].(string))
return
}
And then querying the usernames into a slice:
c := mongodb.DB("mybase").C("mycollection") // Obtain collection
var uns []Username
err = c.Find(nil).Select(bson.M{"username": 1, "_id": 0}).All(&uns)
if err != nil {
fmt.Println(err)
}
fmt.Println(uns)
Note that []Username is not the same as []string, so this may or may not be sufficient to you. Should you need a user name as a value of string instead of Username when processing the result, you can simply convert a Username to string.
Using Query.Iter()
Another way to avoid the slice copying would be to call Query.Iter(), iterate over the results and extract and store the username manually, similarly how the above custom unmarshaling logic does.
This is how it could look like:
var uns []string
it := c.Find(nil).Select(bson.M{"username": 1, "_id": 0}).Iter()
defer it.Close()
for doc := (bson.M{}); it.Next(&doc); {
uns = append(uns, doc["username"].(string))
}
if err := it.Err(); err != nil {
fmt.Println(err)
}
fmt.Println(uns)
I don't see what could be more effective than a simple range loop with appends. Without all the Mongo stuff your code basically is this and that's exactly how I would do this.
package main
import (
"fmt"
)
type User struct {
Username string
}
func main() {
var users []User
users = append(users, User{"John"}, User{"Jane"}, User{"Jim"}, User{"Jean"})
fmt.Println(users)
// Interesting part starts here.
var myUsers []string
for _, user := range users {
myUsers = append(myUsers, user.Username)
}
// Interesting part ends here.
fmt.Println(myUsers)
}
https://play.golang.com/p/qCwENmemn-R

Querying mongodb from golang using the _id stored in an array

So here is my question. I have an array which are stored the _ids of mongodbs objects. Whats the right way to retrieve them all in one query using the mgo and bson package?
So if the array is like that: ids:=["543d171c5b2c12420dd016","543d171c5b2dd016"]
How we make the query ? I tried that but I know its wrong.
query := bson.M{"_id": bson.M{"$in": ids}}
c.Find(query).All()
Thanks in advance
If the documents are stored with string ids, then the code looks correct.
The ids look like hex encoded object ids. If the object identifiers are object ids, then you need to the convert the hex strings to object ids:
oids := make([]bson.ObjectId, len(ids))
for i := range ids {
oids[i] = bson.ObjectIdHex(ids[i])
}
query := bson.M{"_id": bson.M{"$in": oids}}
MongoDB syntax for go.mongodb.org/mongo-driver has been updated, this should work using the official driver.
oids := make([]primitive.ObjectID, len(ids))
for i := range ids {
objID, err := primitive.ObjectIDFromHex(ids[i])
if err == nil {
oids = append(oids, objID)
}
}
This is to convert back to a struct that can be used through out the app
type MongoUser struct {
ID *primitive.ObjectID `json:"id" bson:"_id"`
FirstName string `json:"first_name" bson:"firstName"`
LastName string `json:"last_name" bson:"lastName"`
Email string `json:"email" bson:"email"`
}
This is a helper method that takes your slice of ids and turns it into the object id type.
func formatObjectIdMultiple(hex []string) ([]primitive.ObjectID, error) {
var list []primitive.ObjectID
oids := make([]primitive.ObjectID, len(hex))
for _, i := range hex {
objectId, err := primitive.ObjectIDFromHex(i)
if err != nil {
return nil, err
}
oids = append(oids, objectId)
}
return list, nil
}
Here is my method for the db. Its important you use bson.M for some reason bson.D does not work with this. Also dont forget to close your cursor the defer function will close it at the end of your GetMultipleUser function.
func (mongo *Mongo) GetMultipleUser(ids []string) ([]*MongoUser, error) {
objectIDs, err := formatObjectIdMultiple(ids)
if err != nil {
return nil, err
}
query := bson.M{"_id": bson.M{"$in": objectIDs}}
coll := mongo.Con.Database("dbName").Collection("users")
cursor, err := coll.Find(context.Background(), query)
if err != nil {
return nil, err
}
defer func() {
cursor.Close(context.Background())
}()
var output []*MongoUser
for cursor.Next(context.Background()) {
var temp *MongoUser
cursor.Decode(&temp)
output = append(output, temp)
}
return output, nil
}

Golang mongodb mgo driver Upsert / UpsertId documentation

The mongodb documentation says:
The fields and values of both the and parameters if the parameter contains only update operator expressions. The update creates a base document from the equality clauses in the parameter, and then applies the update expressions from the parameter.
And the mgo documentation says:
Upsert finds a single document matching the provided selector document and modifies it according to the update document. If no document matching the selector is found, the update document is applied to the selector document and the result is inserted in the collection.
But if i do an upsert like this:
session.UpsertId(data.Code, data)
I end up with an entry which have an ObjectID generated automatically by mongodb, instead of data.Code.
this means that UpsertId expect data to be formated with update operators and you can't use a an arbitrary struct? Or what i'm missing here?
Pd. Mongo 2.4.9 mgo v2 golang go version devel +f613443bb13a
EDIT:
This is a sample of what i mean, using the sample code from Neil Lunn:
package main
import (
"fmt"
"gopkg.in/mgo.v2"
// "gopkg.in/mgo.v2/bson"
)
type Person struct {
Code string
Name string
}
func main() {
session, err := mgo.Dial("admin:admin#localhost");
if err != nil {
fmt.Println("Error: ", err)
return
// panic(err)
}
defer session.Close()
session.SetMode(mgo.Monotonic, true)
c := session.DB("test").C("people")
var p = Person{
Code: "1234",
Name: "Bill",
}
_, err = c.UpsertId( p.Code, &p )
result := Person{}
err = c.FindId(p.Code).One(&result)
if err != nil {
fmt.Println("FindId Error: ", err)
return
// panic(err)
}
fmt.Println("Person", result)
}
I found the documentation of the MongoDB was right. The correct way to do this is to wrap the struct to insert into an update operator.
The sample code provided by Neil Lunn, would look like:
package main
import (
"fmt"
"gopkg.in/mgo.v2"
"gopkg.in/mgo.v2/bson"
)
type Person struct {
Code string
Name string
}
func main() {
session, err := mgo.Dial("admin:admin#localhost");
if err != nil {
fmt.Println("Error: ", err)
return
}
defer session.Close()
session.SetMode(mgo.Monotonic, true)
c := session.DB("test").C("people")
var p = Person{
Code: "1234",
Name: "Bill",
}
upsertdata := bson.M{ "$set": p}
info , err2 := c.UpsertId( p.Code, upsertdata )
fmt.Println("UpsertId -> ", info, err2)
result := Person{}
err = c.FindId(p.Code).One(&result)
if err != nil {
fmt.Println("FindId Error: ", err)
return
}
fmt.Println("Person", result)
}
Thank you very much for your interest and help Neil.
You seem to be talking about assigning a struct with a custom _id field here. This really comes down to how you define your struct. Here is a quick example:
package main
import (
"fmt"
"gopkg.in/mgo.v2"
"gopkg.in/mgo.v2/bson"
)
type Person struct {
ID string `bson:"_id"`
Name string
}
func main() {
session, err := mgo.Dial("127.0.0.1");
if err != nil {
panic(err)
}
defer session.Close()
session.SetMode(mgo.Monotonic, true)
c := session.DB("test").C("people")
var p = Person{
ID: "1",
Name: "Bill",
}
_, err = c.UpsertId( p.ID, &p )
result := Person{}
err = c.Find(bson.M{"_id": p.ID}).One(&result)
if err != nil {
panic(err)
}
fmt.Println("Person", result)
}
So in the custom definition here I am mapping the ID field to bson _id and defining it's type as string. As shown in the example this is exactly what happens when serialized via UpsertId and then retrieved.
Now you have elaborated I'll point to the difference on the struct definition.
What I have produces this:
{ "_id": 1, "name": "Bill" }
What you have ( without the same mapping on the struct ) does this:
{ "_id": ObjectId("53cfa557e248860d16e1f7e0"), "code": 1, "name": "Bill" }
As you see, the _id given in the upsert will never match because none of your fields in the struct are mapped to _id. You need the same as I have:
type Person struct {
Code string `bson:"_id"`
Name string
}
That maps a field to the mandatory _id field, otherwise one is automatically produced for you.