Using mgo I'm unable to store any meaningful data. Only the _id gets stored
type Person struct {
name string
age int
}
func main() {
session, err := mgo.Dial("localhost")
if err != nil {
log.Fatal(err)
}
defer session.Close()
p := Person{"Joe", 50}
ppl := session.DB("rest").C("people")
ppl.Insert(p)
}
The result in Mongo is just the _id field - no sign of "Joe".
Using go 1.1.2 on Arch linux, MongoDB 2.4.6.
type Person struct {
name string
age int
}
The mgo package can't access unexported (lowercase) fields of your struct (i.e. no other package than the one the struct is defined in can). You need to export them (first letter must be upper case), like this:
type Person struct {
Name string
Age int
}
If you wish to have the field names in lower case in the DB you must provide a struct tag for them, like this:
type Person struct {
Name string `bson:"name"`
Age int `bson:"age"`
}
See the documentation on names:
Names are as important in Go as in any other language. They even have
semantic effect: the visibility of a name outside a package is
determined by whether its first character is upper case. [...]
EDIT:
Gustavo Niemeyer (author of the mgo and bson packages) noted in the comments that unlike the json package, the bson marshaller will lowercase all struct field names when commiting to the database, effectively making the last step in this answer superfluous.
Related
My data structure is
{
_id: ObjectID,
...
fields: [
{ name: "Aryan" },
{ books : [ 1,2,3 ] },
]
}
In our application a user can define his own fields data structure but with a key value structure. So, we had no way of knowing the structure of the data.
So in our document struct we had
type Document struct {
Fields map[string]interface{}
}
as the second parameter returned by mongo was primitive.A ( []interface{} under the hood ), the individual item could have been an array, map, anything.
But we couldn't range over that for it being.
How can I get the individual values like the book ids [1,2,3] or the name value "Aryan" ?
After a couple of attempts to solving this, I was at a state where my current element it self is an interface{} ( [1,2,3] in this case ) and I couldn't get the individual 1,2,3s.
But finally managed to solve it
for k, val := range doc.Fields {
v := reflect.ValueOf(val)
switch reflect.TypeOf(t).Kind() {
case reflect.Slice:
// getting the individual ids
for i := 0; i < v.Len(); i++ {
fmt.Println(v.Index(i))
}
case reflect.String:
default:
// handle
}
}
Note: v.Index(i) returns a data type of Value .
to change the data type
v.Index(i).Interface().(string) // string
v.Index(i).Interface().(float64) // float
Ideally you should avoid working with interface{} type since it's very error prone and the compiler can't help you. The idiomatic way is to define a struct for your model with BSON tags like in this example`
type MyType struct {
ID primitive.ObjectID `bson:"_id,omitempty"`
Fields []Field `bson:"fields,omitempty"`
}
type Field struct {
Name string `bson:"name,omitempty"`
Books []int `bson:"books,omitempty"`
}
Field here is defined as the reunion of all possible fields which again is not ideal but at least the compiler can help you and developers know what to expect from the database document.
(sorry this question turned out longer than I had thought...)
I'm using Go and MongoDB with the mgo driver. I'm trying to persist and retrieve different structs (implementing a common interface) in the same MongoDB collection. I'm coming from the Java world (where this is very easily done with Spring with literally no config) and I'm having a hard time doing something similar with Go.
I have read every last related article or post or StackExchange question I could find, but still I haven't found a complete solution. This includes:
Unstructured MongoDB collections with mgo
How do you create a new instance of a struct from its type at run time in Go?
Golang reflect: Get Type representation from name?
Here's a simplified setup I use for testing. Suppose two structs S1 and S2, implementing a common interface I. S2 has an implicit field of type S1, meaning that structure-wise S2 embeds a S1 value, and that type-wise S2 implements I.
type I interface {
f1()
}
type S1 struct {
X int
}
type S2 struct {
S1
Y int
}
func (*S1) f1() {
fmt.Println("f1")
}
Now I can save an instance of S1 or S2 easily using mgo.Collection.Insert(), but to properly populate a value using mgo.Collection.Find().One() for example, I need to pass a pointer to an existing value of S1 or S2, which means I already know the type of the object I want to read!!
I want to be able to retrieve a document from the MongoDB collection without knowing if it's a S1, or a S2, or in fact any object that implements I.
Here's where I am so far: instead of directly saving the object I want to persist, I save a Wrapper struct that holds the MongoDB id, an identifier of the type, and the actual value. The type identifier is a concatenation of packageName + "." + typeName, and it is used to lookup the type in a type registry, since there is no native mechanism to map from a type name to a Type object in Go. This means I need to register the types that I want to be able to persist and retrieve, but I could live with that. Here's how it goes:
typeregistry.Register(reflect.TypeOf((*S1)(nil)).Elem())
typeregistry.Register(reflect.TypeOf((*S2)(nil)).Elem())
Here's the code for the type registry:
var types map[string]reflect.Type
func init() {
types = make(map[string]reflect.Type)
}
func Register(t reflect.Type) {
key := GetKey(t)
types[key] = t
}
func GetKey(t reflect.Type) string {
key := t.PkgPath() + "." + t.Name()
return key
}
func GetType(key string) reflect.Type {
t := types[key]
return t
}
The code for saving an object is quite straightforward:
func save(coll *mgo.Collection, s I) (bson.ObjectId, error) {
t := reflect.TypeOf(s)
wrapper := Wrapper{
Id: bson.NewObjectId(),
TypeKey: typeregistry.GetKey(t),
Val: s,
}
return wrapper.Id, coll.Insert(wrapper)
}
The code for retrieving an object is a bit more tricky:
func getById(coll *mgo.Collection, id interface{}) (*I, error) {
// read wrapper
wrapper := Wrapper{}
err := coll.Find(bson.M{"_id": id}).One(&wrapper)
if err != nil {
return nil, err
}
// obtain Type from registry
t := typeregistry.GetType(wrapper.TypeKey)
// get a pointer to a new value of this type
pt := reflect.New(t)
// FIXME populate value using wrapper.Val (type bson.M)
// HOW ???
// return the value as *I
i := pt.Elem().Interface().(I)
return &i, nil
}
This partially works as the returned object is typed correctly, but what i can't figure out is how to populate the value pt with the data retrieved from MongoDB which is stored in wrapper.Val as a bson.M.
I have tried the following but it doesn't work:
m := wrapper.Val.(bson.M)
bsonBytes, _ := bson.Marshal(m)
bson.Unmarshal(bsonBytes, pt)
So basically the remaining problem is: how to populate an unknown structure from a bson.M value? I'm sure there has to be an easy solution...
Thanks in advance for any help.
Here's a Github gist with all the code: https://gist.github.com/ogerardin/5aa272f69563475ba9d7b3194b12ae57
First, you should always check returned errors, always. bson.Marshal() and bson.Unmarshal() return errors which you don't check. Doing so reveals why it doesn't work:
unmarshal can't deal with struct values. Use a pointer
pt is of type reflect.Value (which happens to be a struct), not something you should pass to bson.Unmarshal(). You should pass e.g. a pointer to a struct value you want to unmarshal into (which will be wrapped in an interface{} value). So call Value.Interface() on the value returned by reflect.New():
pt := reflect.New(t).Interface()
You can pass this to bson.Unmarshal():
bsonBytes, err := bson.Marshal(m)
if err != nil {
panic(err)
}
if err = bson.Unmarshal(bsonBytes, pt); err != nil {
panic(err)
}
(In your real code you want to do something else than panic, this is just to show you should always check errors!)
Also note that it is possible to directly convert maps to structs (directly meaning without marshaling and unmarshaling). You may implement it by hand or use a ready 3rd party lib. For details, see Converting map to struct
Also note that there are more clever ways to solve what you want to do. You could store the type in the ID itself, so if you have the ID, you can construct a value of the type to unmarshal into the query result, so you could skip this whole process. It would be a lot more simple and a lot more efficient.
For example you could use the following ID structure:
<type>-<id>
For example:
my.package.S1-123
When fetching / loading this document, you could use reflection to create a value of my.package.S1, and unmarshal into that directly (pass that to Query.One()).
As per #icza 's comments, here's a modified version of getById() that actually works:
func getById(coll *mgo.Collection, id interface{}) (*I, error) {
// read wrapper
wrapper := Wrapper{}
err := coll.Find(bson.M{"_id": id}).One(&wrapper)
if err != nil {
return nil, err
}
// obtain Type from registry
t := typeregistry.GetType(wrapper.TypeKey)
// get a pointer to a new value of this type
pt := reflect.New(t)
// populate value using wrapper.Val
err = mapstructure.Decode(wrapper.V, pt.Interface())
if err != nil {
return nil, err
}
// return the value as *I
i := pt.Elem().Interface().(I)
return &i, nil
}
Conversion from bson.M to the struct is handled by https://github.com/mitchellh/mapstructure instead of marshalling-unmarshaling.
I am working on a project that uses combination of Go and MongoDB. I am stuck at a place where I have a struct like:
type Booking struct {
// booking fields
Id int `json:"_id,omitempty" bson:"_id,omitempty"`
Uid int `json:"uid,omitempty" bson:"uid,omitempty"`
IndustryId int `json:"industry_id,omitempty" bson:"industry_id,omitempty"`
LocationId int `json:"location_id,omitempty" bson:"location_id,omitempty"`
BaseLocationId int `json:"base_location_id,omitempty" bson:"base_location_id,omitempty"`
}
In this struct, the field Id is of int type. But as we know MongoDB's default id is of bsonObject type. Some times, the system generates the default MongoDB id in Id field.
To overcome this I have modified the struct like this:
type Booking struct {
// booking fields
Id int `json:"_id,omitempty" bson:"_id,omitempty"`
BsonId bson.ObjectId `json:"bson_id" bson:"_id,omitempty"`
Uid int `json:"uid,omitempty" bson:"uid,omitempty"`
IndustryId int `json:"industry_id,omitempty" bson:"industry_id,omitempty"`
LocationId int `json:"location_id,omitempty" bson:"location_id,omitempty"`
BaseLocationId int `json:"base_location_id,omitempty" bson:"base_location_id,omitempty"`
}
In above struct, I have mapped the same _id field in two different struct fields Id (type int) and BsonId (type bson.ObjectId). I want if id of integer type comes, it maps under Id otherwise under BsonId.
But this thing is giving following error:
Duplicated key '_id' in struct models.Booking
How can I implement this type of thing with Go Structs ??
Update:
Here is the code I have written for custom marshaling/unmarshaling:
func (booking *Booking) SetBSON(raw bson.Raw) (err error) {
type bsonBooking Booking
if err = raw.Unmarshal((*bsonBooking)(booking)); err != nil {
return
}
booking.BsonId, err = booking.Id
return
}
func (booking *Booking) GetBSON() (interface{}, error) {
booking.Id = Booking.BsonId
type bsonBooking *Booking
return bsonBooking(booking), nil
}
But this is giving type mismatch error of Id and BsonId fields. What should I do now ?
In your original struct you used this tag for the Id field:
bson:"_id,omitempty"
This means if the value of the Id field is 0 (zero value for the int type), then that field will not be sent to MongoDB. But the _id property is mandatory in MongoDB, so in this case the MongoDB server will generate an ObjectId for it.
To overcome this, the easiest is to ensure the Id will is always non-zero; or if the 0 is a valid id, remove the omitempty option from the tag.
If your collection must allow ids of mixed type (int and ObjectId), then the easiest would be to define the Id field with type of interface{} so it can accomodate both (actually all) types of key values:
Id interface{} `json:"_id,omitempty" bson:"_id,omitempty"`
Yes, working with this may be a little more cumbersome (e.g. if you explicitly need the id as int, you need to use type assertion), but this will solve your issue.
If you do need 2 id fields, one with int type and another with ObjectId type, then your only option will be to implement custom BSON marshaling and unmarshaling. This basically means to implement the bson.Getter and / or bson.Setter interfaces (one method for each) on your struct type, in which you may do anything you like to populate your struct or assemble the data to be actually saved / inserted. For details and example, see Accessing MongoDB from Go.
Here's an example how using custom marshaling it may look like:
Leave out the Id and BsonId fields from marshaling (using the bson:"-" tag), and add a 3rd, "temporary" id field:
type Booking struct {
Id int `bson:"-"`
BsonId bson.ObjectId `bson:"-"`
TempId interface{} `bson:"_id"`
// rest of your fields...
}
So whatever id you have in your MongoDB, it will end up in TempId, and only this id field will be sent and saved in MongoDB's _id property.
Use the GetBSON() method to set TempId from the other id fields (whichever is set) before your struct value gets saved / inserted, and use SetBSON() method to "copy" TempId's value to one of the other id fields based on its dynamic type after the document is retrieved from MongoDB:
func (b *Booking) GetBSON() (interface{}, error) {
if b.Id != 0 {
b.TempId = b.Id
} else {
b.TempId = b.BsonId
}
return b, nil
}
func (b *Booking) SetBSON(raw bson.Raw) (err error) {
if err = raw.Unmarshal(b); err != nil {
return
}
if intId, ok := b.TempId.(int); ok {
b.Id = intId
} else bsonId, ok := b.TempId.(bson.ObjectId); ok {
b.BsonId = bsonId
} else {
err = errors.New("invalid or missing id")
}
return
}
Note: if you dislike the TempId field in your Booking struct, you may create a copy of Booking (e.g. tempBooking), and only add TempId into that, and use tempBooking for marshaling / unmarshaling. You may use embedding (tempBooking may embed Booking) so you can even avoid repetition.
Some JSON data I am getting have spaces in the key names. I am using standard encoding/json library to unmarshal the data. However it is unable to understand the keys with spaces in the schema. For e.g. following code:
package main
import (
"encoding/json"
"fmt"
)
func main() {
var jsonBlob = []byte(`[
{"Na me": "Platypus", "Order": "Monotremata"},
{"Na me": "Quoll", "Order": "Dasyuromorphia"}
]`)
type Animal struct {
Name string `json: "Na me"`
Order string `json: "Order,omitempty"`
}
var animals []Animal
err := json.Unmarshal(jsonBlob, &animals)
if err != nil {
fmt.Println("error:", err)
}
fmt.Printf("%+v", animals)
}
Gives the output as:
[{Name: Order:Monotremata} {Name: Order:Dasyuromorphia}]
So in the schema the library removes the space(from Na me) and try to find the key (Name), which is obviously not present. Any suggestion what can I do here?
Your json tag specification is incorrect, that's why the encoding/json library defaults to the field name which is Name. But since there is no JSON field with "Name" key, Animal.Name will remain its zero value (which is the empty string "").
Unmarshaling Order will still work, because the json package will use the field name if json tag specification is missing (tries with both lower and upper-case). Since the field name is identical to the JSON key, it works without extra JSON tag mapping.
You can't have a space in the tag specification after the colon and before the quotation mark:
type Animal struct {
Name string `json:"Na me"`
Order string `json:"Order,omitempty"`
}
With this simple change, it works (try it on the Go Playground):
[{Name:Platypus Order:Monotremata} {Name:Quoll Order:Dasyuromorphia}]
I tried to retrieve a document from my collection with unique id.
I have a collection with fields: name, age, city, and rank. I want to get 'city' results from mongodb using golang.
My struct code
type exp struct {
name string `bson:"name"`
age int `bson:"age"`
city string `bson:"city"`
rank int `bson:"rank"`
}
With the following code to retrieve results from mongodb:
var result []exp //my struct type
err = coll.Find(bson.M{"City":bson.M{}}).Distinct("City",&result)
fmt.Println(result)
With this code I get an empty array as the result. How would I get all the cities?
Try this code
var result []string
err = c.Find(nil).Distinct("city", &result)
if err != nil {
log.Fatal(err)
}
fmt.Println(result)
Due to restrictions in reflection, mgo (as well as encoding/json and other similar packages) is unable to use unexported fields to marshal or unmarshal data. What you need to do is to export your fields by capitalize the first letter:
type exp struct {
Name string `bson:"name"`
Age int `bson:"age"`
City string `bson:"city"`
Rank int `bson:"rank"`
}
A side note: you do not need to specify the bson tags if the desired name is the same as the lowercase field name. The documentation for bson states:
The lowercased field name is used as the key for each exported field,
but this behavior may be changed using the respective field tag.
Edit:
I just realized you did get an empty slice and not a slice with empty struct fields. My answer is then not an actual answer to the question, but it is still an issue that you need to consider.