I have a fairly odd question that I have been trying to wrap my head around and am looking of some pointers as to the best approach. I am use mgo to filter a collection that contains a few different types of structs and am trying to cast from bson.M to the proper struct after the fact. Basically I'd like to be able to filter the collection and look at each result and based on a common field cast to the proper struct.
Here is sample of the structs I am trying to use.
type Action interface {
MyFunc() bool
}
type Struct1 struct {
Id bson.ObjectId `bson:"_id,omitempty"`
Type string `bson:"type"`
Struct1Only string `bson:"struct1only"`
}
func (s Struct1) MyFunc() bool {
return true
}
type Struct2 struct {
Id bson.ObjectId `bson:"_id,omitempty"`
Type string `bson:"type"`
Struct2Only string `bson:"struct2only"`
}
func (s Struct2) MyFunc() bool {
return true
}
My initial idea was to do something like:
var result bson.M{}
err := c.Find(bson.M{nil}).One(&result)
Then switch on the type field and cast to the proper struct, but honestly I am new to go and mongo and am sure there is better way to do this. Any suggestions? Thanks
You don't have to convert bson.M to struct, instead, you directly pass a struct pointer to the One function
var struct2 Struct2
err := c.Find(bson.M{nil}).One(&struct2)
In case of you still want to convert bson.M to struct, use Marshal and Unmarshal
var m bson.M
var s Struct1
// convert m to s
bsonBytes, _ := bson.Marshal(m)
bson.Unmarshal(bsonBytes, &s)
Related
I have a series of collections in a mongodb instance where I want to pull all the collections and decode all documents in the collections so I can mutate them as I go.
I am using go to list the collection names:
...
collections, err := db.ListCollectionNames(context.TODO(), bson.M{})
if err != nil {
log.Fatalf("Failed to get coll names: %v", err)
}
for _, coll := range collections {
collection := db.Collection(coll)
...
This works flawlessly and the next step would be to open a cursor to each collection and loop over the cursor and call decode on each object to its corresponding type.
I have defined all the types in a types package:
type Announcements struct {
Id string `bson:"_id"`
Content string `bson:"content"`
Title string `bson:"title"`
DateModified time.Time `bson:"dateModified,omitempty"`
ModifiedBy string `bson:"modifiedBy,omitempty"`
}
The issue is that I cant define the variable to be used for decoding dynamically.
This code would need to be repeated for every single collection
var obj Announcements // This would need to be dynamic
for cursor.Next(context.Background()) {
err := cursor.Decode(&obj)
if err != nil {
log.Fatal(err)
}
log.Printf("%#v\n", obj)
}
Is there a way to do this without repeating this many times? I realize a dynamically typed language would be better for this. Wanted to ask before I migrate the work to a scripting language.
I have tried using the reflect package and switch statements to instantiate the variable dynamnically and using the interface{} type but that force the decode to use bson types and not my defined structs.
-- edit --
I have tried to use a map to link collection names to types but to no avail.
var Collections = map[string]interface{}{
"announcements": Announcements{},
"badges": Badges{},
"betaUser": BetaUser{},
"bonusLearningItems": BonusLearningItems{},
"books": Books{},
...
}
So that when I use the obj var I tried an assignment like:
var obj types.Collections[coll]
hoping that would give me a variable of type Announcement if I was had looped to the announcements collection. but when I call decode it returns bson types.
I need to dynamically define the obj variable type.
There are many ways you can do this. One of them is by using a callback function:
func Process(cursor *mongo.Cursor, cb func(), out interface{}) {
for cursor.Next(context.Background()) {
err := cursor.Decode(out)
if err != nil {
log.Fatal(err)
}
cb()
log.Printf("%#v\n", obj)
}
}
Then define a map:
type collHandler struct {
New func() interface{}
Process func(interface{})
}
var collections=map[string]func() interface{} {
"announcements": collHandler{
New: func() interface {} {return &Announcement{}},
Process: func(data interface{}) {
processAnnouncements(data.(*Announcement))
},
},
...
}
And then you can:
handler:=collections[collectionName]
obj:=handler.New()
process(cursor,func() {
handler.Process(obj)
},
&obj)
This way, you can place the iteration into a common function, and use closures to deal with type specific logic.
I'm trying to query a view in golang with gorm and save the result in a struct EdgeType containing NodeType. (Basically trying to implement the graphql-relay connection specification).
The view contains 4 fields:
cursor bigint
id bigint
text text
user_id bigint
type NodeType struct {
ID int64
Text string
UserID int64
}
type EdgeType struct {
Cursor int64
Edge NodeType
}
func (receiver EdgeType) TableName() string {
return "connection_view"
}
As this alone doesn't work, so I tried to implement Scanner/Value interface.
Unfortunatelly, Scan isn't executed at all with the following call:
connection := make([]EdgeType, 0)
err := db.GormDB.Find(&connection).Error
This leads to my next problem: I can't debug my Scan/Value functions if they're not called. I've seen another answer pretty much describing my issue but with the advantage to be able to map the Child type to a specific geolocation type in postgres.
func (receiver *NodeType) Scan(src interface{}) error {
// raw := src.(string)
// fmt.Printf("raw: %s", raw)
// maybe parse the src string and set parsed data to receiver?
return nil
}
func (receiver NodeType) Value() (driver.Value, error) {
// what to return for multiple fields?
return receiver, nil
}
What can I return in my Value method to handle multiple fields at once? Or: Is that even possible/supported? Have I declared the Scan function wrongly or why isn't it called?
As mkopriva pointed out, there's a gorm:"embedded" tag.
type NodeType struct {
ID int64
Text string
UserID int64
}
type EdgeType struct {
Cursor int64
Edge NodeType `gorm:"embedded"`
}
This works without any Scan/Value functions.
In a db package, I don't want to expose MongoDB's primitive.ObjectID type as part of its public API. Instead, I want to define type Id interface{} within the db package and expose that. Since the driver doesn't know how to transform the database's ObjectID type into my custom ID type, I registered a type decoder like so:
type Id interface{}
var idType = reflect.TypeOf((*Id)(nil)).Elem()
type User struct {
Id `bson:"_id,omitempty"`
}
func main() {
reg := bson.NewRegistryBuilder().
RegisterTypeDecoder(idType, bsoncodec.ValueDecoderFunc(idDecodeValue)).
Build()
client, err := mongo.Connect(context.TODO(), options.Client().
ApplyURI("...").
SetRegistry(reg))
coll := client.Database("...").Collection("...")
var u0 User
coll.InsertOne(context.TODO(), u0)
var u1 User
coll.FindOne(context.TODO(), bson.D{}).Decode(&u1)
}
func idDecodeValue(dc bsoncodec.DecodeContext, vr bsonrw.ValueReader, val reflect.Value) error {
if !val.IsValid() || val.Type() != idType {
return bsoncodec.ValueDecoderError{
Name: "IdDecodeValue",
Types: []reflect.Type{idType},
Received: val,
}
}
oid, _ := vr.ReadObjectID()
val.Set(reflect.ValueOf(oid))
return nil
}
I saw that there exists a bsoncodec.NewEmptyInterfaceCodec() to use in place of idDecodeValue, but that only seems to work when User.Id's type is exactly interface{}. I wrote idDecodeValue based off of the existing codecs, but I'm not entirely sure what's going on (mostly due to not knowing when val.IsValid() would ever return false). Is all of this the best, most idiomatic way to go about supporting a custom ID type?
Go types like Int64 and String cannot store null values,
so I found I could use sql.NullInt64 and sql.NullString for this.
But when I use these in a Struct,
and generate JSON from the Struct with the json package,
then the format is different to when I use regular Int64 and String types.
The JSON has an additional level because the sql.Null*** is also a Struct.
Is there a good workaround for this,
or should I not use NULLs in my SQL database?
Types like sql.NullInt64 do not implement any special handling for JSON marshaling or unmarshaling, so the default rules apply. Since the type is a struct, it gets marshalled as an object with its fields as attributes.
One way to work around this is to create your own type that implements the json.Marshaller / json.Unmarshaler interfaces. By embedding the sql.NullInt64 type, we get the SQL methods for free. Something like this:
type JsonNullInt64 struct {
sql.NullInt64
}
func (v JsonNullInt64) MarshalJSON() ([]byte, error) {
if v.Valid {
return json.Marshal(v.Int64)
} else {
return json.Marshal(nil)
}
}
func (v *JsonNullInt64) UnmarshalJSON(data []byte) error {
// Unmarshalling into a pointer will let us detect null
var x *int64
if err := json.Unmarshal(data, &x); err != nil {
return err
}
if x != nil {
v.Valid = true
v.Int64 = *x
} else {
v.Valid = false
}
return nil
}
If you use this type in place of sql.NullInt64, it should be encoded as you expect.
You can test this example here: http://play.golang.org/p/zFESxLcd-c
If you use the null.v3 package, you won't need to implement any of the marshal or unmarshal methods. It's a superset of the sql.Null structs and is probably what you want.
package main
import "gopkg.in/guregu/null.v3"
type Person struct {
Name string `json:"id"`
Age int `json:"age"`
NickName null.String `json:"nickname"` // Optional
}
If you'd like to see a full Golang webserver that uses sqlite, nulls, and json you can consult this gist.
Suppose you have a workflow that consists of multiple embedded nodes of different types. Since nodes are of different types, I thought of using Golang interfaces here and came up with following:
type Workflow struct {
CreatedAt time.Time
StartedAt time.Time
CreatedBy string
Nodes []Node
}
type Node interface {
Exec() (int, error)
}
type EmailNode struct {
From string
To string
Subject string
Body string
}
type TwitterNode struct {
Tweet string
Image []byte
}
func (n *EmailNode) Exec() (int, error){
//send email
return 0, nil
}
func (n *TwitterNode) Exec() (int, error) {
//send tweet
return 0, nil
}
These workflows are stored in MongoDB and I have sample seed data in it. Using mgo, when I try to find a workflow (given its ID):
w = &Workflow{}
collection.FindID(bson.ObjectIdHex(id)).One(w)
I get the error - value of type bson.M is not assignable to type Node.
It also feels a bit weird to me that how would mgo unmarshal embedded Node documents into a Go struct without any type information. May be I need to look at the problem from another point of view.
Any suggestions would be highly appreciated.
You cannot use an interface in a document for the reason you noted. The decoder has no information about the type to create.
One way to handle this is to define a struct to hold the type information:
type NodeWithType struct {
Node Node `bson:"-"`
Type string
}
type Workflow struct {
CreatedAt time.Time
StartedAt time.Time
CreatedBy string
Nodes []NodeWithType
}
Implement the SetBSON function on this type. This function should decode the type string, create a value of the correct type based on that string and unmarshal to that value.
func (nt *NodeWithType) SetBSON(r bson.Raw) error {
}
Following Simon Fox's answer regarding the implementation of SetBSON, here is a more precise answer.
Let's take the original piece of code:
type Workflow struct {
CreatedAt time.Time
StartedAt time.Time
CreatedBy string
Nodes []Node
}
type Node interface {
Exec() (int, error)
}
type EmailNode struct {
From string
To string
Subject string
Body string
}
type TwitterNode struct {
Tweet string
Image []byte
}
func (n *EmailNode) Exec() (int, error){
//send email
return 0, nil
}
func (n *TwitterNode) Exec() (int, error) {
//send tweet
return 0, nil
}
What you want to do now is: once you unmarshal the BSON object from Mongo, you want to be able to know if each node is either an EmailNode or a TwitterNode.
As you are storing the nodes as a Node interface, mgo has no way to know what struct to implement, so you have to tell it explicitly. Here comes SetBSON.
In your example, the problem comes from this Workflow.Nodes, which is a slice of Node interface. As it is a simple slice, the best is that you create a custom type that mgo will be able to call when unmarshalling the BSON:
type NodesList []Node
// The updated Workflow struct:
type Workflow struct {
CreatedAt time.Time
StartedAt time.Time
CreatedBy string
Nodes NodesList
}
Now, you can implement SetBSON on this NodesList and describe how it works. Please note that as you are using a pointer, you can define what is contained inside the variable:
// Note that you must use a pointer to the slice
func (list *NodesList) SetBSON(raw raw.BSON) error {
// Now you need to create the objects according to your
// own domain logic and what's contained inside "raw":
if raw.something {
*list = append(*list, &TwitterNode{})
} else {
*list = append(*list, &EmailNode{})
}
return nil
}