I am trying to do a basic TODO list application in Go. I am creating the CRUD operations on my cluster from mongodb atlas. But I have a problem decoding BSON objects. For my model I use a struct which is unimported but it implements a interface which is used in the repo. When trying to read from database I get this error:
panic: no decoder found for interfaces.IToDoItem
I know I should somehow implement a decoder for my interface but can not realize how to do it, without exporting my main struct from model. That would also mean I won't have a privacy in the model and the items in the model can be accessed in any mode all around the program, a thing which I think is wrong.
Here is my code:
model.go
type toDoItem struct{
ItemId int `bson:"itemId,omitempty"`
Title string `bson:"title,omitempty"`
Description string `bson:"description,omitempty"`
}
func New(itemId int,title string,description string) interfaces.IToDoItem {
return toDoItem{
ItemId: itemId,
Title: title,
Description: description,
}
}
func (item toDoItem)GetItemId()int{
return item.ItemId
}
func (item toDoItem)GetTitle()string{
return item.Title
}
func (item toDoItem)GetDescription()string{
return item.Description
}
Interface
type IToDoItem interface {
GetItemId() int
GetTitle() string
GetDescription() string
}
repo function
func (r *Repository)GetAll() []interfaces.IToDoItem{
cursor, err := r.collection.Find(context.TODO(), bson.D{})
if err != nil{
panic(err)
}
defer cursor.Close(context.Background())
var allItems []interfaces.IToDoItem
for cursor.Next(context.Background()){
var result interfaces.IToDoItem
err := cursor.Decode(&result)
if err!= nil{
panic(err)
}
allItems = append(allItems[:],result)
}
fmt.Println(allItems)
return []interfaces.IToDoItem{}
}
For now it does not return anything because I want to resolve the issues at the repo level.
Interfaces are not concrete types, there may be multiple (any number of) types implementing it. The driver rightfully doesn't know how to instantiate it.
Fields of toDoItem are exported, there is no reason to make toDoItem itself unexported. Just export it and use a slice of []ToDoItem (or []*ToDoItem), and all your problems go away.
Also note that there is a Cursor.All() method to get all results, so you don't have to iterate over the documents and call Cursor.Decode() for each.
If you'd ever need to register custom decoders, check out this answer how to do it:
How to ignore nulls while unmarshalling a MongoDB document?
Related
I have a series of collections in a mongodb instance where I want to pull all the collections and decode all documents in the collections so I can mutate them as I go.
I am using go to list the collection names:
...
collections, err := db.ListCollectionNames(context.TODO(), bson.M{})
if err != nil {
log.Fatalf("Failed to get coll names: %v", err)
}
for _, coll := range collections {
collection := db.Collection(coll)
...
This works flawlessly and the next step would be to open a cursor to each collection and loop over the cursor and call decode on each object to its corresponding type.
I have defined all the types in a types package:
type Announcements struct {
Id string `bson:"_id"`
Content string `bson:"content"`
Title string `bson:"title"`
DateModified time.Time `bson:"dateModified,omitempty"`
ModifiedBy string `bson:"modifiedBy,omitempty"`
}
The issue is that I cant define the variable to be used for decoding dynamically.
This code would need to be repeated for every single collection
var obj Announcements // This would need to be dynamic
for cursor.Next(context.Background()) {
err := cursor.Decode(&obj)
if err != nil {
log.Fatal(err)
}
log.Printf("%#v\n", obj)
}
Is there a way to do this without repeating this many times? I realize a dynamically typed language would be better for this. Wanted to ask before I migrate the work to a scripting language.
I have tried using the reflect package and switch statements to instantiate the variable dynamnically and using the interface{} type but that force the decode to use bson types and not my defined structs.
-- edit --
I have tried to use a map to link collection names to types but to no avail.
var Collections = map[string]interface{}{
"announcements": Announcements{},
"badges": Badges{},
"betaUser": BetaUser{},
"bonusLearningItems": BonusLearningItems{},
"books": Books{},
...
}
So that when I use the obj var I tried an assignment like:
var obj types.Collections[coll]
hoping that would give me a variable of type Announcement if I was had looped to the announcements collection. but when I call decode it returns bson types.
I need to dynamically define the obj variable type.
There are many ways you can do this. One of them is by using a callback function:
func Process(cursor *mongo.Cursor, cb func(), out interface{}) {
for cursor.Next(context.Background()) {
err := cursor.Decode(out)
if err != nil {
log.Fatal(err)
}
cb()
log.Printf("%#v\n", obj)
}
}
Then define a map:
type collHandler struct {
New func() interface{}
Process func(interface{})
}
var collections=map[string]func() interface{} {
"announcements": collHandler{
New: func() interface {} {return &Announcement{}},
Process: func(data interface{}) {
processAnnouncements(data.(*Announcement))
},
},
...
}
And then you can:
handler:=collections[collectionName]
obj:=handler.New()
process(cursor,func() {
handler.Process(obj)
},
&obj)
This way, you can place the iteration into a common function, and use closures to deal with type specific logic.
I have the following (simplified) struct:
type newAppRegister struct {
SomeFlag *bool `json:"someflag" validate:"nonnil"`
ComputeLevel string `json:"compute-level" validate:"computelevelvalidator"`
}
And computelevelvalidator is some validation function.
I want that if SomeFlag is false, then ComputeLevel will be required and run his validation function.
A possible solution is to create a static variable, and set it in a custom validation function of SomeFlag, so for example:
var someFlag bool
func someFlagValidator(v interface{}, param string) error {
st := reflect.ValueOf(v)
if st.Kind() != reflect.Bool {
return fmt.Errorf("must be bool")
}
someFlag = st.Bool()
return nil
}
And then in computelevelvalidator it's possible to condition based on his value.
But since this is a REST API, and it may receive plenty of calls per second, I'm not sure if using this static variable will work (I'm worried about race conditions - but I'm not quite sure about it).
I'm using go1.11 and validation package gopkg.in/validator.v2 (version gopkg.in/validator.v2 v2.0.0-20190827175613-1a84e0480e5b).
So first up: Yes, using this static (global) variable will be a problem for concurrent use/access. It's not the way to go.
If your validation logic is contextual (like you said: if the flag is false, then certain restrictions apply on another field), then implementing a simple Validate function is pretty straightforward. I know the package you use supports this type of thing, but an external package will almost always be designed to be fairly generic. At the very least you'll end up performing type assertions. Last I checked, you still need an explicit call for the validation validator.Validate(), so why not move that to a method on your type?
type newAppRegister struct {} // your type
func (n newAppRegister) Validate() error {
if err := validator.Validate(n); err != nil {
return err
}
// at this point, we now the flag field isn't nit, because it passed validation
if !*n.SomeFlag {
// validate ComputeLevel here
}
return nil
}
No need for type assertions, let alone reflection. With just these two fields, I'd even argue there's no need for the validator package at all. You can get the same thing done with just standard JSON tags:
type Foo struct {
SomeFlag *bool `json:"some_flag,omitempty"`
ComputeLevel string `json:"compute_level"`
}
func (f Foo) Validate() error {
if f.SomeFlag == nil {
return ErrSomeFlagRequired
}
if !*f.SomeFlag {
// validate ComputeLevel
}
return nil
}
It's fairly straightforward to use:
f := Foo{}
if err := json.Unmarshal([]byte(payload), &f); err != nil {
// some shady JSON was submitted
}
if err := f.Validate(); err != nil {
// JSON was technically valid, but payload made no sense
}
// handle valid request
Go types like Int64 and String cannot store null values,
so I found I could use sql.NullInt64 and sql.NullString for this.
But when I use these in a Struct,
and generate JSON from the Struct with the json package,
then the format is different to when I use regular Int64 and String types.
The JSON has an additional level because the sql.Null*** is also a Struct.
Is there a good workaround for this,
or should I not use NULLs in my SQL database?
Types like sql.NullInt64 do not implement any special handling for JSON marshaling or unmarshaling, so the default rules apply. Since the type is a struct, it gets marshalled as an object with its fields as attributes.
One way to work around this is to create your own type that implements the json.Marshaller / json.Unmarshaler interfaces. By embedding the sql.NullInt64 type, we get the SQL methods for free. Something like this:
type JsonNullInt64 struct {
sql.NullInt64
}
func (v JsonNullInt64) MarshalJSON() ([]byte, error) {
if v.Valid {
return json.Marshal(v.Int64)
} else {
return json.Marshal(nil)
}
}
func (v *JsonNullInt64) UnmarshalJSON(data []byte) error {
// Unmarshalling into a pointer will let us detect null
var x *int64
if err := json.Unmarshal(data, &x); err != nil {
return err
}
if x != nil {
v.Valid = true
v.Int64 = *x
} else {
v.Valid = false
}
return nil
}
If you use this type in place of sql.NullInt64, it should be encoded as you expect.
You can test this example here: http://play.golang.org/p/zFESxLcd-c
If you use the null.v3 package, you won't need to implement any of the marshal or unmarshal methods. It's a superset of the sql.Null structs and is probably what you want.
package main
import "gopkg.in/guregu/null.v3"
type Person struct {
Name string `json:"id"`
Age int `json:"age"`
NickName null.String `json:"nickname"` // Optional
}
If you'd like to see a full Golang webserver that uses sqlite, nulls, and json you can consult this gist.
I'm new to GO language.
Trying to learn GO by building real web application.
I'm using revel framework.
And here is my resource routes:
GET /resource/:resource Resource.ReadAll
GET /resource/:resource/:id Resource.Read
POST /resource/:resource Resource.Create
PUT /resource/:resource/:id Resource.Update
DELETE /resource/:resource/:id Resource.Delete
for example:
GET /resource/users calls Resource.ReadAll("users")
And this is my Resource controller (it's just a dummy actions for now):
type Resource struct {
*revel.Controller
}
type User struct {
Id int
Username string
Password string
}
type Users struct {}
func (u Users) All() string {
return "All"
}
func (c Resource) ReadAll(resource string) revel.Result {
fmt.Printf("GET %s", resource)
model := reflect.New(resource)
fmt.Println(model.All())
return nil
}
I'm trying get instance of Users struct by converting resource string to object to call All function.
and the error:
cannot use resource (type string) as type reflect.Type in argument to
reflect.New: string does not implement reflect.Type (missing Align
method)
I'm new to GO please don't judge me :)
Your problem is here:
model := reflect.New(resource)
You can't instantiate a type from a string that way. You need to either use a switch there and do stuff depending on the model:
switch resource {
case "users":
model := &Users{}
fmt.Println(model.All())
case "posts":
// ...
}
Or use reflect correctly. Something like:
var types = map[string]reflect.Type{
"users": reflect.TypeOf(Users{}) // Or &Users{}.
}
// ...
model := reflect.New(types[resource])
res := model.MethodByName("All").Call(nil)
fmt.Println(res)
Suppose you have a workflow that consists of multiple embedded nodes of different types. Since nodes are of different types, I thought of using Golang interfaces here and came up with following:
type Workflow struct {
CreatedAt time.Time
StartedAt time.Time
CreatedBy string
Nodes []Node
}
type Node interface {
Exec() (int, error)
}
type EmailNode struct {
From string
To string
Subject string
Body string
}
type TwitterNode struct {
Tweet string
Image []byte
}
func (n *EmailNode) Exec() (int, error){
//send email
return 0, nil
}
func (n *TwitterNode) Exec() (int, error) {
//send tweet
return 0, nil
}
These workflows are stored in MongoDB and I have sample seed data in it. Using mgo, when I try to find a workflow (given its ID):
w = &Workflow{}
collection.FindID(bson.ObjectIdHex(id)).One(w)
I get the error - value of type bson.M is not assignable to type Node.
It also feels a bit weird to me that how would mgo unmarshal embedded Node documents into a Go struct without any type information. May be I need to look at the problem from another point of view.
Any suggestions would be highly appreciated.
You cannot use an interface in a document for the reason you noted. The decoder has no information about the type to create.
One way to handle this is to define a struct to hold the type information:
type NodeWithType struct {
Node Node `bson:"-"`
Type string
}
type Workflow struct {
CreatedAt time.Time
StartedAt time.Time
CreatedBy string
Nodes []NodeWithType
}
Implement the SetBSON function on this type. This function should decode the type string, create a value of the correct type based on that string and unmarshal to that value.
func (nt *NodeWithType) SetBSON(r bson.Raw) error {
}
Following Simon Fox's answer regarding the implementation of SetBSON, here is a more precise answer.
Let's take the original piece of code:
type Workflow struct {
CreatedAt time.Time
StartedAt time.Time
CreatedBy string
Nodes []Node
}
type Node interface {
Exec() (int, error)
}
type EmailNode struct {
From string
To string
Subject string
Body string
}
type TwitterNode struct {
Tweet string
Image []byte
}
func (n *EmailNode) Exec() (int, error){
//send email
return 0, nil
}
func (n *TwitterNode) Exec() (int, error) {
//send tweet
return 0, nil
}
What you want to do now is: once you unmarshal the BSON object from Mongo, you want to be able to know if each node is either an EmailNode or a TwitterNode.
As you are storing the nodes as a Node interface, mgo has no way to know what struct to implement, so you have to tell it explicitly. Here comes SetBSON.
In your example, the problem comes from this Workflow.Nodes, which is a slice of Node interface. As it is a simple slice, the best is that you create a custom type that mgo will be able to call when unmarshalling the BSON:
type NodesList []Node
// The updated Workflow struct:
type Workflow struct {
CreatedAt time.Time
StartedAt time.Time
CreatedBy string
Nodes NodesList
}
Now, you can implement SetBSON on this NodesList and describe how it works. Please note that as you are using a pointer, you can define what is contained inside the variable:
// Note that you must use a pointer to the slice
func (list *NodesList) SetBSON(raw raw.BSON) error {
// Now you need to create the objects according to your
// own domain logic and what's contained inside "raw":
if raw.something {
*list = append(*list, &TwitterNode{})
} else {
*list = append(*list, &EmailNode{})
}
return nil
}