I have a series of collections in a mongodb instance where I want to pull all the collections and decode all documents in the collections so I can mutate them as I go.
I am using go to list the collection names:
...
collections, err := db.ListCollectionNames(context.TODO(), bson.M{})
if err != nil {
log.Fatalf("Failed to get coll names: %v", err)
}
for _, coll := range collections {
collection := db.Collection(coll)
...
This works flawlessly and the next step would be to open a cursor to each collection and loop over the cursor and call decode on each object to its corresponding type.
I have defined all the types in a types package:
type Announcements struct {
Id string `bson:"_id"`
Content string `bson:"content"`
Title string `bson:"title"`
DateModified time.Time `bson:"dateModified,omitempty"`
ModifiedBy string `bson:"modifiedBy,omitempty"`
}
The issue is that I cant define the variable to be used for decoding dynamically.
This code would need to be repeated for every single collection
var obj Announcements // This would need to be dynamic
for cursor.Next(context.Background()) {
err := cursor.Decode(&obj)
if err != nil {
log.Fatal(err)
}
log.Printf("%#v\n", obj)
}
Is there a way to do this without repeating this many times? I realize a dynamically typed language would be better for this. Wanted to ask before I migrate the work to a scripting language.
I have tried using the reflect package and switch statements to instantiate the variable dynamnically and using the interface{} type but that force the decode to use bson types and not my defined structs.
-- edit --
I have tried to use a map to link collection names to types but to no avail.
var Collections = map[string]interface{}{
"announcements": Announcements{},
"badges": Badges{},
"betaUser": BetaUser{},
"bonusLearningItems": BonusLearningItems{},
"books": Books{},
...
}
So that when I use the obj var I tried an assignment like:
var obj types.Collections[coll]
hoping that would give me a variable of type Announcement if I was had looped to the announcements collection. but when I call decode it returns bson types.
I need to dynamically define the obj variable type.
There are many ways you can do this. One of them is by using a callback function:
func Process(cursor *mongo.Cursor, cb func(), out interface{}) {
for cursor.Next(context.Background()) {
err := cursor.Decode(out)
if err != nil {
log.Fatal(err)
}
cb()
log.Printf("%#v\n", obj)
}
}
Then define a map:
type collHandler struct {
New func() interface{}
Process func(interface{})
}
var collections=map[string]func() interface{} {
"announcements": collHandler{
New: func() interface {} {return &Announcement{}},
Process: func(data interface{}) {
processAnnouncements(data.(*Announcement))
},
},
...
}
And then you can:
handler:=collections[collectionName]
obj:=handler.New()
process(cursor,func() {
handler.Process(obj)
},
&obj)
This way, you can place the iteration into a common function, and use closures to deal with type specific logic.
Related
I am trying to do a basic TODO list application in Go. I am creating the CRUD operations on my cluster from mongodb atlas. But I have a problem decoding BSON objects. For my model I use a struct which is unimported but it implements a interface which is used in the repo. When trying to read from database I get this error:
panic: no decoder found for interfaces.IToDoItem
I know I should somehow implement a decoder for my interface but can not realize how to do it, without exporting my main struct from model. That would also mean I won't have a privacy in the model and the items in the model can be accessed in any mode all around the program, a thing which I think is wrong.
Here is my code:
model.go
type toDoItem struct{
ItemId int `bson:"itemId,omitempty"`
Title string `bson:"title,omitempty"`
Description string `bson:"description,omitempty"`
}
func New(itemId int,title string,description string) interfaces.IToDoItem {
return toDoItem{
ItemId: itemId,
Title: title,
Description: description,
}
}
func (item toDoItem)GetItemId()int{
return item.ItemId
}
func (item toDoItem)GetTitle()string{
return item.Title
}
func (item toDoItem)GetDescription()string{
return item.Description
}
Interface
type IToDoItem interface {
GetItemId() int
GetTitle() string
GetDescription() string
}
repo function
func (r *Repository)GetAll() []interfaces.IToDoItem{
cursor, err := r.collection.Find(context.TODO(), bson.D{})
if err != nil{
panic(err)
}
defer cursor.Close(context.Background())
var allItems []interfaces.IToDoItem
for cursor.Next(context.Background()){
var result interfaces.IToDoItem
err := cursor.Decode(&result)
if err!= nil{
panic(err)
}
allItems = append(allItems[:],result)
}
fmt.Println(allItems)
return []interfaces.IToDoItem{}
}
For now it does not return anything because I want to resolve the issues at the repo level.
Interfaces are not concrete types, there may be multiple (any number of) types implementing it. The driver rightfully doesn't know how to instantiate it.
Fields of toDoItem are exported, there is no reason to make toDoItem itself unexported. Just export it and use a slice of []ToDoItem (or []*ToDoItem), and all your problems go away.
Also note that there is a Cursor.All() method to get all results, so you don't have to iterate over the documents and call Cursor.Decode() for each.
If you'd ever need to register custom decoders, check out this answer how to do it:
How to ignore nulls while unmarshalling a MongoDB document?
In a db package, I don't want to expose MongoDB's primitive.ObjectID type as part of its public API. Instead, I want to define type Id interface{} within the db package and expose that. Since the driver doesn't know how to transform the database's ObjectID type into my custom ID type, I registered a type decoder like so:
type Id interface{}
var idType = reflect.TypeOf((*Id)(nil)).Elem()
type User struct {
Id `bson:"_id,omitempty"`
}
func main() {
reg := bson.NewRegistryBuilder().
RegisterTypeDecoder(idType, bsoncodec.ValueDecoderFunc(idDecodeValue)).
Build()
client, err := mongo.Connect(context.TODO(), options.Client().
ApplyURI("...").
SetRegistry(reg))
coll := client.Database("...").Collection("...")
var u0 User
coll.InsertOne(context.TODO(), u0)
var u1 User
coll.FindOne(context.TODO(), bson.D{}).Decode(&u1)
}
func idDecodeValue(dc bsoncodec.DecodeContext, vr bsonrw.ValueReader, val reflect.Value) error {
if !val.IsValid() || val.Type() != idType {
return bsoncodec.ValueDecoderError{
Name: "IdDecodeValue",
Types: []reflect.Type{idType},
Received: val,
}
}
oid, _ := vr.ReadObjectID()
val.Set(reflect.ValueOf(oid))
return nil
}
I saw that there exists a bsoncodec.NewEmptyInterfaceCodec() to use in place of idDecodeValue, but that only seems to work when User.Id's type is exactly interface{}. I wrote idDecodeValue based off of the existing codecs, but I'm not entirely sure what's going on (mostly due to not knowing when val.IsValid() would ever return false). Is all of this the best, most idiomatic way to go about supporting a custom ID type?
I have the following (simplified) struct:
type newAppRegister struct {
SomeFlag *bool `json:"someflag" validate:"nonnil"`
ComputeLevel string `json:"compute-level" validate:"computelevelvalidator"`
}
And computelevelvalidator is some validation function.
I want that if SomeFlag is false, then ComputeLevel will be required and run his validation function.
A possible solution is to create a static variable, and set it in a custom validation function of SomeFlag, so for example:
var someFlag bool
func someFlagValidator(v interface{}, param string) error {
st := reflect.ValueOf(v)
if st.Kind() != reflect.Bool {
return fmt.Errorf("must be bool")
}
someFlag = st.Bool()
return nil
}
And then in computelevelvalidator it's possible to condition based on his value.
But since this is a REST API, and it may receive plenty of calls per second, I'm not sure if using this static variable will work (I'm worried about race conditions - but I'm not quite sure about it).
I'm using go1.11 and validation package gopkg.in/validator.v2 (version gopkg.in/validator.v2 v2.0.0-20190827175613-1a84e0480e5b).
So first up: Yes, using this static (global) variable will be a problem for concurrent use/access. It's not the way to go.
If your validation logic is contextual (like you said: if the flag is false, then certain restrictions apply on another field), then implementing a simple Validate function is pretty straightforward. I know the package you use supports this type of thing, but an external package will almost always be designed to be fairly generic. At the very least you'll end up performing type assertions. Last I checked, you still need an explicit call for the validation validator.Validate(), so why not move that to a method on your type?
type newAppRegister struct {} // your type
func (n newAppRegister) Validate() error {
if err := validator.Validate(n); err != nil {
return err
}
// at this point, we now the flag field isn't nit, because it passed validation
if !*n.SomeFlag {
// validate ComputeLevel here
}
return nil
}
No need for type assertions, let alone reflection. With just these two fields, I'd even argue there's no need for the validator package at all. You can get the same thing done with just standard JSON tags:
type Foo struct {
SomeFlag *bool `json:"some_flag,omitempty"`
ComputeLevel string `json:"compute_level"`
}
func (f Foo) Validate() error {
if f.SomeFlag == nil {
return ErrSomeFlagRequired
}
if !*f.SomeFlag {
// validate ComputeLevel
}
return nil
}
It's fairly straightforward to use:
f := Foo{}
if err := json.Unmarshal([]byte(payload), &f); err != nil {
// some shady JSON was submitted
}
if err := f.Validate(); err != nil {
// JSON was technically valid, but payload made no sense
}
// handle valid request
Go types like Int64 and String cannot store null values,
so I found I could use sql.NullInt64 and sql.NullString for this.
But when I use these in a Struct,
and generate JSON from the Struct with the json package,
then the format is different to when I use regular Int64 and String types.
The JSON has an additional level because the sql.Null*** is also a Struct.
Is there a good workaround for this,
or should I not use NULLs in my SQL database?
Types like sql.NullInt64 do not implement any special handling for JSON marshaling or unmarshaling, so the default rules apply. Since the type is a struct, it gets marshalled as an object with its fields as attributes.
One way to work around this is to create your own type that implements the json.Marshaller / json.Unmarshaler interfaces. By embedding the sql.NullInt64 type, we get the SQL methods for free. Something like this:
type JsonNullInt64 struct {
sql.NullInt64
}
func (v JsonNullInt64) MarshalJSON() ([]byte, error) {
if v.Valid {
return json.Marshal(v.Int64)
} else {
return json.Marshal(nil)
}
}
func (v *JsonNullInt64) UnmarshalJSON(data []byte) error {
// Unmarshalling into a pointer will let us detect null
var x *int64
if err := json.Unmarshal(data, &x); err != nil {
return err
}
if x != nil {
v.Valid = true
v.Int64 = *x
} else {
v.Valid = false
}
return nil
}
If you use this type in place of sql.NullInt64, it should be encoded as you expect.
You can test this example here: http://play.golang.org/p/zFESxLcd-c
If you use the null.v3 package, you won't need to implement any of the marshal or unmarshal methods. It's a superset of the sql.Null structs and is probably what you want.
package main
import "gopkg.in/guregu/null.v3"
type Person struct {
Name string `json:"id"`
Age int `json:"age"`
NickName null.String `json:"nickname"` // Optional
}
If you'd like to see a full Golang webserver that uses sqlite, nulls, and json you can consult this gist.
I can't insert a big array into my collection; I have a problem with the interface{} type.
Here's the code:
collections = getSession().DB("go").C("comments")
func BenchmarkBulkInsert(data interface{}) {
bulk := collections.Bulk()
bulk.Insert(data...)
_, bulkErr := bulk.Run()
if bulkErr != nil {
panic(bulkErr)
}
fmt.Printf("\n - %d comments inserted!", reflect.ValueOf(data).Len())
}
I can't build the app. The error message is this:
cannot use data (type interface {}) as type []interface {} in argument to bulk.Insert: need type assertion
I'm using the "gopkg.in/mgo.v2" package to connect to the Mongo database, and data is an array when I print it to the console.
You should use variadic func
So you have to change the input parameter from this :
func BenchmarkBulkInsert(data interface{})
to this :
func BenchmarkBulkInsert(data ...interface{})