I try to populate the exported fields of a struct with content fetched from a MongoDb-database using the labix.org/v2/mgo package.
mgo uses the labix.org/v2/mgo/bson package to unmarshal the data. But the unmarshaller sets all unexported fields to their zero value.
Is there any way to prevent this behavior?
Working example:
package main
import (
"fmt"
"labix.org/v2/mgo/bson"
)
type Sub struct{ Int int }
type Player struct {
Name string
unexpInt int
unexpPoint *Sub
}
func main() {
dta,err := bson.Marshal(bson.M{"name": "ANisus"})
if err != nil {
panic(err)
}
p := &Player{unexpInt: 12, unexpPoint: &Sub{42}}
fmt.Printf("Before: %+v\n", p)
err = bson.Unmarshal(dta, p)
if err != nil {
panic(err)
}
fmt.Printf("After: %+v\n", p)
}
Output:
Before: &{Name: unexpInt:12 unexpPoint:0xf84005f500}
After: &{Name:ANisus unexpInt:0 unexpPoint:<nil>}
This is not possible. As you can see in the source code, struct values are explicitly being set to their zero value before filling in any fields.
There is no option to disable this behaviour. It is presumably in place to make sure the result of Unmarshal() only depends on the BSON data and not any prior state.
Related
I have a Go struct which contains a slice of strings which I'd like to save as a jsonB object in Postgres with GORM.
I've come accross a solution which requires to use the GORM specific type (postgres.Jsonb) which I'd like to avoid.
When I try to run the AutoMigrate with a slice in my model, it panics and won't start although when I wrap this slice in a struct (which I'm okay doing) it will run without error but won't create the column in postgres.
type User struct {
gorm.Model
Data []string `sql:"type:"jsonb"; json:"data"`
} //Panics
type User struct {
gorm.Model
Data struct {
NestedData []string
} `sql:"type:"jsonb"; json:"data"`
} //Doesn't crash but doesn't create my column
Has anyone been able to manipulate jsonb with GORM without using the postgres.Jsonb type in models ?
The simplest way to use JSONB in Gorm is to use pgtype.JSONB.
Gorm uses pgx as it driver, and pgx has package called pgtype, which has type named pgtype.JSONB.
If you have already install pgx as Gorm instructed, you don't need install any other package.
This method should be the best practice since it using underlying driver and no custom code is needed. It can also be used for any JSONB type beyond []string.
type User struct {
gorm.Model
Data pgtype.JSONB `gorm:"type:jsonb;default:'[]';not null"`
}
Get value from DB
u := User{}
db.find(&u)
var data []string
err := u.Data.AssignTo(&data)
if err != nil {
t.Fatal(err)
}
Set value to DB
u := User{}
err := u.Data.Set([]string{"abc","def"})
if err != nil {
return
}
db.Updates(&u)
Maybe:
type DataJSONB []string
func (dj DataJSONB) Value() (driver.Value, error) {
return json.Marshal(dj)
}
func (dj *DataJSONB) Scan(value interface{}) error {
b, ok := value.([]byte)
if !ok {
return fmt.Errorf("[]byte assertion failed")
}
return json.Unmarshal(b, dj)
}
// Your bit
type User struct {
gorm.Model
Data DataJSONB `sql:"type:"jsonb"; json:"data"`
}
Define a new type:
type Data map[string]interface{}
And implement the Valuer and Scanner interfaces onto them, which allows the field to be converted to a value for the database, and scanned back into the field, respectively:
// Value converts into []byte
func (d Data) Value() (driver.Value, error) {
j, err := json.Marshal(d)
return j, err
}
// Scan puts the []byte back into Data
func (d *Data) Scan(src interface{}) error {
source, ok := src.([]byte)
if !ok {
return errors.New("Type assertion .([]byte) failed.")
}
var i interface{}
if err := json.Unmarshal(source, &i); err != nil {
return err
}
*d, ok = i.(map[string]interface{})
if !ok {
return errors.New("Type assertion .(map[string]interface{}) failed.")
}
return nil
}
Then you can define your field in your model like this:
type User struct {
gorm.Model
Data Data `type: jsonb not null default '{}'::jsonb`
}
Using the underlying map[string]interface{} type is nice too, as you can Unmarshal/Marshal any JSON to/from it.
I'm a beginner in both go and mongodb.
I try to decode a DocumentResult into a struct using bson tags, and it does not work for a custom type wrapping a string.
Can it be done without changing the field's type to a string?
import (
"context"
"github.com/mongodb/mongo-go-driver/mongo"
)
type MyDoc struct {
SomeInt int `bson:"some_int"`
SomeString string `bson:"some_string,omitempty"`
CustomType MyType `bson:"custom_type,omitempty"`
}
type MyType string
const myType MyType = "ABCD"
func main() {
//Connect to db
client, _ := mongo.Connect(context.Background(), "mongodb://localhost:27017", nil)
db := client.Database("example_db")
collection := db.Collection("col")
//Insert document
docToInsert := MyDoc{42, "The Answer", myType}
collection.InsertOne(nil, docToInsert)
//Retrieve document
filterDoc := MyDoc{SomeInt: 42}
resultDoc := &MyDoc{}
result := collection.FindOne(nil, filterDoc)
result.Decode(resultDoc)
println(resultDoc.SomeInt, resultDoc.SomeString, resultDoc.CustomType)
PRINTED RESULT: "42 The Answer" //"ABCD" is missing
Thanks in advance
I try to decode a DocumentResult into a struct using bson tags, and it does not work for a custom type wrapping a string
With your current MyType, the document that would be stored in MongoDB would be as below:
{
"_id": ObjectId("..."),
"some_int": NumberLong("42"),
"some_string": "The Answer",
"custom_type": "ABCD"
}
Even though the underlying type is a string, this could be tricky to decode with the current version of mongo-go-driver (v0.0.12) due to the type wrapping.
However, if you would like to have a custom type as such, you could change the struct into an embedded field instead. For example:
type MyDoc struct {
SomeInt int `bson:"some_int"`
SomeString string `bson:"some_string,omitempty"`
CustomType MyType `bson:"custom_type,omitempty"`
}
type MyType struct {
Value string `bson:"value,omitempty"`
}
var myType = MyType{Value: "ABCD"}
docToInsert := MyDoc{42, "The Answer", "ABCD"}
insertResult, err := collection.InsertOne(nil, docToInsert)
resultDoc := collection.FindOne(context.Background(), nil)
if err != nil {
log.Fatal(err)
}
elem := &MyDoc{}
err = resultDoc.Decode(elem)
if err != nil {
log.Fatal(err)
}
fmt.Println(elem.SomeInt, elem.SomeString, elem.CustomType.Value)
// 42 The Answer ABCD
The document would be stored in MongoDB as below:
{
"_id": ObjectId("..."),
"some_int": NumberLong("42"),
"some_string": "The Answer",
"custom_type": {
"value": "ABCD"
}
}
Otherwise just use string type directly because the resulting document in the database would be the same as the type wrapping version:
type MyDoc struct {
SomeInt int `bson:"some_int"`
SomeString string `bson:"some_string,omitempty"`
CustomType string `bson:"custom_type,omitempty"`
}
You may also find MongoDB Data Modeling a useful reference.
Foreword: Custom types having string as their underlying types are now handled automatically by the driver. This answer predates the driver 1.x versions where this was necessary.
Unfortunately you're out of luck. The current state of the official mongo go driver does not support unmarshaling string values from BSON to a Go value whose type is a custom type having string as its underlying type. This may change in the future, but for now this is not supported.
The way decoding into a struct field is handled is implemented in bson/decode.go, currently line #387:
case 0x2:
str := v.StringValue()
switch containerType {
case tString, tEmpty:
val = reflect.ValueOf(str)
case tJSONNumber:
_, err := strconv.ParseFloat(str, 64)
if err != nil {
return val, err
}
val = reflect.ValueOf(str).Convert(tJSONNumber)
case tURL:
u, err := url.Parse(str)
if err != nil {
return val, err
}
val = reflect.ValueOf(u).Elem()
default:
return val, nil
}
0x02 is the BSON string type. It is only attempted to decode into the struct field if the struct field's type is any of the following: string, interface{}, json.Number or url.URL (or a pointer to these).
Unfortunately implementing bson.Unmarshaler on your custom type does not help either, as it is not checked in case of struct fields, only if the struct itself implements it. But implementing on the struct itself, you would have to duplicate the struct with the field being one of the above listed supported types (or use a map or a bson.Document type).
This is a serious limitation on the library's part which can very easily be solved, so let's hope for the best that they add support for this in the near future.
With the 1.x versions of MongoDB driver for Go (latest version at the time of writing is 1.3.1) it is fully possible to encode and decode aliased types.
Your example now works as expected, given one adjusts the mongo.Connect to match new 1.x API.
package main
import (
"context"
"go.mongodb.org/mongo-driver/mongo"
"go.mongodb.org/mongo-driver/mongo/options"
)
type MyDoc struct {
SomeInt int `bson:"some_int"`
SomeString string `bson:"some_string,omitempty"`
CustomType MyType `bson:"custom_type,omitempty"`
}
type MyType string
const myType MyType = "ABCD"
func main() {
// Connect to db
clientOpts := options.Client().
ApplyURI("mongodb://localhost/example_db")
client, _ := mongo.Connect(context.Background(), clientOpts)
db := client.Database("example_db")
collection := db.Collection("col")
// Insert document
docToInsert := MyDoc{42, "The Answer", myType}
collection.InsertOne(nil, docToInsert)
// Retrieve document
filterDoc := MyDoc{SomeInt: 42}
resultDoc := &MyDoc{}
result := collection.FindOne(nil, filterDoc)
result.Decode(resultDoc)
println(resultDoc.SomeInt, resultDoc.SomeString, resultDoc.CustomType)
}
This returns: 42 The Answer ABCD as expected
What I am looking is equivalent of Document.parse()
in golang, that allows me create bson from json directly? I do not want to create intermediate Go structs for marshaling
The gopkg.in/mgo.v2/bson package has a function called UnmarshalJSON which does exactly what you want.
The data parameter should hold you JSON string as []byte value.
func UnmarshalJSON(data []byte, value interface{}) error
UnmarshalJSON unmarshals a JSON value that may hold non-standard syntax as defined in BSON's extended JSON specification.
Example:
var bdoc interface{}
err = bson.UnmarshalJSON([]byte(`{"id": 1,"name": "A green door","price": 12.50,"tags": ["home", "green"]}`),&bdoc)
if err != nil {
panic(err)
}
err = c.Insert(&bdoc)
if err != nil {
panic(err)
}
mongo-go-driver has a function bson.UnmarshalExtJSON that does the job.
Here's the example:
var doc interface{}
err := bson.UnmarshalExtJSON([]byte(`{"foo":"bar"}`), true, &doc)
if err != nil {
// handle error
}
There is no longer a way to do this directly with supported libraries (e.g. the mongo-go-driver). You would need to write your own converter based on the bson spec.
Edit: here's one that by now has seen a few Terabytes of use in prod.
https://github.com/dustinevan/mongo/blob/main/bsoncv/bsoncv.go
I do not want to create intermediate Go structs for marshaling
If you do want/need to create an intermediate Go BSON structs, you could use a conversion module such github.com/sindbach/json-to-bson-go. For example:
import (
"fmt"
"github.com/sindbach/json-to-bson-go/convert"
"github.com/sindbach/json-to-bson-go/options"
)
func main() {
doc := `{"foo": "buildfest", "bar": {"$numberDecimal":"2021"} }`
opt := options.NewOptions()
result, _ := convert.Convert([]byte(doc), opt)
fmt.Println(result)
}
Will produce output:
package main
import "go.mongodb.org/mongo-driver/bson/primitive"
type Example struct {
Foo string `bson:"foo"`
Bar primitive.Decimal128 `bson:"bar"`
}
This module is compatible with the official MongoDB Go driver, and as you can see it supports Extended JSON formats.
You can also visit https://json-to-bson-map.netlify.app to try the module in action. You can paste a JSON document, and see the Go BSON structs as output.
A simple converter that uses go.mongodb.org/mongo-driver/bson/bsonrw:
func JsonToBson(message []byte) ([]byte, error) {
reader, err := bsonrw.NewExtJSONValueReader(bytes.NewReader(message), true)
if err != nil {
return []byte{}, err
}
buf := &bytes.Buffer{}
writer, _ := bsonrw.NewBSONValueWriter(buf)
err = bsonrw.Copier{}.CopyDocument(writer, reader)
if err != nil {
return []byte{}, err
}
marshaled := buf.Bytes()
return marshaled, nil
}
I was wondering if there is any way to have a stuct field that doesn't get commited to mgo even if it isn't empty.
The only way I have found to do this is to make the field lowercase, which makes it a pain to access. Is there another way?
This is an example, and my goal here is to not commit the SSN into the database but still have it uppercase.
package main
import (
"fmt"
"crypto/sha1"
"encoding/base64"
"labix.org/v2/mgo"
)
type Person struct{
Name string
SSN string
HashedSSN string
}
func main() {
bob := Person{"Bob", "fake_ssn", ""}
hasher := sha1.New()
hasher.Write( []byte(bob.SSN))
sha := base64.URLEncoding.EncodeToString(hasher.Sum(nil))
bob.HashedSSN = sha
mgoSession, err := mgo.Dial("localhost:27017")
if err != nil {
fmt.Println("mongo_config#initMongoSessions : Could not dial to mgoSession", err)
} else {
mgoSession.DB("test").C("person").Insert(bob)
}
}
Thanks,
You can do that by using the field tag as follows:
type T struct {
Field string `bson:"-"`
}
I have a struct that contains math/big.Int fields. I would like to save the struct in mongodb using mgo. Saving the numbers as a strings is good enough in my situation.
I have looked at the available field's tags and nothing seams to allow custom serializer. I was expecting to implement an interface similar to encoding/json.Marshaler but I have found None of such interface in the documentation.
Here is a trivial example of what I want I need.
package main
import (
"labix.org/v2/mgo"
"math/big"
)
type Point struct {
X, Y *big.Int
}
func main() {
session, err := mgo.Dial("localhost")
if err != nil {
panic(err)
}
defer session.Close()
c := session.DB("test").C("test")
err = c.Insert(&Point{big.NewInt(1), big.NewInt(1)})
if err != nil { // should not panic
panic(err)
}
// The code run as expected but the fields X and Y are empty in mongo
}
Thnaks!
The similar interface is named bson.Getter:
http://labix.org/v2/mgo/bson#Getter
It can look similar to this:
func (point *Point) GetBSON() (interface{}, error) {
return bson.D{{"x", point.X.String()}, {"y", point.Y.String()}}, nil
}
And there's also the counterpart interface in the setter side, if you're interested:
http://labix.org/v2/mgo/bson#Setter
For using it, note that the bson.Raw type provided as a parameter has an Unmarshal method, so you could have a type similar to:
type dbPoint struct {
X string
Y string
}
and unmarshal it conveniently:
var dbp dbPoint
err := raw.Unmarshal(&dbp)
and then use the dbp.X and dbp.Y strings to put the big ints back into the real (point *Point) being unmarshalled.