I am looking at this example.
I would never coome up with solution like this,I would go for bson.raw.
type Movie struct {
ID bson.ObjectId `json:"id" bson:"_id,omitempty"`
Name string `json:"name" bson:"name"`
Year string `json:"year" bson:"year"`
Directors []string `json:"directors" bson:"directors"`
Writers []string `json:"writers" bson:"writers"`
BoxOffice BoxOffice `json:"boxOffice" bson:"boxOffice"`
}
GetMovie function reads data from MongoDB and returns JSON
func (db *DB) GetMovie(w http.ResponseWriter, r *http.Request) {
vars := mux.Vars(r)
w.WriteHeader(http.StatusOK)
var movie Movie
err := db.collection.Find(bson.M{"_id": bson.ObjectIdHex(vars["id"])}).One(&movie)
if err != nil {
w.Write([]byte(err.Error()))
} else {
w.Header().Set("Content-Type", "application/json")
response, _ := json.Marshal(movie)
w.Write(response)
}
}
I do not understand how generic map bson.M was created. Why did the author used bson.ObjectIdHex(vars["id"]?
bson.M is a map under the hood:
type M map[string]interface{}
And this:
bson.M{"_id": bson.ObjectIdHex(vars["id"])}
Is a composite literal creating a value of type bson.M. It has a single pair where key is "_id" and the associated value is a bson.ObjectId returned by the function bson.ObjectIdHex().
The document ID to look up and return is most likely coming as a hexadecimal string in vars["id"], and bson.ObjectIdHex() converts (parses) this into an ObjectId.
Tips: to query a document by ID, easier is to use Collection.FindId, e.g.:
err := db.collection.FindId(bson.ObjectIdHex(vars["id"])).One(&movie)
Also to avoid a runtime panic in case an invalid ID is stored in vars["id"], you could use bson.IsObjectIdHex() to check it first. For details, see Prevent runtime panic in bson.ObjectIdHex.
Also, marshaling the result into a byte slice and then writing it to the response is inefficient, the response could be streamed to the output using json.Encoder. For details, see Ouput json to http.ResponseWriter with template.
Related
I would like to know if there's any approach that would allow me to ignore null types while unmarshalling a MongoDB document into a Go struct.
Right now I have some auto-generate Go structs, something like this:
type User struct {
Name string `bson:"name"`
Email string `bson:"email"`
}
Changing the types declared in this struct is not an option, and here's the problem; in a MongoDB database, which I do not have total control, some of the documents have been inserted with null values were originally I was not expecting nulls. Something like this:
{
"name": "John Doe",
"email": null
}
As the string types declared inside my struct are not pointers, they can't receive a nil value, so whenever I try to unmarshall this document in my struct, it returns an error.
Preventing the insertion of this kind of document into the database would be the ideal solution, but for my use case, ignoring the null values would also be acceptable. So after unmarshalling the document my User instance would look like this
User {
Name: "John Doe",
Email: "",
}
I'm trying to find, either some annotation flag, or an option that could be passed to the method Find/FindOne, or maybe even a query parameter to prevent returning any field containing null values from the database. Without any success until now.
Are there any built-in solutions in the mongo-go-driver for this problem?
The problem is that the current bson codecs do not support encoding / decoding string into / from null.
One way to handle this is to create a custom decoder for string type in which we handle null values: we just use the empty string (and more importantly don't report error).
Custom decoders are described by the type bsoncodec.ValueDecoder. They can be registered at a bsoncodec.Registry, using a bsoncodec.RegistryBuilder for example.
Registries can be set / applied at multiple levels, even to a whole mongo.Client, or to a mongo.Database or just to a mongo.Collection, when acquiring them, as part of their options, e.g. options.ClientOptions.SetRegistry().
First let's see how we can do this for string, and next we'll see how to improve / generalize the solution to any type.
1. Handling null strings
First things first, let's create a custom string decoder that can turn a null into a(n empty) string:
import (
"go.mongodb.org/mongo-driver/bson/bsoncodec"
"go.mongodb.org/mongo-driver/bson/bsonrw"
"go.mongodb.org/mongo-driver/bson/bsontype"
)
type nullawareStrDecoder struct{}
func (nullawareStrDecoder) DecodeValue(dctx bsoncodec.DecodeContext, vr bsonrw.ValueReader, val reflect.Value) error {
if !val.CanSet() || val.Kind() != reflect.String {
return errors.New("bad type or not settable")
}
var str string
var err error
switch vr.Type() {
case bsontype.String:
if str, err = vr.ReadString(); err != nil {
return err
}
case bsontype.Null: // THIS IS THE MISSING PIECE TO HANDLE NULL!
if err = vr.ReadNull(); err != nil {
return err
}
default:
return fmt.Errorf("cannot decode %v into a string type", vr.Type())
}
val.SetString(str)
return nil
}
OK, and now let's see how to utilize this custom string decoder to a mongo.Client:
clientOpts := options.Client().
ApplyURI("mongodb://localhost:27017/").
SetRegistry(
bson.NewRegistryBuilder().
RegisterDecoder(reflect.TypeOf(""), nullawareStrDecoder{}).
Build(),
)
client, err := mongo.Connect(ctx, clientOpts)
From now on, using this client, whenever you decode results into string values, this registered nullawareStrDecoder decoder will be called to handle the conversion, which accepts bson null values and sets the Go empty string "".
But we can do better... Read on...
2. Handling null values of any type: "type-neutral" null-aware decoder
One way would be to create a separate, custom decoder and register it for each type we wish to handle. That seems to be a lot of work.
What we may (and should) do instead is create a single, "type-neutral" custom decoder which handles just nulls, and if the BSON value is not null, should call the default decoder to handle the non-null value.
This is surprisingly simple:
type nullawareDecoder struct {
defDecoder bsoncodec.ValueDecoder
zeroValue reflect.Value
}
func (d *nullawareDecoder) DecodeValue(dctx bsoncodec.DecodeContext, vr bsonrw.ValueReader, val reflect.Value) error {
if vr.Type() != bsontype.Null {
return d.defDecoder.DecodeValue(dctx, vr, val)
}
if !val.CanSet() {
return errors.New("value not settable")
}
if err := vr.ReadNull(); err != nil {
return err
}
// Set the zero value of val's type:
val.Set(d.zeroValue)
return nil
}
We just have to figure out what to use for nullawareDecoder.defDecoder. For this we may use the default registry: bson.DefaultRegistry, we may lookup the default decoder for individual types. Cool.
So what we do now is register a value of our nullawareDecoder for all types we want to handle nulls for. It's not that hard. We just list the types (or values of those types) we want this for, and we can take care of all with a simple loop:
customValues := []interface{}{
"", // string
int(0), // int
int32(0), // int32
}
rb := bson.NewRegistryBuilder()
for _, v := range customValues {
t := reflect.TypeOf(v)
defDecoder, err := bson.DefaultRegistry.LookupDecoder(t)
if err != nil {
panic(err)
}
rb.RegisterDecoder(t, &nullawareDecoder{defDecoder, reflect.Zero(t)})
}
clientOpts := options.Client().
ApplyURI("mongodb://localhost:27017/").
SetRegistry(rb.Build())
client, err := mongo.Connect(ctx, clientOpts)
In the example above I registered null-aware decoders for string, int and int32, but you may extend this list to your liking, just add values of the desired types to the customValues slice above.
You can go through the operator $exists and Query for Null or Missing Fields for a detail explanation.
In the mongo-go-driver, you can try below query:
The email => nil query matches documents that either contains the email field whose value is nil or that do not contain the email field.
cursor, err := coll.Find(
context.Background(),
bson.D{
{"email", nil},
})
You have to just add the $ne operator in the above query to get the records that do not have the field email or do not have the value nil in email. For more details about the operator $ne
If you know ahead of time which fields could potentially be null in your mongoDB records, you can use pointers in your structs instead:
type User struct {
Name string `bson:"name"` // Will still fail to decode if null in Mongo
Email *string `bson:"email"` // Will be nil in go if null in Mongo
}
Just remember that now you'll need to code more defensively around anything that uses this value after decoding from mongo, eg:
var reliableVal string
if User.Email != nil {
reliableVal = *user.Email
} else {
reliableVal = ""
}
I am trying to get a slice of json text from mongo using the below code in golang
var a []string
err := col..Find(nil).Select(bson.M{"_id": 0}).All(&a)
I get the error Unsupported document type for unmarshalling: string
May I know the right way to do this?
When you select all but _id, the return will be a document containing only the remaining fields. You can do:
type fieldDoc struct {
Field string `bson:"name"`
}
var a []fieldDoc
err := col.Find(nil).Select(bson.M{"_id": 0}).All(&a)
If you don't know the underlying structure:
var a []bson.M
err := col.Find(nil).Select(bson.M{"_id": 0}).All(&a)
That should give you the documents encoded as bson objects. That is a map[string]interface{}, so you should be able to marshal it to JSON if you want json output:
jsonDocs, err:=json.Marshal(a)
I am fairly new to golang programming and the mongodb interface.
I've got a dbase of records created by another application. I am trying to walk the dbase and examine specific fields of each record. I can decode the full records as bson, but I cannot get the specific values.
This struct defines the 3 fields I would like to extract:
type myDbaseRec struct {
aid string `bson:"pon-util-aid"`
ingressPct string `bson:"ingress-bucket-percent"`
egressPct string `bson:"egress-bucket-percent"`
}
Here is my code to iterate through the cursor after the collection.Find(ctx, queryFilter) and decode the results both as bson and as my struct:
var myResult myDbaseRec
var bsonMResult bson.M
var count int
for cursor.Next(ctx) {
err := cursor.Decode(&myResult)
if err != nil {
fmt.Println("cursor.Next() error:", err)
panic(err)
// If there are no cursor.Decode errors
} else {
fmt.Println("\nresult type:", reflect.TypeOf(myResult))
fmt.Printf("result: %+v\n", myResult)
}
err = cursor.Decode(&bsonMResult)
if err != nil {
fmt.Println("bson decode error:", err)
panic(err)
// If there are no cursor.Decode errors
} else {
fmt.Println("\nresult type:", reflect.TypeOf(bsonMResult))
fmt.Println("\nresult:", bsonMResult)
}
}
Here is an example of one iteration of the loop. The bson decode appears to work, but my struct is empty:
result type: internal.myDbaseRec
result: {aid: ingressPct: egressPct:}
result type: primitive.M
result: map[pon-util-aid:ROLT-1-MONTREAL/1/1/xp2 _id:ObjectID("5d70b4d1b3605301ef72228b")
admitted-assured-upstream-bw:0 admitted-excess-upstream-bw:0 admitted-fixed-upstream-bw:0
assured-upstream-bytes:0 available-excess-upstream-bw:0 available-fixed-upstream-bw:622080
app_counters_key_field:ROLT-1-MONTREAL/1/1/xp2 app_export_time:1567665626 downstream-octets:52639862633214
egress-bucket-bps:8940390198 egress-bucket-percent:91 egress-bucket-seconds:559
excess-upstream-bytes:0 fixed-upstream-bytes:0 ingress-bucket-bps:8253153852
ingress-bucket-percent:84 ingress-bucket-seconds:559 sample-time:0 upstream-octets:48549268162714]
I would have expected to get
result: {aid:"ROLT-1-MONTREAL/1/1/xp2" ingressPct:84 egressPct:91}
Any suggestion on how to properly find these 3 fields from each record?
=== UPDATE: The first comment below answered my question.
First, in Go only fields starting with a (Unicode) upper case letter are exported. See also Exported identifiers. The default decoder will try to decode only to the exported fields. So you should change the struct into:
type myDbaseRec struct {
Aid string `bson:"pon-util-aid"`
IngressPct int32 `bson:"ingress-bucket-percent"`
EgressPct int32 `bson:"egress-bucket-percent"`
}
Also notice that the struct above for IngressPct and EgressPct have type int32. This is because the value in the document is represented in numbers (int/double), and not string. You may change it to other number type accordingly i.e. int16, int64, etc.
I would like to know if there's any approach that would allow me to ignore null types while unmarshalling a MongoDB document into a Go struct.
Right now I have some auto-generate Go structs, something like this:
type User struct {
Name string `bson:"name"`
Email string `bson:"email"`
}
Changing the types declared in this struct is not an option, and here's the problem; in a MongoDB database, which I do not have total control, some of the documents have been inserted with null values were originally I was not expecting nulls. Something like this:
{
"name": "John Doe",
"email": null
}
As the string types declared inside my struct are not pointers, they can't receive a nil value, so whenever I try to unmarshall this document in my struct, it returns an error.
Preventing the insertion of this kind of document into the database would be the ideal solution, but for my use case, ignoring the null values would also be acceptable. So after unmarshalling the document my User instance would look like this
User {
Name: "John Doe",
Email: "",
}
I'm trying to find, either some annotation flag, or an option that could be passed to the method Find/FindOne, or maybe even a query parameter to prevent returning any field containing null values from the database. Without any success until now.
Are there any built-in solutions in the mongo-go-driver for this problem?
The problem is that the current bson codecs do not support encoding / decoding string into / from null.
One way to handle this is to create a custom decoder for string type in which we handle null values: we just use the empty string (and more importantly don't report error).
Custom decoders are described by the type bsoncodec.ValueDecoder. They can be registered at a bsoncodec.Registry, using a bsoncodec.RegistryBuilder for example.
Registries can be set / applied at multiple levels, even to a whole mongo.Client, or to a mongo.Database or just to a mongo.Collection, when acquiring them, as part of their options, e.g. options.ClientOptions.SetRegistry().
First let's see how we can do this for string, and next we'll see how to improve / generalize the solution to any type.
1. Handling null strings
First things first, let's create a custom string decoder that can turn a null into a(n empty) string:
import (
"go.mongodb.org/mongo-driver/bson/bsoncodec"
"go.mongodb.org/mongo-driver/bson/bsonrw"
"go.mongodb.org/mongo-driver/bson/bsontype"
)
type nullawareStrDecoder struct{}
func (nullawareStrDecoder) DecodeValue(dctx bsoncodec.DecodeContext, vr bsonrw.ValueReader, val reflect.Value) error {
if !val.CanSet() || val.Kind() != reflect.String {
return errors.New("bad type or not settable")
}
var str string
var err error
switch vr.Type() {
case bsontype.String:
if str, err = vr.ReadString(); err != nil {
return err
}
case bsontype.Null: // THIS IS THE MISSING PIECE TO HANDLE NULL!
if err = vr.ReadNull(); err != nil {
return err
}
default:
return fmt.Errorf("cannot decode %v into a string type", vr.Type())
}
val.SetString(str)
return nil
}
OK, and now let's see how to utilize this custom string decoder to a mongo.Client:
clientOpts := options.Client().
ApplyURI("mongodb://localhost:27017/").
SetRegistry(
bson.NewRegistryBuilder().
RegisterDecoder(reflect.TypeOf(""), nullawareStrDecoder{}).
Build(),
)
client, err := mongo.Connect(ctx, clientOpts)
From now on, using this client, whenever you decode results into string values, this registered nullawareStrDecoder decoder will be called to handle the conversion, which accepts bson null values and sets the Go empty string "".
But we can do better... Read on...
2. Handling null values of any type: "type-neutral" null-aware decoder
One way would be to create a separate, custom decoder and register it for each type we wish to handle. That seems to be a lot of work.
What we may (and should) do instead is create a single, "type-neutral" custom decoder which handles just nulls, and if the BSON value is not null, should call the default decoder to handle the non-null value.
This is surprisingly simple:
type nullawareDecoder struct {
defDecoder bsoncodec.ValueDecoder
zeroValue reflect.Value
}
func (d *nullawareDecoder) DecodeValue(dctx bsoncodec.DecodeContext, vr bsonrw.ValueReader, val reflect.Value) error {
if vr.Type() != bsontype.Null {
return d.defDecoder.DecodeValue(dctx, vr, val)
}
if !val.CanSet() {
return errors.New("value not settable")
}
if err := vr.ReadNull(); err != nil {
return err
}
// Set the zero value of val's type:
val.Set(d.zeroValue)
return nil
}
We just have to figure out what to use for nullawareDecoder.defDecoder. For this we may use the default registry: bson.DefaultRegistry, we may lookup the default decoder for individual types. Cool.
So what we do now is register a value of our nullawareDecoder for all types we want to handle nulls for. It's not that hard. We just list the types (or values of those types) we want this for, and we can take care of all with a simple loop:
customValues := []interface{}{
"", // string
int(0), // int
int32(0), // int32
}
rb := bson.NewRegistryBuilder()
for _, v := range customValues {
t := reflect.TypeOf(v)
defDecoder, err := bson.DefaultRegistry.LookupDecoder(t)
if err != nil {
panic(err)
}
rb.RegisterDecoder(t, &nullawareDecoder{defDecoder, reflect.Zero(t)})
}
clientOpts := options.Client().
ApplyURI("mongodb://localhost:27017/").
SetRegistry(rb.Build())
client, err := mongo.Connect(ctx, clientOpts)
In the example above I registered null-aware decoders for string, int and int32, but you may extend this list to your liking, just add values of the desired types to the customValues slice above.
You can go through the operator $exists and Query for Null or Missing Fields for a detail explanation.
In the mongo-go-driver, you can try below query:
The email => nil query matches documents that either contains the email field whose value is nil or that do not contain the email field.
cursor, err := coll.Find(
context.Background(),
bson.D{
{"email", nil},
})
You have to just add the $ne operator in the above query to get the records that do not have the field email or do not have the value nil in email. For more details about the operator $ne
If you know ahead of time which fields could potentially be null in your mongoDB records, you can use pointers in your structs instead:
type User struct {
Name string `bson:"name"` // Will still fail to decode if null in Mongo
Email *string `bson:"email"` // Will be nil in go if null in Mongo
}
Just remember that now you'll need to code more defensively around anything that uses this value after decoding from mongo, eg:
var reliableVal string
if User.Email != nil {
reliableVal = *user.Email
} else {
reliableVal = ""
}
I'm working with MGO (cause I didn't found anything better that it).
I'have played with it and have got some result but I don't understand how to get the _id (internal Mongo ObjectId) of document received?
For ex:
type FunnyNumber struct {
Value int
_id string
}
session, err := mgo.Dial("127.0.0.1:27017")
if err != nil {
panic(err)
}
defer session.Close()
// Optional. Switch the session to a monotonic behavior.
session.SetMode(mgo.Monotonic, true)
c := session.DB("m101").C("funnynumbers")
funnynumber := FunnyNumber{}
err = c.Find(bson.M{}).One(&funnynumber)
if err != nil {
log.Fatal(err)
}
fmt.Println("Id one:", funnynumber._id) // Nothing here? WHy? How to get it.
fmt.Println("Value one:", funnynumber.Value) // 62. It's OK!
Could someone help me, please? Ot where might I read some info about it? I haven't found anything in the MGO doc
Schema of my document is:
{ "_id" : ObjectId("50778ce69331a280cf4bcf90"), "value" : 62 }
Thanks!
Change _id variable to uppercase(ID) to make it exportable.
Use bson.ObjectID as its type.
Add tag for struct FunnyNumber Id variable.
field
Above three things should be done to get object Id value.
import "labix.org/v2/mgo/bson"
type FunnyNumber struct {
Value int `json:"value"`
Id bson.ObjectId `bson:"_id,omitempty"`` // only uppercase variables can be exported
}
Take a look at package BSON for more understanding on using bson tags when working with mongodb