How to encode/decode a empty string - encoding

I am using the GOB encoding for my project and i figured out (after a long fight) that empty strings are not encoded/decoded correctly. In my code i use a errormessage (string) to report any problems, this errormessage is most of the time empty. If i encode a empty string, it become nothing, and this gives me a problem with decoding. I don't want to alter the encoding/decoding because these parts are used the most.
How can i tell Go how to encode/decode empty strings?
Example:
Playground working code.
Playground not working code.

The problem isn't the encoding/gob module, but instead the custom MarshalBinary/UnmarshalBinary methods you've declared for Msg, which can't correctly round trip an empty string. There are two ways you could go here:
Get rid of the MarshalBinary/UnmarshalBinary methods and rely on GOB's default encoding for structures. This change alone wont' be enough because the fields of the structure aren't exported. If you're happy to export the fields then this is the simplest option: https://play.golang.org/p/rwzxTtaIh2
Use an encoding that can correctly round trip empty strings. One simple option would be to use GOB itself to encode the struct fields:
func (m Msg) MarshalBinary() ([]byte, error) {
var b bytes.Buffer
enc := gob.NewEncoder(&b)
if err := enc.Encode(m.x); err != nil {
return nil, err
}
if err := enc.Encode(m.y); err != nil {
return nil, err
}
if err := enc.Encode(m.z); err != nil {
return nil, err
}
return b.Bytes(), nil
}
// UnmarshalBinary modifies the receiver so it must take a pointer receiver.
func (m *Msg) UnmarshalBinary(data []byte) error {
dec := gob.NewDecoder(bytes.NewBuffer(data))
if err := dec.Decode(&m.x); err != nil {
return err
}
if err := dec.Decode(&m.y); err != nil {
return err
}
return dec.Decode(&m.z)
}
You can experiment with this example here: https://play.golang.org/p/oNXgt88FtK
The first option is obviously easier, but the second might be useful if your real example is a little more complex. Be careful with custom encoders though: GOB includes a few features that are intended to detect incompatibilities (e.g. if you add a field to a struct and try to decode old data), which are missing from this custom encoding.

Related

Problem with FindOne using mongo-driver/mongo in Golang

Struggling with unmarshalling data in Golang from mongo, may be cause I am new to this. Just started learning golang with MongoDB
Tried with map[string]interface{} to avoid any struct related errors
var data map[string]interface{}
filter := bson.M{"profile.username": username}
singleResult := u.getCollection(client).FindOne(u.ctx, filter)
err := singleResult.Decode(data)
This fails to unmarshall with error cannot Decode to nil value
Tried with exact struct structure too.
var result *models.UserData
filter := bson.M{"profile.username": username}
singleResult := u.getCollection(client).FindOne(u.ctx, filter)
err := singleResult.Decode(result)
Fails with same error cannot Decode to nil value
Tried to find all with map[string]interface{}
var result []models.UserData
cursor, _ := u.getCollection(client).Find(u.ctx, bson.M{})
err := cursor.All(u.ctx, &result)
Works perfectly as expected
Tried to find all with exact struct structure
var data []map[string]interface{}
cursor, _ := u.getCollection(client).Find(u.ctx, bson.M{})
err := cursor.All(u.ctx, &result)
Works perfectly as expected
Now I thought may be I am not finding the data in mongo but then
filter := bson.M{"profile.username": username}
singleResult := u.getCollection(client).FindOne(u.ctx, filter)
raw, _ := singleResult.DecodeBytes()
log.Print("\n\n" + raw.String()+"\n\n")
This prints the data as expected. Although one thing I noticed all non-string values are formatted as {"$numberLong":"1"}. Still don't know if it is correct or cause of the issue.
In your first 2 examples that fail, the data passed to Decode() are both nil:
// data == nil
var data map[string]interface{}
// ...
// result == nil
var result *models.UserData
Try like
var result = &models.UserData{} // init the pointer with a block of valid allocated memory
// ...
err := singleResult.Decode(result)
In order for Decode() to write the document(s) into the passed value, it must be a (non-nil) pointer. Passing any value creates a copy, and if you pass a non-pointer, only the copy could be modified. If you pass a pointer, a copy is still made, but Decode() will modify the pointed value, not the pointer.
In your first 2 examples that fail, you pass a non-pointer (or a nil pointer):
err := singleResult.Decode(result)
Modify it to pass a (non-nil) pointer:
err := singleResult.Decode(&result)
Your last 2 examples work because you're already passing (non-nil) pointers:
err := cursor.All(u.ctx, &result)

How to marshal json string to bson document for writing to MongoDB?

What I am looking is equivalent of Document.parse()
in golang, that allows me create bson from json directly? I do not want to create intermediate Go structs for marshaling
The gopkg.in/mgo.v2/bson package has a function called UnmarshalJSON which does exactly what you want.
The data parameter should hold you JSON string as []byte value.
func UnmarshalJSON(data []byte, value interface{}) error
UnmarshalJSON unmarshals a JSON value that may hold non-standard syntax as defined in BSON's extended JSON specification.
Example:
var bdoc interface{}
err = bson.UnmarshalJSON([]byte(`{"id": 1,"name": "A green door","price": 12.50,"tags": ["home", "green"]}`),&bdoc)
if err != nil {
panic(err)
}
err = c.Insert(&bdoc)
if err != nil {
panic(err)
}
mongo-go-driver has a function bson.UnmarshalExtJSON that does the job.
Here's the example:
var doc interface{}
err := bson.UnmarshalExtJSON([]byte(`{"foo":"bar"}`), true, &doc)
if err != nil {
// handle error
}
There is no longer a way to do this directly with supported libraries (e.g. the mongo-go-driver). You would need to write your own converter based on the bson spec.
Edit: here's one that by now has seen a few Terabytes of use in prod.
https://github.com/dustinevan/mongo/blob/main/bsoncv/bsoncv.go
I do not want to create intermediate Go structs for marshaling
If you do want/need to create an intermediate Go BSON structs, you could use a conversion module such github.com/sindbach/json-to-bson-go. For example:
import (
"fmt"
"github.com/sindbach/json-to-bson-go/convert"
"github.com/sindbach/json-to-bson-go/options"
)
func main() {
doc := `{"foo": "buildfest", "bar": {"$numberDecimal":"2021"} }`
opt := options.NewOptions()
result, _ := convert.Convert([]byte(doc), opt)
fmt.Println(result)
}
Will produce output:
package main
import "go.mongodb.org/mongo-driver/bson/primitive"
type Example struct {
Foo string `bson:"foo"`
Bar primitive.Decimal128 `bson:"bar"`
}
This module is compatible with the official MongoDB Go driver, and as you can see it supports Extended JSON formats.
You can also visit https://json-to-bson-map.netlify.app to try the module in action. You can paste a JSON document, and see the Go BSON structs as output.
A simple converter that uses go.mongodb.org/mongo-driver/bson/bsonrw:
func JsonToBson(message []byte) ([]byte, error) {
reader, err := bsonrw.NewExtJSONValueReader(bytes.NewReader(message), true)
if err != nil {
return []byte{}, err
}
buf := &bytes.Buffer{}
writer, _ := bsonrw.NewBSONValueWriter(buf)
err = bsonrw.Copier{}.CopyDocument(writer, reader)
if err != nil {
return []byte{}, err
}
marshaled := buf.Bytes()
return marshaled, nil
}

Go encoding transform issue

I have a following code in go:
import (
"log"
"net/http"
"code.google.com/p/go.text/transform"
"code.google.com/p/go.text/encoding/charmap"
)
...
res, err := http.Get(url)
if err != nil {
log.Println("Cannot read", url);
log.Println(err);
continue
}
defer res.Body.Close()
The page I load contain non UTF-8 symbols. So I try to use transform
utfBody := transform.NewReader(res.Body, charmap.Windows1251.NewDecoder())
But the problem is, that it returns error even in this simple scenarion:
bytes, err := ioutil.ReadAll(utfBody)
log.Println(err)
if err == nil {
log.Println(bytes)
}
transform: short destination buffer
It also actually sets bytes with some data, but in my real code I use goquery:
doc, err := goquery.NewDocumentFromReader(utfBody)
Which sees an error and fails with not data in return
I tried to pass "chunks" of res.Body to transform.NewReader and figuried out, that as long as res.Body contains no non-UTF8 data it works well. And when it contains non-UTF8 byte it fails with an error above.
I'm quite new to go and don't really understand what's going on and how to deal with this
Without the whole code along with an example URL it's hard to tell what exactly is going wrong here.
That said, I can recommend the golang.org/x/net/html/charset package for this as it supports both char guessing and converting to UTF 8.
func fetchUtf8Bytes(url string) ([]byte, error) {
res, err := http.Get(url)
if err != nil {
return nil, err
}
contentType := res.Header.Get("Content-Type") // Optional, better guessing
utf8reader, err := charset.NewReader(res.Body, contentType)
if err != nil {
return nil, err
}
return ioutil.ReadAll(utf8reader)
}
Complete example: http://play.golang.org/p/olcBM9ughv

Go Interface/Container usage

Newbie Go programmer here. I'm writing a package that reads a JSON configuration file. It uses the built-in JSON decoding, of course. But I want it to be able to include other JSON files as well, by looking for an array of filenames with the key of 'Includes'. I got it working as just a function and passing in a struct for the JSON data that includes a slice of strings labeled 'Includes', but I don't know how to specify this as a package.
Here's the function:
func ReadConfig(filename string, configuration *Configuration) error {
log.Println("reading file", filename)
file, err := os.Open(filename)
if err != nil {
log.Println("Can't read", filename)
return err
}
decoder := json.NewDecoder(file)
if err := decoder.Decode(&configuration); err != nil {
log.Println(err)
return err
}
includes := make([]string, len(configuration.Includes))
copy(includes, configuration.Includes)
config.Includes = configuration.Includes[0:0]
for _, inc := range includes {
log.Println(inc)
if err := ReadConfig(inc, configuration); err != nil {
return err
}
}
return nil
}
Which works with:
type Configuration struct {
Includes []string
.... other defs
}
But, in a package, I want ReadConfig to take any kind Configuration struct, as long as one of its members is 'Includes []string'.
I believe I need to change the ReadConfig def to:
func ReadConfig(filename string, configuration interface{})
But what I don't know is how to access the Includes slice within that.
Just create an interface for it
type Configurable interface {
Configuration() []string
}
And then provide a Configuration method instead of a field for your structs, and change the signature of your function to func ReadConfig(filename string, configuration Configurable).
It'd be much easier to just pass in the slice instead of the struct though.

How to insert math/big.Int in mongo via mgo in golang

I have a struct that contains math/big.Int fields. I would like to save the struct in mongodb using mgo. Saving the numbers as a strings is good enough in my situation.
I have looked at the available field's tags and nothing seams to allow custom serializer. I was expecting to implement an interface similar to encoding/json.Marshaler but I have found None of such interface in the documentation.
Here is a trivial example of what I want I need.
package main
import (
"labix.org/v2/mgo"
"math/big"
)
type Point struct {
X, Y *big.Int
}
func main() {
session, err := mgo.Dial("localhost")
if err != nil {
panic(err)
}
defer session.Close()
c := session.DB("test").C("test")
err = c.Insert(&Point{big.NewInt(1), big.NewInt(1)})
if err != nil { // should not panic
panic(err)
}
// The code run as expected but the fields X and Y are empty in mongo
}
Thnaks!
The similar interface is named bson.Getter:
http://labix.org/v2/mgo/bson#Getter
It can look similar to this:
func (point *Point) GetBSON() (interface{}, error) {
return bson.D{{"x", point.X.String()}, {"y", point.Y.String()}}, nil
}
And there's also the counterpart interface in the setter side, if you're interested:
http://labix.org/v2/mgo/bson#Setter
For using it, note that the bson.Raw type provided as a parameter has an Unmarshal method, so you could have a type similar to:
type dbPoint struct {
X string
Y string
}
and unmarshal it conveniently:
var dbp dbPoint
err := raw.Unmarshal(&dbp)
and then use the dbp.X and dbp.Y strings to put the big ints back into the real (point *Point) being unmarshalled.