I am writing a data structure that needs to hash an arbitrary object. The following function seems to fail if I give an int is the parameter.
func Hash( obj interface{} ) []byte {
digest := md5.New()
if err := binary.Write(digest, binary.LittleEndian, obj); err != nil {
panic(err)
}
return digest.Sum()
}
Calling this on an int results in:
panic: binary.Write: invalid type int
What is the right way to do this?
I found that a good way to do this is to serialize the object using the "gob" package, along the following lines:
var (
digest = md5.New()
encoder = gob.NewEncoder(digest)
)
func Hash(obj interface{}) []byte {
digest.Reset()
if err := encoder.Encode(obj); err != nil {
panic(err)
}
return digest.Sum()
}
Edit: This does not work as intended (see below).
binary.Write writes "a fixed-size value or a pointer to a fixed-size value." Type int is not a fixed size value; int is "either 32 or 64 bits." Use a fixed-size value like int32.
Related
I use Go with PostgreSQL using github.com/lib/pq and able to successfully fetch the records when my structure is known.
Now my query is how to fetch records when my structure changes dynamically?
By rows.columns I am able to fetch the column names, but could you help me with fetching the values of these columns for all the rows. I referred this link answered by #Luke, still, here the person has a structure defined.
Is it possible to retrieve a column value by name using GoLang database/sql
type Person struct {
Id int
Name string
}
Meanwhile I do not have a structure that is fixed, so how will I iterate through all the columns that too again for all rows. My approach would be a pointer to loop through all columns at first, then another one for going to next row.
Still not able to code this, Could you please help me with this, like how to proceed and get the values.
Since you don't know the structure up front you can return the rows as a two dimensional slice of empty interfaces. However for the row scan to work you'll need to pre-allocate the values to the appropriate type and to do this you can use the ColumnTypes method and the reflect package. Keep in mind that not every driver provides access to the columns' types so make sure the one you use does.
rows, err := db.Query("select * from foobar")
if err != nil {
return err
}
defer rows.Close()
// get column type info
columnTypes, err := rows.ColumnTypes()
if err != nil {
return err
}
// used for allocation & dereferencing
rowValues := make([]reflect.Value, len(columnTypes))
for i := 0; i < len(columnTypes); i++ {
// allocate reflect.Value representing a **T value
rowValues[i] = reflect.New(reflect.PtrTo(columnTypes[i].ScanType()))
}
resultList := [][]interface{}{}
for rows.Next() {
// initially will hold pointers for Scan, after scanning the
// pointers will be dereferenced so that the slice holds actual values
rowResult := make([]interface{}, len(columnTypes))
for i := 0; i < len(columnTypes); i++ {
// get the **T value from the reflect.Value
rowResult[i] = rowValues[i].Interface()
}
// scan each column value into the corresponding **T value
if err := rows.Scan(rowResult...); err != nil {
return err
}
// dereference pointers
for i := 0; i < len(rowValues); i++ {
// first pointer deref to get reflect.Value representing a *T value,
// if rv.IsNil it means column value was NULL
if rv := rowValues[i].Elem(); rv.IsNil() {
rowResult[i] = nil
} else {
// second deref to get reflect.Value representing the T value
// and call Interface to get T value from the reflect.Value
rowResult[i] = rv.Elem().Interface()
}
}
resultList = append(resultList, rowResult)
}
if err := rows.Err(); err != nil {
return err
}
fmt.Println(resultList)
This function prints the result of a query without knowing anything about the column types and count. It is a variant of the previous answer without using the reflect package.
func printQueryResult(db *sql.DB, query string) error {
rows, err := db.Query(query)
if err != nil {
return fmt.Errorf("canot run query %s: %w", query, err)
}
defer rows.Close()
cols, _ := rows.Columns()
row := make([]interface{}, len(cols))
rowPtr := make([]interface{}, len(cols))
for i := range row {
rowPtr[i] = &row[i]
}
fmt.Println(cols)
for rows.Next() {
err = rows.Scan(rowPtr...)
if err != nil {
fmt.Println("cannot scan row:", err)
}
fmt.Println(row...)
}
return rows.Err()
}
The trick is that rows.Scan can scan values into *interface{} but you have to wrap it in interface{} to be able to pass it to Scan using ....
i'm trying to convert string of objectid to bson ObjectId format with mgo,
errCheck := d.C("col").FindId(bson.ObjectIdHex(obid[0])).One(&Result)
idk why, but if i give a wrong / invalid input string, my application got runtime panic
how i can prevent that ? thank you
bson.ObjectIdHex() documents that it will panic if you pass an invalid object id:
ObjectIdHex returns an ObjectId from the provided hex representation. Calling this function with an invalid hex representation will cause a runtime panic. See the IsObjectIdHex function.
If you want to avoid this, first check your input string using bson.IsObjectIdHex(), and only proceed to call bson.ObjectIdHex() if your input is valid:
if bson.IsObjectIdHex(obid[0]) {
// It's valid, calling bson.ObjectIdHex() will not panic...
}
As #icza said in the last answer. you should check validity if ObjectId.
And you can use panic recover defer to handle any kind of error in future
package main
import (
"fmt"
"gopkg.in/mgo.v2/bson"
"path/filepath"
"runtime"
"strings"
)
func main() {
r := Result{}
getData(&r)
}
func IdentifyPanic() string {
var name, file string
var line int
var pc [16]uintptr
n := runtime.Callers(3, pc[:])
for _, pc := range pc[:n] {
fn := runtime.FuncForPC(pc)
if fn == nil {
continue
}
file, line = fn.FileLine(pc)
name = fn.Name()
if !strings.HasPrefix(name, "runtime.") {
break
}
}
file = filepath.Base(file)
switch {
case name != "":
return fmt.Sprintf("%v:%v", file, line)
case file != "":
return fmt.Sprintf("%v:%v", file, line)
}
return fmt.Sprintf("pc:%x", pc)
}
type Result struct {
success int
data string
}
func getData(result *Result){
defer func() {
if err := recover(); err != nil {
ip := IdentifyPanic()
errorMessage := fmt.Sprintf("%s Error: %s", ip, err)
fmt.Println(errorMessage)
result.success = 0
}
}()
if bson.IsObjectIdHex(obid[0]) { // this line copied from #icza answer
// It's valid, calling bson.ObjectIdHex() will not panic... // this line copied from #icza answer
errCheck := d.C("col").FindId(bson.ObjectIdHex(obid[0])).One(&res)
result.success = 1
result.data = "your result (res). this is just the exam"
}else{
result.success = 0
}
}
What I am looking is equivalent of Document.parse()
in golang, that allows me create bson from json directly? I do not want to create intermediate Go structs for marshaling
The gopkg.in/mgo.v2/bson package has a function called UnmarshalJSON which does exactly what you want.
The data parameter should hold you JSON string as []byte value.
func UnmarshalJSON(data []byte, value interface{}) error
UnmarshalJSON unmarshals a JSON value that may hold non-standard syntax as defined in BSON's extended JSON specification.
Example:
var bdoc interface{}
err = bson.UnmarshalJSON([]byte(`{"id": 1,"name": "A green door","price": 12.50,"tags": ["home", "green"]}`),&bdoc)
if err != nil {
panic(err)
}
err = c.Insert(&bdoc)
if err != nil {
panic(err)
}
mongo-go-driver has a function bson.UnmarshalExtJSON that does the job.
Here's the example:
var doc interface{}
err := bson.UnmarshalExtJSON([]byte(`{"foo":"bar"}`), true, &doc)
if err != nil {
// handle error
}
There is no longer a way to do this directly with supported libraries (e.g. the mongo-go-driver). You would need to write your own converter based on the bson spec.
Edit: here's one that by now has seen a few Terabytes of use in prod.
https://github.com/dustinevan/mongo/blob/main/bsoncv/bsoncv.go
I do not want to create intermediate Go structs for marshaling
If you do want/need to create an intermediate Go BSON structs, you could use a conversion module such github.com/sindbach/json-to-bson-go. For example:
import (
"fmt"
"github.com/sindbach/json-to-bson-go/convert"
"github.com/sindbach/json-to-bson-go/options"
)
func main() {
doc := `{"foo": "buildfest", "bar": {"$numberDecimal":"2021"} }`
opt := options.NewOptions()
result, _ := convert.Convert([]byte(doc), opt)
fmt.Println(result)
}
Will produce output:
package main
import "go.mongodb.org/mongo-driver/bson/primitive"
type Example struct {
Foo string `bson:"foo"`
Bar primitive.Decimal128 `bson:"bar"`
}
This module is compatible with the official MongoDB Go driver, and as you can see it supports Extended JSON formats.
You can also visit https://json-to-bson-map.netlify.app to try the module in action. You can paste a JSON document, and see the Go BSON structs as output.
A simple converter that uses go.mongodb.org/mongo-driver/bson/bsonrw:
func JsonToBson(message []byte) ([]byte, error) {
reader, err := bsonrw.NewExtJSONValueReader(bytes.NewReader(message), true)
if err != nil {
return []byte{}, err
}
buf := &bytes.Buffer{}
writer, _ := bsonrw.NewBSONValueWriter(buf)
err = bsonrw.Copier{}.CopyDocument(writer, reader)
if err != nil {
return []byte{}, err
}
marshaled := buf.Bytes()
return marshaled, nil
}
I have a struct that contains math/big.Int fields. I would like to save the struct in mongodb using mgo. Saving the numbers as a strings is good enough in my situation.
I have looked at the available field's tags and nothing seams to allow custom serializer. I was expecting to implement an interface similar to encoding/json.Marshaler but I have found None of such interface in the documentation.
Here is a trivial example of what I want I need.
package main
import (
"labix.org/v2/mgo"
"math/big"
)
type Point struct {
X, Y *big.Int
}
func main() {
session, err := mgo.Dial("localhost")
if err != nil {
panic(err)
}
defer session.Close()
c := session.DB("test").C("test")
err = c.Insert(&Point{big.NewInt(1), big.NewInt(1)})
if err != nil { // should not panic
panic(err)
}
// The code run as expected but the fields X and Y are empty in mongo
}
Thnaks!
The similar interface is named bson.Getter:
http://labix.org/v2/mgo/bson#Getter
It can look similar to this:
func (point *Point) GetBSON() (interface{}, error) {
return bson.D{{"x", point.X.String()}, {"y", point.Y.String()}}, nil
}
And there's also the counterpart interface in the setter side, if you're interested:
http://labix.org/v2/mgo/bson#Setter
For using it, note that the bson.Raw type provided as a parameter has an Unmarshal method, so you could have a type similar to:
type dbPoint struct {
X string
Y string
}
and unmarshal it conveniently:
var dbp dbPoint
err := raw.Unmarshal(&dbp)
and then use the dbp.X and dbp.Y strings to put the big ints back into the real (point *Point) being unmarshalled.
I try to populate the exported fields of a struct with content fetched from a MongoDb-database using the labix.org/v2/mgo package.
mgo uses the labix.org/v2/mgo/bson package to unmarshal the data. But the unmarshaller sets all unexported fields to their zero value.
Is there any way to prevent this behavior?
Working example:
package main
import (
"fmt"
"labix.org/v2/mgo/bson"
)
type Sub struct{ Int int }
type Player struct {
Name string
unexpInt int
unexpPoint *Sub
}
func main() {
dta,err := bson.Marshal(bson.M{"name": "ANisus"})
if err != nil {
panic(err)
}
p := &Player{unexpInt: 12, unexpPoint: &Sub{42}}
fmt.Printf("Before: %+v\n", p)
err = bson.Unmarshal(dta, p)
if err != nil {
panic(err)
}
fmt.Printf("After: %+v\n", p)
}
Output:
Before: &{Name: unexpInt:12 unexpPoint:0xf84005f500}
After: &{Name:ANisus unexpInt:0 unexpPoint:<nil>}
This is not possible. As you can see in the source code, struct values are explicitly being set to their zero value before filling in any fields.
There is no option to disable this behaviour. It is presumably in place to make sure the result of Unmarshal() only depends on the BSON data and not any prior state.