How to store Golang big.Int into MongoDB - mongodb

I have a struct referencing a *big.Int. When storing this struct naively into MongoDB (using the official driver) the field turns to be nil when fetching the struct back. What is the proper/best way to store a big.Int into MongoDB?
type MyStruct struct {
Number *big.Int
}
nb := MyStruct{Number: big.NewInt(42)}
r, _ := db.Collection("test").InsertOne(context.TODO(), nb)
result := &MyStruct{}
db.Collection("test").FindOne(context.TODO(), bson.D{{"_id", r.InsertedID}}).Decode(result)
fmt.Println(result) // <== Number will be 0 here
My best idea so far would be to create a wrapper around big.Int that implements MarshalBSON and UnmarshalBSON (which I am not even sure how to do properly to be honest). But that'd be quite inconvenient.

Here's a possible implementation I came up with that stores the big.Int as plain text into MongoDb. It is also possible to easily store as byte array by using methods Bytes and SetBytes of big.Int instead of MarshalText/UnmarshalText.
package common
import (
"fmt"
"math/big"
"go.mongodb.org/mongo-driver/bson"
)
type BigInt struct {
i *big.Int
}
func NewBigInt(bigint *big.Int) *BigInt {
return &BigInt{i: bigint}
}
func (bi *BigInt) Int() *big.Int {
return bi.i
}
func (bi *BigInt) MarshalBSON() ([]byte, error) {
txt, err := bi.i.MarshalText()
if err != nil {
return nil, err
}
a, err := bson.Marshal(map[string]string{"i": string(txt)})
return a, err
}
func (bi *BigInt) UnmarshalBSON(data []byte) error {
var d bson.D
err := bson.Unmarshal(data, &d)
if err != nil {
return err
}
if v, ok := d.Map()["i"]; ok {
bi.i = big.NewInt(0)
return bi.i.UnmarshalText([]byte(v.(string)))
}
return fmt.Errorf("key 'i' missing")
}

the field turns to be nil when fetching the struct back
The reason why it returns 0, is because there is no bson mapping available for big.Int. If you check the document inserted into MongoDB collection you should see something similar to below:
{
"_id": ObjectId("..."),
"number": {}
}
Where there is no value stored in number field.
What is the proper/best way to store a big.Int into MongoDB?
Let's first understand what BigInt is. Big integer data type is intended for use when integer values might exceed the range that is supported by the int data type. The range is -2^63 (-9,223,372,036,854,775,808) to 2^63-1 (9,223,372,036,854,775,807) with storage size of 8 bytes. Generally this is used in SQL.
In Go, you can use a more precise integer types. Available built-in types int8, int16, int32, and int64 (and their unsigned counterparts) are best suited for data. The counterparts for big integer in Go is int64. With range -9,223,372,036,854,775,808 through 9,223,372,036,854,775,807, and storage size of 8 bytes.
Using mongo-go-driver,you can just use int64 which will be converted to bson RawValue.Int64. For example:
type MyStruct struct {
Number int64
}
collection := db.Collection("tests")
nb := MyStruct{Number: int64(42)}
r, _ := collection.InsertOne(context.TODO(), nb)
var result MyStruct
collection.FindOne(context.TODO(), bson.D{{"_id", r.InsertedID}}).Decode(&result)

Related

Go: Unable to unmarshal json value

I have jsonb value stored in postgres columns which I need to convert to object to send the response
I am unable to convert JSON -> struct using json.Unmarshal method
type Result struct {
Id int
Expertise string // because json string
Languages string // because json string
}
// saving dd query data to result struct
e, _ := json.Marshal(data.Expertise)
l, _ := json.Marshal(data.Languages)
row := r.db.QueryRow(`
INSERT INTO filters (user_id, expertise, languages)
VALUES ($1, $2, $3)
ON CONFLICT (user_id) DO
UPDATE SET expertise=$2, languages=$3
RETURNING id, expertise, languages;
`, userid, e, l)
var res Result
err := row.Scan(&res.Id, &res.Expertise, &res.Languages)
Then I take this the Expertise and Languages field and UNMARSHAL THEM
// helper method
func Unmarshal(value string) interface{} {
var obj interface{}
err := json.Unmarshal([]byte(value), &obj)
return obj
}
type CreateFilterResponse struct {
Id int `json:"id"`
Expertise interface{} `json:"expertise"`
Languages interface{} `json:"languages"`
}
response := dto.CreateFilterResponse{
Id: res.Id,
Expertise: Unmarshal(res.Expertise), // πŸ‘ˆ not decoding, still json
Languages: Unmarshal(res.Languages), // πŸ‘ˆ not decoding, still json
}
I would really appreciate the help, I need the Expertise and Languages to be {} and [] respectively
This is what I am getting in response:
{"id":5,"expertise":"[{\"role\":1,\"experience\":5},
{\"role\":3,\"experience\":4}]","languages":"[1,2,3]"}
Note:
Expertise is jsonb column : []{role: int, experience: int}
Languages is jsonb column : []int
Thanks to #Blackgreen comment, the suggestion worked.
As he suggested "I believe the issue is that you are scanning the jsonb bytes into string, which causes it to be interpreted literally. Then when you pass a string into json.Unmarshal it stays a string. Change the type of Expertise and Language to []byte (or json.RawMessage)"
I did the same and it works, here is the code:
// for parsing jsonb coming from db query
type Result struct {
Id int
Expertise json.RawMessage // change
Languages json.RawMessage // change
}
// the database query itself
e, _ := json.Marshal(filters.Expertise)
l, _ := json.Marshal(filters.Languages)
row := r.db.QueryRow(`
INSERT INTO filters (user_id, expertise, languages)
VALUES ($1, $2, $3)
ON CONFLICT (user_id) DO
UPDATE SET expertise=$2, languages=$3
RETURNING id, expertise, languages;
`, userid, e, l)
var res Result
err := row.Scan(&res.Id, &res.Expertise, &res.Languages) // scanning to result
if err != nil {
fmt.Println("Unable to create a new filter ==>", err)
return nil, err
}
Then I Unmarshalled the expertise and languages jsonb values using a helper method and created a Response struct for the client:
// response struct type
type CreateFilterResponse struct {
Id int `json:"id"`
Expertise interface{} `json:"expertise"`
Languages interface{} `json:"languages"`
}
// final response struct
response := dto.CreateFilterResponse{
Id: res.Id,
Expertise: Unmarshal(res.Expertise),
Languages: Unmarshal(res.Languages),
}
// helper method
func Unmarshal(value []byte) interface{} {
var obj interface{}
json.Unmarshal(value, &obj)
return obj
}
This works for now. I still need to find an easy way to do this as this is too much boilerplate code to perform a simple task. like in JS/TS it can be done using single line of code JSON.parse(value)
The problem is in the json output. It is encoding your JSON twice, making it difficult to read and deserialize the json in your structure.
You can start by organizing your structures as follows:
type Expertise struct {
Role int `json:"role"`
Experience int `json:"Experience"`
}
type CreateFilterResponse struct {
Id int `json:"id"`
Expert []Expertise `json:"expertise"`
Languages []int `json:"languages"`
}
I made a cleaner method just to exemplify and facilitate understanding. It will remove the unnecessary quotes and escapes so you can convert your string to []bytes and unmarshal your structure.
func jsonCleaner(quoted string) []byte{
quoted = strings.ReplaceAll(quoted, "\n", "")
quoted = strings.ReplaceAll(quoted, "\\", "")
quoted = strings.ReplaceAll(quoted, "]\"", "]")
quoted = strings.ReplaceAll(quoted, "\"[", "[")
dataBytes := []byte(quoted)
fmt.Println(dataBytes)
return dataBytes
}
Your json message would go from:
{"id":5,"expertise":"[{\"role\":1,\"experience\":5},{\"role\":3,\"experience\":4}]","languages":"[1,2,3]"}
To:
{"id":5,"expertise":[{"role":1,"experience":5},"role":3,"experience":4}],"languages":[1,2,3]}

bson.M {} deepequal does not seem to hande int32

I have a function for comparing two structs and making a bson document as input to mongodb updateOne()
Example struct format
type event struct {
...
Name string
StartTime int32
...
}
Diff function, please ignore that I have not checked for no difference yet.
func diffEvent(e event, u event) (bson.M, error) {
newValues := bson.M{}
if e.Name != u.Name {
newValues["name"] = u.Name
}
if e.StartTime != u.StartTime {
newValues["starttime"] = u.StartTime
}
...
return bson.M{"$set": newValues}, nil
}
Then I generated a test function like so:
func Test_diffEvent(t *testing.T) {
type args struct {
e event
u event
}
tests := []struct {
name string
args args
want bson.M
wantErr bool
}{
{
name: "update startime",
args: args{
e: event{StartTime: 1},
u: event{StartTime: 2},
},
want: bson.M{"$set": bson.M{"starttime": 2}},
wantErr: false,
},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
got, err := diffEvent(tt.args.e, tt.args.u)
if (err != nil) != tt.wantErr {
t.Errorf("diffEvent() error = %v, wantErr %v", err, tt.wantErr)
return
}
if !reflect.DeepEqual(got, tt.want) {
t.Errorf("diffEvent() = %v, want %v", got, tt.want)
}
})
}
}
This fails with a
--- FAIL: Test_diffEvent/update_startime (0.00s)
models_test.go:582: diffEvent() = map[$set:map[starttime:2]], want map[$set:map[starttime:2]]
For me this seem to be the same. I have played around with this and bool fields, string fields, enum fields, and fields as struct or fields as arrays of structs seems to work fine with deepequal, but it gives an error for int32 fields.
As a go beginner; what am I missing here? I would assume that if bool/string works then int32 would too.
This:
bson.M{"starttime": 2}
Sets the "starttime" key to the value of the literal 2. 2 is an untyped integer constant, and since no type is provided, its default type will be used which is int.
And 2 values stored in interface values are only equal if the dynamic value stored in them have identical type and value. So a value 2 with int type cannot be equal to a value 2 of type int32.
Use explicit type to tell you want to specify a value of int32 type:
bson.M{"starttime": int32(2)}

Will MongoDB Go Driver automatically convert β€˜uint64’ to bson type? Overflows error returned

As the title shows, I have a struct defined an uint64 field, but error returned when I set its value as math.MaxUint64.
This is my code:
type MyDoc struct {
Number uint64 `bson:"_id"`
Timestamp int64 `bson:"time"`
}
// I just want to know whether uint64 overflows or not.
func main() {
mydoc := &MyDoc{
Number: math.MaxUint64,
Timestamp: time.Now().UnixNano(),
}
v, err := bson.Marshal(mydoc)
if err != nil {
panic(err)
}
fmt.Println(v)
}
After executed, error is following:
panic: 18446744073709551615 overflows int64 [recovered]
panic: 18446744073709551615 overflows int64
Obviously, uint64 types of data are processed as int64 which is not I expect.
So, how to store an uint64 data but not overflows in MongoDB?? I can not use string type instead, because I need to compare the size of number so that sorts documents.
I am using MongoDB official Go Driver.
Thanks in advance!
You could use a custom encoder and decoder.
With the code you provided, you could do something like that:
// define a custom type so you can register custom encoder/decoder for it
type MyNumber uint64
type MyDoc struct {
Number MyNumber `bson:"_id"`
Timestamp int64 `bson:"time"`
}
// create a custom registry builder
rb := bsoncodec.NewRegistryBuilder()
// register default codecs and encoders/decoders
var primitiveCodecs bson.PrimitiveCodecs
bsoncodec.DefaultValueEncoders{}.RegisterDefaultEncoders(rb)
bsoncodec.DefaultValueDecoders{}.RegisterDefaultDecoders(rb)
primitiveCodecs.RegisterPrimitiveCodecs(rb)
// register custom encoder/decoder
myNumberType := reflect.TypeOf(MyNumber(0))
rb.RegisterTypeEncoder(
myNumberType,
bsoncodec.ValueEncoderFunc(func(_ bsoncodec.EncodeContext, vw bsonrw.ValueWriter, val reflect.Value) error {
if !val.IsValid() || val.Type() != myNumberType {
return bsoncodec.ValueEncoderError{
Name: "MyNumberEncodeValue",
Types: []reflect.Type{myNumberType},
Received: val,
}
}
// IMPORTANT STEP: cast uint64 to int64 so it can be stored in mongo
vw.WriteInt64(int64(val.Uint()))
return nil
}),
)
rb.RegisterTypeDecoder(
myNumberType,
bsoncodec.ValueDecoderFunc(func(_ bsoncodec.DecodeContext, vr bsonrw.ValueReader, val reflect.Value) error {
// IMPORTANT STEP: read sore value in mongo as int64
read, err := vr.ReadInt64()
if err != nil {
return err
}
// IMPORTANT STEP: cast back to uint64
val.SetUint(uint64(read))
return nil
}),
)
// build the registry
reg := rb.Build()
To use this custom registry, you can pass it as option when creating your collection
collection := client.Database("testing").Collection("myCollection", &options.CollectionOptions{
Registry: reg,
})

Golang Postgresql Array

If I have a table that returns something like:
id: 1
names: {Jim, Bob, Sam}
names is a varchar array.
How do I scan that back into a []string in Go?
I'm using lib/pg
Right now I have something like
rows, err := models.Db.Query("SELECT pKey, names FROM foo")
for rows.Next() {
var pKey int
var names []string
err = rows.Scan(&pKey, &names)
}
I keep getting:
panic: sql: Scan error on column index 1: unsupported Scan, storing driver.Value type []uint8 into type *[]string
It looks like I need to use StringArray
https://godoc.org/github.com/lib/pq#StringArray
But, I think I'm too new to Go to understand exactly how to use:
func (a *StringArray) Scan(src interface{})
You are right, you can use StringArray but you don't need to call the
func (a *StringArray) Scan(src interface{})
method yourself, this will be called automatically by rows.Scan when you pass it anything that implements the Scanner interface.
So what you need to do is to convert your []string to *StringArray and pass that to rows.Scan, like so:
rows, err := models.Db.Query("SELECT pKey, names FROM foo")
for rows.Next() {
var pKey int
var names []string
err = rows.Scan(&pKey, (*pq.StringArray)(&names))
}
Long Story Short, use like this to convert pgSQL array to GO array, here 5th column is coming as a array :
var _temp3 []string
for rows.Next() {
// ScanRows scan a row into temp_tbl
err := rows.Scan(&_temp, &_temp0, &_temp1, &_temp2, pq.Array(&_temp3))
In detail :
To insert a row that contains an array value, use the pq.Array function like this:
// "ins" is the SQL insert statement
ins := "INSERT INTO posts (title, tags) VALUES ($1, $2)"
// "tags" is the list of tags, as a string slice
tags := []string{"go", "goroutines", "queues"}
// the pq.Array function is the secret sauce
_, err = db.Exec(ins, "Job Queues in Go", pq.Array(tags))
To read a Postgres array value into a Go slice, use:
func getTags(db *sql.DB, title string) (tags []string) {
// the select query, returning 1 column of array type
sel := "SELECT tags FROM posts WHERE title=$1"
// wrap the output parameter in pq.Array for receiving into it
if err := db.QueryRow(sel, title).Scan(pq.Array(&tags)); err != nil {
log.Fatal(err)
}
return
}
Note: that in lib/pq, only slices of certain Go types may be passed to pq.Array().
Another example in which varchar array in pgSQL in generated at runtime in 5th column, like :
--> predefined_allow false admin iam.create {secrets,configMap}
I converted this as,
Q := "SELECT ar.policy_name, ar.allow, ar.role_name, pro.operation_name, ARRAY_AGG(pro.resource_id) as resources FROM iam.authorization_rules ar LEFT JOIN iam.policy_rules_by_operation pro ON pro.id = ar.operation_id GROUP BY ar.policy_name, ar.allow, ar.role_name, pro.operation_name;"
tx := g.db.Raw(Q)
rows, _ := tx.Rows()
defer rows.Close()
var _temp string
var _temp0 bool
var _temp1 string
var _temp2 string
var _temp3 []string
for rows.Next() {
// ScanRows scan a row into temp_tbl
err := rows.Scan(&_temp, &_temp0, &_temp1, &_temp2, pq.Array(&_temp3))
if err != nil {
return nil, err
}
fmt.Println("Query Executed...........\n", _temp, _temp0, _temp1, _temp2, _temp3)
}
Output :
Query Executed...........
predefined_allow false admin iam.create [secrets configMap]

How do I convert from a slice of interface{} to a slice of my struct type in Go? [duplicate]

This question already has answers here:
Type converting slices of interfaces
(9 answers)
Closed 3 years ago.
func GetFromDB(tableName string, m *bson.M) interface{} {
var (
__session *mgo.Session = getSession()
)
//if the query arg is nil. give it the null query
if m == nil {
m = &bson.M{}
}
__result := []interface{}{}
__cs_Group := __session.DB(T_dbName).C(tableName)
__cs_Group.Find(m).All(&__result)
return __result
}
call
GetFromDB(T_cs_GroupName, &bson.M{"Name": "Alex"}).([]CS_Group)
runtime will give me panic:
panic: interface conversion: interface is []interface {}, not []mydbs.CS_Group
how convert the return value to my struct?
You can't automatically convert between a slice of two different types – that includes []interface{} to []CS_Group. In every case, you need to convert each element individually:
s := GetFromDB(T_cs_GroupName, &bson.M{"Name": "Alex"}).([]interface{})
g := make([]CS_Group, 0, len(s))
for _, i := range s {
g = append(g, i.(CS_Group))
}
You need to convert the entire hierarchy of objects:
rawResult := GetFromDB(T_cs_GroupName, &bson.M{"Name": "Alex"}).([]interface{})
var result []CS_Group
for _, m := range rawResult {
result = append(result,
CS_Group{
SomeField: m["somefield"].(typeOfSomeField),
AnotherField: m["anotherfield"].(typeOfAnotherField),
})
}
This code is for the simple case where the type returned from mgo matches the type of your struct fields. You may need to sprinkle in some type conversions and type switches on the bson.M value types.
An alternate approach is to take advantage of mgo's decoder by passing the output slice as an argument:
func GetFromDB(tableName string, m *bson.M, result interface{}) error {
var (
__session *mgo.Session = getSession()
)
//if the query arg is nil. give it the null query
if m == nil {
m = &bson.M{}
}
__result := []interface{}{}
__cs_Group := __session.DB(T_dbName).C(tableName)
return __cs_Group.Find(m).All(result)
}
With this change, you can fetch directly to your type:
var result []CS_Group
err := GetFromDB(T_cs_GroupName, bson.M{"Name": "Alex"}, &result)
See also: FAQ: Can I convert a []T to an []interface{}?