I am trying to insert an array into a MongoDB instance using Go. I have the [] string slice in Go and want to convert it into a BSON array to pass it to the DB using the github.com/mongodb/mongo-go-driver driver.
var result bson.Array
for _, data := range myData {
value := bson.VC.String(data)
result.Append(value)
}
This loops over each element of my input data and tries to append it to the BSON array. However the line with the Append() fails with panic: document is nil. How should I do this conversion?
Edit: The code in the question and this answer is no longer relevant because the bson.Array type was deleted from the package. At the time of this edit, the bson.A and basic slice operations should be used to construct arrays.
Use the factory function NewArray to create the array:
result := bson.NewArray()
for _, data := range myData {
value := bson.VC.String(data)
result.Append(value)
}
As mentioned by #Cerise bson.Array has since been deleted. I do this with multiple utility functions as follows:
func BSONStringA(sa []string) (result bson.A) {
result = bson.A{}
for_, e := range sa {
result = append(result, e)
}
return
}
func BSONIntA(ia []string) (result bson.A) {
// ...
}
Converting a slice of string (ids) to BSON array
var objIds bson.A
for _, val := range ids {
objIds = append(objIds, val)
}
log.Println(objIds)
Related
I have a struct referencing a *big.Int. When storing this struct naively into MongoDB (using the official driver) the field turns to be nil when fetching the struct back. What is the proper/best way to store a big.Int into MongoDB?
type MyStruct struct {
Number *big.Int
}
nb := MyStruct{Number: big.NewInt(42)}
r, _ := db.Collection("test").InsertOne(context.TODO(), nb)
result := &MyStruct{}
db.Collection("test").FindOne(context.TODO(), bson.D{{"_id", r.InsertedID}}).Decode(result)
fmt.Println(result) // <== Number will be 0 here
My best idea so far would be to create a wrapper around big.Int that implements MarshalBSON and UnmarshalBSON (which I am not even sure how to do properly to be honest). But that'd be quite inconvenient.
Here's a possible implementation I came up with that stores the big.Int as plain text into MongoDb. It is also possible to easily store as byte array by using methods Bytes and SetBytes of big.Int instead of MarshalText/UnmarshalText.
package common
import (
"fmt"
"math/big"
"go.mongodb.org/mongo-driver/bson"
)
type BigInt struct {
i *big.Int
}
func NewBigInt(bigint *big.Int) *BigInt {
return &BigInt{i: bigint}
}
func (bi *BigInt) Int() *big.Int {
return bi.i
}
func (bi *BigInt) MarshalBSON() ([]byte, error) {
txt, err := bi.i.MarshalText()
if err != nil {
return nil, err
}
a, err := bson.Marshal(map[string]string{"i": string(txt)})
return a, err
}
func (bi *BigInt) UnmarshalBSON(data []byte) error {
var d bson.D
err := bson.Unmarshal(data, &d)
if err != nil {
return err
}
if v, ok := d.Map()["i"]; ok {
bi.i = big.NewInt(0)
return bi.i.UnmarshalText([]byte(v.(string)))
}
return fmt.Errorf("key 'i' missing")
}
the field turns to be nil when fetching the struct back
The reason why it returns 0, is because there is no bson mapping available for big.Int. If you check the document inserted into MongoDB collection you should see something similar to below:
{
"_id": ObjectId("..."),
"number": {}
}
Where there is no value stored in number field.
What is the proper/best way to store a big.Int into MongoDB?
Let's first understand what BigInt is. Big integer data type is intended for use when integer values might exceed the range that is supported by the int data type. The range is -2^63 (-9,223,372,036,854,775,808) to 2^63-1 (9,223,372,036,854,775,807) with storage size of 8 bytes. Generally this is used in SQL.
In Go, you can use a more precise integer types. Available built-in types int8, int16, int32, and int64 (and their unsigned counterparts) are best suited for data. The counterparts for big integer in Go is int64. With range -9,223,372,036,854,775,808 through 9,223,372,036,854,775,807, and storage size of 8 bytes.
Using mongo-go-driver,you can just use int64 which will be converted to bson RawValue.Int64. For example:
type MyStruct struct {
Number int64
}
collection := db.Collection("tests")
nb := MyStruct{Number: int64(42)}
r, _ := collection.InsertOne(context.TODO(), nb)
var result MyStruct
collection.FindOne(context.TODO(), bson.D{{"_id", r.InsertedID}}).Decode(&result)
I am trying to insert a struct in mongo database.
type SecretsStruct struct {
UserID string `bson:"userid" json:"userid"`
secretOne string `bson:"secret_one" json:secret_one`
secretTwo string `bson:"secret_two" json:secret_two`
secretThree string `bson:"secret_three" json:secret_three`
}
func (c *SecretsStruct) SetSecrets(userId string, encryptedKeys
[][]byte){
c.UserID = userId
c.secretOne = hex.EncodeToString(encryptedKeys[0])
c.secretTwo = hex.EncodeToString(encryptedKeys[1])
c.secretThree = hex.EncodeToString(encryptedKeys[2])
log.Printf("This is the c %s", c)
}
g := SecretsStruct{}
g.SetSecrets(userStruct.UserID, encryptedKeys)
err = secretCollection.Insert(g)
if err != nil {
panic(err)
}
I have tried inserting the byte arrays corresponding to the secrets but of no help. The result which gets populated to the corresponding insertion operation is :
{'_id': ObjectId('5b80117c118c660aaa0c87c2'),
'userid': 'eb19d220-ef13-43aa-8a7f-f78637718000'}
On the other hand, if I try to insert same data with a map but without struct.
secretCollection.Insert(bson.M{"userid": userStruct.UserID,
"secret_one": encryptedKeys[0],
"secret_two": encryptedKeys[1],
"secret_three": encryptedKeys[2]})
The insertion operation executes successfully.
You have to export your struct fields, so that another package (in this case mgo) can access them:
type SecretsStruct struct {
UserID string `bson:"userid" json:"userid"`
SecretOne string `bson:"secret_one" json:secret_one`
SecretTwo string `bson:"secret_two" json:secret_two`
SecretThree string `bson:"secret_three" json:secret_three`
}
If I have a table that returns something like:
id: 1
names: {Jim, Bob, Sam}
names is a varchar array.
How do I scan that back into a []string in Go?
I'm using lib/pg
Right now I have something like
rows, err := models.Db.Query("SELECT pKey, names FROM foo")
for rows.Next() {
var pKey int
var names []string
err = rows.Scan(&pKey, &names)
}
I keep getting:
panic: sql: Scan error on column index 1: unsupported Scan, storing driver.Value type []uint8 into type *[]string
It looks like I need to use StringArray
https://godoc.org/github.com/lib/pq#StringArray
But, I think I'm too new to Go to understand exactly how to use:
func (a *StringArray) Scan(src interface{})
You are right, you can use StringArray but you don't need to call the
func (a *StringArray) Scan(src interface{})
method yourself, this will be called automatically by rows.Scan when you pass it anything that implements the Scanner interface.
So what you need to do is to convert your []string to *StringArray and pass that to rows.Scan, like so:
rows, err := models.Db.Query("SELECT pKey, names FROM foo")
for rows.Next() {
var pKey int
var names []string
err = rows.Scan(&pKey, (*pq.StringArray)(&names))
}
Long Story Short, use like this to convert pgSQL array to GO array, here 5th column is coming as a array :
var _temp3 []string
for rows.Next() {
// ScanRows scan a row into temp_tbl
err := rows.Scan(&_temp, &_temp0, &_temp1, &_temp2, pq.Array(&_temp3))
In detail :
To insert a row that contains an array value, use the pq.Array function like this:
// "ins" is the SQL insert statement
ins := "INSERT INTO posts (title, tags) VALUES ($1, $2)"
// "tags" is the list of tags, as a string slice
tags := []string{"go", "goroutines", "queues"}
// the pq.Array function is the secret sauce
_, err = db.Exec(ins, "Job Queues in Go", pq.Array(tags))
To read a Postgres array value into a Go slice, use:
func getTags(db *sql.DB, title string) (tags []string) {
// the select query, returning 1 column of array type
sel := "SELECT tags FROM posts WHERE title=$1"
// wrap the output parameter in pq.Array for receiving into it
if err := db.QueryRow(sel, title).Scan(pq.Array(&tags)); err != nil {
log.Fatal(err)
}
return
}
Note: that in lib/pq, only slices of certain Go types may be passed to pq.Array().
Another example in which varchar array in pgSQL in generated at runtime in 5th column, like :
--> predefined_allow false admin iam.create {secrets,configMap}
I converted this as,
Q := "SELECT ar.policy_name, ar.allow, ar.role_name, pro.operation_name, ARRAY_AGG(pro.resource_id) as resources FROM iam.authorization_rules ar LEFT JOIN iam.policy_rules_by_operation pro ON pro.id = ar.operation_id GROUP BY ar.policy_name, ar.allow, ar.role_name, pro.operation_name;"
tx := g.db.Raw(Q)
rows, _ := tx.Rows()
defer rows.Close()
var _temp string
var _temp0 bool
var _temp1 string
var _temp2 string
var _temp3 []string
for rows.Next() {
// ScanRows scan a row into temp_tbl
err := rows.Scan(&_temp, &_temp0, &_temp1, &_temp2, pq.Array(&_temp3))
if err != nil {
return nil, err
}
fmt.Println("Query Executed...........\n", _temp, _temp0, _temp1, _temp2, _temp3)
}
Output :
Query Executed...........
predefined_allow false admin iam.create [secrets configMap]
This question already has answers here:
Type converting slices of interfaces
(9 answers)
Closed 3 years ago.
func GetFromDB(tableName string, m *bson.M) interface{} {
var (
__session *mgo.Session = getSession()
)
//if the query arg is nil. give it the null query
if m == nil {
m = &bson.M{}
}
__result := []interface{}{}
__cs_Group := __session.DB(T_dbName).C(tableName)
__cs_Group.Find(m).All(&__result)
return __result
}
call
GetFromDB(T_cs_GroupName, &bson.M{"Name": "Alex"}).([]CS_Group)
runtime will give me panic:
panic: interface conversion: interface is []interface {}, not []mydbs.CS_Group
how convert the return value to my struct?
You can't automatically convert between a slice of two different types – that includes []interface{} to []CS_Group. In every case, you need to convert each element individually:
s := GetFromDB(T_cs_GroupName, &bson.M{"Name": "Alex"}).([]interface{})
g := make([]CS_Group, 0, len(s))
for _, i := range s {
g = append(g, i.(CS_Group))
}
You need to convert the entire hierarchy of objects:
rawResult := GetFromDB(T_cs_GroupName, &bson.M{"Name": "Alex"}).([]interface{})
var result []CS_Group
for _, m := range rawResult {
result = append(result,
CS_Group{
SomeField: m["somefield"].(typeOfSomeField),
AnotherField: m["anotherfield"].(typeOfAnotherField),
})
}
This code is for the simple case where the type returned from mgo matches the type of your struct fields. You may need to sprinkle in some type conversions and type switches on the bson.M value types.
An alternate approach is to take advantage of mgo's decoder by passing the output slice as an argument:
func GetFromDB(tableName string, m *bson.M, result interface{}) error {
var (
__session *mgo.Session = getSession()
)
//if the query arg is nil. give it the null query
if m == nil {
m = &bson.M{}
}
__result := []interface{}{}
__cs_Group := __session.DB(T_dbName).C(tableName)
return __cs_Group.Find(m).All(result)
}
With this change, you can fetch directly to your type:
var result []CS_Group
err := GetFromDB(T_cs_GroupName, bson.M{"Name": "Alex"}, &result)
See also: FAQ: Can I convert a []T to an []interface{}?
want to save the data of the following format
{"_ibj_id":"1","url_id":'1',"url":{"0":"http://0.com","1":"http:://1.com"}}
Look at my code,
type db_list struct {
Url_id int
Url map[int]string
}
func list(table *mgo.Collection) {
var doc *goquery.Document
var e error
for i := 1628644; i > 1628643; i-- {
if doc, e = goquery.NewDocument("http://www.120ask.com/list/all/" + strconv.Itoa(i)); e != nil {
panic(e.Error())
}
var save_list db_list
save_list.Url_id = i
save_list.Url = make(map[int]string)
//fmt.Println("%s", doc.Text())
doc.Find(".q-quename").Each(func(n int, s *goquery.Selection) {
href, isTrue := s.Attr("href")
if isTrue {
save_list.Url[n] = href
fmt.Println("%D : %s", n, save_list.Url[n])
}
})
fmt.Println("%D", len(save_list.Url))
//save database
table.Insert(save_list)
}
}
The database will eventually save
Please view the picture in the annex, is to save the format of the data, save the URLvalue of the property 1
You're probably after the JSON Unmarshal function in encoding/json
{"_ibj_id":"1","url_id":'1',"url":{"0":"http://0.com","1":"http:://1.com"}} is technically invalid JSON due to the single-quotes around the url_id value( '1' should be "1") but other than that, it should map nicely to the following struct:
{
id string
url_id string
urls []string
}
But you may need to experiment with the types. According to the docs for the Unmarshal function, it will use the following Go types for each JSON type:
bool, for JSON booleans
float64, for JSON numbers
string, for JSON strings
[]interface{}, for JSON arrays
map[string]interface{}, for JSON objects
nil for JSON null
I'd highly recommend reading Andrew Gerrands Blog Post "JSON and Go".
It's unclear to me exactly what you are trying to do. One thing I notice though, is that in your desired format the keys for Url are string values where as in Go they are integers. You could try changing the type of Url to map[string]string and see if that solves your problem.