Problem description
I try to find documents stored in MongoDB using GO
Current state
For testing purposes I created a small test program that inserts data into MongoDB and immediately tries to query:
package main
import (
"fmt"
"gopkg.in/mgo.v2"
"gopkg.in/mgo.v2/bson"
)
type IndexedData struct {
ID bson.ObjectId `json:"id" bson:"_id,omitempty"`
MyID int `json:"myid" bson:"myid"`
Content string `json:"content" bson:"content"`
}
func main() {
// Create a client
session, err := mgo.Dial("localhost")
if err != nil {
panic(err)
}
defer session.Close()
collection := session.DB("test").C("demo")
for index := 0; index < 10; index++ {
data := IndexedData{ID: bson.NewObjectId(), MyID: index, Content: "Some string"}
err = collection.Insert(data)
if nil != err {
panic(err)
} else {
fmt.Println("Successfully inserted")
}
}
for index := 9; index >= 0; index-- {
qry := collection.Find(IndexedData{MyID: index})
cnt, err := qry.Count()
if err != nil {
fmt.Println(fmt.Sprintf("%v - %s", index, err.Error()))
} else {
if cnt == 1 {
fmt.Println("Found")
} else {
if cnt > 1 {
fmt.Println(fmt.Sprintf("%v - Multiple: %v", index, cnt))
} else {
fmt.Println(fmt.Sprintf("%v - Not found", index))
}
}
}
}
qry := collection.Find(nil)
cnt, err := qry.Count()
if err != nil {
panic(err)
}
fmt.Println(fmt.Sprintf("%v items", cnt))
err = collection.DropCollection()
if err != nil {
panic(err)
}
}
Results
Actual
Successfully inserted
Successfully inserted
Successfully inserted
Successfully inserted
Successfully inserted
Successfully inserted
Successfully inserted
Successfully inserted
Successfully inserted
Successfully inserted
9 - Not found
8 - Not found
7 - Not found
6 - Not found
5 - Not found
4 - Not found
3 - Not found
2 - Not found
1 - Not found
0 - Not found
10 items
Expecting
I had expected to get 10 times Found
Update 1
I changed
qry := collection.Find(IndexedData{MyID: index})
to
qry := collection.Find(bson.M{"myid": index})
and I got a working sample.
The documentation states:
The document may be a map or a struct value capable of being marshalled with bson. The map may be a generic one using interface{} for its key and/or values, such as bson.M, or it may be a properly typed map.
I interpreted a properly annotated struct will work.
Question
How do I query for a document's property successfully?
I think you can open a issue.
I compared output of bson.Marshal for bson.M and your struct.
1 is fmt.Printf("%v\n", in) for your 36 row.
2 is fmt.Printf("%v\n", in) for that row
3 is otput println(string(e.out)) for that row
Output for e.out is different for struct and map. I suspect it's a bug.
Also I noticed that there is not some test which test struct. All tests use bson.M.
Thanks for a brilliant format of question!
I don't know anything about go but I would guess that you don't need the nil in the line qry := collection.Find(nil)
Related
I'm using SELECT * in a db.query() to return columns from a table. Typically, I would fmt.Scan() the rows into a pre-declared struct{} for further manipulation, but in this case, the table columns change frequently so I'm not able to use a declared struct{} as part of my Scan().
I've been struggling to figure out how I might dynamically build a struct{} based on the column results of the db.query() which I could subsequently call on the use of for Scan(). I've read a little about reflect but I'm struggling to determine if this is right for my use-case or if I might have to think about something else.
Any pointers would be greatly appreciated.
you can get column names from resulting rowset and prepare a slice for the scan.
Example (https://go.dev/play/p/ilYmEIWBG5S) :
package main
import (
"database/sql"
"fmt"
"log"
"github.com/DATA-DOG/go-sqlmock"
)
func main() {
// mock db
db, mock, err := sqlmock.New()
if err != nil {
log.Fatal(err)
}
columns := []string{"id", "status"}
mock.ExpectQuery("SELECT \\* FROM table").
WillReturnRows(sqlmock.NewRows(columns).AddRow(1, "ok"))
// actual code
rows, err := db.Query("SELECT * FROM table")
if err != nil {
log.Fatal(err)
}
cols, err := rows.Columns()
if err != nil {
log.Fatal(err)
}
data := make([]interface{}, len(cols))
strs := make([]sql.NullString, len(cols))
for i := range data {
data[i] = &strs[i]
}
for rows.Next() {
if err := rows.Scan(data...); err != nil {
log.Fatal(err)
}
for i, d := range data {
fmt.Printf("%s = %+v\n", cols[i], d)
}
}
}
This example reads all columns into strings. To detect column type one can use rows.ColumnTypes method.
I might have found a weird bug..
I do update once in a while my mongodb-driver package as I use for year now I layer I've made a long time ago.
However, today, for some personal reasons, I have refactored my tests and now have e2e full tests suite over my layer. The thing is, while doing my tests, I could reproduce a bug I had in the past, which is when I don't set an ID for my document (_id bson field)
Situation:
For obvious reason, some piece of code are missing but I am quite sure you will be able to guess really easily ;)
The commented out code part is the tests that are creating the weird behavior
If I create a document (really simple struct) and set the _id field before inserting, it works perfectly.
If I create a document (really simple struct) and don't set the _id field before inserting, it inserts something that results as "null" while observing the database using Robo3T.
Code:
To reproduce that bug, there is my tests and the source code of my layer
Tests:
type e2eMongoDBSuite struct {
suite.Suite
_mongodb mongodb.IMongoDB
}
type e2eTestDocument struct {
ID *primitive.ObjectID `json:"_id" bson:"_id"`
ValueStr string `json:"value_str" bson:"value_str"`
ValueInt32 int32 `json:"value_int32" bson:"value_int32"`
}
const (
//// ci
connectionString = "mongodb://localhost:27017"
databaseName = "mongodb-database-e2e-tests"
collection = "mongodb-collection-e2e-tests"
defaultTimeout = 5 * time.Second
)
func (e2eSuite *e2eMongoDBSuite) SetupSuite() {
var err error
if e2eSuite._mongodb, err = mongodb.New(&mongodb.Config{
ConnectionString: connectionString,
DatabaseName: databaseName,
DefaultTimeout: defaultTimeout,
}); err != nil {
panic(fmt.Errorf("couldn't setup suite: %w", err).Error())
}
}
func (e2eSuite *e2eMongoDBSuite) TestInsertOneWithID_OK() {
a := assert.New(e2eSuite.T())
// given
docID := e2eSuite._mongodb.GenerateNewObjectID()
doc := &e2eTestDocument{ID: docID}
// when
insertedID, err := e2eSuite._mongodb.InsertOne(collection, doc)
// expected
a.NotNil(insertedID, "the inserted document ID should be returned")
a.Equal(docID, insertedID, "the inserted document ID should be the same as the given one at creation")
a.Nil(err, "there should be no error for this insert")
// then
_ = e2eSuite._mongodb.DeleteMany(collection, bson.D{})
}
//func (e2eSuite *e2eMongoDBSuite) TestInsertOneWithNoID_OK() {
// a := assert.New(e2eSuite.T())
//
// // given
// doc := &e2eTestDocument{}
//
// // when
// insertedID, err := e2eSuite._mongodb.InsertOne(collection, doc)
//
// // expected
// a.NotNil(insertedID, "the inserted document ID should be returned")
// //a.Equal(docID, insertedID, "the inserted document ID should be the same as the given one at creation")
// a.Nil(err, "there should be no error for this insert")
//
// // then
// _ = e2eSuite._mongodb.DeleteMany(collection, bson.D{})
//}
func (e2eSuite *e2eMongoDBSuite) TestInsertManyWithIDs_OK() {
a := assert.New(e2eSuite.T())
// given
docID1 := e2eSuite._mongodb.GenerateNewObjectID()
docID2 := e2eSuite._mongodb.GenerateNewObjectID()
doc1 := &e2eTestDocument{ID: docID1}
doc2 := &e2eTestDocument{ID: docID2}
// when
insertedIDs, err := e2eSuite._mongodb.InsertMany(collection, []*e2eTestDocument{doc1, doc2})
// expected
a.NotNil(insertedIDs, "the inserted document IDs should be returned")
a.Equal(2, len(insertedIDs), "the inserted document IDs amount should be of two")
a.EqualValues([]*primitive.ObjectID{docID1, docID2}, insertedIDs, "the inserted document IDs should be the same as the given ones at creation")
a.Nil(err, "there should be no error for this insert")
// then
_ = e2eSuite._mongodb.DeleteMany(collection, bson.D{})
}
//func (e2eSuite *e2eMongoDBSuite) TestInsertManyWithNoIDs_OK() {
// a := assert.New(e2eSuite.T())
//
// // given
// doc1 := &e2eTestDocument{}
// doc2 := &e2eTestDocument{}
//
// // when
// insertedIDs, err := e2eSuite._mongodb.InsertMany(collection, []*e2eTestDocument{doc1, doc2})
//
// // expected
// a.NotNil(insertedIDs, "the inserted document IDs should be returned")
// a.Equal(2, len(insertedIDs), "the inserted document IDs amount should be of two")
// a.Nil(err, "there should be no error for this insert")
//
//// then
//_ = e2eSuite._mongodb.DeleteMany(collection, bson.D{})
//}
func (e2eSuite *e2eMongoDBSuite) TearDownSuite() {
_ = e2eSuite._mongodb.DropDatabase()
}
func TestE2eMongoDBSuite(t *testing.T) {
suite.Run(t, new(e2eMongoDBSuite))
}
The layer i've developed:
// InsertOne inserts a document in a given collection and returns the inserted ObjectID.
func (m *mongoDB) InsertOne(collectionName string, document interface{}) (*primitive.ObjectID, error) {
res, err := m.GetDatabase().Collection(collectionName).InsertOne(context.Background(), document)
if err != nil {
return nil, err
}
if oid, ok := res.InsertedID.(primitive.ObjectID); ok {
return &oid, nil
}
return nil, nil
}
// InsertMany inserts documents in a given collection and returns the inserted ObjectIDs.
func (m *mongoDB) InsertMany(collectionName string, documents interface{}) ([]*primitive.ObjectID, error) {
s := reflect.ValueOf(documents)
if s.Kind() != reflect.Slice {
panic("Documents given a non-slice type")
}
slice := make([]interface{}, s.Len())
for i := 0; i < s.Len(); i++ {
slice[i] = s.Index(i).Interface()
}
res, err := m.GetDatabase().Collection(collectionName).InsertMany(context.Background(), slice)
if err != nil {
return nil, err
}
var insertedIDs []*primitive.ObjectID
for _, insertedID := range res.InsertedIDs {
if oid, ok := insertedID.(primitive.ObjectID); ok {
insertedIDs = append(insertedIDs, &oid)
}
}
return insertedIDs, nil
}
Conclusion
I don't know if the behavior is logic or not but an id should be generated if my document as no ID :confused:
Note
A topic as been opened here: https://developer.mongodb.com/community/forums/t/inserting-a-document-in-go-with-no-id-set-results-into-a-weird-behavior/100359 (don't know if it's a bug or not so I thought I should post there too)
Thanks!
Max
If you insert a document containing the _id field, mongodb will use that as the document id, even if it is null. To have the _id auto-generated, you have to insert the document without the _id field. This should work:
type Doc struct {
ID *primitive.ObjectID `json:"_id" bson:"_id,omitempty"`
...
This will not marshal the _id field if it is nil, focing it to be auto-generated.
I use Go with PostgreSQL using github.com/lib/pq and able to successfully fetch the records when my structure is known.
Now my query is how to fetch records when my structure changes dynamically?
By rows.columns I am able to fetch the column names, but could you help me with fetching the values of these columns for all the rows. I referred this link answered by #Luke, still, here the person has a structure defined.
Is it possible to retrieve a column value by name using GoLang database/sql
type Person struct {
Id int
Name string
}
Meanwhile I do not have a structure that is fixed, so how will I iterate through all the columns that too again for all rows. My approach would be a pointer to loop through all columns at first, then another one for going to next row.
Still not able to code this, Could you please help me with this, like how to proceed and get the values.
Since you don't know the structure up front you can return the rows as a two dimensional slice of empty interfaces. However for the row scan to work you'll need to pre-allocate the values to the appropriate type and to do this you can use the ColumnTypes method and the reflect package. Keep in mind that not every driver provides access to the columns' types so make sure the one you use does.
rows, err := db.Query("select * from foobar")
if err != nil {
return err
}
defer rows.Close()
// get column type info
columnTypes, err := rows.ColumnTypes()
if err != nil {
return err
}
// used for allocation & dereferencing
rowValues := make([]reflect.Value, len(columnTypes))
for i := 0; i < len(columnTypes); i++ {
// allocate reflect.Value representing a **T value
rowValues[i] = reflect.New(reflect.PtrTo(columnTypes[i].ScanType()))
}
resultList := [][]interface{}{}
for rows.Next() {
// initially will hold pointers for Scan, after scanning the
// pointers will be dereferenced so that the slice holds actual values
rowResult := make([]interface{}, len(columnTypes))
for i := 0; i < len(columnTypes); i++ {
// get the **T value from the reflect.Value
rowResult[i] = rowValues[i].Interface()
}
// scan each column value into the corresponding **T value
if err := rows.Scan(rowResult...); err != nil {
return err
}
// dereference pointers
for i := 0; i < len(rowValues); i++ {
// first pointer deref to get reflect.Value representing a *T value,
// if rv.IsNil it means column value was NULL
if rv := rowValues[i].Elem(); rv.IsNil() {
rowResult[i] = nil
} else {
// second deref to get reflect.Value representing the T value
// and call Interface to get T value from the reflect.Value
rowResult[i] = rv.Elem().Interface()
}
}
resultList = append(resultList, rowResult)
}
if err := rows.Err(); err != nil {
return err
}
fmt.Println(resultList)
This function prints the result of a query without knowing anything about the column types and count. It is a variant of the previous answer without using the reflect package.
func printQueryResult(db *sql.DB, query string) error {
rows, err := db.Query(query)
if err != nil {
return fmt.Errorf("canot run query %s: %w", query, err)
}
defer rows.Close()
cols, _ := rows.Columns()
row := make([]interface{}, len(cols))
rowPtr := make([]interface{}, len(cols))
for i := range row {
rowPtr[i] = &row[i]
}
fmt.Println(cols)
for rows.Next() {
err = rows.Scan(rowPtr...)
if err != nil {
fmt.Println("cannot scan row:", err)
}
fmt.Println(row...)
}
return rows.Err()
}
The trick is that rows.Scan can scan values into *interface{} but you have to wrap it in interface{} to be able to pass it to Scan using ....
This is function that tries to update the code matching by its value
the res collection as having the code of Marque it will be compared with doc.Marque if it is the case it will be replaced by the value of the marque.
This code is working perfectly in mongoDB CLI, but as i am working with GO.
I tried to convert it into mgo as you may see below but it is not working , i did not find the foreach function in mgo , is there something to be replaced with in this case ? thanks for the help
db.res.find().forEach(function(doc){
var v = db.brands.findOne({code: doc.Marque});
if(v){
db.res.update({"Marque": doc.Marque},
{$set: {"Marque":v.value}}, {multi: true});
}
});
Here is what i tried :
result:=Results{}
pipe:=res.find(bson.M{}).Iter()
for pipe.Next(&result) {
brands:=brands.findOne({code: doc.Marque});
if(v){
pipe.update({"Marque": doc.Marque},
{$set: {"Marque": v.value}}, {multi: true});
}
}
Visit the mgo Godoc may help you understand how it works.
Second, exported types/functions in Golang are begin with a capital letter. So res.find, brands.findOne, ... should be res.Find, brands.FineOne respectively, if such functions exist.
// let's say you have a type like this
type myResult struct {
ID bson.ObjectId `bson:"_id"`
Marque string `bson:"Marque"`
// other fields...
}
// and another type like this
type myCode struct {
Code string `bson:"code"`
// other fields...
}
res := db.C("res")
brands := db.C("brands")
result := myResult{}
// iterate all documents
iter := res.Find(nil).Iter()
for iter.Next(&result) {
var v myCode
err := brands.Find(bson.M{"code": result.Marque}).One(&v)
if err != nil {
// maybe not found or other reason,
// it is recommend to have additional check
continue
}
query := bson.M{"_id": result.ID}
update := bson.M{"Marque": v.value}
if err = res.Update(query, update); err != nil {
// handle error
}
}
if err := iter.Close(); err != nil {
fmt.Println(err)
}
I'm trying to implement a MongoDB update for a Go struct. Stripped down to essentials, it looks something like this:
type MyStruct struct {
Id bson.ObjectId `bson:"_id"`
Fruit string `bson:"fruit"`
}
func TestUpdate(t *testing.T) {
obj1 := MyStruct{Id: bson.NewObjectId(),Fruit: "apple"}
var obj2 MyStruct
session, _ := mgo.Dial("whatever")
col := session.DB("test").C("collection")
col.Insert(&obj1)
obj1.Fruit = "cherry"
if err := col.Update(obj1.Id, bson.M{"$set": &obj1}); err != nil {
t.Errorf(err.Error())
}
if err := col.Find(bson.M{"Id": obj1.Id}).One(&obj2); err != nil {
t.Errorf(err.Error())
}
if obj1.Fruit != obj2.Fruit {
t.Errorf("Expected %s, got %s", obj1.Fruit, obj2.Fruit)
}
}
This generates the error message, indicating that the value wasn't updated. What am I missing?
I understand that just updating one field is possible, but given that this is in a data layer, above which the code doesn't have any knowledge of MongoDB, that would be challenging to implement in a general way. I.e. I really need to make any updates to the Go object, and then update the copy of the object in the backing store. I suppose that I could retrieve the object and do a "diff" manually, constructing a "$set" document, but that doesn't seem like adding a retrieval every time I do an update would be very efficient.
Edit: Trying a map with "_id" deleted
I've tried amending the code to the following:
package testmgo
import (
mgo "gopkg.in/mgo.v2"
"gopkg.in/mgo.v2/bson"
"github.com/fatih/structs"
"testing"
)
type MyStruct struct {
Id bson.ObjectId `bson:"_id"`
Fruit string `bson:"fruit"`
}
func TestUpdate(t *testing.T) {
obj1 := MyStruct{Id: bson.NewObjectId(),Fruit: "apple"}
var obj2 MyStruct
session, _ := mgo.Dial("localhost")
col := session.DB("test").C("collection")
col.Insert(&obj1)
obj1.Fruit = "cherry"
omap := structs.Map(&obj1)
delete(omap, "_id")
if err := col.UpdateId(obj1.Id, bson.M{"$set": bson.M(omap)}); err != nil {
t.Errorf(err.Error())
}
if err := col.Find(bson.M{"Id": obj1.Id}).One(&obj2); err != nil {
t.Errorf(err.Error())
}
if obj1.Fruit != obj2.Fruit {
t.Errorf("Expected %s, got %s", obj1.Fruit, obj2.Fruit)
}
}
and am still receiving the same results (Expected cherry, got apple). Note that the call to UpdateId() is not returning an error.
The problem was that I was using the wrong field as the key. I had mapped "Id" to "_id", but was then asking MongoDB to find a record using the Go attribute name rather than the name. This works correctly:
package testmgo
import (
mgo "gopkg.in/mgo.v2"
"gopkg.in/mgo.v2/bson"
"testing"
)
type MyStruct struct {
Id bson.ObjectId `bson:"_id"`
Fruit string `bson:"fruit"`
}
func TestUpdate(t *testing.T) {
obj1 := MyStruct{Id: bson.NewObjectId(), Fruit: "apple"}
var obj2 MyStruct
session, _ := mgo.Dial("localhost")
col := session.DB("test").C("collection")
col.Insert(&obj1)
obj1.Fruit = "cherry"
if err := col.UpdateId(obj1.Id, bson.M{"$set": &obj1}); err != nil {
t.Errorf(err.Error())
}
if err := col.Find(bson.M{"_id": obj1.Id}).One(&obj2); err != nil {
t.Errorf(err.Error())
}
if obj1.Fruit != obj2.Fruit {
t.Errorf("Expected %s, got %s", obj1.Fruit, obj2.Fruit)
}
}