I need to build a query using comparison operators, equivalent of db.inventory.find( { qty: { $gt: 20 } using the official driver. Any idea how to do that?
Connecting to a server is something like:
client, err := mongo.NewClient("mongodb://foo:bar#localhost:27017")
if err != nil { log.Fatal(err) }
err = client.Connect(context.TODO())
if err != nil { log.Fatal(err) }
Then obtain the inventory mongo.Collection like:
coll := client.Database("baz").Collection("inventory")
Then you can execute your query using Collection.Find() like:
ctx := context.Background()
cursor, err := coll.Find(ctx,
bson.NewDocument(
bson.EC.SubDocumentFromElements("qty",
bson.EC.Int32("$gt", 20),
),
),
)
defer cursor.Close(ctx) // Make sure you close the cursor!
Reading the results using the mongo.Cursor:
doc := bson.NewDocument()
for cursor.Next(ctx) {
doc.Reset()
if err := cursor.Decode(doc); err != nil {
// Handle error
log.Printf("cursor.Decode failed: %v", err)
return
}
// Do something with doc:
log.Printf("Result: %v", doc)
}
if err := cursor.Err(); err != nil {
log.Printf("cursor.Err: %v", err)
}
Note: I used a single bson.Document value to read all documents, and used its Document.Reset() in the beginning of each iteration to clear it and "prepare" it to read a new document into it. If you want to store the documents (e.g. in a slice), then you can obviously not do this. In that case just create a new doc in each iteration like doc := bson.NewDocument().
Related
I'm trying to make a function that watches the database for a certain document with a certain id to update but it does not work. It just stays alive while updating the document while the function should return. I've tried multiple things and the rest of the code works fine. When i remove the id part and listen for all document updates in that collection the function does as it should
func iterateChangeStream(routineCtx context.Context,stream *mongo.ChangeStream, chn chan string) {
defer stream.Close(routineCtx)
for stream.Next(routineCtx) {
var data bson.M
if err := stream.Decode(&data); err != nil {
fmt.Println(err)
}
chn <- "updated"
err := stream.Close(routineCtx)
if err != nil {
return
}
return
}
return
}
func (s Storage) ListenForScannerUpdateById(id primitive.ObjectID) {
ctx, cancel := context.WithTimeout(context.Background(), 5*time.Minute)
defer cancel()
chn := make(chan string)
coll := s.db.Collection("scanners")
scan, err := s.GetScannerById(id)
fmt.Println(scan)
matchPipeline := bson.D{
{
"$match", bson.D{
{"operationType", "update"},
{"fullDocument._id", bson.D{
{"$eq", id},
}},
},
},
}
scannerStream, err := coll.Watch(ctx, mongo.Pipeline{matchPipeline})
if err != nil {
err := scannerStream.Close(ctx)
if err != nil {
panic( err)
}
fmt.Printf("err: %v", err)
}
routineCtx, _ := context.WithCancel(context.Background())
go iterateChangeStream(routineCtx, scannerStream, chn)
msg, _ := <- chn
defer close(chn)
fmt.Println(msg)
return
}
Ok, so after reading the documentation for a seccond time i found this:
For update operations, this field only appears if you configured the change stream with fullDocument set to updateLookup. This field then represents the most current majority-committed version of the document modified by the update operation. This document may differ from the changes described in updateDescription if other majority-committed operations modified the document between the original update operation and the full document lookup.
so after setting the fullDocument option to updateLookup like this it works perfect:
scannerStream, err := coll.Watch(ctx, mongo.Pipeline{matchPipeline}, options.ChangeStream().SetFullDocument(options.UpdateLookup))
I have a collections of 2,000,000 records
> db.events.count(); │
2000000
and I use golang mongodb client to connect to the database
package main
import (
"go.mongodb.org/mongo-driver/bson"
"go.mongodb.org/mongo-driver/mongo"
"go.mongodb.org/mongo-driver/mongo/options"
)
func main() {
ctx, cancel := context.WithTimeout(context.Background(), 10*time.Second)
defer cancel()
client, err := mongo.Connect(ctx, options.Client().ApplyURI("mongodb://localhost:27888").SetAuth(options.Credential{
Username: "mongoadmin",
Password: "secret",
}))
if err != nil {
panic(err)
}
defer func() {
if err = client.Disconnect(ctx); err != nil {
panic(err)
}
}()
collection := client.Database("test").Collection("events")
var bs int32 = 10000
var b = true
cur, err := collection.Find(context.Background(), bson.D{}, &options.FindOptions{
BatchSize: &bs, NoCursorTimeout: &b})
if err != nil {
log.Fatal(err)
}
defer cur.Close(ctx)
s, n := runningtime("retrive db from mongo and publish to kafka")
count := 0
for cur.Next(ctx) {
var result bson.M
err := cur.Decode(&result)
if err != nil {
log.Fatal(err)
}
bytes, err := json.Marshal(result)
if err != nil {
log.Fatal(err)
}
count++
msg := &sarama.ProducerMessage{
Topic: "hello",
// Key: sarama.StringEncoder("aKey"),
Value: sarama.ByteEncoder(bytes),
}
asyncProducer.Input() <- msg
}
But the the program only retrives only about 600,000 records instead of 2,000,000 every times I ran the program.
$ go run main.go
done
count = 605426
nErrors = 0
2020/09/18 11:23:43 End: retrive db from mongo and publish to kafka took 10.080603336s
I don't know why? I want to retrives all 2,000,000 records. Thanks for any help.
Your loop fetching the results may end early because you are using the same ctx context for iterating over the results which has a 10 seconds timeout.
Which means if retrieving and processing the 2 million records (including connecting) takes more than 10 seconds, the context will be cancelled and thus the cursor will also report an error.
Note that setting FindOptions.NoCursorTimeout to true is only to prevent cursor timeout for inactivity, it does not override the used context's timeout.
Use another context for executing the query and iterating over the results, one that does not have a timeout, e.g. context.Background().
Also note that for constructing the options for find, use the helper methods, so it may look as simple and as elegant as this:
options.Find().SetBatchSize(10000).SetNoCursorTimeout(true)
So the working code:
ctx2 := context.Background()
cur, err := collection.Find(ctx2, bson.D{},
options.Find().SetBatchSize(10000).SetNoCursorTimeout(true))
// ...
for cur.Next(ctx2) {
// ...
}
// Also check error after the loop:
if err := cur.Err(); err != nil {
log.Printf("Iterating over results failed: %v", err)
}
Trying to add some json data from an API to a database but get this error when trying
cannot transform type bson.Raw to a BSON Document: length read exceeds number of bytes available. length=259839 bytes=1919951
I know the json is well below mongodb limit of 16mb, ive even tried importing just some small data from this api but get the same error. I was able to import just a test struct to see it was working but my api data doesnt seem to be going through. Is there some type of conversion i need to do with my api data? Here is my golang code
func main(i int) {
url := "http://api.open-notify.org/astros.json"
resp, err := http.Get(url)
if err != nil {
log.Fatalln(err)
}
body, err := ioutil.ReadAll(resp.Body)
if err != nil {
log.Fatalln(err)
}
// _ = body
log.Println(string(body))
clientOptions := options.Client().ApplyURI("mongodb+srv://username:password#cluster0-slmxe.mongodb.net/dbtest?retryWrites=true&w=majority")
// Connect to MongoDB
client, err := mongo.Connect(context.TODO(), clientOptions)
if err != nil {
log.Fatal(err)
}
err = client.Ping(context.TODO(), nil)
if err != nil {
log.Fatal(err)
}
fmt.Println("Connected to database")
collection := client.Database("dbtest").Collection("test")
insertResult, err := collection.InsertOne(context.TODO(), body)
if err != nil {
log.Fatal(err)
}
fmt.Println("Inserted", insertResult.InsertedID)
}
You need to wrap your json with bson.D to able to send the data to Mongodb. This is to build representation for native go types. Example below:
// insert the document {name: "Alice"}
res, err := coll.InsertOne(context.TODO(), bson.D{{"name", "Alice"}})
if err != nil {
log.Fatal(err)
}
Please refer to following documentation:
https://pkg.go.dev/go.mongodb.org/mongo-driver#v1.3.4/mongo?tab=doc#Collection.InsertOne
I checked out the answer here but this uses the old and unmaintained mgo. How can I find all documents in a collection using the mongo-go-driver?
I tried passing a nil filter, but this does not return any documents and instead returns nil. I also checked the documentation but did not see any mention of returning all documents. Here is what I've tried with aforementioned result.
client, err := mongo.Connect(context.TODO(), "mongodb://localhost:27017")
coll := client.Database("test").Collection("albums")
if err != nil { fmt.Println(err) }
// we can assume we're connected...right?
fmt.Println("connected to mongodb")
var results []*Album
findOptions := options.Find()
cursor, err := coll.Find(context.TODO(), nil, findOptions)
if err != nil {
fmt.Println(err) // prints 'document is nil'
}
Also, I'm about confused about why I need to specify findOptions when I've called the Find() function on the collection (or do I not need to specify?).
Here is what I came up with using the official MongoDB driver for golang. I am using godotenv (https://github.com/joho/godotenv) to pass the database parameters.
//Find multiple documents
func FindRecords() {
err := godotenv.Load()
if err != nil {
fmt.Println(err)
}
//Get database settings from env file
//dbUser := os.Getenv("db_username")
//dbPass := os.Getenv("db_pass")
dbName := os.Getenv("db_name")
docCollection := "retailMembers"
dbHost := os.Getenv("db_host")
dbPort := os.Getenv("db_port")
dbEngine := os.Getenv("db_type")
//set client options
clientOptions := options.Client().ApplyURI("mongodb://" + dbHost + ":" + dbPort)
//connect to MongoDB
client, err := mongo.Connect(context.TODO(), clientOptions)
if err != nil {
log.Fatal(err)
}
//check the connection
err = client.Ping(context.TODO(), nil)
if err != nil {
log.Fatal(err)
}
fmt.Println("Connected to " + dbEngine)
db := client.Database(dbName).Collection(docCollection)
//find records
//pass these options to the Find method
findOptions := options.Find()
//Set the limit of the number of record to find
findOptions.SetLimit(5)
//Define an array in which you can store the decoded documents
var results []Member
//Passing the bson.D{{}} as the filter matches documents in the collection
cur, err := db.Find(context.TODO(), bson.D{{}}, findOptions)
if err !=nil {
log.Fatal(err)
}
//Finding multiple documents returns a cursor
//Iterate through the cursor allows us to decode documents one at a time
for cur.Next(context.TODO()) {
//Create a value into which the single document can be decoded
var elem Member
err := cur.Decode(&elem)
if err != nil {
log.Fatal(err)
}
results =append(results, elem)
}
if err := cur.Err(); err != nil {
log.Fatal(err)
}
//Close the cursor once finished
cur.Close(context.TODO())
fmt.Printf("Found multiple documents: %+v\n", results)
}
Try passing an empty bson.D instead of nil:
cursor, err := coll.Find(context.TODO(), bson.D{})
Also, FindOptions is optional.
Disclaimer: I've never used the official driver, but there are a few examples at https://godoc.org/go.mongodb.org/mongo-driver/mongo
Seems like their tutorial is outdated :/
We want our Go application to listen to the data changes on a collection. So, googling in search for a solution, we came across MongoDB's Change Streams. That link also exhibits some implementation snippets for a bunch of languages such as Python, Java, Nodejs etc. Yet, there is no piece of code for Go.
We are using Mgo as a driver but could not find explicit statements on change streams.
Does anyone have any idea on how to watch on Change Streams using that Mgo or any other Mongo driver for Go?
The popular mgo driver (github.com/go-mgo/mgo) developed by Gustavo Niemeyer has gone dark (unmaintained). And it has no support for change streams.
The community supported fork github.com/globalsign/mgo is in much better shape, and has already added support for change streams (see details here).
To watch changes of a collection, simply use the Collection.Watch() method which returns you a value of mgo.ChangeStream. Here's a simple example using it:
coll := ... // Obtain collection
pipeline := []bson.M{}
changeStream := coll.Watch(pipeline, mgo.ChangeStreamOptions{})
var changeDoc bson.M
for changeStream.Next(&changeDoc) {
fmt.Printf("Change: %v\n", changeDoc)
}
if err := changeStream.Close(); err != nil {
return err
}
Also note that there is an official MongoDB Go driver under development, it was announced here: Considering the Community Effects of Introducing an Official MongoDB Go Driver
It is currently in alpha (!!) phase, so take this into consideration. It is available here: github.com/mongodb/mongo-go-driver. It also already has support for change streams, similarly via the Collection.Watch() method (this is a different mongo.Collection type, it has nothing to do with mgo.Collection). It returns a mongo.Cursor which you may use like this:
var coll mongo.Collection = ... // Obtain collection
ctx := context.Background()
var pipeline interface{} // set up pipeline
cur, err := coll.Watch(ctx, pipeline)
if err != nil {
// Handle err
return
}
defer cur.Close(ctx)
for cur.Next(ctx) {
elem := bson.NewDocument()
if err := cur.Decode(elem); err != nil {
log.Fatal(err)
}
// do something with elem....
}
if err := cur.Err(); err != nil {
log.Fatal(err)
}
This example uses the The MongoDB supported driver for Go with stream pipeline (filtering only documents having field1=1 and field2=false):
ctx := context.TODO()
clientOptions := options.Client().ApplyURI(mongoURI)
client, err := mongo.Connect(ctx, clientOptions)
if err != nil {
log.Fatal(err)
}
err = client.Ping(ctx, nil)
if err != nil {
log.Fatal(err)
}
fmt.Println("Connected!")
collection := client.Database("test").Collection("test")
pipeline := mongo.Pipeline{bson.D{
{"$match",
bson.D{
{"fullDocument.field1", 1},
{"fullDocument.field2", false},
},
},
}}
streamOptions := options.ChangeStream().SetFullDocument(options.UpdateLookup)
stream, err := collection.Watch(ctx, pipeline, streamOptions)
if err != nil {
log.Fatal(err)
}
log.Print("waiting for changes")
var changeDoc map[string]interface{}
for stream.Next(ctx) {
if e := stream.Decode(&changeDoc); e != nil {
log.Printf("error decoding: %s", e)
}
log.Printf("change: %+v", changeDoc)
}