Why I get an error "client disconnected" when trying to get documents from mongo collection in go? - mongodb

I have mongo capped collection and a simple API, written on Go. I built and run it. When I try to sent Get request or simply go localhost:8000/logger in browser - my process closes. Debug shows this happens, while executing "find" in collection. It produces error "client is disconnected". Collection has 1 document, and debug shows it is connected with my helper.
Go version 1.13
My code:
func main() {
r := mux.NewRouter()
r.HandleFunc("/logger", getDocs).Methods("GET")
r.HandleFunc("/logger", createDoc).Methods("POST")
log.Fatal(http.ListenAndServe("localhost:8000", r))
}
func getDocs(w http.ResponseWriter, r *http.Request) {
w.Header().Set("Content-Type", "application/json")
var docs []models.Logger
//Connection mongoDB with helper class
collection := helper.ConnectDB()
cur, err := collection.Find(context.TODO(), bson.M{})
if err != nil {
helper.GetError(err, w)
return
}
defer cur.Close(context.TODO())
for cur.Next(context.TODO()) {
var doc models.Logger
err := cur.Decode(&doc)
if err != nil {
log.Fatal(err)
}
docs = append(docs, doc)
}
if err := cur.Err(); err != nil {
log.Fatal(err)
}
json.NewEncoder(w).Encode(docs)
}
func ConnectDB() *mongo.Collection {
client, err := mongo.NewClient(options.Client().ApplyURI("mongodb://127.0.0.1:27017"))
if err != nil {
log.Fatal(err)
}
fmt.Println("Connected to MongoDB!")
logCollection := client.Database("local").Collection("loggerCollection")
return logCollection
}

According to the documentation, the call to mongo.NewClient doesn't ensure that you can connect the Mongo server. You should first call mongo.Client.Ping() to verify if you can connect to the database or not.
client, err := mongo.NewClient(options.Client().ApplyURI("mongodb://127.0.0.1:27017"))
if err != nil {
log.Fatal(err)
}
if err := client.Ping(context.TODO(), readpref.Primary()); err != nil {
// Can't connect to Mongo server
log.Fatal(err)
}
There could be several reasons behind failing to connect, the most obvious one is incorrect setup of ports. Is your mongodb server up and listening on port 27017? Is there any change you're running mongodb with Docker and it's not forwarding to the correct port?

I faced similar issue , read #Jay answer it definitely helped , as I checked my MongoDB was running using "MongoDB Compass" , then I changed the location of my insert statement , previously I was calling before the call of "context.WithTimeout". Below is working code.
package main
import (
"context"
"log"
"time"
"go.mongodb.org/mongo-driver/mongo"
"go.mongodb.org/mongo-driver/mongo/options"
)
type Book struct {
Name string `json:"name,omitempty"`
PublisherID string `json:"publisherid,omitempty"`
Cost string `json:"cost,omitempty"`
StartTime string `json:"starttime,omitempty"`
EndTime string `json:"endtime,omitempty"`
}
func main() {
client, err := mongo.NewClient(options.Client().ApplyURI("mongodb://localhost:27017"))
if err != nil {
log.Fatal(err)
}
ctx, _ := context.WithTimeout(context.Background(), 10*time.Second)
err = client.Connect(ctx)
if err != nil {
log.Fatal(err)
}
defer client.Disconnect(ctx)
testCollection := client.Database("BooksCollection").Collection("BooksRead")
inserRes, err := testCollection.InsertOne(context.TODO(), Book{Name: "Harry Potter", PublisherID: "IBN123", Cost: "1232", StartTime: "2013-10-01T01:11:18.965Z", EndTime: "2013-10-01T01:11:18.965Z"})
log.Println("InsertResponse : ", inserRes)
log.Println("Error : ", err)
}
I can see document inserted in console as well as in "MongoDB Comapass."

In heiper function "ConnectDB" after "NewClient" I must use "client.Connect(context.TODO())"
before any other use of client

Related

How to obtain MongoDB version using Golang library?

I am using Go's MongodDB driver (https://pkg.go.dev/go.mongodb.org/mongo-driver#v1.8.0/mongo#section-documentation) and want to obtain the version of the mongoDB server deployed.
For instance, if it would been a MySQL database, I can do something like below:
db, err := sql.Open("mysql", DbUser+":"+DbPwd+"#tcp("+Host+")/"+DbName)
if err != nil {
log.Printf("Error while connecting to DB: %v", err)
}
defer db.Close()
var dbVersion string
if err := db.QueryRow("SELECT VERSION()").Scan(&dbVersion); err != nil {
dbVersion = "NA"
log.Printf("Couldnt obtain db version: %w", err)
}
fmt.Println("DB Version: ", dbVersion)
I went through the documentation but am not able to find a clue.
I also need to fetch other metadata like Size of a particular database etc.
Any help would be appreciated. Thanks!
The MongoDB version can be acquired by running a command, specifically the buildInfo command.
Using the shell, this is how you could do it:
db.runCommand({buildInfo: 1})
The result is a document whose version property holds the server version, e.g.:
{
"version" : "5.0.6",
...
}
To run commands using the official driver, use the Database.RunCommand() method.
For example:
// Connect to MongoDB and acquire a Database:
ctx := context.Background()
opts := options.Client().ApplyURI("mongodb://localhost")
client, err := mongo.Connect(ctx, opts)
if err != nil {
log.Fatalf("Failed to connect to db: %v", err)
}
defer client.Disconnect(ctx)
db := client.Database("your-db-name")
// And now run the buildInfo command:
buildInfoCmd := bson.D{bson.E{Key: "buildInfo", Value: 1}}
var buildInfoDoc bson.M
if err := db.RunCommand(ctx, buildInfoCmd).Decode(&buildInfoDoc); err != nil {
log.Printf("Failed to run buildInfo command: %v", err)
return
}
log.Println("Database version:", buildInfoDoc["version"])
Based on #icza's answer, here is how to obtain other metadata of the Database:
We need to use dbStats command to obtain metadata.
host := "<your-host-name>:<pot-number>"
url := "mongodb://" + host
credential := options.Credential{
AuthSource: "authentication-database",
Username: "username",
Password: "password",
}
clientOpts := options.Client().ApplyURI(url).SetAuth(credential)
ctx := context.Background()
client, err := mongo.Connect(ctx, clientOpts)
if err != nil {
log.Fatal("Failed to connect to db : %w", err)
}
defer client.Disconnect(ctx)
if err := client.Ping(context.TODO(), readpref.Primary()); err != nil {
panic(err)
}
fmt.Println("Successfully connected and pinged.")
db := client.Database("your-database-name")
dbStatsCmd := bson.D{bson.E{Key: "dbStats", Value: 1}}
var dbStatsDoc bson.M
if err := db.RunCommand(ctx, dbStatsCmd).Decode(&dbStatsDoc); err != nil {
log.Printf("Failed to run dbStats command: %v", err)
return
}
log.Println("\nTotal Used Size in MB: ", dbStatsDoc["totalSize"].(float64) / 1048576 , " ,Total Free size in MB (part of total used size): ", dbStatsDoc["totalFreeStorageSize"].(float64)/1048576)

error from convert ObjectIDFromHex from mongodb

I have a problem finding an objectid through the query param called id.
I can see the id that arrives at the function until the moment of doing the query.
But when I try to use ObjectIDFromHex it returns 00000000000000000000000 and doesn't get the document from mongodb.
I'll leave a screenshot so you can see the full problem.
screenshot with IDE
The code is this.
func RetrieveUser(ID string) (models.User, error) {
ctx, cancel := context.WithTimeout(context.Background(), 15*time.Second)
defer cancel()
db := MongoCN.Database("mydatabase")
col := db.Collection("users")
var user models.User
objID, _ := primitive.ObjectIDFromHex(ID)
fmt.Println(ID)
fmt.Println(objID)
condition := bson.M{
"_id": objID,
}
err := col.FindOne(ctx, condition).Decode(&user)
user.Password = ""
if err != nil {
fmt.Println("User not found" + err.Error())
return user, err
}
return user, nil
}

Mongodb doesn't retrieve all documents in a collection with 2 million records using cursor

I have a collections of 2,000,000 records
> db.events.count(); │
2000000
and I use golang mongodb client to connect to the database
package main
import (
"go.mongodb.org/mongo-driver/bson"
"go.mongodb.org/mongo-driver/mongo"
"go.mongodb.org/mongo-driver/mongo/options"
)
func main() {
ctx, cancel := context.WithTimeout(context.Background(), 10*time.Second)
defer cancel()
client, err := mongo.Connect(ctx, options.Client().ApplyURI("mongodb://localhost:27888").SetAuth(options.Credential{
Username: "mongoadmin",
Password: "secret",
}))
if err != nil {
panic(err)
}
defer func() {
if err = client.Disconnect(ctx); err != nil {
panic(err)
}
}()
collection := client.Database("test").Collection("events")
var bs int32 = 10000
var b = true
cur, err := collection.Find(context.Background(), bson.D{}, &options.FindOptions{
BatchSize: &bs, NoCursorTimeout: &b})
if err != nil {
log.Fatal(err)
}
defer cur.Close(ctx)
s, n := runningtime("retrive db from mongo and publish to kafka")
count := 0
for cur.Next(ctx) {
var result bson.M
err := cur.Decode(&result)
if err != nil {
log.Fatal(err)
}
bytes, err := json.Marshal(result)
if err != nil {
log.Fatal(err)
}
count++
msg := &sarama.ProducerMessage{
Topic: "hello",
// Key: sarama.StringEncoder("aKey"),
Value: sarama.ByteEncoder(bytes),
}
asyncProducer.Input() <- msg
}
But the the program only retrives only about 600,000 records instead of 2,000,000 every times I ran the program.
$ go run main.go
done
count = 605426
nErrors = 0
2020/09/18 11:23:43 End: retrive db from mongo and publish to kafka took 10.080603336s
I don't know why? I want to retrives all 2,000,000 records. Thanks for any help.
Your loop fetching the results may end early because you are using the same ctx context for iterating over the results which has a 10 seconds timeout.
Which means if retrieving and processing the 2 million records (including connecting) takes more than 10 seconds, the context will be cancelled and thus the cursor will also report an error.
Note that setting FindOptions.NoCursorTimeout to true is only to prevent cursor timeout for inactivity, it does not override the used context's timeout.
Use another context for executing the query and iterating over the results, one that does not have a timeout, e.g. context.Background().
Also note that for constructing the options for find, use the helper methods, so it may look as simple and as elegant as this:
options.Find().SetBatchSize(10000).SetNoCursorTimeout(true)
So the working code:
ctx2 := context.Background()
cur, err := collection.Find(ctx2, bson.D{},
options.Find().SetBatchSize(10000).SetNoCursorTimeout(true))
// ...
for cur.Next(ctx2) {
// ...
}
// Also check error after the loop:
if err := cur.Err(); err != nil {
log.Printf("Iterating over results failed: %v", err)
}

mongodb getting 10_000 rows at a time

I'm trying to get 10000 document at a time in mongodb, but i got :
Information :
Driver https://github.com/mongodb/mongo-go-driver
opt.SetBatchSize(15_000)
opt.SetAllowPartialResults(false)
index on timestamp
Code :
package main
import (
"context"
"fmt"
"net/http"
"os"
"time"
"go.mongodb.org/mongo-driver/bson"
"go.mongodb.org/mongo-driver/mongo"
"go.mongodb.org/mongo-driver/mongo/options"
)
var database *mongo.Database
func main() {
ctx, _ := context.WithTimeout(context.Background(), 10*time.Second)
client, err := mongo.Connect(ctx, options.Client().ApplyURI("mongodb://20.20.20.43:27017"))
if err != nil {
panic(err)
}
database = client.Database("chat_data")
chatText := make([]chat, 0)
now := time.Now().Unix()
ctx, _ = context.WithTimeout(context.Background(), 30*time.Second)
// mongodb batch option
opt := options.Find()
opt.SetBatchSize(15_000)
opt.SetAllowPartialResults(false)
// mongodb filter
filter := bson.M{"timestamp": bson.M{"$gte": now - 108000}}
cur, err := database.Collection("chat").Find(ctx, filter, opt)
if err != nil {
// fmt.Fprint(w, err)
fmt.Println(err)
return
}
defer cur.Close(ctx)
for cur.Next(ctx) {
var result chat
err := cur.Decode(&result)
if err != nil {
fmt.Println(err)
continue
}
// do something with result....
// fmt.Println(result)
chatText = append(chatText, result)
}
if err := cur.Err(); err != nil {
// fmt.Fprint(w, cur.Err())
fmt.Println(err)
return
}
fmt.Println("done")
fmt.Println(len(chatText))
}
can i achieve this with mongodb & go driver ?, 30 second timeout are always reached
Edit 1
i try in python (with pymongo) it's only need 0m2.159s to query 36k doc with that filter
Try 7000, if it works try 12000, if it doesn't work try 4000, etc.
Make note of how long these requests take to figure out if the execution time is proportional to batch size.
You are querying on just the timestamp field. If you create an index on that collection with the timestamp field first, you should get faster results, and get a free sort in the process.

How to solve "command find requires authentication" using Golang and MongoDriver

I need help to fix the error. I'm using IBM mongo services.
go version go1.13.6 darwin/amd64
mongo driver version 1.2.1
The connection is working, I can read and write but sometimes it returns
: command find requires authentication and command insert requires authentication
MONGO_DB_URI=mongodb://username:password:port,host/dbname?authSource=admin&replicaSet=replset&connect=direct&alias=default
Connect:
func ConnectDatabase() *mongo.Client {
clientOptions := options.Client().ApplyURI(os.Getenv("MONGO_DB_URI"))
ctx, _ := context.WithTimeout(context.Background(), 30*time.Second)
var err error
client, err = mongo.Connect(ctx, clientOptions)
if err != nil {
log.Fatal(err)
}
ctx, _ = context.WithTimeout(context.Background(), 10*time.Second)
err = client.Ping(ctx, nil)
if err != nil {
log.Fatal(err)
return nil
}
fmt.Println("Connected to MongoDB!")
return client
}
Read:
func FindAll(collectionName string, query bson.M) (*mongo.Cursor, error) {
collection := client.Database("dbname").Collection(collectionName)
singleResult, err := collection.Find(context.TODO(), query)
return singleResult, err
}
Read:
ctx, _ := context.WithTimeout(context.Background(), 20*time.Second)
cur, err := mongo.GetCollection("collection_name").Find(ctx, createQuery())
if err != nil {
log.Println(err.Error())
}
I'm using the same database and the same configurations at our another Python Project. No exceptions.
There is a difference between connecting to a DB and performing operations on the DB.
Mongo lets you connect without authentication because you have to be able to connect to be able to authenticate.
var cred options.Credential
cred.AuthSource = YourAuthSource
cred.Username = YourUserName
cred.Password = YourPassword
// set client options
clientOptions := options.Client().ApplyURI(os.Getenv("MONGO_DB_URI")).SetAuth(cred)
//... the rest of your code
Hope this helps.