My application is using pgx to running database queries in a goroutines. However, I am getting connection busy errors. Is there a way to have a goroutine
func writeDb(dbconn *pgx.Conn) {
sqlWritePost := `QUERY_HERE`
_, err := dbconn.Exec(context.Background(), sqlWritePost, v.Url, v.Content, v.StrippedContent, v.Posthash)
if err != nil {
fmt.Println(err)
os.Exit(1)
}
}
func main() {
var dbconn *pgx.Conn
dbconn, err := pgx.Connect(context.Background(), os.Getenv("database_string"))
if err != nil {
fmt.Println(err)
os.Exit(1)
}
...
go writeDb(dbconn)
...
}
I am receiving errors conn busy. Is there a way to structure my code to avoid this issue?
Thanks!
Related
Context
Currently I have a REST API that manages customer's data in a db. I'm using the following stack:
Go 1.13
github.com/jinzhu/gorm v1.9.1
Postgres 11
I have the following connection settings.
// NewConnection ...
func NewConnection() (*gorm.DB, error) {
config := getConfig()
connStr := "host=xx.xx.xx port=5432 user=chavista-hatter dbname=my-db password=abc sslmode=verify-ca sslrootcert=/path/to/rcert sslcert=/path/to/cert sslkey=/path/to/key connect_timeout=0"
db, err := gorm.Open("postgres", conn)
if err != nil {
return nil, err
}
db.DB().SetMaxOpenConns(25)
db.DB().SetMaxIdleConns(25)
db.DB().SetConnMaxLifetime(5 * time.Minute)
db.SingularTable(true)
if config.LogQueries {
db = db.Debug()
}
return db, nil
}
I get a connection in the main class and inject that connection into a repository class that executes the queries through Gorm (ORM)
Main class
db, err := database.NewConnection()
if err != nil {
panic(fmt.Sprintf("failed to connect database --> %v", err))
}
fmt.Println("database connection established successfully")
defer db.Close()
customerRepo := customer.NewRepository(db)
Repository class
type repository struct {
db *gorm.DB
}
//NewRepository
func NewRepository(db *gorm.DB) Repository {
return &repository{
db: db,
}
}
func (r *repository) Register(customer *models.Customer) (string, error) {
err := r.db.Create(&customer).Error
if err != nil {
return "", err
}
return customer.key, nil
}
Problem
I sending over 500k request (INSERTS) to my db which have 512 connections available and after a few minutes the following error starts to come up repeatedly in postgres log:
unexpected EOF on client connection with an open transaction
could not receive data from client: Connection reset by peer
Any help?
How are you using GetConnection in your code? It's creating a new connection pool every time it's called - ideally you'd only want to call it once, and pass that single connection around wherever it's used.
I would try changing it to this:
var db *gorm.DB
func NewConnection() (*gorm.DB, error) {
if db != nil {
return db, nil
}
config := getConfig()
connStr := "host=xx.xx.xx port=5432 user=chavista-hatter dbname=my-db password=abc sslmode=verify-ca sslrootcert=/path/to/rcert sslcert=/path/to/cert sslkey=/path/to/key connect_timeout=0"
var err error
db, err = gorm.Open("postgres", conn)
if err != nil {
return nil, err
}
db.DB().SetMaxOpenConns(25)
db.DB().SetMaxIdleConns(25)
db.DB().SetConnMaxLifetime(5 * time.Minute)
db.SingularTable(true)
if config.LogQueries {
db = db.Debug()
}
return db, nil
}
and see if it solves the issue.
I'm newbie for MongoDB and I’ve tried to use collection.Watch() from "go.mongodb.org/mongo-driver/mongo" lib. and code from https://github.com/minhthuy30197/change_stream/blob/master/main.go. Then I build and run it stop immediately.
I’ve tried to run again and again and it also stop running. I’ve switch using between go run main.go and ./testStreams and it still stop running
So this is my edited code.
clientOptions := options.Client().
ApplyURI("mongodb://localhost:27017/test")
client, err := mongo.Connect(context.TODO(), clientOptions)
if err != nil {
log.Fatalf("Failed to create the new client: %v", err)
}
ctx := context.Background()
if err := client.Connect(ctx); err != nil {
log.Printf("Failed to open client connection: %v", err)
}
defer client.Disconnect(ctx)
coll := client.Database("test").Collection("streams")
var pipeline interface{}
for {
cur, err := coll.Watch(ctx, pipeline)
if err != nil {
log.Fatalf("Watch error: %v", err)
}
defer cur.Close(ctx)
log.Println(cur)
for cur.Next(ctx) {
elem := CSElem{}
if err := cur.Decode(elem); err != nil {
log.Fatalf("Decode error: %v", err)
}
log.Println(elem)
}
if err := cur.Err(); err != nil {
log.Fatalf("Error detected: %v", err)
}
}
When I edit then error appear
2019/08/07 13:46:39 Failed to open client connection: topology is
connected or connecting exit status 1
How should I fix??
As mentioned on the comment, utilising mongo-go-driver v1+ a new client instance need to be created first before making a connection. For example:
clientOptions := options.Client().ApplyURI("mongodb://localhost:27017/test")
client, err := mongo.NewClient(clientOptions)
if err != nil {log.Fatal(err)}
// Timed out after 10 seconds of trying to connect
ctx, cancel := context.WithTimeout(context.Background(), 10*time.Second)
defer cancel()
err = client.Connect(ctx)
if err != nil { log.Fatal(err)}
I have created a mongodb replica set. I am able to run transactions from the mongo shell. But when I try to do it using mongo-go-driver I always get this error (IllegalOperation) Transaction numbers are only allowed on a replica set member or mongos. I am not sure where I am going wrong. I am using this as a reference https://github.com/simagix/mongo-go-examples/blob/master/examples/transaction_test.go.
I create the client like this
client, err := mongo.NewClient(options.Client().ApplyURI("mongodb://localhost:27017,localhost:27018,localhost:27019?replicaSet=rs"))
I can connect to the individual mongodb instances, just not the replica set.
This is the transaction that I am trying to run
var session mongo.Session
coll := db.Collection("collectionname")
if session, err = client.StartSession(); err != nil {
return "", fmt.Errorf("Could not start session: %q", err)
}
if err = session.StartTransaction(); err != nil {
return "", fmt.Errorf("Could not start Transaction: %q", err)
}
if err = mongo.WithSession(ctx, session, func(sc md.SessionContext) error {
newVal, err = coll.InsertOne(sc, val) // some val that I have
if err != nil {
sc.AbortTransaction(sc)
return fmt.Errorf("Error during New address creation, aborting: %q", err)
}
if err = sc.CommitTransaction(sc); err != nil {
return fmt.Errorf("Error While commiting New address Transaction: %q", err)
}
return nil
}); err != nil {
return "", err
}
session.EndSession(ctx)
Is there something I am missing. Is there some other example maybe that I can reference. Thanks for the help/suggestions.
This is an issue with your connection code - not your transaction implementation most likely. Try using the more modern connection string for connecting to a replica set:
connectionString := "mongodb+srv://USERNAME:PASSWORD#mongoatlas.1mxpg.mongodb.net/?retryWrites=true&w=majority"
var err error
Client, err = mongo.NewClient(options.Client().ApplyURI(ConnectionString))
if err != nil {
log.Fatal(err)
}
I am trying to connect to two remote mongodb servers using ssh port-forwarding in golang which are used by my graphql server for querying. The intermediary host for the two tunnels is same. So let's say the intermediary host is 123.45.678.678 and the two remote mongodb servers are 1.23.45.67 and 1.23.45.78, I create the tunnels like this,
conn, err := ssh.Dial("tcp", 123.45.678.678, config)
if err != nil {
panic(err)
}
remote1, err := conn.Dial("tcp", "1.23.45.67:27017")
if err != nil {
panic(err)
}
remote2, err := conn.Dial("tcp", "1.23.45.78:27017")
if err != nil {
panic(err)
}
local1, err := net.Listen("tcp", "localhost:27018")
if err != nil {
panic(err)
}
local2, err := net.Listen("tcp", "localhost:27019")
if err != nil {
panic(err)
}
Now i forward traffic from local1 to remote1 and local2 to remote2 like this
go func() {
for {
l, err := local1.Accept()
if err != nil {
panic(err)
}
go func() {
_, err := io.Copy(l, remote1)
if err != nil {
panic(err)
}
}()
go func() {
_, err := io.Copy(remote1, l)
if err != nil {
panic(err)
}
}()
}
}()
go func() {
for {
l, err := local2.Accept()
if err != nil {
panic(err)
}
go func() {
_, err := io.Copy(l, remote2)
if err != nil {
panic(err)
}
}()
go func() {
_, err := io.Copy(remote2, l)
if err != nil {
panic(err)
}
}()
}
}()
And I create two mongo sessions using mgo.Dial and export these two sessions to the graphql whenever this function is called. For some queries ( not all queries, only some complex queries ) which need both the sessions, i see the write: broken pipe error
panic: readfrom tcp 127.0.0.1:27019->127.0.0.1:53128: write tcp 127.0.0.1:27019->127.0.0.1:53128: write: broken pipe
When I debugged this, i figured out that this error occurs when the io.copy happens between l and remote2 in the code snippet above which i guess is due to the disconnection of remote2 tunnel.
The tcpdump showed that the intermediary host is sending the finish flag to the remote2 server after sometime which inturn is leading to the termination of the connection. I am wondering how I can resolve this.
My GO version is 1.1.1
the sever recieved messages after connection close, but NoDelay was setted.
Is there something wrong
addr, _ := net.ResolveTCPAddr("tcp", "localhost:5432")
conn, err := net.DialTCP("tcp", nil, addr)
defer conn.Close()
if err != nil {
fmt.Println("connect fail")
return
}
err = conn.SetNoDelay(true)
if err != nil {
fmt.Println(err.Error())
}
for {
var message string
_, err := fmt.Scanln(&message)
if err != nil && err.Error() != "unexpected newline" {
fmt.Println("input finished", err)
break
}
if message == "" {
fmt.Println("no input, end")
break
}
// message = fmt.Sprintf("%s\n",message)
//fmt.Fprintf(conn, message) // send immediately but following message won't send any more
conn.Write([]byte(message)) // won't send until connection close
}
There doesn't seem to be anything vitally wrong with your code so I'm guessing the error is on the server end.
If you create a local TCP server on port 5432 you can test this.
Try running the below server code and then test your client code against it. It just echos all received data to stdout.
package main
import (
"io"
"log"
"net"
"os"
)
func main() {
l, err := net.Listen("tcp", "localhost:5432")
if err != nil {
log.Fatal(err)
}
defer l.Close()
for {
conn, err := l.Accept()
if err != nil {
log.Fatal(err)
}
go func(c net.Conn) {
defer c.Close()
io.Copy(os.Stdout, c)
}(conn)
}
}
You should see each line sent to the client printed (without the newline) as soon as you hit enter.
the problem is on the server end.
func handleConnection(conn net.Conn) {
// I didn't put it in for loop
message, err := bufio.NewReader(conn).ReadString('\n')
}