Golang ssh portforwarding : broken pipe error - mongodb

I am trying to connect to two remote mongodb servers using ssh port-forwarding in golang which are used by my graphql server for querying. The intermediary host for the two tunnels is same. So let's say the intermediary host is 123.45.678.678 and the two remote mongodb servers are 1.23.45.67 and 1.23.45.78, I create the tunnels like this,
conn, err := ssh.Dial("tcp", 123.45.678.678, config)
if err != nil {
panic(err)
}
remote1, err := conn.Dial("tcp", "1.23.45.67:27017")
if err != nil {
panic(err)
}
remote2, err := conn.Dial("tcp", "1.23.45.78:27017")
if err != nil {
panic(err)
}
local1, err := net.Listen("tcp", "localhost:27018")
if err != nil {
panic(err)
}
local2, err := net.Listen("tcp", "localhost:27019")
if err != nil {
panic(err)
}
Now i forward traffic from local1 to remote1 and local2 to remote2 like this
go func() {
for {
l, err := local1.Accept()
if err != nil {
panic(err)
}
go func() {
_, err := io.Copy(l, remote1)
if err != nil {
panic(err)
}
}()
go func() {
_, err := io.Copy(remote1, l)
if err != nil {
panic(err)
}
}()
}
}()
go func() {
for {
l, err := local2.Accept()
if err != nil {
panic(err)
}
go func() {
_, err := io.Copy(l, remote2)
if err != nil {
panic(err)
}
}()
go func() {
_, err := io.Copy(remote2, l)
if err != nil {
panic(err)
}
}()
}
}()
And I create two mongo sessions using mgo.Dial and export these two sessions to the graphql whenever this function is called. For some queries ( not all queries, only some complex queries ) which need both the sessions, i see the write: broken pipe error
panic: readfrom tcp 127.0.0.1:27019->127.0.0.1:53128: write tcp 127.0.0.1:27019->127.0.0.1:53128: write: broken pipe
When I debugged this, i figured out that this error occurs when the io.copy happens between l and remote2 in the code snippet above which i guess is due to the disconnection of remote2 tunnel.
The tcpdump showed that the intermediary host is sending the finish flag to the remote2 server after sometime which inturn is leading to the termination of the connection. I am wondering how I can resolve this.

Related

pgx in a goroutine reporting connection busy

My application is using pgx to running database queries in a goroutines. However, I am getting connection busy errors. Is there a way to have a goroutine
func writeDb(dbconn *pgx.Conn) {
sqlWritePost := `QUERY_HERE`
_, err := dbconn.Exec(context.Background(), sqlWritePost, v.Url, v.Content, v.StrippedContent, v.Posthash)
if err != nil {
fmt.Println(err)
os.Exit(1)
}
}
func main() {
var dbconn *pgx.Conn
dbconn, err := pgx.Connect(context.Background(), os.Getenv("database_string"))
if err != nil {
fmt.Println(err)
os.Exit(1)
}
...
go writeDb(dbconn)
...
}
I am receiving errors conn busy. Is there a way to structure my code to avoid this issue?
Thanks!

Why I get an error "client disconnected" when trying to get documents from mongo collection in go?

I have mongo capped collection and a simple API, written on Go. I built and run it. When I try to sent Get request or simply go localhost:8000/logger in browser - my process closes. Debug shows this happens, while executing "find" in collection. It produces error "client is disconnected". Collection has 1 document, and debug shows it is connected with my helper.
Go version 1.13
My code:
func main() {
r := mux.NewRouter()
r.HandleFunc("/logger", getDocs).Methods("GET")
r.HandleFunc("/logger", createDoc).Methods("POST")
log.Fatal(http.ListenAndServe("localhost:8000", r))
}
func getDocs(w http.ResponseWriter, r *http.Request) {
w.Header().Set("Content-Type", "application/json")
var docs []models.Logger
//Connection mongoDB with helper class
collection := helper.ConnectDB()
cur, err := collection.Find(context.TODO(), bson.M{})
if err != nil {
helper.GetError(err, w)
return
}
defer cur.Close(context.TODO())
for cur.Next(context.TODO()) {
var doc models.Logger
err := cur.Decode(&doc)
if err != nil {
log.Fatal(err)
}
docs = append(docs, doc)
}
if err := cur.Err(); err != nil {
log.Fatal(err)
}
json.NewEncoder(w).Encode(docs)
}
func ConnectDB() *mongo.Collection {
client, err := mongo.NewClient(options.Client().ApplyURI("mongodb://127.0.0.1:27017"))
if err != nil {
log.Fatal(err)
}
fmt.Println("Connected to MongoDB!")
logCollection := client.Database("local").Collection("loggerCollection")
return logCollection
}
According to the documentation, the call to mongo.NewClient doesn't ensure that you can connect the Mongo server. You should first call mongo.Client.Ping() to verify if you can connect to the database or not.
client, err := mongo.NewClient(options.Client().ApplyURI("mongodb://127.0.0.1:27017"))
if err != nil {
log.Fatal(err)
}
if err := client.Ping(context.TODO(), readpref.Primary()); err != nil {
// Can't connect to Mongo server
log.Fatal(err)
}
There could be several reasons behind failing to connect, the most obvious one is incorrect setup of ports. Is your mongodb server up and listening on port 27017? Is there any change you're running mongodb with Docker and it's not forwarding to the correct port?
I faced similar issue , read #Jay answer it definitely helped , as I checked my MongoDB was running using "MongoDB Compass" , then I changed the location of my insert statement , previously I was calling before the call of "context.WithTimeout". Below is working code.
package main
import (
"context"
"log"
"time"
"go.mongodb.org/mongo-driver/mongo"
"go.mongodb.org/mongo-driver/mongo/options"
)
type Book struct {
Name string `json:"name,omitempty"`
PublisherID string `json:"publisherid,omitempty"`
Cost string `json:"cost,omitempty"`
StartTime string `json:"starttime,omitempty"`
EndTime string `json:"endtime,omitempty"`
}
func main() {
client, err := mongo.NewClient(options.Client().ApplyURI("mongodb://localhost:27017"))
if err != nil {
log.Fatal(err)
}
ctx, _ := context.WithTimeout(context.Background(), 10*time.Second)
err = client.Connect(ctx)
if err != nil {
log.Fatal(err)
}
defer client.Disconnect(ctx)
testCollection := client.Database("BooksCollection").Collection("BooksRead")
inserRes, err := testCollection.InsertOne(context.TODO(), Book{Name: "Harry Potter", PublisherID: "IBN123", Cost: "1232", StartTime: "2013-10-01T01:11:18.965Z", EndTime: "2013-10-01T01:11:18.965Z"})
log.Println("InsertResponse : ", inserRes)
log.Println("Error : ", err)
}
I can see document inserted in console as well as in "MongoDB Comapass."
In heiper function "ConnectDB" after "NewClient" I must use "client.Connect(context.TODO())"
before any other use of client

How to solve "command find requires authentication" using Golang and MongoDriver

I need help to fix the error. I'm using IBM mongo services.
go version go1.13.6 darwin/amd64
mongo driver version 1.2.1
The connection is working, I can read and write but sometimes it returns
: command find requires authentication and command insert requires authentication
MONGO_DB_URI=mongodb://username:password:port,host/dbname?authSource=admin&replicaSet=replset&connect=direct&alias=default
Connect:
func ConnectDatabase() *mongo.Client {
clientOptions := options.Client().ApplyURI(os.Getenv("MONGO_DB_URI"))
ctx, _ := context.WithTimeout(context.Background(), 30*time.Second)
var err error
client, err = mongo.Connect(ctx, clientOptions)
if err != nil {
log.Fatal(err)
}
ctx, _ = context.WithTimeout(context.Background(), 10*time.Second)
err = client.Ping(ctx, nil)
if err != nil {
log.Fatal(err)
return nil
}
fmt.Println("Connected to MongoDB!")
return client
}
Read:
func FindAll(collectionName string, query bson.M) (*mongo.Cursor, error) {
collection := client.Database("dbname").Collection(collectionName)
singleResult, err := collection.Find(context.TODO(), query)
return singleResult, err
}
Read:
ctx, _ := context.WithTimeout(context.Background(), 20*time.Second)
cur, err := mongo.GetCollection("collection_name").Find(ctx, createQuery())
if err != nil {
log.Println(err.Error())
}
I'm using the same database and the same configurations at our another Python Project. No exceptions.
There is a difference between connecting to a DB and performing operations on the DB.
Mongo lets you connect without authentication because you have to be able to connect to be able to authenticate.
var cred options.Credential
cred.AuthSource = YourAuthSource
cred.Username = YourUserName
cred.Password = YourPassword
// set client options
clientOptions := options.Client().ApplyURI(os.Getenv("MONGO_DB_URI")).SetAuth(cred)
//... the rest of your code
Hope this helps.

MongoDB collection.Watch() compile and stop running

I'm newbie for MongoDB and I’ve tried to use collection.Watch() from "go.mongodb.org/mongo-driver/mongo" lib. and code from https://github.com/minhthuy30197/change_stream/blob/master/main.go. Then I build and run it stop immediately.
I’ve tried to run again and again and it also stop running. I’ve switch using between go run main.go and ./testStreams and it still stop running
So this is my edited code.
clientOptions := options.Client().
ApplyURI("mongodb://localhost:27017/test")
client, err := mongo.Connect(context.TODO(), clientOptions)
if err != nil {
log.Fatalf("Failed to create the new client: %v", err)
}
ctx := context.Background()
if err := client.Connect(ctx); err != nil {
log.Printf("Failed to open client connection: %v", err)
}
defer client.Disconnect(ctx)
coll := client.Database("test").Collection("streams")
var pipeline interface{}
for {
cur, err := coll.Watch(ctx, pipeline)
if err != nil {
log.Fatalf("Watch error: %v", err)
}
defer cur.Close(ctx)
log.Println(cur)
for cur.Next(ctx) {
elem := CSElem{}
if err := cur.Decode(elem); err != nil {
log.Fatalf("Decode error: %v", err)
}
log.Println(elem)
}
if err := cur.Err(); err != nil {
log.Fatalf("Error detected: %v", err)
}
}
When I edit then error appear
2019/08/07 13:46:39 Failed to open client connection: topology is
connected or connecting exit status 1
How should I fix??
As mentioned on the comment, utilising mongo-go-driver v1+ a new client instance need to be created first before making a connection. For example:
clientOptions := options.Client().ApplyURI("mongodb://localhost:27017/test")
client, err := mongo.NewClient(clientOptions)
if err != nil {log.Fatal(err)}
// Timed out after 10 seconds of trying to connect
ctx, cancel := context.WithTimeout(context.Background(), 10*time.Second)
defer cancel()
err = client.Connect(ctx)
if err != nil { log.Fatal(err)}

golang tcp socket does not send message after write immediately

My GO version is 1.1.1
the sever recieved messages after connection close, but NoDelay was setted.
Is there something wrong
addr, _ := net.ResolveTCPAddr("tcp", "localhost:5432")
conn, err := net.DialTCP("tcp", nil, addr)
defer conn.Close()
if err != nil {
fmt.Println("connect fail")
return
}
err = conn.SetNoDelay(true)
if err != nil {
fmt.Println(err.Error())
}
for {
var message string
_, err := fmt.Scanln(&message)
if err != nil && err.Error() != "unexpected newline" {
fmt.Println("input finished", err)
break
}
if message == "" {
fmt.Println("no input, end")
break
}
// message = fmt.Sprintf("%s\n",message)
//fmt.Fprintf(conn, message) // send immediately but following message won't send any more
conn.Write([]byte(message)) // won't send until connection close
}
There doesn't seem to be anything vitally wrong with your code so I'm guessing the error is on the server end.
If you create a local TCP server on port 5432 you can test this.
Try running the below server code and then test your client code against it. It just echos all received data to stdout.
package main
import (
"io"
"log"
"net"
"os"
)
func main() {
l, err := net.Listen("tcp", "localhost:5432")
if err != nil {
log.Fatal(err)
}
defer l.Close()
for {
conn, err := l.Accept()
if err != nil {
log.Fatal(err)
}
go func(c net.Conn) {
defer c.Close()
io.Copy(os.Stdout, c)
}(conn)
}
}
You should see each line sent to the client printed (without the newline) as soon as you hit enter.
the problem is on the server end.
func handleConnection(conn net.Conn) {
// I didn't put it in for loop
message, err := bufio.NewReader(conn).ReadString('\n')
}