Unable to connect to postgresql using Go and pq - postgresql

I'm trying to connect a Go application with postgresql.
The app import postgresql driver:
"crypto/tls"
"database/sql"
"fmt"
"log"
"os"
"os/signal"
...
_ "github.com/go-sql-driver/mysql"
_ "github.com/lib/pq"
_ "github.com/mattn/go-sqlite3"
and uses like it to connect to the database:
driver, cnxn := dbFromURI(dbURI)
db, err := sql.Open(driver, cnxn)
if err != nil {
panic(err)
}
and the dbFromUri method just split the info
func dbFromURI(uri string) (string, string) {
parts := strings.Split(uri, "://")
return parts[0], parts[1]
}
My URI works locally when i run the command : psql postgresql://user:user#172.20.0.1:5432/lcp
But in go i Go, I got
./lcpserver
2021/03/07 02:00:42 Reading config /root/lcp-server-install/lcp-home/config/config.yaml
panic: pq: SSL is not enabled on the server
I tried this URI for Go without success : psql postgresql://user:user#172.20.0.1:5432/lcp?sslmode=disable
Do you have any idea why i can't connect ?
I tried with my aws rds postgres database, and got same result. Thnaks for the help.
complete code of the server
https://github.com/readium/readium-lcp-server/blob/master/lcpserver/lcpserver.go
I change the typo and the demo provided i succeed the connexion. But in the lcp server i style got the same issue.
postgres://user:user#172.20.0.1:5432/lcp?sslmode=disable
EDIT 2:
The error is due to the fact that the script tries prepare command. I update the minimal example and it fails too.
package main
import (
"database/sql"
"fmt"
"strings"
_ "github.com/lib/pq"
)
func dbFromURI(uri string) (string, string) {
parts := strings.Split(uri, "://")
return parts[0], parts[1]
}
func main() {
driver, cnxn := dbFromURI("postgres://user:user#172.20.0.1:5432/lcp?sslmode=disable")
fmt.Println("The driver " + driver)
fmt.Println("The cnxn " + cnxn)
db, err := sql.Open(driver, cnxn)
_, err = db.Prepare("SELECT id,encryption_key,location,length,sha256,type FROM content WHERE id = ? LIMIT 1")
if err != nil {
fmt.Println("Prepare failed")
fmt.Println(err)
}
if err == nil {
fmt.Println("Successfully connected")
} else {
panic(err)
}
}
I got :
prepare failed
pq: SSL is not enabled on the server
2021/03/07 17:20:13 This panic 3
panic: pq: SSL is not enabled on the server

The first problem is a typo in the connection string: postgresql://user:user#172.20.0.1:5432/lcp?sslmode=disable. In Go code it should be postgres://user:user#172.20.0.1:5432/lcp?sslmode=disable.
We also need to pass the full connection string as the second argument to sql.Open. For now, the dbFromURI function returns user:user#172.20.0.1:5432/lcp?sslmode=disable, but we need postgres://user:user#172.20.0.1:5432/lcp?sslmode=disable, because pq is waiting for this prefix to parse it.
After fixing this, I was able to establish a connection using a minimal postgres client based on your code.
To try this yourself, start the server with the following command:
docker run --rm -p 5432:5432 -e POSTGRES_PASSWORD=some_password postgres
And try to connect using the following client code:
package main
import (
"database/sql"
"fmt"
_ "github.com/lib/pq"
)
func main() {
cnxn := "postgres://postgres:some_password#127.0.0.1:5432/lcp?sslmode=disable"
_, err := sql.Open("postgres", cnxn)
if err != nil {
panic(err)
}
_, err = db.Prepare("SELECT id,encryption_key,location,length,sha256,type FROM content WHERE id = ? LIMIT 1")
if err != nil {
fmt.Println("Prepare failed")
panic(err)
}
}

Related

(GoLang) panic: sql: Register called twice for driver postgres

I have probably spent way to much time on this, so I decided to try here.
I'm having trouble figuring out why my Register is being called twice?
Best I can figure, it seems to be calling once at sql.Register() and again at sqlx.Connect(). But if I remove the sql.Register(), then theres no drivers.
Honestly, I am pretty new to GoLang, I'm hoping for any sort of direction here.
Code - w/o sql.Register
package main
import (
"fmt"
"database/sql"
"github.com/jmoiron/sqlx"
)
const (
host = "localhost"
port = 5432
user = "postgres"
password = "password"
dbname = "sampledb"
)
/*-------------------------------------------*\
|| Functions ||
\*-------------------------------------------*/
// Error Checking Fn
func CheckError(err error, str string) {
if err != nil {
fmt.Printf("Error # : %s\n", str)
panic(err)
}
}
/*-------------------------------------------*\
|| Main() ||
\*-------------------------------------------*/
func main() {
// Open DB Conn
psqlconn := fmt.Sprintf("host=%s port=%d user=%s password=%s dbname=%s sslmode=disable", host, port, user, password, dbname)
// sql.Register("postgres", &pq.Driver{})
fmt.Printf(":: Drivers ::\n%s\n", sql.Drivers())
db, err := sqlx.Connect("postgres", psqlconn)
CheckError(err, "Main: sqlx.connect")
defer db.Close()
}
Error - w/o sql.Register
$ go run name-generator.go
:: Drivers ::
[]
Error # : Main: sqlx.connect
panic: sql: unknown driver "postgres" (forgotten import?)
goroutine 1 [running]:
main.CheckError({0xc84320, 0xc000056680}, {0xc6073f, 0x5})
C:/path/to/program.go:26 +0xa7 <--- Func CheckError(): panic(err)
main.main()
C:/path/to/program.go:40 +0x125 <--- Func Main(): CheckError()
exit status 2
Code - w/ sql.Register
package main
import (
"fmt"
"database/sql"
"github.com/jmoiron/sqlx"
"github.com/lib/pq"
)
const (
host = "localhost"
port = 5432
user = "postgres"
password = "password"
dbname = "sampledb"
)
/*-------------------------------------------*\
|| Functions ||
\*-------------------------------------------*/
// Error Checking Fn
func CheckError(err error, str string) {
if err != nil {
fmt.Printf("Error # : %s\n", str)
panic(err)
}
}
/*-------------------------------------------*\
|| Main() ||
\*-------------------------------------------*/
func main() {
// Open DB Conn
psqlconn := fmt.Sprintf("host=%s port=%d user=%s password=%s dbname=%s sslmode=disable", host, port, user, password, dbname)
sql.Register("postgres", &pq.Driver{})
fmt.Printf(":: Drivers ::\n%s\n", sql.Drivers())
db, err := sqlx.Connect("postgres", psqlconn)
CheckError(err, "Main: sqlx.connect")
defer db.Close()
}
Error - w/ sql.Register
$ go run name-generator.go
panic: sql: Register called twice for driver postgres
goroutine 1 [running]:
database/sql.Register({0xa98bc9, 0x8}, {0xae6680, 0xc8a950})
C:/Program Files/Go/src/database/sql/sql.go:51 +0x13d
main.main()
C:/path/to/program.go:38 +0x11b
exit status 2
Additional Resources
Similar issue, but doesn't solve my problem Link
SQLX Documentation Link
SQL Documentation Link
The package github.com/lib/pq registers it's driver in an init function.
Remove the direct call to register the driver from the application:
sql.Register("postgres", &pq.Driver{}) <-- delete this line
Import github.com/lib/pq for the side effect of executing the init() function:
package main
import (
"fmt"
"database/sql"
"github.com/jmoiron/sqlx"
_ "github.com/lib/pq" // <-- add this line
)

How to connect to mongoDB via ssl using .crt file in Go

I am trying to connect to a mongo database hosted in azure using the .crt file.
I am successfully able to connect from my linux machine terminal using command:
mongo mongodb://username:password#prod-replicaset-0.com:27017,prod-replicaset-1.com:27017,prod-replicaset-2.com:27017/ --tls --tlsCAFile rootca.crt --tlsAllowInvalidCertificates
I am also able to connect from mongo UI client like robo3T by setting "Use SSL protocol" and using Auth Mechanism as "SCRAM-SHA-256".
[If I set Auth Mechanism to any other value, results in Authentication Failure]
But I am not able to connect to that database in Go lang code.
Here is a sample of code I am using:
package main
import (
"crypto/tls"
"crypto/x509"
"io/ioutil"
"log"
"net"
"github.com/globalsign/mgo"
)
func InitMongo() error {
rootCerts := x509.NewCertPool()
ca, err := ioutil.ReadFile("./rootca.crt")
if err != nil {
log.Fatalf("failed to read file : %s", err.Error())
return err
}
success := rootCerts.AppendCertsFromPEM(ca)
if !success {
log.Printf("rootcert failed")
}
connStr := "mongodb://username:password#prod-replicaset-0.com:27017,prod-replicaset-1.com:27017,prod-replicaset-2.com:27017/?ssl=true"
dbDialInfo, err := mgo.ParseURL(connStr)
if err != nil {
log.Fatal("unable to parse url - " + err.Error())
}
dbDialInfo.DialServer = func(addr *mgo.ServerAddr) (net.Conn, error) {
return tls.Dial("tcp", addr.String(), &tls.Config{
RootCAs: rootCerts,
InsecureSkipVerify: true,
})
}
// dbDialInfo.Mechanism = "SCRAM-SHA-256"
_session, err := mgo.DialWithInfo(dbDialInfo)
if err != nil {
log.Fatalf("failed to creating db session : %s", err.Error())
return err
}
log.Printf("Created session - %v", _session)
return nil
}
When I run this code, I get error:
failed to creating db session : "server returned error on SASL authentication step: Authentication failed."
If I set [dbDialInfo.Mechanism = "SCRAM-SHA-256"] before creating session, I get error:
failed to creating db session : "SASL support not enabled during build (-tags sasl)"
Please let me know what is causing this issue, how can I connect to the database.
Currently I am using "github.com/globalsign/mgo", if it required to use any other library, that's totally fine for me.
I just want to get connected to the db.
rootca.crt file looks something like:
-----BEGIN CERTIFICATE-----
MIIGLjCCBBagAwIBAgIUbxINX1qe6W+7kolWGp+MX8NbYj8wDQYJKoZIhvcNAQEL
<blah> <blah> <blah> <blah> <blah> <blah> <blah> <blah> <blah>
jCZAGGHmbrR3zeIsOY8yKau0IXqRp5Wy6NQ0poOTcma9BfwNUVc4/ixsCkEVYbgW
eMs=
-----END CERTIFICATE-----
Thank you.
After researching a lot, I was not able to find a way to connect to mongodb using .crt file using globalsign library.
However I was successfully able to do this using mongo-driver library.
here connection string can be of format:
mongodb://user:password#replicaset-0.com:27017,replicaset-1.com:27017,replicaset-2.com:27017/?ssl=true&tlsCAFile=./ca.crt&tlsCertificateKeyFile=./ca.pem&authSource=admin&replicaSet=replicaset
Sample code:
import (
"context"
"log"
"os"
// "github.com/globalsign/mgo"
mgo "go.mongodb.org/mongo-driver/mongo"
mongoOptions "go.mongodb.org/mongo-driver/mongo/options"
)
func InitMongo() (error) {
connStr := os.Getenv("MONGODB_CONN_STR")
dbName := os.Getenv("MONGODB_DATABASE")
clientOpts := mongoOptions.Client().ApplyURI(connStr)
if err := clientOpts.Validate(); err != nil {
log.Print("unable to parse url")
log.Fatal(err)
}
client, err := mgo.Connect(context.TODO(), clientOpts)
if err != nil {
log.Print("unable to connect into database")
log.Fatal(err)
}
if err := client.Ping(context.TODO(), nil); err != nil {
log.Print("database ping failed")
log.Fatal(err)
}
//client.Database(dbName)
return nil
}

Go Mongo DB doesn't connect

I'm trying to connect to my mongo db atlas database, but I'm getting an error that I can't solve it within mongo+go forums.
The code:
package main
import (
"context"
"log"
"time"
"fmt"
"go.mongodb.org/mongo-driver/mongo"
"go.mongodb.org/mongo-driver/mongo/options"
"go.mongodb.org/mongo-driver/mongo/readpref"
"go.mongodb.org/mongo-driver/bson"
)
func main(){
client, err := mongo.NewClient(options.Client().ApplyURI("mongodb://my-user:<my-pass>#datalake0-lesz0.a.query.mongodb.net/my-db?ssl=true&authSource=admin"))
if err != nil {
log.Fatal(err)
}
ctx, _ := context.WithTimeout(context.Background(), 10*time.Second)
err = client.Connect(ctx)
if err != nil {
log.Fatal(err)
}
defer client.Disconnect(ctx)
err = client.Ping(ctx, readpref.Primary())
if err != nil {
log.Fatal(err)
}
databases, err := client.ListDatabaseNames(ctx, bson.M{})
if err != nil {
log.Fatal(err)
}
fmt.Println(databases)
}
The error:
connection() error occured during connection handshake: auth error: sasl conversation error: unable to authenticate using mechanism "SCRAM-SHA-1": (AuthenticationFailed) authentication failed, correlationID = 167bc5ba18415510a4144b7a
exit status 1
If the URI in the connect string is verbatim, then this is your problem:
mongodb://my-user:<my-pass> is incorrect, <my-pass> should be substituted with your password
You've probably cut and pasted the provided URI on completion of DB setup but this connect string does not include items like your password, also you gave my-user which, unless you set up the username my-user that also needs changing.

golang-migrate unable to find postgres driver

In my internal/platform/database/database.go
import (
"github.com/golang-migrate/migrate"
"github.com/jmoiron/sqlx"
_ "github.com/lib/pq"
)
func RunMigrations() error {
m, err := migrate.New(
"file://schema",
"postgres://postgres:postgres#localhost:5432/postgres?sslmode=disable")
if err != nil {
return errors.Wrap(err, "error creating migrations object")
}
This function is invoked from my cmd/my-api/main.go as follows:
import (
_ "github.com/golang-migrate/migrate/v4/database/postgres"
_ "github.com/golang-migrate/migrate/v4/source/file"
"github.com/jmoiron/sqlx"
_ "github.com/lib/pq"
"github.com/myrepo/myproject/internal/platform/database"
)
// =========================================================================
// Executing migrations
if err := database.RunMigrations(); err != nil {
log.Fatal(err)
}
Although I am importing postgres driver in both files, _ "github.com/lib/pq"
running the program fails as follows:
error creating migrations object: source driver: unknown driver file (forgotten import?)
exit status 1
Why is that?
It seems that golang-migrate needs its own version of the corresponding driver (?)
The following import solved it for me
_ "github.com/golang-migrate/migrate/v4/database/postgres"
When you import the following, postgres driver init function triggered and this function register the postgres driver.
_ "github.com/golang-migrate/migrate/v4/database/postgres"
You can inspect this.
https://www.calhoun.io/why-we-import-sql-drivers-with-the-blank-identifier/

Golang SSH tunneling connection to a remote postgres DB

I have looked at various resources. I'm not sure how to accomplish this task. I can connect locally no problem, but I cannot connect to the remote easily. I need to pass on an RSA .pem key, and I'm not quite sure how to do this w/o forcing a connection unsecured
package main
import (
"database/sql"
"fmt"
"os"
_ "github.com/lib/pq"
)
var (
dbUser = os.Getenv("DB_USER")
dbPass = os.Getenv("DB_PASS")
dbName = os.Getenv("DB_NAME")
dbHost = os.Getenv("DB_HOST")
dbPort = os.Getenv("DB_PORT")
sslMode = os.Getenv("SSLMODE")
)
// ConnectDb is a short cut function that takes parameters through
// CLI that returns a pointer to a sql.DB connection.
// It takes no arguments.
func ConnectDb() (*sql.DB, error) {
db, err := sql.Open("postgres", getDbInfo())
CheckErr(err, "Unable to connecto tthe DB")
if err := db.Ping(); err != nil {
return nil, err
}
return db, nil
}
func getDbInfo() string {
var dbInfo string
if dbName != "" {
dbInfo += fmt.Sprintf("dbname=%s ", dbName)
} else {
dbInfo += fmt.Sprintf("dbname=%s ", "development")
}
// checks for nil value
if dbUser != "" {
dbInfo += fmt.Sprintf("dbuser=%s ", "user")
}
// checks for nil value
if dbPass != "" {
dbInfo += fmt.Sprintf("dbpass=%s ", dbPass)
}
if sslMode != "" {
dbInfo += fmt.Sprintf("sslmode=%s", sslMode)
} else {
dbInfo += fmt.Sprintf("sslmode=disable")
}
return dbInfo
}
It is my understanding that you need to open connection to postgre database. I don't know if a native postgre ssh tunneling support exists. So, this answer about SSH tunneling to DB machine.
I have not tested postgre this way, but I have had this model used in some proprietary server connections.
The process goes like that:
open ssh connection to db machine
establish tunnel from local port to remote db port
open db connection on local port
You can accomplish #1 and #2 with ssh client like OpenSSH or putty. You should probably do that 1st. If external client works, then you can attempt to put it all into go language code without external SSH client.
In go you would use
"golang.org/x/crypto/ssh"
package.
There are tutorials out there how to use GO ssh tunneling. Below is not a tested sample without error checking:
var buffer []byte
var err error
buffer, err = ioutil.ReadFile(sshKeyFile)
var key ssh.Signer
key, err = ssh.ParsePrivateKey(buffer)
var authMethod ssh.AuthMethod
authMethod = ssh.PublicKeys(key)
sshConfig = &ssh.ClientConfig{
User: "user_id",
Auth: []ssh.AuthMethod{authMethod},
}
conn, err := ssh.Dial("tcp", endpoint, sshConfig)
// open connection on postgre:
dbConn, err = conn.Dial("tcp", dbEndpoint)
The last line above is not exactly tunneling, but an TCP connection open to DB server. You might be able to pass that connection into db library. If not, you would have to setup a tunnel.