Golang SSH tunneling connection to a remote postgres DB - postgresql

I have looked at various resources. I'm not sure how to accomplish this task. I can connect locally no problem, but I cannot connect to the remote easily. I need to pass on an RSA .pem key, and I'm not quite sure how to do this w/o forcing a connection unsecured
package main
import (
"database/sql"
"fmt"
"os"
_ "github.com/lib/pq"
)
var (
dbUser = os.Getenv("DB_USER")
dbPass = os.Getenv("DB_PASS")
dbName = os.Getenv("DB_NAME")
dbHost = os.Getenv("DB_HOST")
dbPort = os.Getenv("DB_PORT")
sslMode = os.Getenv("SSLMODE")
)
// ConnectDb is a short cut function that takes parameters through
// CLI that returns a pointer to a sql.DB connection.
// It takes no arguments.
func ConnectDb() (*sql.DB, error) {
db, err := sql.Open("postgres", getDbInfo())
CheckErr(err, "Unable to connecto tthe DB")
if err := db.Ping(); err != nil {
return nil, err
}
return db, nil
}
func getDbInfo() string {
var dbInfo string
if dbName != "" {
dbInfo += fmt.Sprintf("dbname=%s ", dbName)
} else {
dbInfo += fmt.Sprintf("dbname=%s ", "development")
}
// checks for nil value
if dbUser != "" {
dbInfo += fmt.Sprintf("dbuser=%s ", "user")
}
// checks for nil value
if dbPass != "" {
dbInfo += fmt.Sprintf("dbpass=%s ", dbPass)
}
if sslMode != "" {
dbInfo += fmt.Sprintf("sslmode=%s", sslMode)
} else {
dbInfo += fmt.Sprintf("sslmode=disable")
}
return dbInfo
}

It is my understanding that you need to open connection to postgre database. I don't know if a native postgre ssh tunneling support exists. So, this answer about SSH tunneling to DB machine.
I have not tested postgre this way, but I have had this model used in some proprietary server connections.
The process goes like that:
open ssh connection to db machine
establish tunnel from local port to remote db port
open db connection on local port
You can accomplish #1 and #2 with ssh client like OpenSSH or putty. You should probably do that 1st. If external client works, then you can attempt to put it all into go language code without external SSH client.
In go you would use
"golang.org/x/crypto/ssh"
package.
There are tutorials out there how to use GO ssh tunneling. Below is not a tested sample without error checking:
var buffer []byte
var err error
buffer, err = ioutil.ReadFile(sshKeyFile)
var key ssh.Signer
key, err = ssh.ParsePrivateKey(buffer)
var authMethod ssh.AuthMethod
authMethod = ssh.PublicKeys(key)
sshConfig = &ssh.ClientConfig{
User: "user_id",
Auth: []ssh.AuthMethod{authMethod},
}
conn, err := ssh.Dial("tcp", endpoint, sshConfig)
// open connection on postgre:
dbConn, err = conn.Dial("tcp", dbEndpoint)
The last line above is not exactly tunneling, but an TCP connection open to DB server. You might be able to pass that connection into db library. If not, you would have to setup a tunnel.

Related

pgx tls connection throws client cert invalid error for valid cert

I'm trying to use pgx to make a TLS connection to a postgres 10 db.
My connection string is similar to: "host='my-host.com' port='5432' dbname='my-db' user='my-db-user' sslmode='verify-full' sslcert='/path/to/db_user.crt' sslkey='/path/to/db_user.key' sslrootcert='/path/to/ca_roots.pem'"
When I run this directly with psql on the command-line, it works, so the cert and key files must be valid. db_user.crt and db_user.key are both PEM files. (the command-line also works with sslmode='verify-full', so the rootcert should also be ok)
But when I initialize a pgx pool with that connection string, it fails with:
FATAL: connection requires a valid client certificate (SQLSTATE 28000)
Is go expecting something other than PEM? Or is there a different way ssl cert and key pair is supposed to be initialized with pgx?
Code
import (
"context"
"fmt"
"os"
"github.com/jackc/pgx/v4"
"github.com/jackc/pgx/v4/pgxpool"
)
type mockLogger struct{}
func (ml *mockLogger) Log(ctx context.Context, level pgx.LogLevel, msg string, data map[string]interface{}) {
fmt.Printf("[%s] %s : %+v\n", level.String(), msg, data)
}
func connect() error {
connStr := "host='my-host.com' port='5432' dbname='my-db' user='my-db-user' sslmode='verify-full' sslcert='/path/to/db_user.crt' sslkey='/path/to/db_user.key' sslrootcert='/path/to/ca_roots.pem'"
poolCfg, err := pgxpool.ParseConfig(connStr)
if err != nil {
return err
}
poolCfg.ConnConfig.Logger = &mockLogger{}
poolCfg.ConnConfig.LogLevel = pgx.LogLevelTrace
fmt.Printf("using connection string: \"%s\"\n", poolCfg.ConnString())
connPool, err := pgxpool.ConnectConfig(context.TODO(), poolCfg)
if err != nil {
return err
}
connPool.Close()
return nil
}
func main() {
if err := connect(); err != nil {
fmt.Printf("%+v\n", err)
os.Exit(1)
}
}
Output from calling connect():
using connection string: "host='my-host.com' port='5432' dbname='my-db' user='my-db-user' sslmode='require' sslcert='/path/to/db_user.crt' sslkey='/path/to/db_user.key' sslrootcert='/path/to/ca_roots.pem'"
[info] Dialing PostgreSQL server : map[host:my-host.com]
[error] connect failed : map[err:failed to connect to `host=my-host.com user=my-db-user database=my-db`: server error (FATAL: connection requires a valid client certificate (SQLSTATE 28000))]
failed to connect to `host=my-host.com user=my-db-user database=my-db`: server error (FATAL: connection requires a valid client certificate (SQLSTATE 28000))
Summary
Turns out for go, the cert pointed to by sslcert needed to contain the full client cert chain.
When /path/to/db_user.crt contained the client cert followed by client cert chain, the pgx connection worked.
Whereas the psql command worked in both cases:
when sslcert was just the leaf client cert without the chain
when sslcert contained client cert + chain
Not sure why psql was fine without the full chain, but it works now.
Details
Under-the-hood, pgx uses the pgconn module to create the connection. That, in turn, is just calling tls.X509KeyPair on the contents of the sslcert and sslkey files.
pgconn/config.go:
func configTLS(settings map[string]string, thisHost string, parseConfigOptions ParseConfigOptions) ([]*tls.Config, error) {
[...]
sslcert := settings["sslcert"]
sslkey := settings["sslkey"]
[...]
if sslcert != "" && sslkey != "" {
[...]
certfile, err := ioutil.ReadFile(sslcert)
if err != nil {
return nil, fmt.Errorf("unable to read cert: %w", err)
}
cert, err := tls.X509KeyPair(certfile, pemKey)
if err != nil {
return nil, fmt.Errorf("unable to load cert: %w", err)
}
tlsConfig.Certificates = []tls.Certificate{cert}

(GoLang) panic: sql: Register called twice for driver postgres

I have probably spent way to much time on this, so I decided to try here.
I'm having trouble figuring out why my Register is being called twice?
Best I can figure, it seems to be calling once at sql.Register() and again at sqlx.Connect(). But if I remove the sql.Register(), then theres no drivers.
Honestly, I am pretty new to GoLang, I'm hoping for any sort of direction here.
Code - w/o sql.Register
package main
import (
"fmt"
"database/sql"
"github.com/jmoiron/sqlx"
)
const (
host = "localhost"
port = 5432
user = "postgres"
password = "password"
dbname = "sampledb"
)
/*-------------------------------------------*\
|| Functions ||
\*-------------------------------------------*/
// Error Checking Fn
func CheckError(err error, str string) {
if err != nil {
fmt.Printf("Error # : %s\n", str)
panic(err)
}
}
/*-------------------------------------------*\
|| Main() ||
\*-------------------------------------------*/
func main() {
// Open DB Conn
psqlconn := fmt.Sprintf("host=%s port=%d user=%s password=%s dbname=%s sslmode=disable", host, port, user, password, dbname)
// sql.Register("postgres", &pq.Driver{})
fmt.Printf(":: Drivers ::\n%s\n", sql.Drivers())
db, err := sqlx.Connect("postgres", psqlconn)
CheckError(err, "Main: sqlx.connect")
defer db.Close()
}
Error - w/o sql.Register
$ go run name-generator.go
:: Drivers ::
[]
Error # : Main: sqlx.connect
panic: sql: unknown driver "postgres" (forgotten import?)
goroutine 1 [running]:
main.CheckError({0xc84320, 0xc000056680}, {0xc6073f, 0x5})
C:/path/to/program.go:26 +0xa7 <--- Func CheckError(): panic(err)
main.main()
C:/path/to/program.go:40 +0x125 <--- Func Main(): CheckError()
exit status 2
Code - w/ sql.Register
package main
import (
"fmt"
"database/sql"
"github.com/jmoiron/sqlx"
"github.com/lib/pq"
)
const (
host = "localhost"
port = 5432
user = "postgres"
password = "password"
dbname = "sampledb"
)
/*-------------------------------------------*\
|| Functions ||
\*-------------------------------------------*/
// Error Checking Fn
func CheckError(err error, str string) {
if err != nil {
fmt.Printf("Error # : %s\n", str)
panic(err)
}
}
/*-------------------------------------------*\
|| Main() ||
\*-------------------------------------------*/
func main() {
// Open DB Conn
psqlconn := fmt.Sprintf("host=%s port=%d user=%s password=%s dbname=%s sslmode=disable", host, port, user, password, dbname)
sql.Register("postgres", &pq.Driver{})
fmt.Printf(":: Drivers ::\n%s\n", sql.Drivers())
db, err := sqlx.Connect("postgres", psqlconn)
CheckError(err, "Main: sqlx.connect")
defer db.Close()
}
Error - w/ sql.Register
$ go run name-generator.go
panic: sql: Register called twice for driver postgres
goroutine 1 [running]:
database/sql.Register({0xa98bc9, 0x8}, {0xae6680, 0xc8a950})
C:/Program Files/Go/src/database/sql/sql.go:51 +0x13d
main.main()
C:/path/to/program.go:38 +0x11b
exit status 2
Additional Resources
Similar issue, but doesn't solve my problem Link
SQLX Documentation Link
SQL Documentation Link
The package github.com/lib/pq registers it's driver in an init function.
Remove the direct call to register the driver from the application:
sql.Register("postgres", &pq.Driver{}) <-- delete this line
Import github.com/lib/pq for the side effect of executing the init() function:
package main
import (
"fmt"
"database/sql"
"github.com/jmoiron/sqlx"
_ "github.com/lib/pq" // <-- add this line
)

conn closed when trying to make an insert into postgresql

I try to insert a user into postgres database and get three parameters back for futher processing but always get the error 'conn closed':
package db
import (
"context"
"os"
"github.com/jackc/pgx/v4"
)
const (
insertSql = "INSERT into users (name, email, created, status, role, password, activation_code) VALUES ($1, $2, $3, $4, $5, $6, $7) RETURNING name, email, activation_code;"
)
type userRepository struct {
connect *pgx.Conn
}
func NewUserRepository(c *pgx.Conn) domain.UserRepository {
return &userRepository{
connect: c,
}
}
func (r *userRepository) Save(u *domain.User) (string, string, string, *resterrors.RestErr) {
var name, email, activation_code string
if r.connect == nil {
os.Exit(1)
}
if err := r.connect.QueryRow(context.Background(), insertSql, u.Name, u.Email, u.Created, u.Status, u.Role, u.Password, u.ActivationCode).Scan(&name, &email, &activation_code); err != nil {
logger.Error("error saving user to the database: " + err.Error())
return "", "", "", resterrors.NewIntenalServerError(resterrors.DB_ERROR)
}
return name, email, activation_code, nil
}
connect is inserted in application.go when respository is created
repository := db.NewUserRepository(postgresql.Connect)
userHandler := controller.NewUserHandler(service.NewUserService(repository))
Connect is globally declared in postgresql package
package postgresql
import (
"context"
"fmt"
"net/url"
"github.com/jackc/pgx/v4"
)
var (
Connect *pgx.Conn
username = "postgres"
password = "***********"
host = "127.0.0.1"
port = "5432"
dbName = "test"
)
func init() {
//prepare URL to connect to database
var err error
datasourceName := fmt.Sprintf("postgres://%s:%s#%s:%s/%s", username, password, host, port, dbName)
Connect, err = pgx.Connect(context.Background(), datasourceName)
if err != nil {
logger.Error("unable to connect to database: " + err.Error())
panic(fmt.Errorf("unable to connect to database: %w", err))
}
logger.Info("successsfully connected to postgres database")
// to close DB connection
defer Connect.Close(context.Background())
}
What could be wrong with it?
A single DB Connection is not concurrency safe and has no mechanism for reestablishing connections. Your issue is likely that concurrent access borked the connection, or it was closed for some other reason.
You want a DB Connection Pool so that it can create new connections for concurrent operations (requests for net/http servers handle connections concurrently). From https://pkg.go.dev/github.com/jackc/pgx/v4#hdr-Connection_Pool:
*pgx.Conn represents a single connection to the database and is not concurrency safe. Use sub-package pgxpool for a concurrency safe connection pool.
You should have better results with pgxpool - see https://pkg.go.dev/github.com/jackc/pgx/v4#v4.14.1/pgxpool#hdr-Establishing_a_Connection
The reason for the conn closed error was that the connection was closed within init() function straight after the application start.
defer Connect.Close(context.Background())

Unable to connect to postgresql using Go and pq

I'm trying to connect a Go application with postgresql.
The app import postgresql driver:
"crypto/tls"
"database/sql"
"fmt"
"log"
"os"
"os/signal"
...
_ "github.com/go-sql-driver/mysql"
_ "github.com/lib/pq"
_ "github.com/mattn/go-sqlite3"
and uses like it to connect to the database:
driver, cnxn := dbFromURI(dbURI)
db, err := sql.Open(driver, cnxn)
if err != nil {
panic(err)
}
and the dbFromUri method just split the info
func dbFromURI(uri string) (string, string) {
parts := strings.Split(uri, "://")
return parts[0], parts[1]
}
My URI works locally when i run the command : psql postgresql://user:user#172.20.0.1:5432/lcp
But in go i Go, I got
./lcpserver
2021/03/07 02:00:42 Reading config /root/lcp-server-install/lcp-home/config/config.yaml
panic: pq: SSL is not enabled on the server
I tried this URI for Go without success : psql postgresql://user:user#172.20.0.1:5432/lcp?sslmode=disable
Do you have any idea why i can't connect ?
I tried with my aws rds postgres database, and got same result. Thnaks for the help.
complete code of the server
https://github.com/readium/readium-lcp-server/blob/master/lcpserver/lcpserver.go
I change the typo and the demo provided i succeed the connexion. But in the lcp server i style got the same issue.
postgres://user:user#172.20.0.1:5432/lcp?sslmode=disable
EDIT 2:
The error is due to the fact that the script tries prepare command. I update the minimal example and it fails too.
package main
import (
"database/sql"
"fmt"
"strings"
_ "github.com/lib/pq"
)
func dbFromURI(uri string) (string, string) {
parts := strings.Split(uri, "://")
return parts[0], parts[1]
}
func main() {
driver, cnxn := dbFromURI("postgres://user:user#172.20.0.1:5432/lcp?sslmode=disable")
fmt.Println("The driver " + driver)
fmt.Println("The cnxn " + cnxn)
db, err := sql.Open(driver, cnxn)
_, err = db.Prepare("SELECT id,encryption_key,location,length,sha256,type FROM content WHERE id = ? LIMIT 1")
if err != nil {
fmt.Println("Prepare failed")
fmt.Println(err)
}
if err == nil {
fmt.Println("Successfully connected")
} else {
panic(err)
}
}
I got :
prepare failed
pq: SSL is not enabled on the server
2021/03/07 17:20:13 This panic 3
panic: pq: SSL is not enabled on the server
The first problem is a typo in the connection string: postgresql://user:user#172.20.0.1:5432/lcp?sslmode=disable. In Go code it should be postgres://user:user#172.20.0.1:5432/lcp?sslmode=disable.
We also need to pass the full connection string as the second argument to sql.Open. For now, the dbFromURI function returns user:user#172.20.0.1:5432/lcp?sslmode=disable, but we need postgres://user:user#172.20.0.1:5432/lcp?sslmode=disable, because pq is waiting for this prefix to parse it.
After fixing this, I was able to establish a connection using a minimal postgres client based on your code.
To try this yourself, start the server with the following command:
docker run --rm -p 5432:5432 -e POSTGRES_PASSWORD=some_password postgres
And try to connect using the following client code:
package main
import (
"database/sql"
"fmt"
_ "github.com/lib/pq"
)
func main() {
cnxn := "postgres://postgres:some_password#127.0.0.1:5432/lcp?sslmode=disable"
_, err := sql.Open("postgres", cnxn)
if err != nil {
panic(err)
}
_, err = db.Prepare("SELECT id,encryption_key,location,length,sha256,type FROM content WHERE id = ? LIMIT 1")
if err != nil {
fmt.Println("Prepare failed")
panic(err)
}
}

golang client fails to connect to mongo db server - sslv3 alert bad certificate

I'm trying to connect a go client to mongodb server running with ssl enabled. I get a clear error message indicating that the hand shake failed due to ssl error. I use a self signed certificate on the client side.
Got below from the mongodb server:
2017-05-13T04:38:53.910+0000 I NETWORK [thread1] connection accepted from 172.17.0.1:51944 #10 (1 connection now open)
2017-05-13T04:38:53.911+0000 E NETWORK [conn10] SSL: error:14094412:SSL routines:SSL3_READ_BYTES:sslv3 alert bad certificate
2017-05-13T04:38:53.911+0000 I - [conn10] end connection
Error from Go client:
Could not connect to mongodb_s1.dev:27017 x509: certificate signed by unknown authority (possibly because of "crypto/rsa: verification error" while trying to verify candidate authority certificate "XYZ")
Tried multiple options, but didn't help
You can skip TLS security checks using InsecureSkipVerify = true. This allows you to use self-signed certificates. See the code from compose help below.
Instead of skipping security checks, it is advisable to add the CA used to sign your certificates to the list of trusted CAs of the system.
package main
import (
"crypto/tls"
"fmt"
"net"
"os"
"strings"
"gopkg.in/mgo.v2"
)
func main() {
uri := os.Getenv("MONGODB_URL")
if uri == "" {
fmt.Println("No connection string provided - set MONGODB_URL")
os.Exit(1)
}
uri = strings.TrimSuffix(uri, "?ssl=true")
Here:
tlsConfig := &tls.Config{}
tlsConfig.InsecureSkipVerify = true
dialInfo, err := mgo.ParseURL(uri)
if err != nil {
fmt.Println("Failed to parse URI: ", err)
os.Exit(1)
}
And here:
dialInfo.DialServer = func(addr *mgo.ServerAddr) (net.Conn, error) {
conn, err := tls.Dial("tcp", addr.String(), tlsConfig)
return conn, err
}
session, err := mgo.DialWithInfo(dialInfo)
if err != nil {
fmt.Println("Failed to connect: ", err)
os.Exit(1)
}
defer session.Close()
dbnames, err := session.DB("").CollectionNames()
if err != nil {
fmt.Println("Couldn't query for collections names: ", err)
os.Exit(1)
}
fmt.Println(dbnames)
}