I am trying to perform inter-service communication between microservices. I followed the documentation and it was successful. Then, I tried to establish the same with echo framework. But that gives me an invalid memory address when trying to call the gRPC registered method.
rpc error: code = Unavailable desc = connection closedpanic:
runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x7eacad]
Server config:
e := echo.New()
e.Use(middleware.Logger())
e.Use(middleware.Recover())
e.Logger.Fatal(e.Start(":1323"))
s := grpc.NewServer()
hellopb.RegisterHelloServiceServer(s, NewServer())
e.GET("/", func(c echo.Context) error {
return c.JSON(http.StatusOK, echo.Map{"status": "success"})
})
s.Serve(e.Listener)
Client panics:
resp, err := client.Hello(context.Background(), request)
if err != nil {
fmt.Printf("%+v", err)
}
Hello is the function registered with gRPC.
Related
I'm trying to use pgx to make a TLS connection to a postgres 10 db.
My connection string is similar to: "host='my-host.com' port='5432' dbname='my-db' user='my-db-user' sslmode='verify-full' sslcert='/path/to/db_user.crt' sslkey='/path/to/db_user.key' sslrootcert='/path/to/ca_roots.pem'"
When I run this directly with psql on the command-line, it works, so the cert and key files must be valid. db_user.crt and db_user.key are both PEM files. (the command-line also works with sslmode='verify-full', so the rootcert should also be ok)
But when I initialize a pgx pool with that connection string, it fails with:
FATAL: connection requires a valid client certificate (SQLSTATE 28000)
Is go expecting something other than PEM? Or is there a different way ssl cert and key pair is supposed to be initialized with pgx?
Code
import (
"context"
"fmt"
"os"
"github.com/jackc/pgx/v4"
"github.com/jackc/pgx/v4/pgxpool"
)
type mockLogger struct{}
func (ml *mockLogger) Log(ctx context.Context, level pgx.LogLevel, msg string, data map[string]interface{}) {
fmt.Printf("[%s] %s : %+v\n", level.String(), msg, data)
}
func connect() error {
connStr := "host='my-host.com' port='5432' dbname='my-db' user='my-db-user' sslmode='verify-full' sslcert='/path/to/db_user.crt' sslkey='/path/to/db_user.key' sslrootcert='/path/to/ca_roots.pem'"
poolCfg, err := pgxpool.ParseConfig(connStr)
if err != nil {
return err
}
poolCfg.ConnConfig.Logger = &mockLogger{}
poolCfg.ConnConfig.LogLevel = pgx.LogLevelTrace
fmt.Printf("using connection string: \"%s\"\n", poolCfg.ConnString())
connPool, err := pgxpool.ConnectConfig(context.TODO(), poolCfg)
if err != nil {
return err
}
connPool.Close()
return nil
}
func main() {
if err := connect(); err != nil {
fmt.Printf("%+v\n", err)
os.Exit(1)
}
}
Output from calling connect():
using connection string: "host='my-host.com' port='5432' dbname='my-db' user='my-db-user' sslmode='require' sslcert='/path/to/db_user.crt' sslkey='/path/to/db_user.key' sslrootcert='/path/to/ca_roots.pem'"
[info] Dialing PostgreSQL server : map[host:my-host.com]
[error] connect failed : map[err:failed to connect to `host=my-host.com user=my-db-user database=my-db`: server error (FATAL: connection requires a valid client certificate (SQLSTATE 28000))]
failed to connect to `host=my-host.com user=my-db-user database=my-db`: server error (FATAL: connection requires a valid client certificate (SQLSTATE 28000))
Summary
Turns out for go, the cert pointed to by sslcert needed to contain the full client cert chain.
When /path/to/db_user.crt contained the client cert followed by client cert chain, the pgx connection worked.
Whereas the psql command worked in both cases:
when sslcert was just the leaf client cert without the chain
when sslcert contained client cert + chain
Not sure why psql was fine without the full chain, but it works now.
Details
Under-the-hood, pgx uses the pgconn module to create the connection. That, in turn, is just calling tls.X509KeyPair on the contents of the sslcert and sslkey files.
pgconn/config.go:
func configTLS(settings map[string]string, thisHost string, parseConfigOptions ParseConfigOptions) ([]*tls.Config, error) {
[...]
sslcert := settings["sslcert"]
sslkey := settings["sslkey"]
[...]
if sslcert != "" && sslkey != "" {
[...]
certfile, err := ioutil.ReadFile(sslcert)
if err != nil {
return nil, fmt.Errorf("unable to read cert: %w", err)
}
cert, err := tls.X509KeyPair(certfile, pemKey)
if err != nil {
return nil, fmt.Errorf("unable to load cert: %w", err)
}
tlsConfig.Certificates = []tls.Certificate{cert}
Folks,
I'm trying to run gRPC python server with go stub.
It's working properly locally and as dockerized, however, when I'm converting to deployments and run it on kubernetes GKE cluster, I'm getting frequent "connection refused" error as below
I'm client! --> working
Retrieving ... --> working
id:1 username:"admin" email:"admin#admin.admin" --> working
id:2 username:"user1" --> working
I'm client! --> working
Retrieving ... --> working
2021/11/27 00:55:22 Retrive error: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing dial tcp 10.96.3.78:50051: connect: connection refused"
Also tried to add WithKeepaliveParams() dial option and close connection explicitly at the end without defer but still getting the same error.
Here is my stub code
package main
import (
"context"
"fmt"
// "io"
"log"
"time"
"github.com/nurhun/grpc_django_go_client/accountpb"
"google.golang.org/grpc"
"google.golang.org/grpc/keepalive"
)
func main() {
var kacp = keepalive.ClientParameters{
Time: 10 * time.Second, // send pings every 10 seconds if there is no activity
Timeout: time.Second, // wait 1 second for ping back
PermitWithoutStream: true, // send pings even without active streams
}
for {
fmt.Println("I'm client!")
cc, err := grpc.Dial("crud:50051", grpc.WithInsecure(), grpc.WithDisableHealthCheck(), grpc.WithKeepaliveParams(kacp))
if err != nil {
log.Fatal("Connection error", err)
}
// defer cc.Close()
c := accountpb.NewUserControllerClient(cc)
// ##### Using Retrive with User ID to get user data #####
fmt.Println("Retrieving ...")
res1, err := c.Retrieve(context.Background(), &accountpb.UserRetrieveRequest{Id: 1})
if err != nil {
log.Fatal("Retrive error: ", err)
}
fmt.Println(res1)
res2, err := c.Retrieve(context.Background(), &accountpb.UserRetrieveRequest{Id: 2})
if err != nil {
log.Fatal("Retrive error: ", err)
}
fmt.Println(res2)
time.Sleep(10 * time.Second)
cc.Close()
}
}
Any idea where is this error comes from ?
I am trying to connect to my GAE Postgres SQL db using Go+Gin+PGX. I have the Postgres SQL api activated and this program runs on my local machine but does not run on GAE. I think it is not connecting the db via pgx but I am not sure.
main.go works on my local machine and instance of psql but after deploying to GAE cannot connect to db via the Public IP in GCP SQL instance. I have modified my working code to fix the MWE.
package main
import (
"fmt"
"os"
"github.com/gin-gonic/gin"
"github.com/jackc/pgx/v4"
"golang.org/x/net/context"
)
func main() {
conn, err := connectDB()
if err != nil {
os.Exit(1)
}
defer conn.Close(context.Background())
router := gin.Default()
router.GET("/", func(c *gin.Context) {
c.JSON(200, gin.H{
"message": "pong",
})
})
router.Run(":8080")
}
func connectDB() (c *pgx.Conn, err error) {
postgres := "postgresql://postgres:root#35.236.60.144:5432/goapi" //35.236.60.144
conn, err := pgx.Connect(context.Background(), postgres)
if err != nil {
fmt.Fprintf(os.Stderr, "Unable to connect to database:\n\t%v\n", err.Error())
return nil, err
}
err = conn.Ping(context.Background())
if err != nil {
fmt.Fprintf(os.Stderr, "Unable to ping:\n\t%v\n", err.Error())
return nil, err
}
return conn, err
}
If I use Postman to perform a GET to GAE website then it I get
<html>
<head>
<meta http-equiv="content-type" content="text/html;charset=utf-8">
<title>500 Server Error</title>
</head>
<body text=#000000 bgcolor=#ffffff>
<h1>Error: Server Error</h1>
<h2>The server encountered an error and could not complete your request.<p>Please try again in 30 seconds.</h2>
<h2></h2>
</body>
</html>
The corresponding GAE error stack trace sample is
runtime error: invalid memory address or nil pointer dereference [signal SIGSEGV: segmentation violation code=0x1 addr=0x8 pc=0xaabec7]
at
github.com/jackc/pgx/v4.(*Conn).exec (conn.go:413)
at
github.com/jackc/pgx/v4.(*Conn).Exec (conn.go:396)
at
github.com/jackc/pgx/v4.(*Conn).Ping (conn.go:347)
at
main.connectDB (main.go:41)
at
main.main (main.go:15)
When I view the log I see...
Unable to connection to database: failed to connect to `host=35.236.60.144 user=postgres database=goapi`: dial error (dial tcp 35.236.60.144:5432: connect: connection timed out)
There are two go apps, one is stapi listening on port 8050 and providing RESTful APIs, another is client to consume those APIs.
Both are running on different servers, client is often getting error when calling APIs with HTTP POST method. Below are few lines from client log (real IP replaced with imaginary one)
2018/02/17 11:42:58 ERROR: [DoLogin] API Error: [Post https://123.123.123.123:8050/v1/st/verifyuser: dial tcp 123.123.123.123:8050: getsockopt: connection refused]
2018/02/17 11:47:14 ERROR: [CreateAttempt] Error: [Post https://123.123.123.123:8050/v1/userattempts/createattempt: dial tcp 123.123.123.123:8050: getsockopt: connection refused]
It is intermittent and making the app unreliable, out of approx 1k request i got such error for approx 50+ request.
Initially stapi was listening on all IPs
httpSrv := http.Server{
Addr: ":8050",
Handler: router, // < gin router
...
}
But after reading the workaroung in Golang HTTP Post error: connection refused i modified the stapi app and make it listening on different IPs, as shown below
$ sudo lsof -i -P -n | grep LISTEN
stapi 4775 samtech 10u IPv4 2388179 0t0 TCP 123.123.123.123:8050 (LISTEN)
stapi 4775 samtech 11u IPv6 2388181 0t0 TCP [::1]:8050 (LISTEN)
stapi 4775 samtech 12u IPv4 2388183 0t0 TCP 127.0.0.1:8050 (LISTEN)
But still the issue is same, what else i should check and fix ? Please suggest.
API is protected with JWT, here is how client is making POST requests
func (w *OST) DoLogin(c *gin.Context) {
...
ud := stapimodels.UserLogin{}
err := c.BindJSON(&ud)
...
//call api to save user response
url := config.AppConfig.APIBaseURL + "st/verifyuser"
res, err := api.JwtApi.APIPost(url, &ud)
if err != nil {
g.Logger.Errorm("DoLogin", "Error: %v", err)
t.Error("Error", err.Error())
return
}
...
}
//APIPost - call given apiurl with POST method and pass data
func (j *JwtAPI) APIPost(apiurl string, postdata interface{}) (*APIResult, error) {
if postdata == nil {
return nil, fmt.Errorf("postdata is nil")
}
jsondata, err := toJSON(postdata)
if err != nil {
return nil, err
}
resp, err := j.makeRequest(http.MethodPost, apiurl, jsondata)
if err != nil {
return nil, err
}
defer resp.Body.Close()
res := APIResult{}
json.NewDecoder(resp.Body).Decode(&res)
return &res, nil
}
//makeRequest makes http request for given url with given method
// also inject Authorization Header
func (j *JwtAPI) makeRequest(method, apiurl string, body io.Reader) (*http.Response, error) {
retry := 0
//Create []byte buffer from body - so it can be passed in further retries
var buf []byte
if body != nil {
buf, _ = ioutil.ReadAll(body)
}
r, err := http.NewRequest(method, apiurl, bytes.NewReader(buf))
if err != nil {
return nil, err
}
r.Header.Set("Authorization", "bearer "+j.token.AccessToken)
r.Header.Set("Content-Type", "application/json")
client := j.getClient()
resp, err := client.Do(r)
if err != nil {
return nil, err
}
return resp, nil
}
func (j *JwtAPI) getClient() *http.Client {
// default timeout (if not set by client)
timeoutInSec := 10
if j.Timeout.Seconds() > 0 {
// client sets timeout, so use it
timeoutInSec = int(j.Timeout.Seconds())
}
client := &http.Client{
Timeout: time.Second * time.Duration(timeoutInSec),
}
return client
}
To make your code more resilient you should add some retries with back-offs, so even when the connection was refused it is still working.
Connection refused means that the port is not opened. Is there any firewall or proxies in between? The authentication part shouldn't matter here because it doesn't even get to this point.
Some things that you can check:
Make sure the service is running
Check for firewall configuration
Implement retries for resilience
Is the IP-Address fixed? Is Dynamic DNS used and maybe not updated?
Package for back-off retrying
As for implementing the back-off you might try this package:
https://github.com/cenkalti/backoff
It is listing examples on how to use it and it's pretty much exactly what you need:
// An operation that may fail.
operation := func() error {
// do the request here and check the response code (or check the response body depending on your need) . e.g. above 500 should retry, above 400 and below 500, it should be a client side error and retrying might not help much
return nil // or an error
}
err := Retry(operation, NewExponentialBackOff())
if err != nil {
// Handle error.
return
}
// Operation is successful.
I'm trying to connect a go client to mongodb server running with ssl enabled. I get a clear error message indicating that the hand shake failed due to ssl error. I use a self signed certificate on the client side.
Got below from the mongodb server:
2017-05-13T04:38:53.910+0000 I NETWORK [thread1] connection accepted from 172.17.0.1:51944 #10 (1 connection now open)
2017-05-13T04:38:53.911+0000 E NETWORK [conn10] SSL: error:14094412:SSL routines:SSL3_READ_BYTES:sslv3 alert bad certificate
2017-05-13T04:38:53.911+0000 I - [conn10] end connection
Error from Go client:
Could not connect to mongodb_s1.dev:27017 x509: certificate signed by unknown authority (possibly because of "crypto/rsa: verification error" while trying to verify candidate authority certificate "XYZ")
Tried multiple options, but didn't help
You can skip TLS security checks using InsecureSkipVerify = true. This allows you to use self-signed certificates. See the code from compose help below.
Instead of skipping security checks, it is advisable to add the CA used to sign your certificates to the list of trusted CAs of the system.
package main
import (
"crypto/tls"
"fmt"
"net"
"os"
"strings"
"gopkg.in/mgo.v2"
)
func main() {
uri := os.Getenv("MONGODB_URL")
if uri == "" {
fmt.Println("No connection string provided - set MONGODB_URL")
os.Exit(1)
}
uri = strings.TrimSuffix(uri, "?ssl=true")
Here:
tlsConfig := &tls.Config{}
tlsConfig.InsecureSkipVerify = true
dialInfo, err := mgo.ParseURL(uri)
if err != nil {
fmt.Println("Failed to parse URI: ", err)
os.Exit(1)
}
And here:
dialInfo.DialServer = func(addr *mgo.ServerAddr) (net.Conn, error) {
conn, err := tls.Dial("tcp", addr.String(), tlsConfig)
return conn, err
}
session, err := mgo.DialWithInfo(dialInfo)
if err != nil {
fmt.Println("Failed to connect: ", err)
os.Exit(1)
}
defer session.Close()
dbnames, err := session.DB("").CollectionNames()
if err != nil {
fmt.Println("Couldn't query for collections names: ", err)
os.Exit(1)
}
fmt.Println(dbnames)
}