Stream DB data as soon as table is changed with gRPC - postgresql

I am making Task List using Go, gRPC and Postgres.
How can I automatically stream the data as soon as PostItem is called to insert the new data? Do I need to subscribe the Postgres or can I accomplish this without subscription or pub-sub?
// ProtoBuf schema
syntax = "proto3";
package tasklist;
import "google/protobuf/empty.proto";
service TodoList {
rpc GetTasks(google.protobuf.Empty) returns (stream GetTasksResponse) {}
rpc PostItem(PostItemRequest) returns (PostTaskRequest) {}
}
message Task {
int64 id = 1;
string name = 2;
}
message GetTasksResponse {
Task task = 1;
}
message PostTaskRequest {
Task Task = 1;
}
message PostItemResponse {
bool result = 1;
}
// Postgres Table Schema
create table Task (
id integer not null PRIMARY KEY,
name varchar(10) not null
);
// Go
func (s *server) GetTasks(_ *empty.Empty, stream pb.TaskList_GetTasksServer) error {
// How can I steam data as soon as `PostTask` is called to update db? <- <-
for _, r := range s.requests {
// stream data
}
}
func (s *server) PostTask(ctx context.Context, r *pb.PostTaskRequest) (*pb.PostTaskResponse, error) {
// update Postgres here
return &pb.PostItemResponse{Result: true}, nil
}

I guess s.requests is something like chan Task. So after successful // update Postgres here you might send your request in chan.
func (s *server) PostTask(ctx context.Context, r *pb.PostTaskRequest) (*pb.PostTaskResponse, error) {
postTask := toDomain(r)
err := s.service.UpdateTask(ctx, postTask)
if err != nil {
return nil, status.Error(codes.Internal, err.Error())
}
s.requests <- postTask
return &pb.PostItemResponse{Result: true}, nil
}

Related

pgx preparedStatement execution does not return results (unlike fmt.Sprintf)

For some reason, the prepareStatement I built in go using pgx is not returning any results/rows (see scenario1). In scenario 2, if I build the same query using Sprintf and $, the results are also not returned.
But, as shown in scenario 3, if I build the query with a Sprintf with %s placeholders and execute it, it will return results/rows.
I would like to build/execute the query using the "logic" of scenario 1 (with prepared statement instead with Sprintf). Any ideas?
I tested the query directly on the DB (with pgadmin) and it has data/results.
Assumption:
I suspect it might have something to do with:
in the WHERE condition: the uuid data type or with the string in the IN condition
the query result that returns a uuid and a jsonb
Details:
The DB is postgres and the table t has, among others, the field types:
t.str1 and t.properties are string
t.my_uuid is uuid.UUID (https://github.com/satori/go.uuid)
t.C is jsonb
the pgx version I'm using is: github.com/jackc/pgx v3.6.2+incompatible
scenario 1
func prepareQuery(cp *pgx.ConnPool) (string, error) {
prep := `SELECT DISTINCT
A,B,C
FROM table t
WHERE t.str1 = $1
AND t.properties IN ($2)
AND t.my_uuid = $3`
_, err := cp.Prepare("queryPrep", prep)
if err != nil {
return "some error", err
}
return prep, nil
}
func x() {
//...
q, err := prepareQuery(cPool)
if err != nil {
return fmt.Errorf("failed to prepare query %s: %w", q, err)
}
rows, err := cPool.Query("queryPrep", "DumyStr", "'prop1','prop2'", myUUID) //myUUID is a valid satori uuid
if err != nil {
//return error...
}
for rows.Next() { // zero rows
//Scan...
}
//...
}
scenario 2
func makeQueryStr()string {
return fmt.Sprintf(`SELECT DISTINCT
A,B,C
FROM table t
WHERE t.str1 = $1
AND t.properties IN ($2)
AND t.my_uuid = $3`)
}
func x() {
//...
myQuery := makeQueryStr()
rows, err := cPool.Query(myQuery, "DumyStr", "'prop1','prop2'", myUUID) //myUUID is a valid satori uuid
if err != nil {
//return error...
}
for rows.Next() { // zero rows
//Scan...
}
//...
}
scenario 3
func makeQueryStr(par1 string, par2 string, par3 uuid.UUID)string {
return fmt.Sprintf(`SELECT DISTINCT
A,B,C
FROM table t
WHERE t.str1 = '%s'
AND t.properties IN (%s)
AND t.my_uuid = '%s'`, par1, par2, par3)
}
func x() {
//...
myQuery := makeQueryStr("DumyStr", "'prop1','prop2'", myUUID) //myUUID is a valid satori uuid
rows, err := cPool.Query(myQuery)
if err != nil {
//return error...
}
for rows.Next() { // 10 rows
//Scan...
}
//...
}

Why is Postgres complaining 'FATAL #53300 sorry, too many clients already' when I am using only one client?

I have a global instance to Postgres database which is being used throughout the code.
There are 500 goroutines inserting some data into the database using the same instance.
So when there is just one instance (client) why does it fail with panic: FATAL #53300 sorry, too many clients already?
My understanding is that, there is just one instance of the database which is being created by ConnectToDB() method when it is called for the first time & on the subsequent calls, it merely returns this instance instead of creating a new one.
Finally when main ends, that one single instance is closed.
I am not able to understand how could it possibly create too many instances as indicated by the error message.
package main
import (
"github.com/go-pg/pg/v10"
"github.com/go-pg/pg/v10/orm"
"sync"
)
var db *pg.DB
var once sync.Once
type DummyTable struct {
IdOne int
IdTwo int
}
func main() {
var wg sync.WaitGroup
db := ConnectToDB()
defer db.Close()
for i := 0; i < 500; i++ {
wg.Add(1)
go insertIntoDB(&wg)
}
wg.Wait()
}
func insertIntoDB(wg *sync.WaitGroup) {
defer wg.Done()
localDb := ConnectToDB()
_, err := localDb.Model(&DummyTable{
IdOne: 2,
IdTwo: 3,
}).Insert()
if err != nil {
panic(err)
}
}
func createSchema(db *pg.DB) error {
models := []interface{}{
(*DummyTable)(nil),
}
for _, model := range models {
err := db.Model(model).CreateTable(&orm.CreateTableOptions{
Temp: false,
})
if err != nil {
return err
}
}
return nil
}
func ConnectToDB() *pg.DB {
once.Do(func() {
db = pg.Connect(&pg.Options{
User: "username",
Database: "dbName",
})
err := createSchema(db)
if err != nil {
panic(err)
}
})
return db
}

Go Gin Setting and Accessing context value from middleware

I am trying to set my user context, in the middleware then trying to check if user have permission in other handler functions. But for some reason when I try to access the user from context it is coming back as nils. The middleware code seems to be working, when I pass a valid jwt token, it is showing the user is being set in context in the middleware function. But as soon as I hit getCurrentUser function it says it's nil.
Here is the code:
Middleware
// Middleware wraps the request with auth middleware
func Middleware(path string, sc *cfg.Server, orm *orm.ORM) gin.HandlerFunc {
logger.Info("[Auth.Middleware] Applied to path: ", path)
return gin.HandlerFunc(func(c *gin.Context) {
t, err := ParseToken(c, sc)
if err != nil {
authError(c, err)
} else {
if claims, ok := t.Claims.(jwt.MapClaims); ok {
if claims["exp"] != nil {
issuer := claims["iss"].(string)
userid := claims["jti"].(string)
email := claims["email"].(string)
if claims["aud"] != nil {
audiences := claims["aud"].(interface{})
logger.Warnf("\n\naudiences: %s\n\n", audiences)
}
if claims["alg"] != nil {
algo := claims["alg"].(string)
logger.Warnf("\n\nalgo: %s\n\n", algo)
}
if user, err := orm.FindUserByJWT(email, issuer, userid); err != nil {
authError(c, ErrForbidden)
} else {
if user != nil {
c.Request = addToContext(c, consts.ProjectContextKeys.UserCtxKey, user)
logger.Debug("User: ", user.ID)
}
c.Next()
}
} else {
authError(c, ErrMissingExpField)
}
} else {
authError(c, err)
}
}
})
}
routes
// User routes
func User(sc *cfg.Server, r *gin.Engine, orm *orm.ORM) error {
// OAuth handlers
mw := auth.Middleware(sc.VersionedEndpoint("/user/:id"), sc, orm)
g := r.Group(sc.VersionedEndpoint("/user"))
g.Use(mw)
g.GET("/:id", mw, user.Get(orm))
g.PUT("/:id", mw, user.Update(orm))
g.POST("/", user.Create(orm))
return nil
}
handler
func Get(orm *orm.ORM) gin.HandlerFunc {
return func(ctx *gin.Context) {
cu := getCurrentUser(ctx)
if ok, err := cu.HasPermission(consts.Permissions.Create, consts.EntityNames.Users); !ok || err != nil {
ctx.String(http.StatusUnauthorized, "BAD")
}
}
}
addToContext:
func addToContext(c *gin.Context, key consts.ContextKey, value interface{}) *http.Request {
return c.Request.WithContext(context.WithValue(c.Request.Context(), key, value))
}
getCurrentUser:
func getCurrentUser(ctx context.Context) *dbm.User {
cu := ctx.Value(utils.ProjectContextKeys.UserCtxKey).(*dbm.User)
logger.Debugf("currentUser: %s - %s", cu.Email, cu.ID)
return cu
}
The problem is that you're storing the user in one context but then you're attempting to retrieve the user from another context. The value *gin.Context and the value *gin.Context.Request.Context are two separate context values.
You're using the Request's context to store the user:
c.Request.WithContext(context.WithValue(c.Request.Context(), key, value))
And then you're using the gin context to retrieve the user:
func getCurrentUser(ctx context.Context) *dbm.User {
cu := ctx.Value(utils.ProjectContextKeys.UserCtxKey).(*dbm.User)
// ...
func Get(orm *orm.ORM) gin.HandlerFunc {
return func(ctx *gin.Context) {
cu := getCurrentUser(ctx) // here you're passing *gin.Context to the function.
// ...
So to fix that change the value that's passed in to the getCurrentUser call to:
func Get(orm *orm.ORM) gin.HandlerFunc {
return func(ctx *gin.Context) {
cu := getCurrentUser(ctx.Request.Context())
if ok, err := cu.HasPermission(consts.Permissions.Create, consts.EntityNames.Users); !ok || err != nil {
ctx.String(http.StatusUnauthorized, "BAD")
}
}
}

MutableSideEffect() panics when setting second value

We're in the process of writing a .NET Cadence client and are a bit confused with how MutableSideEffect() is supposed to work. We've been thinking of the ID being passed as essentially a variable name and that developers should be able to update mutable values in a workflow. When we try this though, the second MutableSideEffect() call fails with this panic:
panic: adding duplicate decision DecisionType: Marker, ID: MutableSideEffect_value-1, state=Created, isDone()=false, history=[Created]
We munged the greetings workflow sample to make these calls:
package main
import (
"fmt"
"math/rand"
"time"
"go.uber.org/cadence/activity"
"go.uber.org/cadence/workflow"
"go.uber.org/zap"
)
/**
* This greetings sample workflow executes 3 activities in sequential. It gets greeting and name from 2 different activities,
* and then pass greeting and name as input to a 3rd activity to generate final greetings.
*/
// ApplicationName is the task list for this sample
const ApplicationName = "greetingsGroup"
// This is registration process where you register all your workflows
// and activity function handlers.
func init() {
workflow.Register(SampleGreetingsWorkflow)
activity.Register(getGreetingActivity)
activity.Register(getNameActivity)
activity.Register(sayGreetingActivity)
}
// SampleGreetingsWorkflow Workflow Decider.
func SampleGreetingsWorkflow(ctx workflow.Context) error {
// Get Greeting.
ao := workflow.ActivityOptions{
ScheduleToStartTimeout: time.Minute,
StartToCloseTimeout: time.Minute,
HeartbeatTimeout: time.Second * 20,
}
ctx = workflow.WithActivityOptions(ctx, ao)
logger := workflow.GetLogger(ctx)
var greetResult string
err := workflow.ExecuteActivity(ctx, getGreetingActivity).Get(ctx, &greetResult)
if err != nil {
logger.Error("Get greeting failed.", zap.Error(err))
return err
}
f := func(ctx workflow.Context) interface{} {
return rand.Intn(100)
}
e := func(a, b interface{}) bool {
if a == b {
return true
}
return false
}
var result int
sideEffectValue := workflow.MutableSideEffect(ctx, "value-1", f, e)
err = sideEffectValue.Get(&result)
if err != nil {
panic(err)
}
logger.Debug("MutableSideEffect-1", zap.Int("Value", result))
//************** THIS CALL FAILS **************
sideEffectValue = workflow.MutableSideEffect(ctx, "value-1", f, e)
err = sideEffectValue.Get(&result)
if err != nil {
panic(err)
}
logger.Debug("MutableSideEffect-2", zap.Int("Value", result))
// Get Name.
var nameResult string
err = workflow.ExecuteActivity(ctx, getNameActivity).Get(ctx, &nameResult)
if err != nil {
logger.Error("Get name failed.", zap.Error(err))
return err
}
// Say Greeting.
var sayResult string
err = workflow.ExecuteActivity(ctx, sayGreetingActivity, greetResult, nameResult).Get(ctx, &sayResult)
if err != nil {
logger.Error("Marshalling failed with error.", zap.Error(err))
return err
}
logger.Info("Workflow completed.", zap.String("Result", sayResult))
return nil
}
// Get Name Activity.
func getNameActivity() (string, error) {
return "Cadence", nil
}
// Get Greeting Activity.
func getGreetingActivity() (string, error) {
return "Hello", nil
}
// Say Greeting Activity.
func sayGreetingActivity(greeting string, name string) (string, error) {
result := fmt.Sprintf("Greeting: %s %s!\n", greeting, name)
return result, nil
}
Are we thinking about this correctly?
This is a bug in the Go client library. It happens when a MutableSideEffect with the same id is used multiple times during a single decision.
If you force a separate decision by putting workflow.Sleep(ctx, time.Second) just before the second MutableSideEffect call the problem disappears.
I filed an issue to get this fixed.
Thanks a lot for reporting!

Golang: fail to connect with MongoDB

I am trying to connect Go application with MongoDB server while running the database using docker.
I'm able to connect to the DB using the shell and perform different actions. However, the Go app fails when connecting to the DB. I'm using mgo driver and below is the code in use where am trying to implement a db middleware that can be used by all routes:
middleware code:
package db
import (
"net/http"
"os"
"github.com/gorilla/context"
"github.com/urfave/negroni"
mgo "gopkg.in/mgo.v2"
)
const key = "dbkey"
func GetDb(r *http.Request) *mgo.Database {
if rv := context.Get(r, key); rv != nil {
return rv.(*mgo.Database)
}
return nil
}
func SetDb(r *http.Request, val *mgo.Database) {
context.Set(r, key, val)
}
func MongoMiddleware() negroni.HandlerFunc {
database := os.Getenv("DB_NAME")
session, err := mgo.Dial("127.0.0.1:27017")
if err != nil {
println(err) // error message below
}
return negroni.HandlerFunc(func(rw http.ResponseWriter, r *http.Request, next http.HandlerFunc) {
reqSession := session.Clone()
defer reqSession.Close()
db := reqSession.DB(database)
SetDb(r, db)
next(rw, r)
})
}
the error am getting is :
panic: runtime error: invalid memory address or nil pointer dereference
route and main package code:
package main
import (
gmux "github.com/gorilla/mux"
"github.com/urfave/negroni"
"github.com/mypro/db"
"github.com/mypro/hub"
)
func main() {
router := gmux.NewRouter()
router.HandleFunc("/name", hub.Create).
Methods("GET")
n := negroni.Classic()
n.Use(db.MongoMiddleware())
n.UseHandler(router)
n.Run(":9000")
}
method that consume the db middleware to find a collection:
type Name struct {
Id bson.ObjectId `bson:"_id"`
Name string `bson:"name"`
}
func Create(w http.ResponseWriter, r *http.Request) {
var aName Name
db := db.GetDb(r)
names := db.C("x")
err := names.Find(bson.M{"name": "sam"}).One(&aName)
if err != nil {
log.Print(err)
}
fmt.Println(&aName)
json.NewEncoder(w).Encode(&aName)
}