MutableSideEffect() panics when setting second value - cadence-workflow

We're in the process of writing a .NET Cadence client and are a bit confused with how MutableSideEffect() is supposed to work. We've been thinking of the ID being passed as essentially a variable name and that developers should be able to update mutable values in a workflow. When we try this though, the second MutableSideEffect() call fails with this panic:
panic: adding duplicate decision DecisionType: Marker, ID: MutableSideEffect_value-1, state=Created, isDone()=false, history=[Created]
We munged the greetings workflow sample to make these calls:
package main
import (
"fmt"
"math/rand"
"time"
"go.uber.org/cadence/activity"
"go.uber.org/cadence/workflow"
"go.uber.org/zap"
)
/**
* This greetings sample workflow executes 3 activities in sequential. It gets greeting and name from 2 different activities,
* and then pass greeting and name as input to a 3rd activity to generate final greetings.
*/
// ApplicationName is the task list for this sample
const ApplicationName = "greetingsGroup"
// This is registration process where you register all your workflows
// and activity function handlers.
func init() {
workflow.Register(SampleGreetingsWorkflow)
activity.Register(getGreetingActivity)
activity.Register(getNameActivity)
activity.Register(sayGreetingActivity)
}
// SampleGreetingsWorkflow Workflow Decider.
func SampleGreetingsWorkflow(ctx workflow.Context) error {
// Get Greeting.
ao := workflow.ActivityOptions{
ScheduleToStartTimeout: time.Minute,
StartToCloseTimeout: time.Minute,
HeartbeatTimeout: time.Second * 20,
}
ctx = workflow.WithActivityOptions(ctx, ao)
logger := workflow.GetLogger(ctx)
var greetResult string
err := workflow.ExecuteActivity(ctx, getGreetingActivity).Get(ctx, &greetResult)
if err != nil {
logger.Error("Get greeting failed.", zap.Error(err))
return err
}
f := func(ctx workflow.Context) interface{} {
return rand.Intn(100)
}
e := func(a, b interface{}) bool {
if a == b {
return true
}
return false
}
var result int
sideEffectValue := workflow.MutableSideEffect(ctx, "value-1", f, e)
err = sideEffectValue.Get(&result)
if err != nil {
panic(err)
}
logger.Debug("MutableSideEffect-1", zap.Int("Value", result))
//************** THIS CALL FAILS **************
sideEffectValue = workflow.MutableSideEffect(ctx, "value-1", f, e)
err = sideEffectValue.Get(&result)
if err != nil {
panic(err)
}
logger.Debug("MutableSideEffect-2", zap.Int("Value", result))
// Get Name.
var nameResult string
err = workflow.ExecuteActivity(ctx, getNameActivity).Get(ctx, &nameResult)
if err != nil {
logger.Error("Get name failed.", zap.Error(err))
return err
}
// Say Greeting.
var sayResult string
err = workflow.ExecuteActivity(ctx, sayGreetingActivity, greetResult, nameResult).Get(ctx, &sayResult)
if err != nil {
logger.Error("Marshalling failed with error.", zap.Error(err))
return err
}
logger.Info("Workflow completed.", zap.String("Result", sayResult))
return nil
}
// Get Name Activity.
func getNameActivity() (string, error) {
return "Cadence", nil
}
// Get Greeting Activity.
func getGreetingActivity() (string, error) {
return "Hello", nil
}
// Say Greeting Activity.
func sayGreetingActivity(greeting string, name string) (string, error) {
result := fmt.Sprintf("Greeting: %s %s!\n", greeting, name)
return result, nil
}
Are we thinking about this correctly?

This is a bug in the Go client library. It happens when a MutableSideEffect with the same id is used multiple times during a single decision.
If you force a separate decision by putting workflow.Sleep(ctx, time.Second) just before the second MutableSideEffect call the problem disappears.
I filed an issue to get this fixed.
Thanks a lot for reporting!

Related

Is there a way with GRPC to notify the stream of CRUD operations to give realtime updates to the client?

I am new to GRPC and I am trying to implement a basic CRUD + listing. I use unary rpc's for the CRUD and a server stream for the listing. What I would like to do however is update the client whenever someone changes a record in the database that you are listing.
So for example user A is listing 10 companies. And user B is updating one of those companies. I want user A's client to be updated once the update rpc is called.
This is what I have for now
func RegisterCompanyServer(l hclog.Logger, gs *grpc.Server) {
r := postgres.NewPostgresCompanyRepository()
cs := NewCompanyServer(l, r)
pb.RegisterCompanyServiceServer(gs, cs)
}
type CompanyServer struct {
logger hclog.Logger
repo repo.CompanyRepository
pb.UnimplementedCompanyServiceServer
}
func NewCompanyServer(l hclog.Logger, r repo.CompanyRepository) *CompanyServer {
return &CompanyServer{
logger: l,
repo: r,
}
}
func (c *CompanyServer) ListCompany(req *pb.CompanyListRequest, stream pb.CompanyService_ListCompanyServer) error {
//Somehow listen to CreateCompany() and update the client
companies, err := c.repo.List(req.Query)
if err != nil {
return err
}
for _, c := range companies {
bytes, err := json.Marshal(c)
if err != nil {
return err
}
out := &pb.Company{}
if err = jsonEnc.Unmarshal(bytes, out); err != nil {
return err
}
res := &pb.CompanyListResponse{
Company: out,
}
err = stream.Send(res)
if err != nil {
return err
}
}
return nil
}
func (c *CompanyServer) CreateCompany(context context.Context, req *pb.CompanyCreateRequest) (*pb.CompanyCreateResponse, error) {
input := req.GetCompany()
if input == nil {
return nil, errors.New("Parsing Error")
}
bytes, err := jsonEnc.Marshal(input)
if err != nil {
return nil, err
}
company := &myCompany.Company{}
if err = json.Unmarshal(bytes, company); err != nil {
return nil, err
}
result, err := c.repo.Create(company)
if err != nil {
return nil, err
}
res := &pb.CompanyCreateResponse{
Id: result,
}
//Somehow notify the stream that a company was created
return res, nil
}
Is this even feasable with GRPC? What techniques are out there to do this? I am currently working with a postgresql database.

Go Gin Setting and Accessing context value from middleware

I am trying to set my user context, in the middleware then trying to check if user have permission in other handler functions. But for some reason when I try to access the user from context it is coming back as nils. The middleware code seems to be working, when I pass a valid jwt token, it is showing the user is being set in context in the middleware function. But as soon as I hit getCurrentUser function it says it's nil.
Here is the code:
Middleware
// Middleware wraps the request with auth middleware
func Middleware(path string, sc *cfg.Server, orm *orm.ORM) gin.HandlerFunc {
logger.Info("[Auth.Middleware] Applied to path: ", path)
return gin.HandlerFunc(func(c *gin.Context) {
t, err := ParseToken(c, sc)
if err != nil {
authError(c, err)
} else {
if claims, ok := t.Claims.(jwt.MapClaims); ok {
if claims["exp"] != nil {
issuer := claims["iss"].(string)
userid := claims["jti"].(string)
email := claims["email"].(string)
if claims["aud"] != nil {
audiences := claims["aud"].(interface{})
logger.Warnf("\n\naudiences: %s\n\n", audiences)
}
if claims["alg"] != nil {
algo := claims["alg"].(string)
logger.Warnf("\n\nalgo: %s\n\n", algo)
}
if user, err := orm.FindUserByJWT(email, issuer, userid); err != nil {
authError(c, ErrForbidden)
} else {
if user != nil {
c.Request = addToContext(c, consts.ProjectContextKeys.UserCtxKey, user)
logger.Debug("User: ", user.ID)
}
c.Next()
}
} else {
authError(c, ErrMissingExpField)
}
} else {
authError(c, err)
}
}
})
}
routes
// User routes
func User(sc *cfg.Server, r *gin.Engine, orm *orm.ORM) error {
// OAuth handlers
mw := auth.Middleware(sc.VersionedEndpoint("/user/:id"), sc, orm)
g := r.Group(sc.VersionedEndpoint("/user"))
g.Use(mw)
g.GET("/:id", mw, user.Get(orm))
g.PUT("/:id", mw, user.Update(orm))
g.POST("/", user.Create(orm))
return nil
}
handler
func Get(orm *orm.ORM) gin.HandlerFunc {
return func(ctx *gin.Context) {
cu := getCurrentUser(ctx)
if ok, err := cu.HasPermission(consts.Permissions.Create, consts.EntityNames.Users); !ok || err != nil {
ctx.String(http.StatusUnauthorized, "BAD")
}
}
}
addToContext:
func addToContext(c *gin.Context, key consts.ContextKey, value interface{}) *http.Request {
return c.Request.WithContext(context.WithValue(c.Request.Context(), key, value))
}
getCurrentUser:
func getCurrentUser(ctx context.Context) *dbm.User {
cu := ctx.Value(utils.ProjectContextKeys.UserCtxKey).(*dbm.User)
logger.Debugf("currentUser: %s - %s", cu.Email, cu.ID)
return cu
}
The problem is that you're storing the user in one context but then you're attempting to retrieve the user from another context. The value *gin.Context and the value *gin.Context.Request.Context are two separate context values.
You're using the Request's context to store the user:
c.Request.WithContext(context.WithValue(c.Request.Context(), key, value))
And then you're using the gin context to retrieve the user:
func getCurrentUser(ctx context.Context) *dbm.User {
cu := ctx.Value(utils.ProjectContextKeys.UserCtxKey).(*dbm.User)
// ...
func Get(orm *orm.ORM) gin.HandlerFunc {
return func(ctx *gin.Context) {
cu := getCurrentUser(ctx) // here you're passing *gin.Context to the function.
// ...
So to fix that change the value that's passed in to the getCurrentUser call to:
func Get(orm *orm.ORM) gin.HandlerFunc {
return func(ctx *gin.Context) {
cu := getCurrentUser(ctx.Request.Context())
if ok, err := cu.HasPermission(consts.Permissions.Create, consts.EntityNames.Users); !ok || err != nil {
ctx.String(http.StatusUnauthorized, "BAD")
}
}
}

Post Request with PostgreSQL and json-api returns an empty body

After a POST request, i was expecting to have a last inserted record marshalled into json, but instead returns an empty body. What am i not doing well?
package models
import (
"encoding/json"
"errors"
"flag"
"fmt"
"log"
"net/http"
"strconv"
"github.com/go-chi/chi"
"github.com/google/jsonapi"
"github.com/thedevsaddam/renderer"
"github.com/xo/dburl"
)
var rnd = renderer.New()
var flagVerbose = flag.Bool("v", false, "verbose")
var FlagURL = flag.String("url", "postgres://postgres:#127.0.0.1/sweb", "url")
// Page represents a row from 'public.pages'.
type Page struct {
Tag string `jsonapi:"attr,tag"` // tag
Body string `jsonapi:"attr,body"` // body
Slug string `jsonapi:"attr,slug"` // slug
Title string `jsonapi:"attr,title"` // title
ID int `jsonapi:"primary,pages"` // id
Link string `jsonapi:"attr,link"` // link
// xo fields
_exists, _deleted bool
}
func (page Page) JSONAPILinks() *jsonapi.Links {
return &jsonapi.Links{
"self": fmt.Sprintf("https://%d", page.ID),
}
}
I believe this is the culprit. After inserting a record, it should return the last inserted record as specified.
func (p *Page) PInsert(db XODB) (*Page, error) {
var err error
// if already exist, bail
if p._exists {
return p, errors.New("insert failed: already exists")
}
// sql insert query, primary key provided by sequence
const sqlstr = `INSERT INTO public.pages (` +
`tag, body, slug, title` +
`) VALUES (` +
`$1, $2, $3, $4` +
`) RETURNING id, tag, body, title`
// run query
XOLog(sqlstr, p.Tag, p.Body, p.Slug, p.Title)
err = db.QueryRow(sqlstr, p.Tag, p.Body, p.Slug, p.Title).Scan(&p.ID, &p.Tag, &p.Body, &p.Title)
if err != nil {
return p, err
}
// set existence
p._exists = true
return p, nil
}
Update updates the Page in the database and return last inserted records.
The same should apply for the Update function
func (p *Page) Update(db XODB) (*Page, error) {
var err error
// if doesn't exist, bail
if !p._exists {
return p, errors.New("update failed: does not exist")
}
// if deleted, bail
if p._deleted {
return p, errors.New("update failed: marked for deletion")
}
// sql query
const sqlstr = `UPDATE public.pages SET (` +
`tag, body, slug, title` +
`) = ( ` +
`$1, $2, $3, $4` +
`) WHERE id = $5`
// run query
XOLog(sqlstr, p.Tag, p.Body, p.Slug, p.Title, p.ID)
_, err = db.Exec(sqlstr, p.Tag, p.Body, p.Slug, p.Title, p.ID)
return p, err
}
func (p *Page) PSave(db XODB) (*Page, error) {
if p.Exists() {
return p.Update(db)
}
return p.PInsert(db)
}
func NewPage(w http.ResponseWriter, r *http.Request) {
db, err := dburl.Open(*FlagURL)
defer db.Close()
if err != nil {
log.Fatal(err)
}
var page Page
//page := new(Page)
if err := jsonapi.UnmarshalPayload(r.Body, &page); err != nil {
http.Error(w, err.Error(), http.StatusInternalServerError)
return
}
p, err := page.PSave(db)
if err != nil {
fmt.Println(err)
if err := jsonapi.MarshalPayload(w, p); err != nil {
http.Error(w, err.Error(), http.StatusInternalServerError)
fmt.Println(err)
}
}
w.Header().Set("Content-Type", jsonapi.MediaType)
w.WriteHeader(http.StatusCreated)
}
This is the last function i believe the issue is happening from. the last inserted record supposed to be marshalled into json.
Your last section of code contains a number of mistakes. The relevant section (without the useless and obfuscating Printlns) is:
p, err := page.PSave(db)
if err != nil {
if err := jsonapi.MarshalPayload(w, p); err != nil {
http.Error(w, err.Error(), http.StatusInternalServerError)
}
}
w.Header().Set("Content-Type", jsonapi.MediaType)
w.WriteHeader(http.StatusCreated)
And the primary mistake is that json.MarshalPayload is only called when err != nil. In other words, you only serialize the page if you failed to save it.
The secondary mistake is that jsonapi.MarshalPayload will call Write on the http.ResponseWriter. This turns all subsequent calls to Header().Set and WriteHeader into no-ops.
More correct code would look like this.
// 1. Save the page in the database, bail on error
p, err := page.PSave(db)
if err != nil {
http.Error(w, err.Error(), http.StatusInternalServerError)
return
}
// 2. Marshal the page into an intermediate buffer, bail on error
var buf bytes.Buffer
if err := jsonapi.MarshalPayload(&buf, p); err != nil {
http.Error(w, err.Error(), http.StatusInternalServerError)
return
}
// 3. Write the entire response; failures to write the intermediate buffer
// cannot be communicated over HTTP
w.Header().Set("Content-Type", jsonapi.MediaType)
w.WriteHeader(http.StatusCreated)
if _, err := buf.WriteTo(w); err != nil {
log.Printf("failed to write response: %v", err)
return
}

Convert type from struct table to base.FixedDataGrid in GO

i'm having a trouble converting my struct table to fixedDataGrid, because i need my data to be a fixedDataGrid so that i can use machine learning methods from GoLearn lib.
My struct is like this:
type dataStruct struct{
Sepal_length string
Sepal_width string
Petal_length string
Petal_width string
Species string
}
So when i get my data from my mongo db, i get them like this:
var results []dataStruct
err := col.Find(nil).All(&results)
Is there a way to convert my "results" from []dataStruct type to base.FixedDataGrid ??
CreateModel function:
func CreateModel(c echo.Context) error {
fmt.Println("====> Entry CreateModel function");
//var results []dataStruct
var Success bool = false
Db := db.MgoDb{}
Db.Init()
defer Db.Close()
col := Db.C(db.TrainingDataCollection)
var results dataStruct
if err := col.Find(nil).All(results); err != nil {
fmt.Println("ERROR WHILE GETTING THE TRAINING DATA")
} else {
//fmt.Println("Results All: ", results)
Success = true
}
fmt.Println("=============",results)
//Initialises a new KNN classifier
cls := knn.NewKnnClassifier("euclidean", "linear", 2)
//Do a training-test split
trainData, testData := base.InstancesTrainTestSplit(results, 0.55)
cls.Fit(trainData)
//Calculates the Euclidean distance and returns the most popular label
predictions, err := cls.Predict(testData)
if err != nil {
panic(err)
}
fmt.Println(predictions)
// Prints precision/recall metrics
confusionMat, err := evaluation.GetConfusionMatrix(testData, predictions)
if err != nil {
panic(fmt.Sprintf("Unable to get confusion matrix: %s", err.Error()))
}
fmt.Println(evaluation.GetSummary(confusionMat))
return c.JSON(http.StatusOK, Success)
}
Thank you in advance for your help !
Here is how i solved the issue: Actually there is a function InstancesFromMat64(row int, col int, matrix) than creates instances from a float64 matrix, and this is what i used:
func CreateModel(c echo.Context) error {
fmt.Println("====> Entry CreateModel function");
var Success bool = false
Db := db.MgoDb{}
Db.Init()
defer Db.Close()
col := Db.C(db.TrainingDataCollection)
var results dataStruct
if err := col.Find(nil).All(&results); err != nil {
fmt.Println("ERROR WHILE GETTING THE TRAINING DATA")
} else {
Success = true
}
Data := make([]float64, len(results*nbAttrs)
/**** Filling the Data var with my dataset data *****/
mat := mat64.NewDense(row,nbAttrs,Data)
inst := base.InstancesFromMat64(row,nbAttrs,mat)
//Selecting the class attribute for our instance
attrs := inst.AllAttributes()
inst.AddClassAttribute(attrs[4])
//Initialise a new KNN classifier
cls := knn.NewKnnClassifier("manhattan","linear",3)
//Training-tessting split
trainData, testData := base.InstancesTrainTestSplit(inst,0.7)
/******* Continue the Model creation ******/
I'll be glad if my answer helps someone.
Thanks a lot #mkopriva for your help !
base.FixedDataGrid is an interface, so what you need to do is to implement that interface, that is, implement all of its methods, on the type you want to use as FixedDataGrid.
Since you want to use []dataStruct, a slice of dataStructs, which is an unnamed type, as FixedDataGrid you will have to declare a new type to be able to add methods to it because you can add methods only to named types. For example something like this:
type dataStructList []dataStruct
Now, if you take a look at the documentation, you can see that the FixedDataGrid interface declares two methods RowString and Size but also embeds another interface, the base.DataGrid interface, which means you need to implement the methods declared by DataGrid as well. So, given your new dataStructList type, you can do something like this:
func (l dataStructList) RowString(int) string { /* ... */ }
func (l dataStructList) Size() (int, int) { /* ... */ }
func (l dataStructList) GetAttribute(base.Attribute) (base.AttributeSpec, error) { /* ... */ }
func (l dataStructList) AllAttributes() []base.Attribute { /* ... */ }
func (l dataStructList) AddClassAttribute(base.Attribute) error { /* ... */ }
func (l dataStructList) RemoveClassAttribute(base.Attribute) error { /* ... */ }
func (l dataStructList) AllClassAttributes() []base.Attribute { /* ... */ }
func (l dataStructList) Get(base.AttributeSpec, int) []byte { /* ... */ }
func (l dataStructList) MapOverRows([]base.AttributeSpec, func([][]byte, int) (bool, error)) error { /* ... */ }
After you've implemented the /* ... */ parts you can then start using dataStructList as a FixedDataGrid, so something like this:
var results []dataStruct
err := col.Find(nil).All(&results)
fdg := dataStructList(results) // you can use fdg as FixedDataGrid
Or
var results dataStructList // you can use results as FixedDataGrid
err := col.Find(nil).All(&results)
Update:
After you've implemented all of those methods on the dataStructList all you need is the type of the results variable inside your function:
func CreateModel(c echo.Context) error {
fmt.Println("====> Entry CreateModel function")
//var results []dataStruct
var Success bool = false
Db := db.MgoDb{}
Db.Init()
defer Db.Close()
col := Db.C(db.TrainingDataCollection)
var results dataStructList // <--- use the type that implements the interface
if err := col.Find(nil).All(&results); err != nil { // <-- pass a pointer to results
fmt.Println("ERROR WHILE GETTING THE TRAINING DATA")
} else {
//fmt.Println("Results All: ", results)
Success = true
}
fmt.Println("=============", results)
//Initialises a new KNN classifier
cls := knn.NewKnnClassifier("euclidean", "linear", 2)
//Do a training-test split
trainData, testData := base.InstancesTrainTestSplit(results, 0.55) // <-- this will work because results if of type dataStructList, which implements the base.FixedDataGrid interface.
cls.Fit(trainData)
//Calculates the Euclidean distance and returns the most popular label
predictions, err := cls.Predict(testData)
if err != nil {
panic(err)
}
fmt.Println(predictions)
// Prints precision/recall metrics
confusionMat, err := evaluation.GetConfusionMatrix(testData, predictions)
if err != nil {
panic(fmt.Sprintf("Unable to get confusion matrix: %s", err.Error()))
}
fmt.Println(evaluation.GetSummary(confusionMat))
return c.JSON(http.StatusOK, Success)
}

How to create elements of a struct with a Mutex field

I have a Get() function:
func Get(url string) *Response {
res, err := http.Get(url)
if err != nil {
return &Response{}
}
// res.Body != nil when err == nil
defer res.Body.Close()
body, err := ioutil.ReadAll(res.Body)
if err != nil {
log.Fatalf("ReadAll: %v", err)
}
reflect.TypeOf(body)
return &Response{sync.Mutex(),string(body), res.StatusCode}
}
as well as a Read() function:
func Read(url string, timeout time.Duration) (res *Response) {
done := make(chan bool)
go func() {
res = Get(url)
done <- true
}()
select { // As soon as either
case <-done: // done is sent on the channel or
case <-time.After(timeout): // timeout
res = &Response{"Gateway timeout\n", 504}
}
return
}
the Response type returned by the functions is defined as:
type Response struct {
Body string
StatusCode int
}
This read function makes use of the Get() function and also implements a timeout. The problem is that a data race can occur if the timeout occurs and the Get() response is written to res at the same time in Read().
I have a plan for how to solve this. It is to use Mutex. To do this, I would add a field to the Response struct:
type Response struct {
mu sync.Mutex
Body string
StatusCode int
}
so that the Response can be locked. However, I'm not sure how to fix this in the other parts of the code.
My attempt looks like this, for the Get():
func Get(url string) *Response {
res, err := http.Get(url)
if err != nil {
return &Response{}
}
// res.Body != nil when err == nil
defer res.Body.Close()
body, err := ioutil.ReadAll(res.Body)
if err != nil {
log.Fatalf("ReadAll: %v", err)
}
reflect.TypeOf(body)
return &Response{sync.Mutex(),string(body), res.StatusCode} // This line is changed.
}
and for the Read():
func Read(url string, timeout time.Duration) (res *Response) {
done := make(chan bool)
res = &Response{sync.Mutex()} // this line has been added
go func() {
res = Get(url)
done <- true
}()
select {
case <-done:
case <-time.After(timeout):
res.mu.Lock()
res = &Response{sync.Mutex(), "Gateway timeout\n", 504} // And mutex was added here.
}
defer res.mu.Unlock()
return
}
This "solution" generates these errors:
./client.go:54: missing argument to conversion to sync.Mutex: sync.Mutex()
./client.go:63: missing argument to conversion to sync.Mutex: sync.Mutex()
./client.go:63: too few values in struct initializer
./client.go:73: missing argument to conversion to sync.Mutex: sync.Mutex()
./client.go:95: cannot use "Service unavailable\n" (type string) as type sync.Mutex in field value
./client.go:95: cannot use 503 (type int) as type string in field value
./client.go:95: too few values in struct initializer
What is the correct way of using Mutex in this case?
While your answer with Volker's guidance is good, you might want to consider using a non default http.Client so that you can set a Timeout on the client making the request (then you don't have to worry about handling the timeouts yourself).
I followed Volker's suggestion and used a channel to solve the problem.
func Read(url string, timeout time.Duration) (res *Response) {
done := make(chan bool) // A channel
resChan := make(chan *Response)
go func() {
resChan <- Get(url)
done <- true
}()
select {
case <-done:
res = &Response{}
case <-time.After(timeout):
res = &Response{"Gateway timeout\n", 504}
}
return
}
Now, there can be no simultaneous writes to res. It's going to be either the timeout or the returned value of Get(url).