How to create elements of a struct with a Mutex field - rest

I have a Get() function:
func Get(url string) *Response {
res, err := http.Get(url)
if err != nil {
return &Response{}
}
// res.Body != nil when err == nil
defer res.Body.Close()
body, err := ioutil.ReadAll(res.Body)
if err != nil {
log.Fatalf("ReadAll: %v", err)
}
reflect.TypeOf(body)
return &Response{sync.Mutex(),string(body), res.StatusCode}
}
as well as a Read() function:
func Read(url string, timeout time.Duration) (res *Response) {
done := make(chan bool)
go func() {
res = Get(url)
done <- true
}()
select { // As soon as either
case <-done: // done is sent on the channel or
case <-time.After(timeout): // timeout
res = &Response{"Gateway timeout\n", 504}
}
return
}
the Response type returned by the functions is defined as:
type Response struct {
Body string
StatusCode int
}
This read function makes use of the Get() function and also implements a timeout. The problem is that a data race can occur if the timeout occurs and the Get() response is written to res at the same time in Read().
I have a plan for how to solve this. It is to use Mutex. To do this, I would add a field to the Response struct:
type Response struct {
mu sync.Mutex
Body string
StatusCode int
}
so that the Response can be locked. However, I'm not sure how to fix this in the other parts of the code.
My attempt looks like this, for the Get():
func Get(url string) *Response {
res, err := http.Get(url)
if err != nil {
return &Response{}
}
// res.Body != nil when err == nil
defer res.Body.Close()
body, err := ioutil.ReadAll(res.Body)
if err != nil {
log.Fatalf("ReadAll: %v", err)
}
reflect.TypeOf(body)
return &Response{sync.Mutex(),string(body), res.StatusCode} // This line is changed.
}
and for the Read():
func Read(url string, timeout time.Duration) (res *Response) {
done := make(chan bool)
res = &Response{sync.Mutex()} // this line has been added
go func() {
res = Get(url)
done <- true
}()
select {
case <-done:
case <-time.After(timeout):
res.mu.Lock()
res = &Response{sync.Mutex(), "Gateway timeout\n", 504} // And mutex was added here.
}
defer res.mu.Unlock()
return
}
This "solution" generates these errors:
./client.go:54: missing argument to conversion to sync.Mutex: sync.Mutex()
./client.go:63: missing argument to conversion to sync.Mutex: sync.Mutex()
./client.go:63: too few values in struct initializer
./client.go:73: missing argument to conversion to sync.Mutex: sync.Mutex()
./client.go:95: cannot use "Service unavailable\n" (type string) as type sync.Mutex in field value
./client.go:95: cannot use 503 (type int) as type string in field value
./client.go:95: too few values in struct initializer
What is the correct way of using Mutex in this case?

While your answer with Volker's guidance is good, you might want to consider using a non default http.Client so that you can set a Timeout on the client making the request (then you don't have to worry about handling the timeouts yourself).

I followed Volker's suggestion and used a channel to solve the problem.
func Read(url string, timeout time.Duration) (res *Response) {
done := make(chan bool) // A channel
resChan := make(chan *Response)
go func() {
resChan <- Get(url)
done <- true
}()
select {
case <-done:
res = &Response{}
case <-time.After(timeout):
res = &Response{"Gateway timeout\n", 504}
}
return
}
Now, there can be no simultaneous writes to res. It's going to be either the timeout or the returned value of Get(url).

Related

Consuming Paginated REST API

I am trying to consume an API that on each successful request has a json key like the following:
{
"hasNextPage": true,
"endCursor": "some_guid_present_here"
}
The way the API works (I have used it many times in Python but am trying with Go for separate use case) is to basically check if there is a next page, then use the appropriate cursor to continue to paginate.
However, every time I use this structure, it will sometimes keep looping even though the response.Paginations.HasNextPage will be false.
I am trying to understand if it is my structure of consuming a paginated API like this that is causing this or else.
Also, I have say 4-5 requests to start off with, which I sent separately via goroutines. I am not sure if this is causing an issue but I've attached that as well in etl.go.
The main request making structs are in api.go.
I've confirmed that I do receive responses and they are unmarshaling properly, but am trying to wrangle this uncertain behavior.
api.go
package models
import (
"encoding/json"
"io/ioutil"
"net/http"
"fmt"
)
type Request struct {
Url string
ApiKey string
}
type Response struct {
...some fields...
Paginations Pagination `json:"pagination"`
}
type Pagination struct {
EndCursor string `json:"endCursor"`
HasNextPage bool `json:"hasNextPage"`
}
func (request *Request) Get() ([]Response, error) {
var responses []Response
var response Response
// Set up new request
req, err := http.NewRequest("GET", request.Url, nil)
if err != nil {
fmt.Println("Error creating request...")
return responses, err
}
// Add request headers
req.Header = http.Header{
"accept": {"application/json"},
"authorization": {"Bearer " + request.ApiKey},
}
// Get our initial response from the API and capture status code
resp, _ := http.DefaultClient.Do(req)
response.Status = resp.StatusCode
// Read the response body and Unmarshal into struct
respBody, err := ioutil.ReadAll(resp.Body)
json.Unmarshal(respBody, &response)
// If there was a parsing error, log it
if err != nil {
fmt.Println(err)
}
defer resp.Body.Close()
// This field will be in the response payload
// It is verified to be of type bool (not string)
fmt.Printf("Has Next Page? %t\n", resp.Paginations.HasNextPage)
// Append response to our slice of responses
responses = append(responses, response)
// If paginations are available, continue to loop through until all paginations are exhausted
for response.Paginations.HasNextPage == true {
req, err := http.NewRequest("GET", request.Url, nil)
if err != nil {
return responses, err
}
// Append "after" cursor to query in order to redirect to paginated response
qry := req.URL.Query()
qry.Set("after", response.Paginations.EndCursor)
req.URL.RawQuery = qry.Encode()
fmt.Println("Paginated request query: ", req.URL.String())
// Make request
resp, err := http.DefaultClient.Do(req)
response.Status = resp.StatusCode
fmt.Printf("Status Code: %d\n", response.Status)
// Read response and deserialize it
respBody, err := ioutil.ReadAll(resp.Body)
json.Unmarshal(respBody, &response)
fmt.Println("Pagination Cursor: ", response.Paginations.EndCursor)
fmt.Printf("Has Next Page? %t\n", response.Paginations.HasNextPage)
// If there was a parsing error, log it
if err != nil {
fmt.Println(err)
}
defer resp.Body.Close()
// Append response to our slice of responses
responses = append(responses, response)
}
return responses, nil
}
etl.go
package models
import (
"fmt"
"sync"
)
type Etl struct {
Requests []Request
}
func (etl *Etl) Extract() [][]Response {
var wg sync.WaitGroup
ch := make(chan []Response)
defer close(ch)
for _, req := range etl.Requests {
wg.Add(1) // Increment
fmt.Println("Incremented wait group")
go func(i Request) {
defer wg.Done() // Decrement
resp, err := req.Get()
if err != nil {
fmt.Println(err)
}
ch <- resp
fmt.Println("Decremented wait group")
}(req)
}
results := make([][]Response, len(etl.Requests))
for i, _ := range results {
results[i] = <-ch
//fmt.Println(results[i])
}
wg.Wait()
return nil
}
I believe I found the issue. In my pagination loop for response.Paginations.HasNextPage == true I was creating a new request object (http.NewRequest) on each iteration which did not have headers added from the prior (initial request).
This caused a 401 unauthorized error to return and continuous querying of the API since it was not receiving a new response.Paginations.HasNextPage value.
My solution was to simply alter the for loop a bit like so:
package models
import (
"encoding/json"
"io/ioutil"
"net/http"
"fmt"
)
type Request struct {
Url string
ApiKey string
}
type Response struct {
...some fields...
Paginations Pagination `json:"pagination"`
}
type Pagination struct {
EndCursor string `json:"endCursor"`
HasNextPage bool `json:"hasNextPage"`
}
func (request *Request) Get() ([]Response, error) {
var responses []Response
var response Response
// Set up new request
req, err := http.NewRequest("GET", request.Url, nil)
if err != nil {
fmt.Println("Error creating request...")
return responses, err
}
// Add request headers
req.Header = http.Header{
"accept": {"application/json"},
"authorization": {"Bearer " + request.ApiKey},
}
// Get our initial response from the API and capture status code
resp, _ := http.DefaultClient.Do(req)
response.Status = resp.StatusCode
// Read the response body and Unmarshal into struct
respBody, err := ioutil.ReadAll(resp.Body)
json.Unmarshal(respBody, &response)
// If there was a parsing error, log it
if err != nil {
fmt.Println(err)
}
defer resp.Body.Close()
// This field will be in the response payload
// It is verified to be of type bool (not string)
fmt.Printf("Has Next Page? %t\n", resp.Paginations.HasNextPage)
// Append response to our slice of responses
responses = append(responses, response)
// If paginations are available, continue to loop through until all paginations are exhausted
for response.Paginations.HasNextPage == true {
// Append "after" cursor to query in order to redirect to paginated response
qry := req.URL.Query()
qry.Set("after", response.Paginations.EndCursor)
req.URL.RawQuery = qry.Encode()
fmt.Println("Paginated request query: ", req.URL.String())
// Make request
resp, err := http.DefaultClient.Do(req)
response.Status = resp.StatusCode
fmt.Printf("Status Code: %d\n", response.Status)
// Read response and deserialize it
respBody, err := ioutil.ReadAll(resp.Body)
json.Unmarshal(respBody, &response)
fmt.Println("Pagination Cursor: ", response.Paginations.EndCursor)
fmt.Printf("Has Next Page? %t\n", response.Paginations.HasNextPage)
// If there was a parsing error, log it
if err != nil {
fmt.Println(err)
}
defer resp.Body.Close()
// Append response to our slice of responses
responses = append(responses, response)
}
return responses, nil
}

golang & mongo: How to implement Zeroer interface

I'm trying to figure out how to implement bsoncoded.Zeroer interface.
These is my straightforward struct:
type WorkContract struct {
ID primitive.ObjectID `bson:"_id,omitempty"`
Code string `bson:"code,omitempty"`
}
func (m *WorkContract) IsZero() bool {
log.Println("zero called")
return len(m.Code) == 0
}
I've tried to implement bsoncodec.Zeroer interface.
Nevertheless, this is never reached.
My repository's SaveNew method is:
func (repository *WorkContractRepository) SaveNew(ctx context.Context, workContract *model.WorkContract) (*model.WorkContract, error) {
result, err := repository.mongoCollection.InsertOne(ctx, workContract)
if err != nil {
return nil, err
}
if oid, ok := result.InsertedID.(primitive.ObjectID); ok {
modelWorkContract.ID = oid
} else {
//TODO Handle it, probably ignore it
// Not objectid.ObjectID, do what you want
}
return workContract, nil
}
I don't quite figure out what I'm doing wrong.
Any ideas?

Custom BSON marshal and unmarshal using mongo-driver

I have a struct field like this below. I also store raw protobuf of the same struct in db. Now every time fetch or save data to mongo. I have to update ReallyBigRaw, from the proto when I want to save to DB and when fetch I have to unmarshal ReallyBigRaw to ReallyBigObj to give out responses. Is there a way I can implement some interface or provide some callback functions so that the mongo driver does this automatically before saving or fetching data from DB.
Also, I am using the offical golang mongo driver not mgo, I have read some answers where can be done in mgo golang library.
import (
"github.com/golang/protobuf/jsonpb"
"go.mongodb.org/mongo-driver/bson"
"go.mongodb.org/mongo-driver/bson/primitive"
proto "github.com/dinesh/api/go"
)
type ReallyBig struct {
ID string `bson:"_id,omitempty"`
DraftID string `bson:"draft_id,omitempty"`
// Marshaled ReallyBigObj proto to map[string]interface{} stored in DB
ReallyBigRaw map[string]interface{} `bson:"raw,omitempty"`
ReallyBigObj *proto.ReallyBig `bson:"-"`
CreatedAt primitive.DateTime `bson:"created_at,omitempty"`
UpdatedAt primitive.DateTime `bson:"updated_at,omitempty"`
}
func (r *ReallyBig) GetProto() (*proto.ReallyBig, error) {
if r.ReallyBigObj != nil {
return r.ReallyBigObj, nil
}
Obj, err := getProto(r.ReallyBigRaw)
if err != nil {
return nil, err
}
r.ReallyBigObj = Obj
return r.ReallyBigObj, nil
}
func getRaw(r *proto.ReallyBig) (map[string]interface{}, error) {
m := jsonpb.Marshaler{}
b := bytes.NewBuffer([]byte{})
// marshals proto to json format
err := m.Marshal(b, r)
if err != nil {
return nil, err
}
var raw map[string]interface{}
// unmarshal the raw data to an interface
err = json.Unmarshal(b.Bytes(), &raw)
if err != nil {
return nil, err
}
return raw, nil
}
func getProto(raw map[string]interface{}) (*proto.ReallyBig, error) {
b, err := json.Marshal(raw)
if err != nil {
return nil, err
}
u := jsonpb.Unmarshaler{}
var reallyBigProto proto.ReallyBig
err = u.Unmarshal(bytes.NewReader(b), &recipeProto)
if err != nil {
return nil, err
}
return &reallyBigProto, nil
}
I implemented the Marshaler and Unmarshaler interface. Since mongo driver calls MarshalBSON and UnmarshalBSON if the type implements Marshaler and Unmarshaler we also end up in infinite loop. To avoid that we create a Alias of the type. Alias in Golang inherit only the fields not the methods so we end up calling normal bson.Marshal and bson.Unmarshal
func (r *ReallyBig) MarshalBSON() ([]byte, error) {
type ReallyBigAlias ReallyBig
reallyBigRaw, err := getRaw(r.ReallyBigObj)
if err != nil {
return nil, err
}
r.ReallyBigRaw = reallyBigRaw
return bson.Marshal((*ReallyBigAlias)(r))
}
func (r *ReallyBig) UnmarshalBSON(data []byte) error {
type ReallyBigAlias ReallyBig
err := bson.Unmarshal(data, (*ReallyBigAlias)(r))
if err != nil {
return err
}
reallyBigProto, err := getProto(r.ReallyBigRaw)
if err != nil {
return err
}
r.ReallyBigObj = reallyBigProto
return nil
}

MutableSideEffect() panics when setting second value

We're in the process of writing a .NET Cadence client and are a bit confused with how MutableSideEffect() is supposed to work. We've been thinking of the ID being passed as essentially a variable name and that developers should be able to update mutable values in a workflow. When we try this though, the second MutableSideEffect() call fails with this panic:
panic: adding duplicate decision DecisionType: Marker, ID: MutableSideEffect_value-1, state=Created, isDone()=false, history=[Created]
We munged the greetings workflow sample to make these calls:
package main
import (
"fmt"
"math/rand"
"time"
"go.uber.org/cadence/activity"
"go.uber.org/cadence/workflow"
"go.uber.org/zap"
)
/**
* This greetings sample workflow executes 3 activities in sequential. It gets greeting and name from 2 different activities,
* and then pass greeting and name as input to a 3rd activity to generate final greetings.
*/
// ApplicationName is the task list for this sample
const ApplicationName = "greetingsGroup"
// This is registration process where you register all your workflows
// and activity function handlers.
func init() {
workflow.Register(SampleGreetingsWorkflow)
activity.Register(getGreetingActivity)
activity.Register(getNameActivity)
activity.Register(sayGreetingActivity)
}
// SampleGreetingsWorkflow Workflow Decider.
func SampleGreetingsWorkflow(ctx workflow.Context) error {
// Get Greeting.
ao := workflow.ActivityOptions{
ScheduleToStartTimeout: time.Minute,
StartToCloseTimeout: time.Minute,
HeartbeatTimeout: time.Second * 20,
}
ctx = workflow.WithActivityOptions(ctx, ao)
logger := workflow.GetLogger(ctx)
var greetResult string
err := workflow.ExecuteActivity(ctx, getGreetingActivity).Get(ctx, &greetResult)
if err != nil {
logger.Error("Get greeting failed.", zap.Error(err))
return err
}
f := func(ctx workflow.Context) interface{} {
return rand.Intn(100)
}
e := func(a, b interface{}) bool {
if a == b {
return true
}
return false
}
var result int
sideEffectValue := workflow.MutableSideEffect(ctx, "value-1", f, e)
err = sideEffectValue.Get(&result)
if err != nil {
panic(err)
}
logger.Debug("MutableSideEffect-1", zap.Int("Value", result))
//************** THIS CALL FAILS **************
sideEffectValue = workflow.MutableSideEffect(ctx, "value-1", f, e)
err = sideEffectValue.Get(&result)
if err != nil {
panic(err)
}
logger.Debug("MutableSideEffect-2", zap.Int("Value", result))
// Get Name.
var nameResult string
err = workflow.ExecuteActivity(ctx, getNameActivity).Get(ctx, &nameResult)
if err != nil {
logger.Error("Get name failed.", zap.Error(err))
return err
}
// Say Greeting.
var sayResult string
err = workflow.ExecuteActivity(ctx, sayGreetingActivity, greetResult, nameResult).Get(ctx, &sayResult)
if err != nil {
logger.Error("Marshalling failed with error.", zap.Error(err))
return err
}
logger.Info("Workflow completed.", zap.String("Result", sayResult))
return nil
}
// Get Name Activity.
func getNameActivity() (string, error) {
return "Cadence", nil
}
// Get Greeting Activity.
func getGreetingActivity() (string, error) {
return "Hello", nil
}
// Say Greeting Activity.
func sayGreetingActivity(greeting string, name string) (string, error) {
result := fmt.Sprintf("Greeting: %s %s!\n", greeting, name)
return result, nil
}
Are we thinking about this correctly?
This is a bug in the Go client library. It happens when a MutableSideEffect with the same id is used multiple times during a single decision.
If you force a separate decision by putting workflow.Sleep(ctx, time.Second) just before the second MutableSideEffect call the problem disappears.
I filed an issue to get this fixed.
Thanks a lot for reporting!

Convert type from struct table to base.FixedDataGrid in GO

i'm having a trouble converting my struct table to fixedDataGrid, because i need my data to be a fixedDataGrid so that i can use machine learning methods from GoLearn lib.
My struct is like this:
type dataStruct struct{
Sepal_length string
Sepal_width string
Petal_length string
Petal_width string
Species string
}
So when i get my data from my mongo db, i get them like this:
var results []dataStruct
err := col.Find(nil).All(&results)
Is there a way to convert my "results" from []dataStruct type to base.FixedDataGrid ??
CreateModel function:
func CreateModel(c echo.Context) error {
fmt.Println("====> Entry CreateModel function");
//var results []dataStruct
var Success bool = false
Db := db.MgoDb{}
Db.Init()
defer Db.Close()
col := Db.C(db.TrainingDataCollection)
var results dataStruct
if err := col.Find(nil).All(results); err != nil {
fmt.Println("ERROR WHILE GETTING THE TRAINING DATA")
} else {
//fmt.Println("Results All: ", results)
Success = true
}
fmt.Println("=============",results)
//Initialises a new KNN classifier
cls := knn.NewKnnClassifier("euclidean", "linear", 2)
//Do a training-test split
trainData, testData := base.InstancesTrainTestSplit(results, 0.55)
cls.Fit(trainData)
//Calculates the Euclidean distance and returns the most popular label
predictions, err := cls.Predict(testData)
if err != nil {
panic(err)
}
fmt.Println(predictions)
// Prints precision/recall metrics
confusionMat, err := evaluation.GetConfusionMatrix(testData, predictions)
if err != nil {
panic(fmt.Sprintf("Unable to get confusion matrix: %s", err.Error()))
}
fmt.Println(evaluation.GetSummary(confusionMat))
return c.JSON(http.StatusOK, Success)
}
Thank you in advance for your help !
Here is how i solved the issue: Actually there is a function InstancesFromMat64(row int, col int, matrix) than creates instances from a float64 matrix, and this is what i used:
func CreateModel(c echo.Context) error {
fmt.Println("====> Entry CreateModel function");
var Success bool = false
Db := db.MgoDb{}
Db.Init()
defer Db.Close()
col := Db.C(db.TrainingDataCollection)
var results dataStruct
if err := col.Find(nil).All(&results); err != nil {
fmt.Println("ERROR WHILE GETTING THE TRAINING DATA")
} else {
Success = true
}
Data := make([]float64, len(results*nbAttrs)
/**** Filling the Data var with my dataset data *****/
mat := mat64.NewDense(row,nbAttrs,Data)
inst := base.InstancesFromMat64(row,nbAttrs,mat)
//Selecting the class attribute for our instance
attrs := inst.AllAttributes()
inst.AddClassAttribute(attrs[4])
//Initialise a new KNN classifier
cls := knn.NewKnnClassifier("manhattan","linear",3)
//Training-tessting split
trainData, testData := base.InstancesTrainTestSplit(inst,0.7)
/******* Continue the Model creation ******/
I'll be glad if my answer helps someone.
Thanks a lot #mkopriva for your help !
base.FixedDataGrid is an interface, so what you need to do is to implement that interface, that is, implement all of its methods, on the type you want to use as FixedDataGrid.
Since you want to use []dataStruct, a slice of dataStructs, which is an unnamed type, as FixedDataGrid you will have to declare a new type to be able to add methods to it because you can add methods only to named types. For example something like this:
type dataStructList []dataStruct
Now, if you take a look at the documentation, you can see that the FixedDataGrid interface declares two methods RowString and Size but also embeds another interface, the base.DataGrid interface, which means you need to implement the methods declared by DataGrid as well. So, given your new dataStructList type, you can do something like this:
func (l dataStructList) RowString(int) string { /* ... */ }
func (l dataStructList) Size() (int, int) { /* ... */ }
func (l dataStructList) GetAttribute(base.Attribute) (base.AttributeSpec, error) { /* ... */ }
func (l dataStructList) AllAttributes() []base.Attribute { /* ... */ }
func (l dataStructList) AddClassAttribute(base.Attribute) error { /* ... */ }
func (l dataStructList) RemoveClassAttribute(base.Attribute) error { /* ... */ }
func (l dataStructList) AllClassAttributes() []base.Attribute { /* ... */ }
func (l dataStructList) Get(base.AttributeSpec, int) []byte { /* ... */ }
func (l dataStructList) MapOverRows([]base.AttributeSpec, func([][]byte, int) (bool, error)) error { /* ... */ }
After you've implemented the /* ... */ parts you can then start using dataStructList as a FixedDataGrid, so something like this:
var results []dataStruct
err := col.Find(nil).All(&results)
fdg := dataStructList(results) // you can use fdg as FixedDataGrid
Or
var results dataStructList // you can use results as FixedDataGrid
err := col.Find(nil).All(&results)
Update:
After you've implemented all of those methods on the dataStructList all you need is the type of the results variable inside your function:
func CreateModel(c echo.Context) error {
fmt.Println("====> Entry CreateModel function")
//var results []dataStruct
var Success bool = false
Db := db.MgoDb{}
Db.Init()
defer Db.Close()
col := Db.C(db.TrainingDataCollection)
var results dataStructList // <--- use the type that implements the interface
if err := col.Find(nil).All(&results); err != nil { // <-- pass a pointer to results
fmt.Println("ERROR WHILE GETTING THE TRAINING DATA")
} else {
//fmt.Println("Results All: ", results)
Success = true
}
fmt.Println("=============", results)
//Initialises a new KNN classifier
cls := knn.NewKnnClassifier("euclidean", "linear", 2)
//Do a training-test split
trainData, testData := base.InstancesTrainTestSplit(results, 0.55) // <-- this will work because results if of type dataStructList, which implements the base.FixedDataGrid interface.
cls.Fit(trainData)
//Calculates the Euclidean distance and returns the most popular label
predictions, err := cls.Predict(testData)
if err != nil {
panic(err)
}
fmt.Println(predictions)
// Prints precision/recall metrics
confusionMat, err := evaluation.GetConfusionMatrix(testData, predictions)
if err != nil {
panic(fmt.Sprintf("Unable to get confusion matrix: %s", err.Error()))
}
fmt.Println(evaluation.GetSummary(confusionMat))
return c.JSON(http.StatusOK, Success)
}