I am using Newrelic to get insights on my golang app. I am trying to test a middleware that will log whenever a request comes with a proper new relic header. ( "Newrelic":"eyXXXXXXX" ).
This is my test :
func TestGetNewRelicTraceID(t *testing.T) {
w := httptest.NewRecorder()
req := httptest.NewRequest("GET", "/test", nil)
req.Header.Add("Newrelic", "eyJ2IjpbMCwxXSwiZCI6eyJ0eSI6IkFwcCIsImFwIjoiNDk1Njg4OTcwIiwiYWMiOiIxMzA5OTAiLCJ0eCI6IjE3MGNmYjRiNTBiMTQ2MGIiLCJpZCI6IjQ1NGY0MTFmOWNjYjA1MDgiLCJ0ciI6IjE3MGNmYjRiNTBiMTQ2MGI0MmQ0N2ZkZmQ3MTg2NzM3IiwicHIiOjEuMTI3NTUxLCJzYSI6dHJ1ZSwidGkiOjE2MjEwMTAwMjcwMjIsInRrIjoiMzQ2MDgwIn19")
app, _ := newrelic.NewApplication(
newrelic.ConfigAppName("test"),
newrelic.ConfigLicense("1TI35kweH5xJjYLvDgp6gX1LGbYvJ130n0E5Jecs"),
newrelic.ConfigDistributedTracerEnabled(true),
func(cfg *newrelic.Config) {
cfg.ErrorCollector.RecordPanics = true
},
)
_, fn := newrelic.WrapHandleFunc(app, "/test", func(w http.ResponseWriter, r *http.Request) {
txn2 := newrelic.FromContext(r.Context())
nrTraceID := fmt.Sprintf("%s", txn2.GetTraceMetadata().TraceID)
w.Write([]byte(nrTraceID))
})
fn(w, req)
assert.Equal(t, http.StatusOK, w.Code)
assert.Equal(t, "170cfb4b50b1460b42d47fdfd7186737", string(w.Body.Bytes()))
}
No matter what I do, the test never passes as every run creates a new trace id, instead of using the one coming with the header.
What am I doing incorrectly?
Well, I found my problem. So it seems that I have to wait for the connection to the New Relic server to actually take place before running the test. Which means I had to replace 1TI35kweH5xJjYLvDgp6gX1LGbYvJ130n0E5Jecs with a REAL key. This is very important it seems. Additionally, I had to add this too to the test:
err := app.WaitForConnection(time.Second * 10)
require.Nil(t, err)
And now it works as expected.
Related
I am working on a SaaS based project on which Merchants can subscribe to set up their online store.
Project Overview
I am using Golang (backend), Mongodb database service and Angular4 (frontend) to build the system. I have multiple merchants that can set up their store. Each merchant has its own url (having its business name as subdomain in the url) to connect to his database.
For Routing, I am using Golang's Gin framework at back end.
Problem
I want to run the cron jobs for the merchant-specific database. In these cron jobs there are some operations that need to connect to the database. But in my routing, until a route for an API is called, the database won't be set. And ultimately, the cron does not run with proper data.
Code
cron.go
package cron
import (
"gopkg.in/robfig/cron.v2"
"controllers"
)
func RunCron(){
c := cron.New()
c.AddFunc("#every 0h1m0s", controllers.ExpireProviderInvitation)
c.Start()
}
Controller function
func ExpireProviderInvitation() {
bookingAcceptTimeSetting, _ := models.GetMerchantSetting(bson.M{"section": "providers", "option_name": "bookings_accept_time"})
if bookingAcceptTimeSetting.OptionValue != nil{
allInvitations, _ := models.GetAllBookingInvitations(bson.M{ "status": 0, "send_type": "invitation", "datetime": bson.M{"$le": float64(time.Now().Unix()) - bookingAcceptTimeSetting.OptionValue.(float64)} })
if len(allInvitations) > 0 {
for _, invitationData := range allInvitations {
_ = GetNextAvailableProvider(invitationData.Bid, invitationData.Pid)
}
}
}
}
router.go
func NewRouter() {
router := gin.Default()
router.Use(gin.Recovery())
router.Use(SetMerchantDatabase)
public := router.Group("/api/v1")
for _, route := range publicRoutes{
switch route.Method {
case "GET" : public.GET(route.Pattern, route.HandlerFunc)
case "POST" : public.POST(route.Pattern, route.HandlerFunc)
case "PUT" : public.PUT(route.Pattern, route.HandlerFunc)
case "DELETE": public.DELETE(route.Pattern, route.HandlerFunc)
default : public.GET(route.Pattern, func(c *gin.Context){
c.JSON(200, gin.H{
"result": "Specify a valid http method with this route.",
})
})
}
}
router.NoRoute(controllers.UnauthorizedAccessResponse)
router.Run(":8080")
}
func SetMerchantDatabase(c *gin.Context){
subdomain := strings.Split(c.Request.Host, ".")
if len(subdomain) > 0{
config.Database = subdomain[0]
config.CurrentBusinessName = subdomain[0]
}else{
errMsg := "Failed: Invalid domain in headers."
response := controllers.ResponseController{
config.FailureCode,
config.FailureFlag,
errMsg,
nil,
}
controllers.GetResponse(c, response)
c.Abort()
}
c.Next()
}
main.go
package main
import (
"cron"
)
func main(){
cron.RunCron()
NewRouter()
}
Explanation of above code
An example route can be:
Route{ "AddCustomer", "POST", "/customer", controllers.SaveCustomer },
An example API url can be:
http://business-name.main-domain.com/api/v1/customer
Where "business-name" is the database which is set whenever an API is called.
I want to run my cron without calling an API route.
Alternative approach
In Shell script, we can run cron by hitting url as a command. For this, I can create a url to run it as a command. But this is my theoratical approach. Also I don't know how will I get different merchant databases.
I am not sure if this approach will work. Any kind of help will be greatly appreciated.
You need to adapt SetMerchantDatabase to work independently of your router. Then you can have it set things for Cron just as well.
I am currently working on a small script where I use the channels, select and goroutine and I really don't understand why it doesn't run as I think.
I have 2 channels that all my goroutines listen to.
I pass the channels to each goroutine where there is a select which must choose between the 2 depending on where the data comes first.
The problem is that no goroutine falls into the second case. I can have received 100 jobs one after the other, I see everything in the log. It does well what is requested in the first case and after that it sent the work in the second channel (still if it does well ...) I do not have any more logs.
I just don't understand why...
If someone can enlighten me :)
package main
func main() {
wg := new(sync.WaitGroup)
in := make(chan *Job)
out := make(chan *Job)
results := make(chan *Job)
for i := 0; i < 50; i++ {
go work(wg, in, out, results)
}
wg.Wait()
// Finally we collect all the results of the work.
for elem := range results {
fmt.Println(elem)
}
}
func Work(wg *sync.WaitGroup, in chan *Job, out chan *Job, results chan *Job) {
wg.Add(1)
defer wg.Done()
for {
select {
case job := <-in:
ticker := time.Tick(10 * time.Second)
select {
case <-ticker:
// DO stuff
if condition is true {
out <- job
}
case <-time.After(5 * time.Minute):
fmt.Println("Timeout")
}
case job := <-out:
ticker := time.Tick(1 * time.Minute)
select {
case <-ticker:
// DO stuff
if condition is true {
results <- job
}
case <-quitOut:
fmt.Println("Job completed")
}
}
}
}
I create a number of workers who listen to 2 channels and send the final results to the 3rd.
It does something with the received job and if it validates a given condition, it passes this job to the next channel and if it validates a condition it passes the job into the result channel.
So, in my head I had a pipeline like this for 5 workers for example: 3 jobs in the channel IN, directly 3 workers takes them, if the 3 job validates the condition, they are sent in the channel OUT. Directly 2 workers takes them and the 3rd job is picked up by one of the first 3 workers ...
Now I hope you have a better understanding for my first code. But in my code, I never get to the second case.
I think your solution might be a bit over complicated. Here is a simplified version. Bare in mind that there are numerous implementations. A good article to read
https://medium.com/smsjunk/handling-1-million-requests-per-minute-with-golang-f70ac505fcaa
Or even better right from the Go handbook
https://gobyexample.com/worker-pools (which I think maybe is what you were aiming for)
Anyway, below serves as a different type of example.. There are a few ways to go about solving this problem.
package main
import (
"context"
"log"
"os"
"sync"
"time"
)
type worker struct {
wg *sync.WaitGroup
in chan job
quit context.Context
}
type job struct {
message int
}
func main() {
numberOfJobs := 50
ctx, cancel := context.WithTimeout(context.Background(), 3*time.Second)
defer cancel()
w := worker{
wg: &sync.WaitGroup{},
in: make(chan job),
quit: ctx,
}
for i := 0; i < numberOfJobs; i++ {
go func(i int) {
w.in <- job{message: i}
}(i)
}
counter := 0
for {
select {
case j := <-w.in:
counter++
log.Printf("Received job %+v\n", j)
// DO SOMETHING WITH THE RECEIVED JOB
// WORKING ON IT
x := j.message * j.message
log.Printf("job processed, result %d", x)
case <-w.quit.Done():
log.Printf("Recieved quit, timeout reached. Number of jobs queued: %d, Number of jobs complete: %d\n", numberOfJobs, counter)
os.Exit(0)
default:
// TODO
}
}
}
Your quitIn and quitOut channels are basically useless: You create them and try to receive from them. Which you cannot as nobody can write to these channels because nobody even knows about their existence. I cannot say more because I do not understand what the code is supposed to do.
Because your function is "Work" and you are calling "work".
I'm trying to wrap my head around test driven development with Go and having an issue testing my CRUD functions since they are written for my production database. I'm coming from Ruby on Rails so I am used to using a test database, but Go doesn't seem to be too friendly in this regard.
So, how does one go about testing CRUD with Go?
main.go
package main
import (
"database/sql"
)
type book struct {
id int `json:"id"`
isbn string `json:"isbn"`
title string `json:"title"`
author string `json:"author"`
price float32 `json:"price"`
}
// type Books []*Book
// CRUD functions for Book
func (b *book) getBook(db *sql.DB) error {
return db.QueryRow("SELECT * FROM books WHERE id=$1", b.id).Scan(&b)
}
app.go
func (a *App) Initialize(dbname string) {
var err error
a.DB, err = sql.Open("postgres", "postgresql://localhost:5432/bookstore?sslmode=disable")
if err != nil {
log.Fatal(err)
}
}
my test
func TestGetBook(t *testing.T) {
clearTable()
addBook(1)
req, _ := http.NewRequest("GET", "/book/1", nil)
response := executeRequest(req)
checkResponseCode(t, http.StatusOK, response.Code)
}
The problem is that this keeps on looking at the books table in my DB, not the books_test table I'd like to use for testing. How can I go about making ONLY the tests use the books_test DB?
You should create a dev/test database which should be a complete copy of your production database. You will never want to run test directly against your production database since too many unexpected issues could happen.
A workaround would be starting up your app first, which creates the connection to your database, then run the test. You can use IntelliJ to achieve this.
TDD in my opinion is great for developing business logic layer code since new models and business processes can have unexpected impacts on existing ones.
#Godzilla74, there'are 2 solutions: enable SSL for test DB (try to check database settings or ask your system administrator) of have completely different setting for test:
func (a *App) Initialize(dbname string) {
var err error
pgsettings := os.Getenv("PGSETTINGS")
if pgsettins == "" {
// default options if not overridden
pgsettins := "postgresql://localhost:5432/bookstore?sslmode=disable"
}
a.DB, err = sql.Open("postgres", pgsettins)
if err != nil {
log.Fatal(err)
}
}
So you can run set environment setting to any required value and run app, like so:
export PGSETTINGS="postgresql://localhost:5432/bookstore_test?sslmode=disable"
go run main.go
In this post, it is pointed out that response.Body should be closed to avoid resource leak. It is also shown in the overview examples in http package godoc.
In my test code, I send multiple requests to try an API with
resp, err := http.DefaultClient.Do(req)
multiple times in the same function. Is this a bad practice? In this case, do I write defer resp.Body.Close() after each of them, or just once?
url := server.URL + "/ticket/add"
reader = strings.NewReader(`{"id": "test1", "detail": "test1"}`)
req, err := http.NewRequest("POST", url, reader)
assert.Nil(t, err)
resp, err := http.DefaultClient.Do(req)
assert.Nil(t, err)
defer resp.Body.Close()
assert.Equal(t, http.StatusCreated, resp.StatusCode)
// add a ticket with same id
reader = strings.NewReader(`{"id": "test1"}`)
req, err = http.NewRequest("POST", url, reader)
assert.Nil(t, err)
resp, err = http.DefaultClient.Do(req)
assert.Nil(t, err)
assert.Equal(t, http.StatusInternalServerError, resp.StatusCode)
A related question, on the server side, i.e., inside the func(w http.ResponseWriter, r *http.Request), is it necessary to close the request body as well?
Yes, you need to close both responses. Deferring one call to resp.Body.Close does not somehow effect the other. The *http.Response is different in each case, and they both can be deferred.
On the server side, you do not need to close the Request.Body -- from the http.Request documentation:
// The Server will close the request body. The ServeHTTP
// Handler does not need to.
Bad practice
If you did not reuse resp variable, it would be obvious that you are dealing with different response instances each of which must be closed.
Do sends an HTTP request and returns an HTTP response (instance)...
So, the bad practice is not to do several requests but to reuse the same variable for several responses. It causes code unobviousness. And produces unreachable objects which never be finalized.
Deferred execution
A defer statement pushes a function call onto a list. The list of saved calls is executed after the surrounding function returns.
If you have scheduled single or several deferred executions with reference to the same variable only the last assigned to the variable object's method will be executed (several times for several defers).
Playground with an example
Closing Response.Body
From Response documentation:
It is the caller's responsibility to
close Body.
So, typically you must close each Response.Body.
Garbage collecting and finalization (edit)
Garbage collector invokes bind to collecting objects finalizers to close files, connections and to do other cleanup actions. And there is no finalizers bind to Body object by default.
Your improved snippet:
// ...a lot of code
resp_one, err := http.DefaultClient.Do(req)
// BTW, `assert.Nil` just returns whether the assertion was successful (bool) not terminates a test.
if assert.Nil(t, err) == true {
defer resp_one.Body.Close()
}
// ...a lot of code
resp_two, err = http.DefaultClient.Do(req)
if assert.Nil(t, err) == true {
defer resp_two.Body.Close()
}
I have the following bit of code where I'm using the RESTful framework for Go called sleepy.
I can successfully start the service at: http://localhost:3000, however when I try to access http://localhost:3000/temperature I'm expecting my SparkCore function dht to execute.
I'm using the Gobot.io Spark platform to execute this function based on this example, which I've implemented in my own code.
The problem is that the code doesn't get past the gobot.Start() method inside the Get() function so I can't actually return the result data.
I'm setting the data value hoping that I can do the:
return 200, data, http.Header{"Content-type": {"application/json"}}
But it never gets called becuase of the gobot.Start().
I'm very new to Go so any help would be greatly appreciated.
package main
import (
"net/url"
"net/http"
"fmt"
"github.com/dougblack/sleepy"
"github.com/hybridgroup/gobot"
"github.com/hybridgroup/gobot/platforms/spark"
)
var gbot = gobot.NewGobot()
var sparkCore = spark.NewSparkCoreAdaptor("spark", "device_id", "auth_token")
type Temperature struct {}
func (temperature Temperature) Get(values url.Values, headers http.Header) (int, interface{}, http.Header) {
work := func() {
if result, err := sparkCore.Function("dht", ""); err != nil {
fmt.Println(err)
} else {
data := map[string]string{"Temperature": result}
fmt.Println("result from \"dht\":", result)
}
}
robot := gobot.NewRobot("spark",
[]gobot.Connection{sparkCore},
work,
)
gbot.AddRobot(robot)
gbot.Start()
return 200, data, http.Header{"Content-type": {"application/json"}}
}
func main() {
api := sleepy.NewAPI()
temperatureResource := new(Temperature)
api.AddResource(temperatureResource, "/temperature")
fmt.Println("Listening on http://localhost:3000/")
api.Start(3000)
}
gbot.Start() is a blocking call.
In this context, you are expected to call it as:
go gbot.Start()
This will launch it in a goroutine (think thread) and then let your app continue.
When you look at the gobot example app, they don't run in the background since it is the main function. If main runs everything in the background and doesn't wait for anything, the app exits immediately with no apparent effect.